A Privileged Basis for the Recurrent State of SSMs
This post describes ongoing research. Preliminary code is available in ssm-interp and ssm-interp-runs, and the per-layer statistics are on Hugging Face. Most of the tools we use to look inside …
State_space_models
Can LLM Finetuning be Stateful? Small states for Large Models - Part 2
The code for this project can be found at: https://github.com/omanshuthapliyal/HRM-adapter. And the paper can be found at: https://arxiv.org/pdf/2606.26290. In the previous post, I introduced the …
Can LLM Finetuning be Stateful? Small states for Large Models - Part 1
In the last post I talked about Hankel operators of simple linear dynamical systems. Such operators, and either eigenvalues, “can be used as a proxy for how much memory a task actually requires …
State Space Models: How Control Theory can echo Machine Learning
I wrote in a previous post how transformers are kind of like system identification methods applied to some sequence in a state space. In this post we try to understand if the other way round is true. …