Linear_algebra on Omanshu Thapliyal

Can LLM Finetuning be Stateful? Small states for Large Models - Part 2

Fri, 07 Aug 2026 11:57:21 -0700

The code for this project can be found at: https://github.com/omanshuthapliyal/HRM-adapter. And the paper can be found at: https://arxiv.org/pdf/2606.26290.

In the previous post, I introduced the basic idea behind a stateful adapter. Instead of asking a small trainable module to only modify the current transformer representation, we allow it to maintain a state: $$ s_t = \bar{A}s_{t-1} + \bar{B}h_t, $$ and use that state to generate a residual correction, $$ r_t = Cs_t. $$ The important unresolved question was: if this adapter gets a small amount of memory, what should it remember? The answer I want to explore is the same one that appeared in the discussion of Hankel singular values:

Can LLM Finetuning be Stateful? Small states for Large Models - Part 1

Fri, 31 Jul 2026 11:57:21 -0700

In the last post I talked about Hankel operators of simple linear dynamical systems. Such operators, and either eigenvalues, “can be used as a proxy for how much memory a task actually requires in a state-space model”. And later we tied their input-output relations empirically tie the memory requirements of a post-trained state space model (SSM) to its Hankel singular value (HSV) rates. Further, HSV-based balanced truncation can offer us to selectively choose ‘some’ eigenvalues to keep that involve higher input-to-output signal strength over time. And so, we used Hankel singular values to decide which states of a dynamical system are worth keeping. However, a language model also processes a sequence of tokens over time. We saw task-dependent HSV decay rates for various tasks for different SSM-based models. Similarly, could we give the model a small state, one deliberately built around that same notion of “worth keeping”, without retraining the entire model?

State Space Models: How Control Theory can echo Machine Learning

Mon, 06 Oct 2025 18:57:21 -0700

I wrote in a previous post how transformers are kind of like system identification methods applied to some sequence in a state space. In this post we try to understand if the other way round is true.

We already saw previously ¹ how predicting words/tokens is a sequence to sequence mapping problem, which is the subject of a lot of prediction, estimation, and controls problems. In fact, we leveraged a transformer’s sequence mapping ability and applied it to a controls problem of trajectory prediction for a simple 2D car model ¹. Can we do the other way round? Turns out: yes we can!

On Eigenvalues, Homotopy, and Root Loci

Fri, 05 Sep 2025 11:57:21 -0700

Consider an $$n+1$$ dimensional matrix as follows:

$$ A = \begin{pmatrix} \frac{\pi^2}{6} & 1 & \frac{1}{4} & ... & \frac{1}{n^2} \\ 1 & \frac{\pi^2}{6} & \frac{1}{4} & ... & \frac{1}{n^2} \\ \vdots & & \ddots & & \vdots\\ 1 & \frac{1}{4} & ... & \frac{1}{n^2} &\frac{\pi^2}{6} \\ \end{pmatrix} $$

Is the matrix positive-definite? The problem itself is a dramatization of how my professor would ask us to write arbitrary real matrices and quickly determine its positive-definiteness (most of the times) as a trick, seemingly like in-class wizardry!