Blog · Experiment 4 · Second post
Inside our LeapQuant implementation: how an 8-bit state is stored, read, folded and served
An architecture note for one experiment. Our first post reported what our LeapQuant implementation measures. This one explains how it works: what is stored, what happens on every token, what happens on every sixteenth token, and how the pieces sit inside a serving engine. It assumes no background in linear attention.
The method is from the paper by Yi Pan and colleagues [1]. The implementation described here is ours: a reference core, GPU kernels, and an integration into the SGLang serving engine. All numbers are for Qwen3.5-9B.
| State | The running memory a linear-attention head keeps for one request. A grid of 128 by 128 numbers. |
|---|---|
| Head | One of 32 independent units inside a layer. Each has its own state. |
| Key, update, query | Three vectors computed for every token. The key and the update change the state. The query reads it. |
| Gate | A number between 0 and 1 that shrinks the old state a little on every token. |
| Rank-one piece | The simplest kind of grid: one column of numbers times one row of numbers. |
| Window | The most recent tokens, at most 16, whose changes have not yet been rounded into the stored state. |
| Fold | The step that rounds a full window into the stored state. |
| Slot | The memory reserved in the server for the state of one request. |
The problem: a state that every token touches
Language models used to remember the past in one way: an attention cache that grows with every token. Newer models replace most attention layers with linear attention. A linear-attention head does not keep the past tokens. It keeps a summary of them, of fixed size: the state.
Each new token does three things to a head’s state.
- It shrinks the old state a little. The factor is the gate.
- It adds one new piece. The piece is built from two vectors of the token, its key and its update. It is a rank-one piece: the update as a column, times the key as a row.
- It reads the state. The state is multiplied by the token’s query, and the result is the head’s output.
In fp32, the usual format, one state is 65,536 bytes. A request has 768 of them: 50.33 MB. A server with 512 running requests holds about 26 GB of state, and every decode step moves all of it through the GPU’s memory. At high load that movement is a large part of the time of a step (technical note 2).
Fewer bytes per number would mean less to move and more requests in the same memory. That is the reason to compress the state.
Why plain rounding fails
The obvious fix is to store every number in 8 bits. It does not work, for two reasons.
- The error is carried forward. The state is updated from itself. A rounding error made on one token is still in the state on the next, and the next rounding adds to it. After thousands of tokens the state has drifted. Our own replay shows it, and the paper’s accuracy table shows the model’s scores collapsing.
- A few large numbers spoil the rest. Eight bits give 256 levels. The levels must span the largest number in the grid. A state has a few rows and columns far larger than the rest, so most numbers land on only a handful of levels.
LeapQuant answers the first with a window and the second with two repairs before rounding.
What is stored
The 8-bit residual. The 128 by 128 grid, one byte per number, after the two repairs below have been applied.
Compensator tokens. Four rank-one pieces kept at 16-bit precision. They hold the largest structures of the state, the ones that would otherwise set the range for everything else. They are taken out before rounding and added back on every read.
Scales. After the compensators are removed, some rows are still much larger than others. Each row is divided by its own scale before rounding, so that all rows use the 256 levels well. The scales are kept, to undo the division on a read.
The window buffer. For each of the last 16 tokens at most, its key, its update and its gate, at 16-bit precision. These are the changes that have not been rounded yet.
| Format | Bytes for one head | Bytes per number |
|---|---|---|
| fp32 | 65,536 | 4.00 |
| 16-bit | 32,768 | 2.00 |
| LeapQuant, stored state | 19,456 | 1.19 |
| LeapQuant, window buffer of a running request | 8,224 more |
A running request needs 21.26 MB, 16% less than with a 16-bit state. A parked copy of a state, such as a cached prompt prefix, needs only the stored part: 14.94 MB, about 40% less than 16-bit.
Every token: the decode step
The important point is what does not happen. The stored 8-bit state is not written. It stays exactly as the last fold left it. Everything the recent tokens changed is in the window, as rank-one pieces.
So a read has three parts.
- One large multiplication. The 8-bit grid times a vector, with the scales applied.
- Four small corrections. One for each compensator token.
- Up to sixteen small corrections. One for each token in the window, each shrunk by the gates of the tokens after it.
A compensator token has the same shape as a token in the window. The kernel handles both in the same way, as extra entries of the window. That is why the paper calls them tokens.
The step reads twice. The update of a token depends on what the state already holds for its key, so the first read uses the key. The second read uses the query and gives the output.
At a batch of 512 requests, one decode step for one layer takes 0.252 ms in our kernel.
Every sixteenth token: the fold
When a request’s window is full, its state is folded.
- Rebuild. The full state is computed in high precision: the stored residual with its scales, plus the compensators, plus the 16 pieces of the window.
- Fit new compensators. Four rank-one pieces are fitted to the rebuilt state. The fit passes over the grid several times. Our exact kernel makes eight passes, and the fast kernels make two.
- Smooth. One scale per row is computed from what is left.
- Round. What is left is divided by the scales and rounded to 8 bits.
- Store and clear. The new residual, compensators and scales replace the old ones. The window is emptied.
Rounding now happens once per 16 tokens instead of once per token. Between two folds the output is computed from one fixed stored state and exact additions, so no new error enters.
The fold is the expensive step. For 512 requests of one layer it takes 2.36 ms in our Triton version and 1.27 ms in our CUDA version, by our own timing. Spread over 16 tokens that is 0.15 ms or 0.08 ms per token. With the Triton fold we measured 0.400 ms per token for the layer in total. With the CUDA fold we estimate 0.332 ms; the full cycle has not been timed yet.
The window length is a trade. A longer window means fewer rounding events, a larger buffer, more corrections on every read, and a longer pause when the fold comes. We use the paper’s choice of 16.
The pieces of the implementation
The reference core. A C++ program that defines the arithmetic: what a decode step and a fold must produce, number for number. It runs on a CPU and is slow. Everything else is checked against it.
The decode kernel. GPU code that runs the decode step for all heads of a layer and all requests of a batch at once. It is written in Triton and uses the GPU’s tensor cores for the large multiplication.
The fold kernel. GPU code for the fold. We first wrote it in Triton and then by hand in CUDA.
The state pool. Memory reserved when the server starts, divided into slots. A slot holds the stored state of all 768 heads, and the window buffers while a request is running. In our matched runs the server was configured with 560 slots for fp32, 1,100 for 16-bit and 1,310 for LeapQuant.
The fold scheduler. It decides which requests fold on which step. Section 8 explains why that matters.
SGLang. The serving engine: the API, the scheduler that batches requests, and the model’s forward pass. It runs each decode step as a CUDA graph, a recorded sequence of GPU work that is replayed, so that starting each kernel costs almost nothing (technical note 8). Our kernels run inside that graph.
The life of a request
- A request arrives.
- The scheduler takes a free slot from the state pool. If the server has already seen the beginning of this prompt, it copies the parked state of that prefix into the slot and skips that part of the work.
- The prompt is processed. The state advances token by token, with a fold every 16 tokens.
- For each new token, every linear-attention layer runs one decode step.
- The layers’ outputs flow on through the rest of the model.
- The token goes back to the client.
- On every sixteenth step of this request, its 768 states are folded.
- At the end, the server may keep a copy of the state for its prefix cache. The slot is freed.
Parking and restoring states is where a recurrent model differs most from an attention model. A cached prefix is a copy of a whole state, not a range of a cache. Copying a slot is real work: in four test cells, copying the slot in one fused operation gave 0.7% to 4.1% more throughput than copying it field by field.
The fold schedule and the tail
A fold touches all 24 linear layers. If 512 requests fold on the same step, that step carries 24 folds of 512 requests each: about 57 ms of extra work with the Triton fold and about 30 ms with the CUDA fold. The other 15 steps carry none.
Our benchmark starts all requests together with prompts of equal length, so their windows fill on the same step. That is the likely source of our tail latency: a p99 time between tokens of about 70 ms where a 16-bit state has 51 ms.
Staggering gives the requests different phases, so that each step folds about one sixteenth of them. In the first test this cut the p99 from 69.9 ms to 49.0 ms for about 2% of throughput. On a second request shape, and with the prefix cache on, it helped less.
A schedule that changes the length of a window changes the arithmetic slightly. So each schedule needs its own accuracy check, not only a latency check.
How we check it
We record the states and inputs of linear layers while the real model generates text. Each recording covers one layer for 4,095 tokens. The recording is then played through the reference core and through the GPU kernels.
Two things are compared. After every fold, the stored 8-bit numbers must agree. On every token, the outputs must agree within a stated tolerance. The exact kernel matches the reference bit for bit on every fold. The fast kernels do their arithmetic in a different order, so they are held to the tolerance. In the latest run all nine displayed cases of the fast kernels pass.
This check says that the kernels compute what the reference defines. It does not say that the model’s answers are unchanged. That is a separate test, on tasks, and our first post reports where it stands.
How this differs from the authors’ release
| The authors’ open-source release | Our implementation | |
|---|---|---|
| Serving engine | vLLM, as a plugin | SGLang |
| Kernels | Written with TileLang | Triton for the decode step, CUDA for the fold |
| Models | Qwen and GLM families | Qwen3.5-9B |
| Checks in the repository | None | Reference core, verification on recorded states, accuracy replay, task and serving runs |
The two have not been run side by side on one GPU. That comparison is on our list.
Open design questions
- The fold is still too slow for our own target. It needs to go from 1.27 ms to about 0.85 ms.
- One deep layer is less accurate than a 16-bit state. We are testing a rotation of the state before rounding, and a different state format for the deep layers only.
- Low load. At 1 to 4 users our implementation is slower than both baselines, and we have not found the cause.
- The schedule with the prefix cache on. Staggering helped less there.
- Memory in bytes. Our slot counts are configured values. We still owe a full account of GPU memory per run.
Answers will be announced on the blog as they come, and recorded in research note 4.
References
- Y. Pan, H. Xi, K. Zhu, et al. LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization. arXiv:2609.38166, 2026.
The method is the authors’. The implementation and the measurements in this post are ours. Write to us about this experiment.