FastPathLabs
Blog and announcements

Blog · Experiment 4 · Second post

Inside our LeapQuant implementation: how an 8-bit state is stored, read, folded and served

Date
14 November 2026
Experiment
Line 4 of the research program: recurrent-state quantization, at the runtime layer
Model
Qwen3.5-9B in SGLang
Companion
The first post reports the results. This one explains how the implementation works.

An architecture note for one experiment. Our first post reported what our LeapQuant implementation measures. This one explains how it works: what is stored, what happens on every token, what happens on every sixteenth token, and how the pieces sit inside a serving engine. It assumes no background in linear attention.

The method is from the paper by Yi Pan and colleagues [1]. The implementation described here is ours: a reference core, GPU kernels, and an integration into the SGLang serving engine. All numbers are for Qwen3.5-9B.

Words used in this post
StateThe running memory a linear-attention head keeps for one request. A grid of 128 by 128 numbers.
HeadOne of 32 independent units inside a layer. Each has its own state.
Key, update, queryThree vectors computed for every token. The key and the update change the state. The query reads it.
GateA number between 0 and 1 that shrinks the old state a little on every token.
Rank-one pieceThe simplest kind of grid: one column of numbers times one row of numbers.
WindowThe most recent tokens, at most 16, whose changes have not yet been rounded into the stored state.
FoldThe step that rounds a full window into the stored state.
SlotThe memory reserved in the server for the state of one request.

The problem: a state that every token touches

Language models used to remember the past in one way: an attention cache that grows with every token. Newer models replace most attention layers with linear attention. A linear-attention head does not keep the past tokens. It keeps a summary of them, of fixed size: the state.

Each new token does three things to a head’s state.

  • It shrinks the old state a little. The factor is the gate.
  • It adds one new piece. The piece is built from two vectors of the token, its key and its update. It is a rank-one piece: the update as a column, times the key as a row.
  • It reads the state. The state is multiplied by the token’s query, and the result is the head’s output.
Diagram of where the recurrent state lives. The model has 32 layers: 24 linear-attention layers and 8 attention layers. One linear-attention layer has 32 heads, each with its own state. One head holds a 128 by 128 state, 65,536 bytes in fp32. One request: 24 layers times 32 heads is 768 states, 50.33 MB in fp32, and every token reads and writes all of them.
Figure 1. Qwen3.5-9B has 768 such states per request.

In fp32, the usual format, one state is 65,536 bytes. A request has 768 of them: 50.33 MB. A server with 512 running requests holds about 26 GB of state, and every decode step moves all of it through the GPU’s memory. At high load that movement is a large part of the time of a step (technical note 2).

Fewer bytes per number would mean less to move and more requests in the same memory. That is the reason to compress the state.

Why plain rounding fails

The obvious fix is to store every number in 8 bits. It does not work, for two reasons.

  • The error is carried forward. The state is updated from itself. A rounding error made on one token is still in the state on the next, and the next rounding adds to it. After thousands of tokens the state has drifted. Our own replay shows it, and the paper’s accuracy table shows the model’s scores collapsing.
  • A few large numbers spoil the rest. Eight bits give 256 levels. The levels must span the largest number in the grid. A state has a few rows and columns far larger than the rest, so most numbers land on only a handful of levels.

LeapQuant answers the first with a window and the second with two repairs before rounding.

What is stored

Diagram of what LeapQuant keeps for one head. The stored state, kept for every slot, is 19,456 bytes: an 8-bit residual of 128 by 128 entries, 16,384 bytes; compensator tokens, 4 pairs of 16-bit vectors, 2,048 bytes; and 32-bit scales, one set per window, 1,024 bytes. Separately, for active requests only, a window buffer of up to 16 token records holding the key, update and gate of each recent token in 16 bits, 8,224 bytes.
Figure 2. The stored state is what every slot keeps. The window buffer exists only while a request is running.

The 8-bit residual. The 128 by 128 grid, one byte per number, after the two repairs below have been applied.

Compensator tokens. Four rank-one pieces kept at 16-bit precision. They hold the largest structures of the state, the ones that would otherwise set the range for everything else. They are taken out before rounding and added back on every read.

Scales. After the compensators are removed, some rows are still much larger than others. Each row is divided by its own scale before rounding, so that all rows use the 256 levels well. The scales are kept, to undo the division on a read.

The window buffer. For each of the last 16 tokens at most, its key, its update and its gate, at 16-bit precision. These are the changes that have not been rounded yet.

Table 1. Bytes for one head, by format.
FormatBytes for one headBytes per number
fp3265,5364.00
16-bit32,7682.00
LeapQuant, stored state19,4561.19
LeapQuant, window buffer of a running request8,224 more
Bar chart of megabytes of state for one running request of Qwen3.5-9B. fp32 state 50.33 MB. 16-bit state 25.17 MB. LeapQuant 21.26 MB, of which 14.94 MB is stored state and 6.32 MB is the window.
Figure 3. State of one running request. The stored part alone is what a parked copy needs.

A running request needs 21.26 MB, 16% less than with a 16-bit state. A parked copy of a state, such as a cached prompt prefix, needs only the stored part: 14.94 MB, about 40% less than 16-bit.

Every token: the decode step

Flow diagram of one decode step for one head, in five boxes: 1, a token arrives with its key, value, query and gate; 2, read with the key from the 8-bit state, compensators and window records; 3, form the update, what this token adds to the state; 4, append the key, update and gate to the window; 5, read with the query to give the layer's output for this token. The 8-bit state is not written during a window; only the window grows.
Figure 4. The five parts of a decode step.

The important point is what does not happen. The stored 8-bit state is not written. It stays exactly as the last fold left it. Everything the recent tokens changed is in the window, as rank-one pieces.

So a read has three parts.

  • One large multiplication. The 8-bit grid times a vector, with the scales applied.
  • Four small corrections. One for each compensator token.
  • Up to sixteen small corrections. One for each token in the window, each shrunk by the gates of the tokens after it.

A compensator token has the same shape as a token in the window. The kernel handles both in the same way, as extra entries of the window. That is why the paper calls them tokens.

The step reads twice. The update of a token depends on what the state already holds for its key, so the first read uses the key. The second read uses the query and gives the output.

At a batch of 512 requests, one decode step for one layer takes 0.252 ms in our kernel.

Every sixteenth token: the fold

Flow diagram of the fold, every 16th token, in five boxes: 1, rebuild the full state in high precision; 2, fit compensators, 4 pairs of vectors that hold the largest parts; 3, smooth, one scale per key row of what is left; 4, round the rest to 8 bits; 5, store the new state and empty the window. Rounding error enters once per 16 tokens, not once per token.
Figure 5. The five parts of a fold.

When a request’s window is full, its state is folded.

  • Rebuild. The full state is computed in high precision: the stored residual with its scales, plus the compensators, plus the 16 pieces of the window.
  • Fit new compensators. Four rank-one pieces are fitted to the rebuilt state. The fit passes over the grid several times. Our exact kernel makes eight passes, and the fast kernels make two.
  • Smooth. One scale per row is computed from what is left.
  • Round. What is left is divided by the scales and rounded to 8 bits.
  • Store and clear. The new residual, compensators and scales replace the old ones. The window is emptied.

Rounding now happens once per 16 tokens instead of once per token. Between two folds the output is computed from one fixed stored state and exact additions, so no new error enters.

The fold is the expensive step. For 512 requests of one layer it takes 2.36 ms in our Triton version and 1.27 ms in our CUDA version, by our own timing. Spread over 16 tokens that is 0.15 ms or 0.08 ms per token. With the Triton fold we measured 0.400 ms per token for the layer in total. With the CUDA fold we estimate 0.332 ms; the full cycle has not been timed yet.

The window length is a trade. A longer window means fewer rounding events, a larger buffer, more corrections on every read, and a longer pause when the fold comes. We use the paper’s choice of 16.

The pieces of the implementation

Architecture diagram. The SGLang server, with its API, scheduler, batching, model forward pass and CUDA graphs, sits above the state pool and the fold scheduler, which are linked to each other. The state pool, holding slots, cached-prefix copies and slot copy, feeds the decode kernel, written in Triton with tensor cores and run on every token. The fold scheduler, which decides which requests fold on which step, feeds the fold kernel, written in CUDA and run for a request every 16th token. Both kernels feed verification. Separately, the C++ reference core, which defines the arithmetic, feeds verification of the kernels against the reference on recorded states, which feeds reports on kernels, serving, accuracy and tasks.
Figure 6. The parts and how they depend on each other.

The reference core. A C++ program that defines the arithmetic: what a decode step and a fold must produce, number for number. It runs on a CPU and is slow. Everything else is checked against it.

The decode kernel. GPU code that runs the decode step for all heads of a layer and all requests of a batch at once. It is written in Triton and uses the GPU’s tensor cores for the large multiplication.

The fold kernel. GPU code for the fold. We first wrote it in Triton and then by hand in CUDA.

The state pool. Memory reserved when the server starts, divided into slots. A slot holds the stored state of all 768 heads, and the window buffers while a request is running. In our matched runs the server was configured with 560 slots for fp32, 1,100 for 16-bit and 1,310 for LeapQuant.

The fold scheduler. It decides which requests fold on which step. Section 8 explains why that matters.

SGLang. The serving engine: the API, the scheduler that batches requests, and the model’s forward pass. It runs each decode step as a CUDA graph, a recorded sequence of GPU work that is replayed, so that starting each kernel costs almost nothing (technical note 8). Our kernels run inside that graph.

The life of a request

Sequence diagram with four participants: client, scheduler, state pool and kernels. 1, the client sends a request to the scheduler. 2, the scheduler takes a slot from the state pool, or copies a cached prefix. 3, the prompt runs on the kernels as decode steps with a fold every 16 tokens. Repeated for every token: 4, a decode step for the next token; 5, the output of the layer returns to the scheduler; 6, the token goes back to the client; 7, on this request's 16th step, a fold. 8, at the end, the scheduler keeps a copy for the prefix cache and frees the slot.
Figure 7. Sequence of one request through the server.
  1. A request arrives.
  2. The scheduler takes a free slot from the state pool. If the server has already seen the beginning of this prompt, it copies the parked state of that prefix into the slot and skips that part of the work.
  3. The prompt is processed. The state advances token by token, with a fold every 16 tokens.
  4. For each new token, every linear-attention layer runs one decode step.
  5. The layers’ outputs flow on through the rest of the model.
  6. The token goes back to the client.
  7. On every sixteenth step of this request, its 768 states are folded.
  8. At the end, the server may keep a copy of the state for its prefix cache. The slot is freed.

Parking and restoring states is where a recurrent model differs most from an attention model. A cached prefix is a copy of a whole state, not a range of a cache. Copying a slot is real work: in four test cells, copying the slot in one fused operation gave 0.7% to 4.1% more throughput than copying it field by field.

The fold schedule and the tail

A fold touches all 24 linear layers. If 512 requests fold on the same step, that step carries 24 folds of 512 requests each: about 57 ms of extra work with the Triton fold and about 30 ms with the CUDA fold. The other 15 steps carry none.

Our benchmark starts all requests together with prompts of equal length, so their windows fill on the same step. That is the likely source of our tail latency: a p99 time between tokens of about 70 ms where a 16-bit state has 51 ms.

Diagram of 512 requests over 16 steps. With folds on one step, steps 1 to 15 carry no folds and step 16 carries all 512. With folds staggered, each of the 16 steps carries 32.
Figure 8. The same work, placed differently.

Staggering gives the requests different phases, so that each step folds about one sixteenth of them. In the first test this cut the p99 from 69.9 ms to 49.0 ms for about 2% of throughput. On a second request shape, and with the prefix cache on, it helped less.

A schedule that changes the length of a window changes the arithmetic slightly. So each schedule needs its own accuracy check, not only a latency check.

How we check it

Diagram of how the kernels are checked. States and inputs recorded from the real model, one linear layer at a time with 4,095 tokens per trace, go both to the C++ reference core on the CPU and to the GPU decode and fold kernels. The two are compared on the stored 8-bit entries after every fold, and on the outputs.
Figure 9. Verification of the kernels.

We record the states and inputs of linear layers while the real model generates text. Each recording covers one layer for 4,095 tokens. The recording is then played through the reference core and through the GPU kernels.

Two things are compared. After every fold, the stored 8-bit numbers must agree. On every token, the outputs must agree within a stated tolerance. The exact kernel matches the reference bit for bit on every fold. The fast kernels do their arithmetic in a different order, so they are held to the tolerance. In the latest run all nine displayed cases of the fast kernels pass.

This check says that the kernels compute what the reference defines. It does not say that the model’s answers are unchanged. That is a separate test, on tasks, and our first post reports where it stands.

How this differs from the authors’ release

Table 2. The authors’ open-source release and our implementation.
The authors’ open-source releaseOur implementation
Serving enginevLLM, as a pluginSGLang
KernelsWritten with TileLangTriton for the decode step, CUDA for the fold
ModelsQwen and GLM familiesQwen3.5-9B
Checks in the repositoryNoneReference core, verification on recorded states, accuracy replay, task and serving runs

The two have not been run side by side on one GPU. That comparison is on our list.

Open design questions

  • The fold is still too slow for our own target. It needs to go from 1.27 ms to about 0.85 ms.
  • One deep layer is less accurate than a 16-bit state. We are testing a rotation of the state before rounding, and a different state format for the deep layers only.
  • Low load. At 1 to 4 users our implementation is slower than both baselines, and we have not found the cause.
  • The schedule with the prefix cache on. Staggering helped less there.
  • Memory in bytes. Our slot counts are configured values. We still owe a full account of GPU memory per run.

Answers will be announced on the blog as they come, and recorded in research note 4.

References

  1. Y. Pan, H. Xi, K. Zhu, et al. LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization. arXiv:2609.38166, 2026.

The method is the authors’. The implementation and the measurements in this post are ours. Write to us about this experiment.