FastPathLabs
All experiments

Research note 4

Recurrent-state quantization for linear attention: from a first kernel to first serving results

Date
13 November 2026. Updated 14 November 2026.
Status
In progress. Kernel rewritten and inside SGLang. First serving results, mostly one run per arm. The 16-bit baseline is not yet qualified on accuracy.
Hardware
1 × NVIDIA Hopper-class GPU. The kernel reports record an H100 PCIe; the later serving reports do not record the GPU. CPU for the reference and the replay.
Model
Qwen3.5-9B, whose linear-attention layers are Gated DeltaNet
Baseline
SGLang’s linear-attention kernels and serving modes: the default fp32 state, and the 16-bit state with a 16-token window

Abstract

Hybrid models replace part of their attention with linear attention, which keeps a fixed-size recurrent state for each head in place of a growing KV cache. For Qwen3.5-9B that state is 50 MB per request in fp32, and it is read and rewritten at every token. LeapQuant [1] stores it in 8 bits by rounding once per window of tokens and keeping the largest errors in a few high-precision compensators. We built a reference implementation and a first GPU kernel. The reference reproduces the method’s arithmetic. The kernel is 7.2 times slower than SGLang’s window kernel at batch 512, because the fold that closes each window takes 73 ms against a budget near 1 ms. We report the profile and the redesign.

Update of 14 November. A rewritten kernel takes 0.400 ms per token at batch 512, 3.0 times faster than SGLang’s fp32 kernel and 4.6% slower than its 16-bit window kernel. Inside SGLang at 512 concurrent requests it gave about 45% more output tokens per second than the fp32 default and about 4% more than the 16-bit window state, with a worse tail. Section 7 gives the results and the status of each gate.

One line of five. This note is one experiment in our research program, at the runtime layer. Sections 1 to 6 are the record of 13 November and are left as published. Section 7 adds the results of 14 November. A shorter account is on the blog.

Background

A standard attention layer keeps every past token and consults all of them for each new one. A linear-attention layer keeps instead one matrix per head that summarizes everything seen so far [2]. That matrix is the recurrent state. In Qwen3.5-9B, 24 of the 32 layers are of this kind, each with 32 heads, and each head holds a matrix of 128 by 128 numbers. One request therefore owns 24 × 32 × 16,384 numbers, which is 50.3 MB in fp32.

For every generated token each head decays the matrix, reads what it holds for the current key, forms a correction, writes the correction back as a rank-one update, and reads the output. The arithmetic is small. The cost is moving the matrix, which is loaded from GPU memory and stored back at every token. The kernel is bound by memory bandwidth and not by compute (technical note 2).

Storing the state in fewer bytes attacks that cost directly, and it also raises the number of requests and cached prefixes that fit in memory.

Why rounding the state is difficult

A KV cache entry is written once and only read thereafter, so it can be rounded once [4]. A recurrent state is rewritten at every token. If it is rounded after every token, each token starts from an already rounded matrix and adds a new error, and the errors compound over thousands of tokens. The LeapQuant paper measures the effect: with a bf16 state rounded at every token, the AIME score of Qwen3.5-9B falls from 87.9 to 72.1 [1].

LeapQuant makes two changes. It rounds less often: the low-precision state is held fixed for a window of 16 tokens, each token’s change is kept as a small record in higher precision, and outputs are computed from the fixed state together with the records. At the end of the window a fold applies the records and rounds once. And it rounds less harmfully: before rounding, the largest outliers of the state are set aside as a few rank-one compensators kept in higher precision, and the remainder is smoothed. A related method reaches lower bit widths by a different route [3].

Per-token quantization The state is rounded after each of the 16 tokens. Each rounding starts from the last one’s error. Per-window quantization Fold The rounded state is held for 16 tokens. Updates are kept as records. One fold rounds once.
Figure 1. One window of 16 tokens. Each open cell is a token. Each solid mark is a rounding of the whole state.

What we built

  • A reference implementation in C++, written to be read. It follows the equations of the paper and defines the numbers a GPU kernel must reproduce.
  • A recorder and a replay tool. Nine traces of 4,095 tokens each were recorded from three layers under three kinds of prompt, and can be replayed through the reference and through a kernel.
  • A GPU kernel with two parts: an append step, which adds one record and computes the output, and a fold, which rebuilds the state, selects compensators, rescales, rounds and empties the buffer.

Results

Numerical behavior

All nine traces pass the reference’s self-checks. On six of them the 8-bit state gives lower output error than an ideal bf16 state held over the same window. On the remaining three it gives higher error, and all three are from layer 30, where the worst head reaches 15.4% output error. In layer 0 the four compensators reduce the error about a hundredfold. In layer 30 they change it from 3.63% to 3.13%.

One trace has been verified on the GPU. Across its 510 folds the kernel’s state matched the reference in every one of 267,386,880 entries.

Speed

Table 1. Mean time of one linear-attention layer per decode step, in milliseconds. All states are fp32 in these timings.
BatchSGLang, packed kernelSGLang, window kernelOur kernelOur kernel, with gate arithmeticRatio to window kernel
320.08210.05830.35230.598110.3
1280.30700.20101.41371.56537.8
5121.20630.78695.34835.68007.2

The ratio compares our kernel with gate arithmetic to SGLang’s window kernel.

Our kernel is slower than SGLang’s window kernel at every batch size recorded.

Where the time goes

A step is defined as fifteen appends and one fold, divided by sixteen. At batch 512 the profiler attributes 73.4 ms to one fold. Spread over sixteen tokens that is 4.59 ms of a 5.44 ms step, or 84%. The fitting of the compensators takes 57.6 ms of the fold, 78.5%.

The same definition gives the fold a budget. At batch 512 an append takes 0.714 ms and the window kernel averages 0.787 ms, so to equal the window kernel a fold may cost 1.17 ms more than an append. The recorded figure is 74.2 ms, about 64 times the budget. At batch 128 the append alone is already slower than the window kernel.

Fewer sweeps would not be enough. If the compensator fit vanished entirely, the step would still take about 1.84 ms, more than twice the baseline.

Table 2. Causes of the fold’s cost, largest first.
CauseMechanism
Bank conflicts in shared memoryEach lane of a warp walks one row of the matrix, so the 32 lanes read addresses 128 words apart. These fall in one memory bank and are served one after another. A pass costs about 1.21 ms in place of 0.145 ms, and a fold makes 44 such passes: about 47 of the 73 ms.
Number of passesFour compensators are found one at a time, with eight sweeps each: 72 walks over the matrix.
Loads inside the inner loopThe rebuild reads from global GPU memory twenty times for each matrix entry.
Serial sumsEach sum is taken by one thread in a fixed order while the others wait.
BarriersA fold passes about 211 points at which all threads wait for each other.
OccupancyA block needs about 65 KB of shared memory, so at most three blocks run on one streaming multiprocessor.
DivisionThe rounding divides twice for each entry.

The bank-conflict estimate is derived from the code and the profile. It awaits confirmation by a new run.

The fit itself has a weakness independent of speed. It starts from the longest row of the matrix on the assumption that this row overlaps the strongest pattern. A counterexample shows otherwise: scored against the strongest pattern, that start captures 16 of 291, where a block fit from a random start captures all 291.

Redesign

The following changes were planned on 13 November. Section 7 reports what has come of them.

  • Lay the matrix out in shared memory with a stride of 129 words, which by our arithmetic removes the bank conflict.
  • Fit the compensators together as a block, starting from the previous window’s compensators.
  • Determine why layer 30 loses, and set the policy for each layer separately.
  • Make the compressed state the only state of an active request. At present the fp32 pool is kept beside it, which adds about 46% to state memory where it should save.
  • Measure serving speed and task accuracy together. Neither has been measured yet.

The first target is a mean layer time at least 20% below the fastest baseline that passes the quality checks, with gate arithmetic and complete folds included. On the figures above that is 0.63 ms or less at batch 512, against 0.787 ms.

Update of 14 November: the second kernel and first serving results

Kernel

The fold was rewritten in Triton and the whole cycle runs inside a CUDA graph. The timings below are for one linear-attention layer at batch 512, counting one fold every 16 steps. Unlike Table 1, each SGLang mode is timed with its own state format.

Table 3. Kernel time per token, one layer, batch 512, in milliseconds.
KernelTimeEvidence
SGLang, fp32 state1.198Measured
SGLang, fp32 state with window0.695Measured
SGLang, 16-bit state0.611Measured
SGLang, 16-bit state with window0.382Measured
Ours, Triton fold0.400Measured
Ours, CUDA fold0.332Estimate: 0.2524 + 1.27 / 16

The measured kernel is 3.0 times faster than SGLang’s fp32 kernel and 4.6% slower than its 16-bit window kernel. A fold written in CUDA takes 1.27 ms for 512 requests, against 2.36 ms in Triton. If the rest of the step is unchanged, the layer would take 0.332 ms, about 13% less time than the 16-bit window kernel. The full cycle with the CUDA fold has not been timed on one build. Part of the fold’s gain since 13 November is less work: the first figure used eight fitting sweeps, the fast folds use two.

Serving

Table 4. Output tokens per second at 512 concurrent requests. 128 tokens in, 512 out, prefix cache off. One run per arm in each of two snapshots.
StateRun 1Run 2p99 between tokens
SGLang default, fp327,8597,931not reported
SGLang, 16-bit with window11,02511,032about 51 ms
Ours, 8-bit11,49311,463about 70 ms

That is 1.46 and 1.45 times the fp32 default, and 4.3% and 3.9% above the 16-bit state, with a p99 time between tokens about 36% higher. With 1,000 tokens in and 1,000 out, the 8-bit state was 6 to 7% slower than the default at 1 to 4 users, level at 16, and 11 to 41% faster from 64 users up; against the 16-bit state it ranged from 4% behind to 4% ahead.

The tail appears to come from requests that started together folding on the same step. Staggering the folds cut the p99 from 69.9 ms to 49.0 ms in the decode-heavy shape, below the 16-bit state’s 51.8 ms, for about 2% of throughput. In the second shape it cut the tail but stayed above the 16-bit state, and with the prefix cache on a staggered run was about 6% above it. Staggering is a candidate, not a default.

A run at 1,024 users showed a 53% gain over fp32, but there the fp32 server had 560 state slots against 1,100 for the others, so the comparison mixes speed with capacity. We do not use it.

Accuracy

In single-layer replay the 8-bit state is at least as accurate as the 16-bit window state on 6 of 9 traces, far better in layer 0 and worse in layer 30. On tasks it is noninferior to fp32 on MMLU-Pro at a one-point margin (67.0% against 65.6%, interval −0.82 to +3.62 points). On GSM8K the result is inconclusive (93.5% against 94.0%, interval −1.81 to +0.81).

The paper reports that a 16-bit state rounded per token loses 11.5 points on average on AIME 2026, BeyondAIME, GPQA-Diamond and LiveCodeBench v6 [1]. Our 16-bit control rounds once per window, and our tests use short answers; our one AIME run had 29 of 30 answers cut at the token limit. If the windowed 16-bit state fails the paper’s tasks, the fair baseline is fp32 and the gain is about 45%. If it passes, the gain is about 4% and the case rests on memory. That run is the first quality task.

Status of the gates

Table 5. The acceptance gates of the plan, and where each stands.
GateStatus
G0 EvidenceNot met. GPU manifests, measured slot bytes, the latency target and repetitions are missing.
G1 NumericalPartly. Nine of nine displayed fast cases pass. Layer 30 trails the 16-bit state in replay.
G2 KernelNot shown. Measured 4.6% slower than the 16-bit window kernel; estimated 13% less time with the CUDA fold. Target: 20% less.
G3 StatePartly. Two-turn cache checks pass. Lifecycle tests and byte accounting are missing.
G4 ProductNot shown. Raw throughput 3.9 to 4.3% above the 16-bit state, with a worse tail. No goodput at a stated target. Target: 15%.
G5 ReleaseOpen.

What the paper reports

The paper measures B200, RTX PRO 6000 and RTX 5090 GPUs, and reports a kernel speedup of 2.05 to 3.70 times over an fp32 kernel, a decode-step speedup of 1.22 to 1.37 times for whole models at batch 512, an end-to-end speedup of 1.47 times, and 41 to 56% less memory with prefix caching [1]. It has no measurement on the H100 generation. We therefore do not take its kernel gains as an acceptance figure for our hardware. Our target is set against the fastest baseline that passes the quality checks on our workload.

Prefix caching for models of this kind raises its own questions, since a recurrent state cannot be truncated to an earlier position as a KV cache can [5].

Limitations

  • Most serving numbers are one run per arm. Two runs of one point differed by several percent.
  • The 0.332 ms kernel figure is an estimate. The full cycle with the CUDA fold has not been timed.
  • The 16-bit baseline has not been tested on the paper’s long-reasoning tasks.
  • Slot counts are configured, not measured in bytes.
  • Replay error is an arithmetic measure. It is not task accuracy.
  • This is not a reproduction of the paper, whose results are on other GPUs and, for serving, in vLLM.

References

  1. Y. Pan, H. Xi, K. Zhu, et al. LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization. 2026.
  2. S. Yang, J. Kautz, A. Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. 2024.
  3. B. Yao, H. Xu, H. Lin, et al. STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization. 2026.
  4. Z. Liu, J. Yuan, H. Jin, et al. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. 2024.
  5. R. Pan, Z. Wang, Z. Jia, et al. Marconi: Prefix Caching for the Era of Hybrid LLMs. 2024.

Write to us about this note.