FastPathLabs
Blog and announcements

Blog · From research note 4

~3× faster linear-attention kernels than packed FP32 at batch 512

45% more serving throughput than SGLang’s FP32 default, with our implementation of LeapQuant: 8-bit quantization of the recurrent state of linear attention.

Date
14 November 2026
Experiment
Line 4 of the research program: recurrent-state quantization, at the runtime layer
Model
Qwen3.5-9B in SGLang, one GPU
Baselines
SGLang’s default fp32 state, and SGLang’s 16-bit state with a window
Status
In progress. Most numbers are one run per arm.

One experiment among five. FastPath studies how a request becomes tokens, one mechanism at a time, each against a bar set before the run. This post reports one of those experiments. It is not a product announcement, and it does not yet clear the bar we set for it. We publish it now, with what does not look good, because that is how we publish every result.

Newer language models mix two kinds of layer. Attention layers keep a cache that grows with every token. Linear-attention layers keep a running state of fixed size for every request, and every decode step reads and writes all of it. In Qwen3.5-9B that state is a 128 by 128 matrix per head, in 24 of the model’s 32 layers.

LeapQuant, by Yi Pan and colleagues [1], stores this state in 8 bits. We implemented it from the paper, wrote our own GPU kernels, put it inside the SGLang serving engine, and measured it with Qwen3.5-9B on one GPU. This post reports what we measured, including the parts that do not look good yet.

The result in one paragraph

At 512 concurrent requests, LeapQuant produced about 45% more output tokens per second than SGLang’s default, the fp32 state. Against SGLang’s own faster setting, a 16-bit state with a window, it was 4% ahead, and its tail latency was worse. So most of the gain over the default comes from leaving fp32, and the rest is small so far. Whether the 16-bit setting is a fair baseline depends on its accuracy, which we have not yet tested on hard tasks.

What LeapQuant does

Rounding a running state to 8 bits after every token destroys it, because each rounding error feeds the next step. The paper avoids that in three ways.

  • It rounds once per window. Recent token updates are buffered at higher precision. The state is rounded only when the window is full. We use a window of 16 tokens and call that step the fold.
  • It keeps the outliers apart. A few high-precision vectors, the compensator tokens, hold the largest parts of the state.
  • It smooths the rest before rounding. Scales per row keep a few large entries from setting the step size for all the others.

The stored state takes 1.19 bytes per entry, against 2 for a 16-bit state and 4 for fp32. Our second post explains in detail how the state is stored, read, folded and served. Technical note 9 explains why a recurrent state is harder to round than a KV cache.

Serving results

Bar chart of output tokens per second at 512 concurrent requests in two runs. First run: fp32 default 7,859, 16-bit with window 11,025, LeapQuant 11,493. Second run: 7,931, 11,032 and 11,463.
Figure 1. Output tokens per second at 512 concurrent requests. 128 tokens in, 512 out, prefix cache off. One run per arm in each of two snapshots.

In two runs LeapQuant reached 11,493 and 11,463 output tokens per second. The fp32 default reached 7,859 and 7,931. The 16-bit state reached 11,025 and 11,032. That is 1.46 and 1.45 times the default, and 4.3% and 3.9% above the 16-bit state.

The gain has a cost. LeapQuant’s p99 time between tokens was about 70 ms against 51 ms for the 16-bit state. A throughput chart alone would hide that, which is why we judge results on goodput at a latency target (research note 5).

At the same hourly price, a throughput gain of s lowers the cost per token by 1 − 1/s. So 4% more throughput is about 3.8% lower cost per output token, and 45% more is about 31% lower. These are arithmetic on our measured ratios, not a measured bill.

The gain depends on load

Bar chart of LeapQuant's throughput relative to the fp32 default and to the 16-bit window state by concurrent users. 1,000 in and 1,000 out: 0.94 to 1.41 against fp32 and 0.96 to 1.04 against 16-bit, from 1 to 512 users. 8,192 in and 1,024 out: 0.92 to 1.22 against fp32 and 0.96 to 1.05 against 16-bit, from 1 to 256 users.
Figure 2. LeapQuant’s throughput relative to each baseline on two request shapes. One run per point; some points are carried over from an earlier round.

With 1,000 tokens in and 1,000 out, LeapQuant was 6% to 7% slower than the default at 1 to 4 users, level at 16, and 11% to 41% faster from 64 users up. Against the 16-bit state it ranged from 4% behind at low load to 4% ahead at high load.

An 8-bit state pays when many requests share the GPU, because that is when the state is a large part of what each step moves. At low load our implementation is slower than both baselines, and we do not yet know why.

The kernel

Bar chart of kernel time per token for one linear-attention layer at batch 512. SGLang fp32 1.198 ms, fp32 with window 0.695, 16-bit 0.611, 16-bit with window 0.382, LeapQuant 0.400 measured, LeapQuant with the CUDA fold 0.332 estimated.
Figure 3. Kernel time per token for one linear-attention layer at batch 512. Five measured rows and one estimate, hatched.

Measured inside a CUDA graph, our Triton kernels take 0.400 ms per token for one layer at batch 512, counting one fold every 16 steps. That is 3.0 times faster than SGLang’s fp32 kernel. It is also 4.6% slower than SGLang’s 16-bit window kernel. Two days earlier our first kernel was 7.2 times slower than the window kernel; research note 4 records where that time went.

We have since written the fold in CUDA. It takes 1.27 ms for 512 requests where the Triton fold took 2.36 ms. If the rest of the step is unchanged, the layer takes about 0.332 ms per token, which would be 15% faster than the 16-bit window kernel. That figure is an estimate. We have not yet timed the full cycle on one build.

Tail latency and fold scheduling

Every 16th token a request folds. Requests that start together fold on the same step, and that step takes longer than the others. We believe that is where the tail comes from, though our aggregate tables cannot prove it.

Bar chart of p99 time between tokens at 512 concurrent requests. 128 in, 512 out: 16-bit 51.8 ms, LeapQuant folds on one step 69.9 ms, staggered 49.0 ms. 1,000 in, 1,000 out: 59.5, 79.5 and 61.7 ms.
Figure 4. p99 time between tokens at 512 concurrent requests, with the folds on one step and staggered. One run per arm.

Staggering the folds is the obvious remedy, and the first test is encouraging. On the decode-heavy shape it cut LeapQuant’s p99 from 69.9 ms to 49.0 ms, below the 16-bit state’s 51.8 ms, for about 2% of throughput. On the second shape it cut the tail too but stayed above the 16-bit state. With the prefix cache on, a staggered run was still about 6% above 16-bit. Each of these is one run.

Accuracy

Bar chart of single-layer replay error against the fp32 state, mean of three prompts. Layer 0: 16-bit window 0.89%, LeapQuant 0.02%. Layer 13: 1.29% and 0.69%. Layer 30: 0.60% and 1.21%.
Figure 5. Reported output error of one layer on states recorded from the real model. An arithmetic diagnostic, not task accuracy.

On recorded states, replayed one layer at a time, LeapQuant is at least as accurate as the 16-bit state on 6 of 9 traces. It is far better in the early layers and worse in layer 30.

On tasks, LeapQuant passes our one-point noninferiority test against fp32 on MMLU-Pro: 67.0% against 65.6%. On GSM8K the result is inconclusive: 93.5% against 94.0%, with an interval too wide to decide. Inconclusive is not the same as equivalent.

One thing matters more than the rest. The paper reports that a plain 16-bit state loses 11.5 points on average on hard reasoning and coding tasks, while LeapQuant matches fp32. Our short tests show no such gap for the 16-bit state with a window. We have not run the paper’s tasks. If the windowed 16-bit state also loses accuracy there, it is not a fair baseline, and the comparison that counts is the one against fp32. That run decides the headline of this experiment, so it is the first quality task.

What we are not claiming

  • A reproduction of the paper. Its kernel results are on B200, RTX PRO 6000 and RTX 5090 GPUs, and its serving results are in vLLM. Ours are in SGLang on one GPU with short fixed-length requests.
  • A repeated result. Most numbers here come from one run per arm. Two runs of one point differed by several percent.
  • An official benchmark result. Two request shapes follow public benchmarks. The model, the system and the method are ours.
  • Fully documented hardware. Our earlier reports record an H100 PCIe. The later ones do not record the GPU.
  • Production readiness or unchanged quality.

What comes next

  • Time the full kernel cycle with the CUDA fold on one build.
  • Run the paper’s tasks on four state formats: fp32, 16-bit, 16-bit with a window, and LeapQuant.
  • Repeat the serving comparisons as paired runs, with every byte of GPU memory accounted for.
  • Measure quality at long context: loss by position, exact recall, and code completion with a repository in context.
  • Find the cost at low load, and settle the fold schedule with the prefix cache on and off.

Each of these will be announced on the blog when its evidence exists, and folded into research note 4. The other four lines of the program continue alongside it.

References

  1. Y. Pan, H. Xi, K. Zhu, et al. LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization. arXiv:2609.38166, 2026. Project page: leapquant.github.io.

The method is the authors’. The implementation and the measurements in this post are ours. Write to us about this experiment.