Experiments
Each experiment is written up as a research note: the question, the method, the result, its limits, and the references it rests on. Notes are published whether or not the result cleared its bar, and updated as runs come in. Shorter accounts and announcements are on the blog.
-
Research note 1
10 November 2026
Engine settings on a competent baseline: a pre-registered search
Thirty combinations of four vLLM settings, measured on one H100. The best raised goodput by a factor of 1.14 against a bar of 1.5, so the experiment stopped. Two settings produced the whole gain.
- Status
- Completed
- Hardware
- 1 × H100
- Model
- Qwen3-8B, FP8
-
Research note 2
10 November 2026
Cache-aware routing for multi-turn chat
A router that returns each turn to the worker holding the conversation, compared with round-robin and least-loaded routing on four GPUs. The gain is real, and bounded by the prefill that would otherwise be repeated.
- Status
- Preliminary
- Hardware
- 4 × RTX A6000
- Model
- Qwen2.5-7B
-
Research note 3
11 November 2026
Speculative decoding: tokens per verification and the cost of a round
On a small lab model, fewer forward passes per token did not become more tokens per second until the whole round was compiled into one graph. The note separates the small-model and large-model regimes and gives the design of the run on an H100.
- Status
- Lab result; GPU run specified
- Model
- 16.7M-parameter lab model
-
Research note 4
13 November 2026
Updated 14 NovemberRecurrent-state quantization for linear attention: from a first kernel to first serving results
An implementation of LeapQuant, which stores the recurrent state of linear-attention layers in 8 bits. The first kernel was 7.2 times slower than SGLang’s window kernel; the rewrite is 4.6% slower. In SGLang at 512 concurrent requests it gave about 45% more throughput than the fp32 default and about 4% more than the 16-bit window state, with a worse tail. Whether the 16-bit state is a fair baseline is the next test.
- Status
- In progress
- Hardware
- 1 × Hopper-class GPU
- Model
- Qwen3.5-9B
-
Research note 5
10 November 2026
Measuring goodput: calibration and four guards
How capacity is defined, what calibration showed about throughput and attainment, and four faults in our own measurement that were each larger than the effects under study.
- Status
- In use
- Hardware
- 1 × H100
- Model
- Qwen3-8B, FP8