FastPathLabs

A research lab for large language model inference.

FastPath studies how a request becomes tokens: how it is routed, scheduled, executed and held in memory. We work one experiment at a time. Each mechanism is measured on real hardware against a bar fixed before the run, and the result is published whether or not it clears the bar. No single result is the company; the program is. The work is directed toward an inference stack of our own.

Inside an 8-bit state: one minute on how our LeapQuant implementation stores, reads and folds a recurrent state. Read the post
Subject
Inference for large language models
Layers
Router, scheduler, engine, runtime
Method
Pre-registered experiments, judged on goodput at a latency target
In use
vLLM and SGLang, on NVIDIA H100, A100 and RTX A6000

Latest

One experiment’s news. The other lines continue.

  1. 14 November 2026

    Experiment 4First serving results for an 8-bit recurrent state

    Our implementation of LeapQuant in SGLang, measured against two baselines, with what does not look good yet: a worse tail, a slowdown at low load, and an accuracy baseline still to qualify.

All announcements

The inference stack

Where the work sits. A request travels down seven layers and the tokens return the same way.

Applications

Chat, agents and batch jobs. Each sends a prompt and expects tokens within a latency target.

  • Chat
  • Agents
  • Batch jobs

Gateway

Authenticates the caller, applies limits, and holds the streaming connection.

  • Authentication
  • Rate limits
  • Streaming

Scope of our research

Router

Chooses the replica for each request. The right choice returns a conversation to the replica that already holds its cached state.

  • Replica choice
  • Cache-aware routing
  • Load balancing

Scheduler

Decides what runs in each step: which requests share a batch, how much prefill is admitted, and who waits.

  • Queue
  • Continuous batching
  • Chunked prefill

Engine

Executes the model for a batch. It reads each prompt once (prefill), then writes one token per step (decode).

  • PrefillDecodeSampling
  • KV cache
  • Speculation

Runtime

Turns a model step into GPU work: the compiled graph, the kernels, and the number format of weights and state.

  • Compiled graphs
  • Kernels
  • Quantization

Hardware

The GPU. Its memory capacity, memory bandwidth and compute bound every layer above.

  • Compute
  • Memory bandwidth
  • Interconnect
Figure 1. The layers of an inference deployment. The outlined region is the subject of our experiments. The layers outside it are held fixed and reported with each result.

Findings to date

Stated with their limits. Each links to the note that supports it.

  1. Engine defaults are a competent baseline

    Searching thirty combinations of four vLLM settings on one H100 raised goodput from 3,494 to 3,978 output tokens per second, a factor of 1.14 against a pre-registered bar of 1.5. Two settings produced the whole gain.

    Research note 1

  2. A router can recover only repeated prefill

    On four GPUs, cache-aware routing completed the same chats in 16 to 27% less time than round-robin, and 3 to 6% less than least-loaded routing, which already kept most turns on a warm cache. The saving is bounded by the prefill that would otherwise be repeated.

    Research note 2

  3. Fewer forward passes do not by themselves mean more tokens

    In lab runs a two-token draft brought the cost to 0.43 forward passes per token. It drew level with plain decoding only after verification, acceptance, cache trimming and the next draft were compiled into one graph.

    Research note 3

  4. An 8-bit state pays under load, and the baseline decides by how much

    In SGLang at 512 concurrent requests, our 8-bit recurrent state gave about 45% more output tokens per second than the fp32 default and about 4% more than the 16-bit window state, with a worse tail. At low load it was slower than both. Which of the two is the fair baseline depends on an accuracy test not yet run. One run per arm.

    Research note 4 · Blog post

  5. Throughput alone misleads

    One server produced 3,277 tokens per second with 128 concurrent users and with 256. In the first case 76% of requests met the latency target. In the second, none did.

    Research note 5

Papers

The experiments build on published work. The reading list gives the papers and articles we rely on, grouped by subject, with links to each.

Papers and reading list

Correspondence

We welcome correspondence from those who operate models at scale and from those who study the same questions.