Prompt processing vs token generation

Two different speeds: how fast a model reads your prompt (pp) versus how fast it writes the answer (tg). Long context and RAG bottleneck on the first.

4 min read Reviewed July 2026

TL;DR

Local-LLM speed is two numbers, not one. Token generation (tg) is the model writing its answer — memory-bandwidth bound, and the tok/s figure people usually quote. Prompt processing (pp) is the model reading your prompt first — compute bound, fast per token, but it scales with prompt length. Short chats barely notice pp; long context and RAG bottleneck on it.

Local-LLM speed is really two numbers, and confusing them leads to surprises. They are usually written pp and tg — also known as prefill and decode.

Prompt processing — parallel one batch Token generation — serial t1 t2 t3 t4 each token waits for the one before it
Prefill ingests the whole prompt in parallel; decode then writes the answer one token at a time

Two speeds, two bottlenecks

Token generation (tg / decode) is the model writing its answer, one token at a time. Each step reads through the (active) weights once, so it is memory-bandwidth bound — this is the tok/s figure people usually quote, and the one the bandwidth rule of thumb predicts. (For an MoE model, each step reads only the active weights — see MoE vs dense.)

Prompt processing (pp / prefill) is the model reading your prompt before it writes anything — the system prompt, the chat history, and any retrieved documents. Unlike decoding, prefill can chew through all those tokens in parallel, so it is compute (FLOPS) bound rather than bandwidth bound, and per token it is usually much faster than generation.

But pp scales with prompt length. Feed in a few thousand tokens of context and you pay a real prefill cost before the first output token appears — the "time to first token".

Prompt processing (pp)

  • What it is: reading the prompt — system prompt, history, documents — before any output
  • Bound by: compute (FLOPS) and kernel quality; tokens are processed in parallel
  • Scales with: prompt length — paid up front, before the first token
  • When it dominates: long context, RAG, large system prompts

Token generation (tg)

  • What it is: writing the answer, one token at a time
  • Bound by: memory bandwidth — the weights are read through every token
  • Scales with: answer length — paid per output token
  • When it dominates: short prompts with long answers — most everyday chat

Why do long context and RAG bottleneck on ingestion?

A short chat prompt makes prefill negligible. But push thousands of tokens through prefill on every request and pp dominates the wait. That happens whenever the prompt itself is the heavy part:

  • Retrieval-augmented generation (RAG) — every request front-loads retrieved documents.
  • Long documents pasted in for summarising or questioning.
  • Large system prompts and long-running chat histories.

At that point a card with strong compute and FlashAttention-class kernels pulls ahead — which is largely an NVIDIA/CUDA advantage today (see GPU software stacks). The KV cache those tokens create also grows the memory bill (Context and the KV cache).

How does this site report the two?

Estimated and measured numbers mean different things here, so the catalog keeps them apart.

Why does the first token take ages, then the rest stream quickly?

That opening wait is prefill. The whole prompt is processed before any output appears; once decoding starts, each token costs about the same. A long pause followed by fast streaming means pp — not tg — was your bottleneck.

Which number matters when choosing hardware?

For everyday chat, decode (tg) — it tracks memory bandwidth. If you run RAG, long documents, or big system prompts, prompt processing matters just as much, and compute plus kernel support come into play (GPU software stacks).