↑ Top

How LLM Inference Works

How an LLM
answers you

You type a prompt and hit enter. What follows is a fall — from an abstract idea of a model, down through the numbers it is made of, down through the software that serves it, all the way to electrons moving through silicon. Scroll to follow one token the whole way down.

6 parts · about 45 minutes · no ML background needed — the math stays in optional asides; every interactive is skippable

Begin the descent
0

Part 0 · The prompt

What just happened when I hit enter?

You typed something and hit enter. A fraction of a second later, words appeared — and it felt like the machine understood you. What actually happened is stranger, and more mechanical.

One token at a time

The model did exactly one thing, very fast, over and over: it read everything so far and predicted the most plausible next token — a whole word, or a fragment of one. Then it appended that token to the text and did it again, and again, until it picked a token that means stop.

Pick a prompt and predict its next token. Everything below this point — the transformer, the weights, the serving stack (the software that runs the model), the silicon — exists to answer that one question, fast: given everything so far, what comes next? You’ll meet each of them on the way down.

A prompt is just text the model continues — one token at a time.

Pick a prompt
The capital of France is?

That is the whole trick — but how did it pick that token? Everything below is the answer.

Pick a prompt, predict its next token, then see what the model actually picked — that gap is the whole trick.

The weights arrive frozen — everything below is inference, the running of a model that was already trained, not the training itself. So how does it pick that token? Follow it down. This is a descent.

1

Part 1 · The transformer

What is the model?

The model that answered you is a transformer. Strip away the branding and it is a tall stack of the same simple operations, applied over and over to a list of numbers. Let’s follow one token through it — from the word you typed to the next token the model chose to say.

Tokenization

Before the model can do anything, it chops your text into pieces it recognizes. Those pieces are tokens — not quite words, not quite letters, but subword chunks learned from data (byte-pair encoding: start from raw bytes and repeatedly merge the most frequent adjacent pair). A common word like the is usually one token; a rarer one like tokenization splits into several. Spaces and punctuation count too.

Each chunk maps to an integer id from a fixed vocabulary of tens of thousands — sometimes a few hundred thousand — entries. From here on the model never sees your letters — only this sequence of ids.

Tokenizer

Text is split into subword tokens, each mapped to an integer id. Edit the sentence and watch the pieces — and their ids — change.

The·transformer·reads·tokenization·as·subword·pieces.
21 tokens/53 chars≈ 2.5 chars / token

Toy splitter — real BPE averages ~4 chars / token, and spaces ride along with the words they precede.

wordsubwordpunctuationspace
Text → subword tokens → integer ids. Edit the text to re-tokenize.
∇ Why models can't count the r's in strawberry

The model never sees letters — strawberry arrives as just one or two opaque ids, so counting its r’s means recalling spelling facts about a chunk, not reading characters off a page. That is why letter-counting and rhyming trip up systems that otherwise write flawless code.

The same design decides who pays more per sentence. Vocabularies are learned mostly from English-heavy text, so a common English word is a single token while the same sentence in Tamil or Burmese can shatter into several tokens per word — costing more context and more money, since APIs bill per token. Numbers are quirky too: 1234 might be one chunk while 1235 splits in two, part of why digit-by-digit arithmetic comes out awkward. Try it above — misspell a word and watch it break into pieces.

Embeddings

An id is just an arbitrary index; on its own it means nothing. So the model’s first move is to look each id up in a big table and replace it with a vector — a list of a few thousand numbers. This is the embedding, where meaning begins.

Tokens with similar meanings get similar vectors, so they sit near each other. More strikingly, directions carry relationships: the step from king to queen is about the same as from man to woman. Click a word to see its nearest neighbors.

One thing a lookup can’t capture is order — the cat and cat the would map to the very same set of vectors. So the model also records where each token sits in the sequence. Modern models fold this into attention itself; older ones stamped a position onto the embedding before the layers begin — see the aside.

Select a word to highlight its nearest neighbours, or toggle the analogy.
A 2-D shadow of embedding space: similar words cluster; directions encode analogies.
∇ 2-D is a cartoon

Real embeddings live in hundreds or thousands of dimensions (e.g. dmodel=4096d_{\text{model}} = 4096). We can’t see that, so the picture above projects down to two dimensions — which inevitably distorts distances. The intuition (neighbors and directions carry meaning) survives even though the exact geometry does not.

∇ How word order gets in

(Attention — and the query and key vectors — arrives in the next few sections; this aside runs a little ahead.) Attention by itself is order-blind — shuffle the tokens and it returns the same answer. Models fix this with positional encoding. Most modern LLMs use RoPE (rotary position embeddings): each token’s query and key vectors are rotated by an angle proportional to its position, so the attention score between two tokens depends on their relative distance. Earlier models simply added a fixed or learned position vector to each embedding. Because RoPE encodes position as a rotation angle, a model trained at one length can be stretched to a longer context by rescaling those angles (position interpolation, YaRN-style) plus a little fine-tuning — which is how 8K-context models grew into 128K-context ones without retraining from scratch.

The stack

With every token now a vector, the real work begins. The model is a stack of N layers of the same shape — dozens in a large model, each with its own learned weights (the numbers fixed during training that don’t change while you chat — a model’s billions of parameters are exactly these weights). Each layer does two things in turn: an attention sublayer, where tokens exchange information, and a feed-forward sublayer, which transforms each token on its own.

Two details keep a stack this deep trainable. A residual connection wraps each sublayer, adding its input back onto its output — so each sublayer only has to learn a small change to the vector, and the original always has a clear path through. And RMSNorm runs before each sublayer — “pre-norm” — to keep the numbers from ballooning or vanishing. The vectors flow down the stack, getting richer at every layer.

12 layers · dModel 768 · 12 heads · 85M params

Counts attention + FFN weights only; embeddings and the LM head are excluded.

token embeddings in
Thecatsaton
Layer 1pre-norm block
Layer 2pre-norm block
Layer 12pre-norm block
final RMSNorm → next-token logits
One pre-norm layer: norm → attention → add, then norm → FFN → add, repeated N times.
∇ Pre-norm and RMSNorm

Pre-norm computes x+Sublayer(Norm(x))x + \text{Sublayer}(\text{Norm}(x)) rather than Norm(x+Sublayer(x))\text{Norm}(x + \text{Sublayer}(x)) — a clean residual highway that makes very deep stacks stable to train. RMSNorm rescales by the root-mean-square, RMSNorm(x)=x1d∑ixi2+ϵ⊙g\text{RMSNorm}(x) = \dfrac{x}{\sqrt{\frac{1}{d}\sum_i x_i^2 + \epsilon}}\odot g — a cheaper cousin of LayerNorm that drops mean-centering.

Attention

Attention is the mechanism that makes transformers work. At each layer, every token looks back at the tokens before it (itself included) and decides how much to borrow from each — it can’t peek ahead, and that one restriction is exactly what lets the model generate left to right: when it’s choosing the next token, the tokens after it don’t exist yet, so looking only backward means reading your prompt and extending it work the very same way. A pronoun reaches back to the noun it refers to; a verb pulls in its subject. These attention weights are soft — spread across many tokens, more on some, less on others.

Click a token in the fan to make it the one doing the looking. Warmer, thicker lines show where its attention flows; switch heads — each head is the same attention run with different learned weights, so it picks up a different kind of relationship (the next section builds one up) — to see a completely different pattern.

Head:
Query: mat · Each token reaches back to related earlier words — noun←adjective, verb←subject.hover or focus a key token
Every token looks only at the tokens before it, leaning on some far more than others — line warmth and width show where its attention lands.

Queries, keys, values — and many heads

How does a token decide where to look? It projects its vector into three — each projection just multiplies the vector by a grid of learned weights (a matrix) to get a new list: a query (what am I looking for?), a key (what do I offer?), and a value (what will I hand over?). It scores its query against every key with a dot product (one number for how well they match), scales the scores down, and runs them through a softmax to turn them into attention weights — then takes the weighted sum of the values.

One such computation is a head; a layer runs many in parallel, each specializing. A modern twist — GQA/MQA — lets several query heads share one set of keys and values, shrinking the KV cache — the running store of keys and values for every token so far, which we size in Part 2 and exploit in Part 3. Remember this; it pays off later.

Head:Q = K projection — each query attends to tokens like itself.
Pick a query token — each token is projected into a Query, Key and Value (d = 4).
QKV
QKV
QKV
QKV
Query tea · scores = (Q·Kᵀ)/√d → softmax → Σ wᵢ·Vᵢ
she+0.005%
poured+2.0034%
the+0.005%
tea+2.5056%
output =[0.14, 1.25, 0.14, 1.47]weighted blend of the V's
KV sharing:
Query heads share K/V in groups.
K/V 0
Q0Q1Q2Q3
K/V 1
Q4Q5Q6Q7

8 query heads · 2 K/V sets → 4× smaller KV cache (the saving pays off in Part 3).

Q·Kᵀ → scale → softmax → weighted sum of V, per head. Toggle GQA to share K/V across heads.
∇ Dot products and softmax, intuitively

A dot product lines two vectors up, multiplies them position by position, and adds it all into one number — large when they point the same way, near zero when they don’t — so the score reads as how well a query matches a key. A softmax then squashes a row of those raw scores into positive weights that sum to 1, so they behave like proportions: a soft blend that leans hardest on the closest matches.

∇ The attention equation

For a head with key dimension dkd_k: Attention(Q,K,V)=softmax ⁣(QK⊤dk)V.\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V. The dk\sqrt{d_k} divisor stops the dot products from growing with dimension and saturating the softmax — piling nearly all the weight onto a single token (a near one-hot spike: one entry close to 1, the rest near 0). That flattens the function locally, leaving training almost no signal to learn from.

The feed-forward network — and Mixture-of-Experts

Between attention sublayers sits the feed-forward network: a couple of big matrix multiplies (those grids of weights again) with a nonlinearity between — a simple bend that lets the layer learn more than a straight line could — run on each token independently. This is where most of a model’s parameters live, and much of its stored knowledge with them.

Mixture-of-Experts replaces that single FFN with many expert FFNs plus a small router that, per token, picks only the top few. So a model can carry an enormous total parameter count while running a small active fraction per token — say a few billion active of a hundred-plus billion total.

Mixture-of-experts router

The router scores every expert; only the top-k actually run for this token. Most parameters sit dormant.

Input token
Expert 0 · gate 0.6% · dormantExpert 1 · gate 0.9% · dormantExpert 2 · gate 1.0% · dormantExpert 3 · gate 0.9% · dormantExpert 4 · gate 0.6% · dormantExpert 5 · gate 0.4% · dormantExpert 6 · gate 0.2% · dormantExpert 7 · gate 0.2% · dormantExpert 8 · gate 0.2% · dormantExpert 9 · gate 0.3% · dormantExpert 10 · gate 0.4% · dormantExpert 11 · gate 2.3% · ACTIVEExpert 12 · gate 25.3% · ACTIVEExpert 13 · gate 1.4% · dormantExpert 14 · gate 0.2% · dormantExpert 15 · gate 0.1% · dormantExpert 16 · gate 0.1% · dormantExpert 17 · gate 0.2% · dormantExpert 18 · gate 0.3% · dormantExpert 19 · gate 0.5% · dormantExpert 20 · gate 0.6% · dormantExpert 21 · gate 0.6% · dormantExpert 22 · gate 0.5% · dormantExpert 23 · gate 0.4% · dormantExpert 24 · gate 0.3% · dormantExpert 25 · gate 0.3% · dormantExpert 26 · gate 0.4% · dormantExpert 27 · gate 0.6% · dormantExpert 28 · gate 0.9% · dormantExpert 29 · gate 1.0% · dormantExpert 30 · gate 0.9% · dormantExpert 31 · gate 0.6% · dormantExpert 32 · gate 0.4% · dormantExpert 33 · gate 0.2% · dormantExpert 34 · gate 0.2% · dormantExpert 35 · gate 0.2% · dormantExpert 36 · gate 0.3% · dormantExpert 37 · gate 0.4% · dormantExpert 38 · gate 0.4% · dormantExpert 39 · gate 0.4% · dormantExpert 40 · gate 0.3% · dormantExpert 41 · gate 0.2% · dormantExpert 42 · gate 0.1% · dormantExpert 43 · gate 0.1% · dormantExpert 44 · gate 0.2% · dormantExpert 45 · gate 0.3% · dormantExpert 46 · gate 1.6% · dormantExpert 47 · gate 13.3% · ACTIVEExpert 48 · gate 2.1% · dormantExpert 49 · gate 0.5% · dormantExpert 50 · gate 0.4% · dormantExpert 51 · gate 0.3% · dormantExpert 52 · gate 0.3% · dormantExpert 53 · gate 0.4% · dormantExpert 54 · gate 0.6% · dormantExpert 55 · gate 0.9% · dormantExpert 56 · gate 1.1% · dormantExpert 57 · gate 0.9% · dormantExpert 58 · gate 0.6% · dormantExpert 59 · gate 0.4% · dormantExpert 60 · gate 0.2% · dormantExpert 61 · gate 0.2% · dormantExpert 62 · gate 0.2% · dormantExpert 63 · gate 0.3% · dormantExpert 64 · gate 0.4% · dormantExpert 65 · gate 0.5% · dormantExpert 66 · gate 0.4% · dormantExpert 67 · gate 0.3% · dormantExpert 68 · gate 0.2% · dormantExpert 69 · gate 0.1% · dormantExpert 70 · gate 0.1% · dormantExpert 71 · gate 0.2% · dormantExpert 72 · gate 0.3% · dormantExpert 73 · gate 0.4% · dormantExpert 74 · gate 0.6% · dormantExpert 75 · gate 0.6% · dormantExpert 76 · gate 0.5% · dormantExpert 77 · gate 0.3% · dormantExpert 78 · gate 0.3% · dormantExpert 79 · gate 0.3% · dormantExpert 80 · gate 0.4% · dormantExpert 81 · gate 0.6% · dormantExpert 82 · gate 0.9% · dormantExpert 83 · gate 1.1% · dormantExpert 84 · gate 0.9% · dormantExpert 85 · gate 0.6% · dormantExpert 86 · gate 0.4% · dormantExpert 87 · gate 0.7% · dormantExpert 88 · gate 2.5% · ACTIVEExpert 89 · gate 0.7% · dormantExpert 90 · gate 0.3% · dormantExpert 91 · gate 0.4% · dormantExpert 92 · gate 0.5% · dormantExpert 93 · gate 0.4% · dormantExpert 94 · gate 0.3% · dormantExpert 95 · gate 0.2% · dormantExpert 96 · gate 0.1% · dormantExpert 97 · gate 0.1% · dormantExpert 98 · gate 0.2% · dormantExpert 99 · gate 0.3% · dormantExpert 100 · gate 0.4% · dormantExpert 101 · gate 0.6% · dormantExpert 102 · gate 0.5% · dormantExpert 103 · gate 0.4% · dormantExpert 104 · gate 0.3% · dormantExpert 105 · gate 0.3% · dormantExpert 106 · gate 0.3% · dormantExpert 107 · gate 0.4% · dormantExpert 108 · gate 0.7% · dormantExpert 109 · gate 1.0% · dormantExpert 110 · gate 1.1% · dormantExpert 111 · gate 0.9% · dormantExpert 112 · gate 0.6% · dormantExpert 113 · gate 0.4% · dormantExpert 114 · gate 0.3% · dormantExpert 115 · gate 0.2% · dormantExpert 116 · gate 0.3% · dormantExpert 117 · gate 0.4% · dormantExpert 118 · gate 0.5% · dormantExpert 119 · gate 0.5% · dormantExpert 120 · gate 0.4% · dormantExpert 121 · gate 0.2% · dormantExpert 122 · gate 0.2% · dormantExpert 123 · gate 0.1% · dormantExpert 124 · gate 0.1% · dormantExpert 125 · gate 0.2% · dormantExpert 126 · gate 0.3% · dormantExpert 127 · gate 0.4% · dormant
128 experts — router gate weight
Experts per token (top-k)
routed →E1258%E4731%E886%E115%

(gate weights renormalized over the chosen k)

Active params / token
~5.1B
4 of 128 experts run
Total params stored
~117B
4.4% active per token
share of params active per token
The router scores experts for a token and activates only the top-k; the rest stay dormant.
∇ What the FFN actually computes

Most current models use a gated feed-forward block (SwiGLU) rather than the classic two-matrix one: FFN(x)=(Swish(xWgate)⊙xWup) Wdown,\text{FFN}(x) = \big(\text{Swish}(xW_{\text{gate}}) \odot xW_{\text{up}}\big)\,W_{\text{down}}, three projections, with the gate path supplying the nonlinearity. The hidden dimension is several times dmodeld_{\text{model}} — which is why this block holds the lion’s share of the parameters.

From logits to a token

At the top of the stack, the vector at the last position — the one that has been gathering context across the whole prompt — is projected back out to the whole vocabulary, giving one score — a logit — for every possible next token. A softmax turns those scores into probabilities, just as it did inside attention. But we rarely just take the most likely one; that gets repetitive.

Temperature flattens or sharpens the distribution. Top-k keeps only the k most likely candidates. Top-p (nucleus) keeps the smallest set covering probability p. Then we sample one token — drawn at random, each weighted by its probability — from what survives. Drag the sliders and watch the distribution reshape.

The cat sat on the ___12 of 12 tokens kept
  • mat46.6%
  • floor22.0%
  • sofa11.8%
  • rug7.1%
  • couch4.9%
  • chair3.0%
  • bed1.8%
  • lap1.2%
  • table0.8%
  • windowsill0.4%
  • grass0.3%
  • roof0.1%
draws one token under the current distribution
The next-token distribution, reshaped by temperature / top-k / top-p before sampling.
∇ Temperature, precisely

Sampling starts from a softmax with a temperature TT: pi=ezi/T∑jezj/T.p_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}. As T→0T \to 0 the distribution collapses onto the single most-likely token (greedy decoding); T=1T = 1 is the model’s raw distribution; T>1T > 1 flattens it toward uniform. Top-k and top-p then truncate this distribution — discarding the unlikely tail — before the draw.

Autoregression

Now the loop. Append the chosen token to the sequence and feed the whole thing back in to predict the next — then again, and again, one token at a time, until the model emits a stop token or hits a limit. “Generating” is just this same forward pass — one full sweep of the tokens down through every layer to a next-token distribution — run over a sequence that grows by one each step.

Notice the waste: every step re-reads the entire context from scratch. The fix — caching what has already been computed — is the whole drama of Part 3.

AUTOREGRESSIONone token per step
step 3 / 7
Context fed in this step — 5 tokens re-read
Onceuponatimethere
transformeremits
there
append the new token → feed the longer context back in → repeat

The context grows by one token every step, so the work per step grows too.

Append the sampled token, feed it back, repeat. The context grows by one each step.
∇ Why there's a context window

The loop can’t run forever. Every model has a context window — a hard ceiling on how many tokens it can hold at once, from a few thousand in early models to hundreds of thousands or a million today. Everything competes for that one budget: the system prompt, your whole conversation so far, any documents you paste, and every token the model has generated — which is why a very long chat eventually gets truncated or seems to forget how it began.

The ceiling has two causes. Positions are only trained — or later stretched — up to a certain length, so attention degrades past it. And the running store of keys and values grows with every token held, so both memory and the per-step reading cost climb as the window fills — Parts 2 and 3 put numbers on exactly that. When an app says context limit exceeded, this is the window overflowing.

So that is the model: tokens become vectors, vectors flow through a stack of attention and feed-forward layers, the top of the stack produces a distribution, and we sample from it on a loop. But we have been waving our hands at those “vectors” and “parameters.” What are they, underneath? Just numbers — billions of them. Time to descend into how they are stored.

2

Part 2 · Weights as numbers

How are the weights represented?

We keep calling the model “numbers.” This part takes that literally. Underneath the abstractions of Part 1, a model is just billions of stored values, and the question now is how to make them all fit: first by writing each number more cheaply, then by a different route that trains a smaller model in the first place.

Zoom into a weight

Zoom out and a large model is a few hundred matrices stacked into layers. Zoom into one layer and you find the attention and feed-forward weight matrices. Zoom into one matrix and it is a grid of numbers. Zoom into one cell and it is a single value — a weight, like 0.0123.

There are billions of them, and “running” the model is mostly multiplying these matrices by the vectors flowing through — each output number a weighted blend of the whole input vector. So the question for this whole part is concrete: how do you store one of these numbers, cheaply, without breaking the model?

It is all just numbers

A model is billions of numbers in matrices. Zoom in until a single weight is just one float.

12 layers

A model is a tall stack of layers — billions of numbers in total, nothing more.

1 / 4
Zoom in: model → layer → matrix → one weight. It is numbers all the way down.

Floating point

The default answer is a floating-point number: a sign bit, an exponent (the scale), and a mantissa (the significant digits). It is just scientific notation in binary — the mantissa holds the significant digits and the exponent is the power of two that slides the point, the way 6.02×10236.02\times10^{23} splits a number into digits and a scale. Full precision is 32 bits — the original full-precision baseline, and overkill for inference: once the weights are fixed, running them to generate text tolerates far coarser numbers than training them did. Sixteen bits halves it, in two flavors. FP16 spends 5 bits on exponent and 10 on mantissa; BF16 keeps FP32’s full 8-bit exponent — its huge dynamic range, the span from the tiniest to the largest magnitude it can represent — and trades mantissa bits for it.

That trade is why BF16 is the modern default: for deep nets, range matters more than the last digits of precision — a weight rounded a hair off rarely hurts, but one that overflows to infinity or underflows to zero breaks the math. Newer hardware pushes further still — to 8-bit floats for the heaviest matmuls (matrix multiplications, the bulk of the work): FP8 comes as E4M3 (4 exponent bits, 3 mantissa) or E5M2 (the reverse trade). And FP8 is no longer only a compute format: by 2026 the datacenter routinely stores and serves the weights themselves at E4M3 — one byte per weight, so a 70B model needs roughly 70 GB — which is the serving precision Parts 4 and 5 quietly assume. Flip the bits below and watch the value change.

Format

Click any bit to flip it

sign(1 bit)exponent(8 bits)mantissa(23 bits)
Value0x3E200000
0.15625FP32 · normal
Load
Exponent sets the range, mantissa the detail — flip bits, or switch FP16 ↔ BF16, to see which digits survive.
∇ How a float becomes a value

A floating-point number is read as (−1)sign×1.mantissa×2 exp−bias(-1)^{\text{sign}} \times 1.\text{mantissa} \times 2^{\,\text{exp} - \text{bias}}. The field widths set the trade-off: FP32 is 1/8/23, FP16 is 1/5/10, BF16 is 1/8/7. BF16 keeps FP32’s 8 exponent bits — and therefore its range — while dropping mantissa bits, so it covers the same magnitudes with coarser steps.

Quantization

We can go further by leaving floating point behind. Quantization stores each weight in fewer bits — 8, 4, even 2 — mapping the range of values onto a small set of levels, with a single scale factor stretching those few levels to cover the real spread of the weights. You lose precision, but a surprising rule holds: a bigger model at lower precision usually beats a smaller model at higher precision, and the quality loss from quantizing is smaller on bigger models — they carry more redundancy, so rounding any single weight matters less.

Formats like k-quants (in GGUF, the file format behind most run-it-on-your-laptop models) even mix precision per layer, spending bits where they matter. Step the precision down and watch the distribution stair-step — and the model size fall with it.

Weight quantization

Lower precision snaps every weight onto a few discrete levels — smaller files, coarser values.

weight value →↑ count
Precision
Model size
On-disk size
3.50 GB
0.5 bytes / param
Quality (illustrative)
98.3
proxy — not a benchmark
Drop the precision: the smooth distribution stair-steps into discrete levels, and the size falls.
∇ The quantization map

Integer quantization stores a weight as q=round⁡(w/s)+zq = \operatorname{round}(w/s) + z — a scale ss and an optional zero-point zz — and recovers it as w^=s (q−z)\hat{w} = s\,(q - z). Fewer bits means fewer levels (2b2^b of them), so the step ss is coarser and the rounding error larger. The whole craft is choosing ss, and how finely to share it, so the levels land where the weights actually sit.

Block scaling: how 4-bit survives

Four-bit numbers — a plain FP4 float (E2M1: one sign bit, two exponent, one mantissa) — have a problem: a single scale for an entire tensor — one whole weight matrix, the grid from the zoom — wastes most of its range wherever values are small. The block formats built on FP4 — MXFP4 / NVFP4 — fix it with block scaling: chop the tensor into small blocks of 32 values and give each block its own scale factor. Each block now spends its few bits on its own local range, so error drops sharply where magnitudes differ.

Toggle below between one shared scale and per-block scales to watch the error collapse.

∇ What MXFP4 actually stores

MXFP4 stores each value as a 4-bit float (E2M1) and, per block of 32, a shared 8-bit power-of-two scale (E8M0); NVFP4 uses 16-value blocks with an FP8 scale. The shared scale costs a fraction of a bit per value but recovers most of the accuracy of formats several bits wider.

Block scaling (MXFP4 / NVFP4)

Low-bit formats give each block of values its own shared scale instead of one scale for the whole tensor. Cells are colored by quantization error — warmer means more error.

Scale granularity
lowhigh

Shown: 4 blocks of 8 values. Real MXFP4 blocks hold 32 values.

Per-tensor error
0.0080
mean abs error
Per-block error
0.0031
mean abs error

Block scaling cuts mean error by 61% here, because each block's scale tracks its own local magnitude instead of being dragged coarse by the loudest region of the tensor.

One scale for the whole tensor wastes range; a scale per block recovers it.
∇ Reading a GGUF filename

That Q4_K_M suffix on a model you download decodes piece by piece. Q4 means roughly four bits per weight; K means the k-quant scheme — blocks of weights sharing scales, grouped into 256-value super-blocks whose scales are themselves quantized: block scaling, exactly as above; and S, M, or L says how generously the sensitive tensors (attention and output layers) keep higher precision. In practice a Q4_K_M 7B file averages ~4.5 bits per weight — 4–5 GB, matching the ~0.6 rule below — and is the community’s default quality-for-size sweet spot; Q8_0 is near-lossless at twice the size, Q2_K audibly degraded. The newer IQ (“i-quant”) family adds an importance matrix computed from calibration text, spending its bits on the weights that matter most.

The memory budget

Now the budget. Weight memory is just params × bytes per weight: 7 billion params at FP16 (2 bytes) is 14 GB; at Q4 (4-bit, half a byte each) it is ~3.5 GB in theory — though real 4-bit formats also store per-block scales, which nudges that up. A safe rule of thumb is a model’s size in billions times ~0.6 ≈ its Q4 footprint in gigabytes (so 7B ≈ 4 GB).

But weights are not the whole story: the KV cache — the running scratchpad of keys and values the model keeps for every token so far, so it needn’t recompute them (Part 3) — sits on top and grows with context length and batch (how many separate sequences run through the model at once). The bar stacks weights and KV cache against a card’s VRAM — the memory soldered onto the GPU itself, separate from your system RAM and all you get — pick the knobs and see whether it fits.

Memory budget — does it fit in VRAM?

Weights plus a growing KV cache, stacked against your GPU’s capacity. Overflow reads hot.

16 GBRTX 5060 Ti
weightsKV cacheover capacity
Model size
Precision
GPU

RTX 5060 Ti — entry consumer GPU

Weights
3.50 GB
KV cache
0.54 GB
Total
4.04 GB
Fits — headroom
12.0 GB

KV cache held at FP16, single sequence (batch 1) — the precision above applies to weights only.

Rule of thumb: sizeB × 0.6 ≈ Q4 GB  →  7.0B × 0.6 = 4.2 GB (weights only; the rule’s extra ~0.1 byte/param over raw 4-bit is the per-block scales real formats store)

Illustrative capacities, ~2026 — not vendor specs.

Weights plus KV cache have to fit under the card's VRAM — pick a model, precision, context, and card to see if they do.
∇ Sizing the KV cache

The KV cache stores a key and value vector for every token seen so far: KV bytes≈2×nlayers×nkv×dhead×seqlen×batch×bytes.\text{KV bytes} \approx 2 \times n_\text{layers} \times n_\text{kv} \times d_\text{head} \times \text{seqlen} \times \text{batch} \times \text{bytes}. The leading 2 is K and V. It grows linearly with context length and batch — and because it scales with nkvn_\text{kv} (the number of key/value heads), this is exactly the term GQA/MQA shrink, back in Part 1.

∇ Quantizing the KV cache too

Everything above quantized the weights, but the KV cache is stored bytes too — and at long context and high batch it can outgrow them. Serving engines increasingly keep keys and values at FP8 instead of FP16: the bytes\text{bytes} factor in the sizing formula above drops from 2 to 1, halving the cache — which buys double the batch or double the context on the same card. Keys are touchier than values — their dot products against queries amplify rounding error — so engines sometimes hold keys a notch higher, or give them finer scales. The engines in Part 3 expose this as a one-line flag; the sandboxes here and in Part 5 assume FP16 KV, the conservative default.

A different route: distillation

Quantization shrinks a model by storing it more cheaply. Distillation is a different route entirely: train a small student to imitate a big teacher’s outputs — often its step-by-step reasoning traces. The student never sees the teacher’s weights, only its behavior — sometimes just its sampled text, sometimes its full next-token probabilities (the dark knowledge in the options it didn’t pick). And yet a surprisingly small number of examples, sometimes ~1,000, can transfer a lot of capability.

Quantization compresses; distillation transfers. They are often combined: distill, then quantize.

Distillation — the other route to small models

Train a small student on a big teacher’s answers and reasoning traces. This transfers capability — distinct from quantization, which only compresses an existing model’s numbers.

Teacher70B paramsbig model1K outputs / reasoning tracesStudent7B params
Capability transferred (illustrative)
50%
illustrative curve — saturating, with diminishing returns. Not a benchmark.
A different route: a small student learns from a big teacher's outputs, not its weights.

Between them, lower precision and distillation are how a frontier-scale model gets onto a single GPU — or your laptop. We now know what the weights are and how much room they take. Next: the software that actually runs them, fast.

3

Part 3 · Inference: software

How is it served?

Between the math and the metal sits the inference engine — the software that actually runs the model for you. It is, underneath, a memory manager and a scheduler: its whole job is to squeeze a memory-bound workload — one that spends its time waiting on memory rather than on arithmetic, generated one token at a time — onto hardware that craves big parallel batches. Its central drama is an asymmetry.

Prefill vs decode

Running the model has two separate costs that pull apart here: reading each weight from memory, and doing arithmetic with it once it is loaded. Reading your prompt — prefill — happens in one big parallel pass over all the prompt tokens at once, so each weight, once read, is reused across every token. That makes it compute-bound — the arithmetic is the bottleneck, not the reading — and it sets the time to first token. Generating the answer — decode — emits one token at a time, each step depending on the last. It is memory-bound: producing a single token still means reading every one of the model’s weights (or, for a Mixture-of-Experts model, all of its active ones) but doing only a little arithmetic with each, so the limit is how fast they stream from memory, not the chip’s raw FLOP/s (floating-point operations per second — its arithmetic rate) — and this sets the speed between tokens. That is why a chat answer types — the app streams each token to your screen the instant decode produces it, so the typing rate you watch is this gap made visible.

To avoid recomputing the past at every step, the engine keeps a KV cache: the attention keys and values for every token already seen. Without it, each new token would reprocess the whole sequence — so writing an n-token reply is work that grows with the square of its length (O(n²)). The cache means each step runs the network on just the one new token instead of re-running all n, dropping that recompute to a constant per step (O(1)); only the attention scan over the cached keys and values still grows with the sequence. That cache is the memory pressure — it swells with context and batch — which is why GQA / MQA (sharing keys and values across heads, as Part 1 promised) — and MLA, which compresses them into a smaller shared latent — pay off right here.

PREFILLCOMPUTE-BOUND
frame 0 / 6
Prompt — processed in one parallel pass
Thecatsatonthe
Generated — one token per step (autoregressive loop)
— not started —
KV cache · rows = layers, cols = tokensreused each step
token-processings to reach here0cached · O(n) · 1 new token / step

32 query heads, 32 KV heads — full multi-head attention

KV cache 2.15 GBfull MHA would be 2.15 GB→ full multi-head attention (no sharing)
Prefill fills the KV cache in parallel; decode ticks one token at a time. Toggle the cache off to see the waste.
∇ The three numbers on every latency dashboard

You just met all three of a serving dashboard’s numbers without their names. TTFT — time to first token — is the prefill bill: how long the compute-bound pass over your prompt takes before anything appears, growing with prompt length and shrinking when a cached prefix skips the work. ITL — inter-token latency, also written TPOT — is decode’s memory-bound tick: the gap between tokens, set by how fast weights and KV stream from memory. Multiply across the whole batch and you get throughput — but an operator can juice throughput by packing the batch until every user’s ITL degrades, so serious dashboards track goodput: throughput that still meets a latency target. Every trick in this part moves one of these dials.

∇ The reasoning tax — when the model thinks before it speaks

A reasoning model answers your one-line question by first generating a hidden chain of thought — often thousands of tokens you never see, every one produced by decode, the slow memory-bound phase. That inverts the classic serving profile: instead of long-prompt-in, short-answer-out, reasoning workloads are decode-heavy, so the bandwidth law, the swelling KV cache, and every decode trick in this part matter more, not less. You pay for the invisible tokens too — providers bill thinking tokens at output rates — and your effective TTFT, the wait before the first visible token, stretches from milliseconds to minutes, because your answer stands in line behind the entire hidden trace. This is why 2026 pricing pages break out reasoning tokens as their own line item, and why models ship “effort” knobs: the knob is literally how much decode you are willing to buy.

Batching & scheduling

A single memory-bound stream barely keeps the GPU busy — its arithmetic units mostly idle, waiting on the next read of weights — so engines run many sequences at once, sharing each weight read across all of them (which tips decode back toward compute-bound). A GPU is happiest with a big batch anyway. But sequences finish at different times, and naive static batching makes the whole batch wait for its slowest member, leaving finished slots idle. Continuous batching schedules per iteration: the instant a sequence finishes, a waiting one takes its slot. Utilization climbs.

Two tricks ride along. Chunked prefill breaks a long prompt into pieces and interleaves them with ongoing decodes, so one big prefill doesn’t stall everyone. And prefix caching reuses the KV of a shared prompt across requests — a system prompt (the fixed instructions silently prepended to every chat) is computed once, not per user.

utilization52%
static
52%
continuous
93%
continuous packs +41 points of slot utilization
rows = slots · columns = decode iterationsiter 9 / 9
seq 0·5 itseq 1·2 itseq 2·1 itseq 3·4 itseq 4·2 itidle

Chunked prefill interleaves long prompt prefills with ongoing decodes so a big prompt never stalls the batch; prefix caching reuses the shared prompt’s KV across requests.

Static batching idles finished slots until the whole batch drains; continuous batching refills them at once.
∇ Disaggregated serving — splitting prefill from decode

Prefill is compute-bound and decode is memory-bound, so newer systems (NVIDIA Dynamo, Moonshot AI’s Mooncake) run them on separate pools of GPUs — a prefill tier and a decode tier — and stream the KV cache between them over fast interconnect. Each tier is then sized and batched for its own bottleneck instead of compromising on one box. The same instinct drives KV-cache offloading: spill cold cache down to CPU RAM or SSD and page it back, serving far more concurrent users than VRAM alone would hold.

PagedAttention

The KV cache has a memory-management problem: sequences grow unpredictably, so reserving a contiguous block of max length per sequence wastes enormous space. PagedAttention borrows the operating system’s answer — virtual memory. The cache is cut into fixed-size blocks; a block table maps each sequence’s logical positions to physical blocks, handed out on demand.

Fragmentation nearly vanishes, and two sequences that share a prefix can point at the very same physical blocks. Switch between contiguous and paged to watch the wasted blocks collapse.

Paged KV cache

4 tokens per block · block table maps logical → physical

Seq 05 tok
·
·
·
·
Seq 110 tok
·
·
·
·
Seq 23 tok
·
·
·
·
Physical block pool · 12 blocksreserved up front
internal fragmentation6whole blocks reserved but never used (of 12)
Fixed-size blocks + a block table map logical positions to physical blocks — fragmentation collapses, prefixes share.

The engines

All of this is packaged into inference engines, each with a personality. llama.cpp is the portable one — runs the GGUF format anywhere, ideal for a single user on local hardware. vLLM is the production workhorse — PagedAttention and continuous batching for many concurrent users. SGLang leans into prefix caching (RadixAttention) and structured, agentic workloads — long chains of tool-calling steps, and outputs forced into a fixed shape like JSON, where big shared prefixes repeat across requests. TensorRT-LLM squeezes maximum throughput from NVIDIA hardware — a PyTorch-first runtime now, with an optional ahead-of-time engine build for the last increment of speed. And when one box isn’t enough, NVIDIA Dynamo spreads serving across many GPUs — the disaggregated prefill/decode split from the aside above.

Inference engines — the landscape

A model is just numbers until an engine runs it. A handful dominate, each shaped by a different priority. Pick one to compare its character.

llama.cppillustrative ratings, not benchmarks
Best for
Single-user, local & on-device inference
Key tech
GGUF format + portable C/C++ (CPU, Metal, many backends)
  • Portability
    5
  • Multi-user
    1
  • Prefix caching
    2
  • Throughput
    2
A handful of engines, each with a personality. Pick one to see what it is for.
∇ Forcing JSON — masking the logits

That “forced into a fixed shape like JSON” has a concrete mechanism, and it lives at the sampling step you met in Part 1. At every decode step the model proposes logits over the whole vocabulary; a grammar engine masks out every token that would break the required format before the draw, so the model cannot emit an illegal one. Your JSON schema or regex is compiled into a state machine that tracks where in the grammar the output stands and produces each step’s legal-token mask — this is what libraries like Outlines and XGrammar do, and what SGLang builds in. Because the mask lands on logits the model already computed, well-formed output costs almost nothing extra on the GPU; the catch is that masking bends the distribution — the model can be railroaded into continuations it never ranked highly.

Where it runs

Those engines are one option among three. The same open-weights model can run locally on your own machine (Ollama, llama.cpp, LM Studio, MLX) — private and free after the download, but capped by your RAM and its memory bandwidth — how fast that memory feeds the chip (Part 4 makes this precise); self-hosted, a serving engine on GPUs you rent or own (vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo) — full control and the best cost at scale, but you own the ops; or behind a managed API, where someone else runs the GPU and you just call an endpoint (Together, Fireworks, Groq, Bedrock) — zero ops and instant scale, paid per token. Pick a lane and weigh what you’re trading.

Where it runs

The same model, three places to run it — trading control for convenience.

Localillustrative, not a ranking — ~2026

Run it on your own machine — private, offline, free after the download. Capped by your RAM and memory bandwidth.

Representative tools
  • Ollamaone-command models; now an MLX backend on Apple Silicon
  • llama.cppportable C/C++ core; the GGUF format; runs almost anywhere
  • LM Studioa desktop GUI over llama.cpp / MLX
  • Apple MLXfastest on Apple Silicon; bandwidth-bound on big models
  • Control / privacy
    5
  • Convenience
    4
  • Cost at scale
    2
  • Latency
    3
  • Model choice
    4
Local buys privacy, self-hosted the best cost at scale, the API zero ops — pick what you're optimizing for.

Speculative decoding

Decode is memory-bound, so the GPU’s compute sits half-idle waiting on memory — spare capacity we can spend. Speculative decoding does exactly that: a small, cheap draft model guesses the next k tokens, and the big target verifies all k in a single forward pass. It can, because the k guesses are already in hand: checking them all at once is one parallel, compute-bound pass (just like prefill) that costs about as much as generating a single token — and it runs on the compute decode was leaving idle. Where the draft guessed right, the tokens are kept for free; at the first miss, we resample from the target and continue.

The win depends on the acceptance rate and on the draft being much cheaper than the target — push either the wrong way and the speedup can dip below 1.

∇ Why it stays correct, and how much it gains

Rejection sampling makes the output provably identical to sampling from the target alone — same distribution, fewer passes. With per-token acceptance probability α\alpha and draft length kk, the expected tokens produced per target pass is E=1−α k+11−α\mathbb{E} = \dfrac{1 - \alpha^{\,k+1}}{1 - \alpha}. Variants like Medusa, EAGLE, and MTP fold the draft into the target itself.

Speculative decoding — draft + verify

A cheap draft model proposes k tokens; the big target verifies all k in one forward pass. Rejection sampling keeps the accepted prefix (preserving the target’s exact distribution), resamples the first miss, and adds a free bonus token when every draft token survives.

Draft proposes 5 tokens
the model drafts a few
Target verifies in one pass
the model draftsresample
Accepted prefix: 3 of 5 — first miss resampled by the target.
Expected tokens / target pass
2.94
includes the bonus token (1 → 6 possible)
Speedup vs. plain decode
1.68×
draft costs 15% of a target pass; < 1× means speculation hurts.

Variants make the draft nearly free: Medusa adds extra prediction heads, EAGLE drafts in feature space, and MTP (multi-token prediction) trains the target to propose its own continuations. Lower draft cost or higher α both push the speedup up.

A cheap draft guesses k tokens; the target verifies all k in one pass. Slide the acceptance rate.

The engine, then, is a memory manager and a scheduler in a trench coat — bending a memory-bound, autoregressive workload onto hardware that wants big parallel batches. But what is that hardware, and why does it want what it wants? Down one more level: the silicon.

4

Part 4 · Inference: hardware

How does the silicon run it?

We have reached the floor. Everything above — the transformer, the weights, the serving engine — ultimately runs as electrons moving through one chip. A GPU is a wall of tensor cores, thousands of tiny matrix-multiply units, fed by a steep hierarchy of memory. Almost every performance story down here reduces to one tension: the cores can compute far faster than memory can feed them. Keeping them fed is the whole game.

The GPU

Open one up and you find a grid of streaming multiprocessors (SMs), each packed with tensor cores built to multiply small matrices. Around them sits memory in tiers, and the tiers trade size against speed brutally. HBM — the GPU’s main memory, the headline “192 GB” — is large but comparatively slow: it delivers bytes at roughly 8 TB/s, its bandwidth. This is the VRAM you budgeted against in Part 2, the same pool: its size decides what fits, its bandwidth decides how fast decode runs. On-chip SRAM is roughly ten to thirty times faster, but measured in kilobytes per SM, not gigabytes.

A value in HBM is far away; a value in SRAM is right next to the core. Every fast kernel — a routine that runs on the GPU — is, at heart, a scheme for hauling the right data up this pyramid and reusing it before it falls back.

A GPU — many cores, fed by a steep memory hierarchy

B200: 148 SMs × 4 tensor cores = 592 cores, behind 192 GB of HBM. The higher up the memory pyramid, the faster — and the smaller. Keeping the cores fed is the whole game.

HBMHBM
148 SMs · tensor cores hinted as dot clusters
HBM192 GB · 1xL2 cache96 MB · 4xSRAM / shared228 KB · 20xregisters1 KB · 100xfaster ↑ smallerslower ↓ bigger
HBM ~8 TB/s baseline · SRAM ~10-30x faster, KBs not GBs
Inspect a layer
HBMsize 192 GBvs HBM 1xbandwidth ~8 TB/s
Vast off-chip pool — where the weights and KV cache live. GBs, but the slow road.
A grid of SMs and tensor cores, fed by a memory pyramid: vast slow HBM at the base, tiny fast SRAM at the tip.

The accelerator landscape

You just toured one GPU; here is the field of them. Strip away the marketing and every accelerator comes down to two numbers: memory capacity — does the model even fit? — and memory bandwidth — how fast it can decode, since each token streams the active weights once from memory. The frontier datacenter parts (NVIDIA’s Blackwell, AMD’s MI355X, Google’s inference-first TPU) cluster at the top-right: vast HBM and ~8 TB/s. A Mac’s unified memory — one pool of RAM that its CPU and GPU share — sits far to the right but low: huge, but slow. And a different bet sits off the chart entirely: SRAM machines like Groq and Cerebras trade capacity for blistering speed.

Pick a chip: does a 70B model fit, and how fast would it decode? Both follow from the bottleneck Part 3 named — decode is memory-bound. Capacity decides the fit; bandwidth decides the speed, and the exact formula lands by the end of this floor.

The accelerator landscape

Two specs decide everything — capacity (x) says whether a model fits; bandwidth (y) sets the decode ceiling, since tok/s ≈ bandwidth ÷ bytes read per token.

010020030040050002468memory capacity — GBmemory bandwidth — TB/sM3 Ultra — 512 GB, 0.819 TB/sM3 UltraRTX 5090 — 32 GB, 1.8 TB/sRTX 5090H200 — 141 GB, 4.8 TB/sH200Trainium3 — 144 GB, 4.9 TB/sTrainium3MI300X — 192 GB, 5.3 TB/sMI300XTPU v7 — 192 GB, 7.4 TB/sTPU v7B200 — 192 GB, 8 TB/sB200MI355X — 288 GB, 8 TB/sMI355XB300 — 288 GB, 8 TB/sB300H100 — 80 GB, 3.35 TB/sH100
Accelerator
Reference weights precision
Vendor
NVIDIA
H100
Capacity
80 GB
does it fit?
Bandwidth
3.35 TB/s
how fast it decodes
Decode ceiling
~48 tok/s
single stream
Reference — a 70B model at FP8 ≈ 70 GB of weights. Fits — 70 GB ≤ 80 GB, so it loads on one H100, and bandwidth caps decode near ~48 tok/s.
A different bet, off this chart — SRAM, tiny memory, blistering single-stream speed:
  • Groq LPU (Groq) — SRAM-only, deterministic dataflow — lowest latency
  • Cerebras WSE-3 (Cerebras) — wafer-scale SRAM — thousands of tok/s on big models

illustrative public specs, ~2026 — not benchmarks.

Today's inference chips by the two specs that decide everything — capacity sets what fits, bandwidth sets how fast it decodes.

GEMM on tensor cores: the workhorse matmul

The workhorse is the matrix multiply (GEMM) — the Q/K/V projections, the FFN, and the final vocabulary projection are all matmuls. A matrix multiply combines two grids of numbers into a third: each output number is one row of the left grid dotted with one column of the right — the very dot product from Part 1, multiply the paired numbers and add. Tensor cores don’t multiply whole matrices at once; they chew through tiles. Each output tile reads a strip of rows from one input and a strip of columns from the other, multiplies them in fast memory, and writes one block of the result.

Bigger tiles reuse each loaded value across more multiplications — more arithmetic per byte fetched, which is exactly what the hardware wants. And tensor cores do this natively in low precision: FP16, FP8, even FP4 — the formats from Part 2.

Tiled matmul on tensor cores

Q/K/V, the FFN, and the LM head are all GEMMs: C = A·B. The GPU computes the output one tile at a time — each tile multiplies a row-strip of A by a column-strip of B.

Tiled GEMM: highlighted output tile, A row-strip, and B column-stripB [K×N]A [M×K]C [M×N]
3264128
Tensor-core precision
Arithmetic intensity
32
FLOPs/byte
Reuse vs smallest tile
2.0×
per loaded value

Bigger tiles reuse each loaded value across more multiply-accumulates, so FLOPs/byte rises — pushing the kernel off the memory wall toward compute-bound. Narrower elements (FP8 / FP4) lift it further. Total work for this GEMM: 134.2 MFLOP.

tile 1,1 of 4×4

Tensor cores compute each tile natively in low precision (FP16 / FP8 / FP4), accumulating in higher precision. Shown: a 256×256 output over K=1024.

Matmul runs as tiles: each output tile reads a row-strip and a column-strip. Bigger tiles reuse more.

FlashAttention

Attention has a memory problem the matmuls don’t. Done naively, it builds the full n×n matrix of attention scores — one score for every pair of tokens in a length-n sequence — writes it to HBM, reads it back to softmax, then reads it again to multiply by V — round-trips to slow memory that grow with the square of the sequence length (O(n²)). For long sequences, that traffic, not the math, is the bottleneck.

FlashAttention refuses to ever write that matrix. It tiles Q, K, and V into blocks that fit in SRAM and streams through them, keeping a running maximum and running sum so the softmax is computed online — incrementally, as the blocks stream past, never assembled in full. The n×n matrix never exists: the score matrix’s O(n²) round-trips to HBM vanish, and what’s left is streaming Q, K and V past the cores — the same answer, a fraction of the memory movement.

∇ Online softmax

Softmax exponentiates each score and divides by the sum of those exponentials — so to normalize you must carry that running sum, and because exe^{x} explodes for large xx you first subtract the running max (it cancels in the ratio, leaving the answer unchanged). So softmax can be accumulated incrementally. Carry a running max mm and running normalizer ℓ\ell; when a new block arrives with its own max, rescale the partial output by emold−mnewe^{m_\text{old} - m_\text{new}} and fold in the block’s contribution. The end result is bit-for-bit the ordinary softmax — just never materialized in full.

FlashAttention · naive vs fused

n = 2,048 tokens · head dim d = 64 · fp16. Attention is memory-bound: what costs is bytes moved through HBM.

Naivescores live in HBM
n × n = 4,194,304 scores
write → HBM → read
HBM trafficO(n²)17.0 MB
Fusedtiled through SRAM
kept in SRAM, tile by tile
K/V tiles → SRAM3/3
2.6
5.3
6
online softmaxmax 6.0Σ 1.53
HBM trafficstreams Q, K, V1.0 MB
HBM traffic, to scale16.9× more for naive
naive
fused
Naive attention writes the whole n×n score matrix to HBM; FlashAttention streams through SRAM and never stores it.

The roofline

Why is prefill fast per token and decode slow? The roofline answers it with one ratio. Plot a workload by its arithmetic intensity — the number of FLOPs (floating-point operations — a raw count here, not the per-second rate) it performs per byte moved from memory — against the FLOP/s (that rate: operations per second) it can attain. Low intensity and you are memory-bound: the math finishes long before the next bytes arrive, so bandwidth caps you. High intensity and you are compute-bound, pinned at the chip’s peak. The crossover is the ridge point. Plotted, the limit looks like a roof — a sloped line that climbs with intensity until it hits the flat ceiling of the chip’s peak.

Prefill, processing many tokens against the same weights, sits high and compute-bound. Decode at batch 1 reloads all the weights to produce a single token — dreadful intensity, deeply memory-bound. Batching is the lever: more sequences reuse each loaded weight, raising intensity and sliding the point toward the roof. Drag the batch size and watch decode climb.

∇ The roofline, precisely

Attainable performance is min⁡ ⁣(peak FLOP/s,  arithmetic intensity×bandwidth)\min\!\big(\text{peak FLOP/s},\; \text{arithmetic intensity} \times \text{bandwidth}\big) — a flat compute roof and a sloped memory roof. They meet at the ridge=peak/bandwidth\text{ridge} = \text{peak} / \text{bandwidth}: below it you are memory-bound, above it compute-bound.

Roofline — what bounds the kernel?

Arithmetic intensity (FLOPs/byte) decides the bottleneck. Below the ridge: memory-bound. Above it: compute-bound. Raise the decode batch to slide right toward the roof.

1101001,00010T100T1Parithmetic intensity — FLOPs/byteattainable — FLOP/sridge 299prefillcompute-bounddecode ×32
Arithmetic intensity
32 FLOPs/byte
Attainable
107 TFLOP/s
Bottleneck
memory-bound
roofline (memory + compute roofs)prefill — compute-bounddecode — memory-bound at low batch

Decode reuses each loaded weight for just one token per sequence, so batch 1 sits at ~1 FLOPs/byte — starved for bandwidth. Bigger batches amortise the same weight load over more tokens, sliding decode up the memory roof toward the ridge, where the compute roof takes over.

Arithmetic intensity vs the compute roof. Drag batch size — decode climbs from memory-bound toward the ridge.

Splitting across GPUs

When a model is too big for one GPU, you split it — three ways. Tensor parallel shards every weight matrix across devices, so each holds a slice; an all-reduce stitches the partial results back together every layer, which is why TP wants the fat GPU-to-GPU links inside one server — NVLink, far faster than the network between servers. Pipeline parallel instead gives each device a range of layers; activations — the vectors passing from one layer to the next — flow device to device like a pipeline, efficient but with bubbles of idle time at the ends, shrunk by feeding more microbatches so that while a late device finishes one chunk an early one is already on the next. Expert parallel spreads a MoE (Mixture-of-Experts, Part 1) model’s experts across devices and routes each token to its experts with an all-to-all — every device shipping each token across to whichever device holds the expert it needs. Those three all answer one problem: the model is too big for a single GPU. When it does fit and you just need more tokens per second, you split nothing — you run whole copies of the model on separate GPUs and spread requests across them (data parallelism). Real deployments stack all four at once.

∇ The wires are part of the computer

TP’s every-layer all-reduce only pays off because of NVLink: direct GPU-to-GPU links inside one server, on the order of 1.8 TB/s per Blackwell GPU — within sight of HBM bandwidth itself. Step outside the box and you drop to InfiniBand or Ethernet at roughly 50–100 GB/s per link, an order of magnitude slower. That gap is the real rule behind this beat: chatty TP stays inside the scale-up domain — one server’s NVLink island — while PP’s point-to-point handoffs and EP’s all-to-all are built to cross the scale-out network between servers. Rack-scale systems move the boundary outward: NVIDIA’s NVL72 wires 72 Blackwell GPUs into a single NVLink domain, a whole rack behaving like one enormous GPU, and trillion-parameter MoE serving is designed around exactly that island. Seen this way the memory pyramid from the top of this floor keeps going below HBM — NVLink, then the datacenter network, each tier another step further from the cores.

Splitting a model across devices

Three axes of parallelism. Each shards a different unit and pays a different communication cost.

Strategy
dev 0dev 1dev 2dev 3all-reduce activations · every layer
shards
each weight matrix, sharded across devices
communication
all-reduce
bound by
intra-node interconnect (NVLink)

Every device holds a slice of every layer and computes a partial activation; an all-reduce each layer sums the slices. Chatty — keep it inside one node.

TP shards matrices (all-reduce per layer); PP splits layers (pipeline bubbles); EP spreads experts (all-to-all).

Putting a number on it: tokens per second

Everything on this floor has pointed at one number: how fast a token comes out — and for decode it just tracks each chip’s bandwidth. Here is the whole field at once. Switch to the MoE and watch every bar leap: reading only its few active experts per token, it decodes like a small model on hardware that would crawl through a dense one its size. But a fast bar is no use if the weights overrun the card — those chips grey out.

Decode speed across the field

Decode is memory-bound: tok/s ≈ bandwidth ÷ bytes/token. Pick a model and a precision — each bar is that chip's decode ceiling.

Model
Weights precision
  • B200192 GB · 8 TB/s
    114tok/s
  • MI355X288 GB · 8 TB/s
    114tok/s
  • B300288 GB · 8 TB/s
    114tok/s
  • TPU v7192 GB · 7.4 TB/s
    106tok/s
  • MI300X192 GB · 5.3 TB/s
    76tok/s
  • Trainium3144 GB · 4.9 TB/s
    70tok/s
  • H200141 GB · 4.8 TB/s
    69tok/s
  • H10080 GB · 3.35 TB/s
    48tok/s
  • RTX 509032 GB · 1.8 TB/s
    won't fit26tok/s
  • M3 Ultra512 GB · 0.819 TB/s
    12tok/s

70B dense at FP8 streams ~70 GB per token. Grey bars are chips the full weights overflow — fast, but it won't load. Illustrative, ~2026 — not benchmarks.

Tokens per second for the chosen model on each chip — the bandwidth law made literal. Grey bars overflow the chip's memory.

Step back and one law rules this floor: for decode, tokens per second ≈ memory bandwidth ÷ bytes read per token. The compute is almost incidental; what you wait on is weights streaming from memory. That single fact explains the landscape — why unified-memory machines (an Apple-Silicon Mac, say), slow but vast, run enormous models that would overflow most fast GPUs; why MoE feels like cheating, since only the few active experts’ weights are read per token; and why every trick in Parts 2 and 3 — quantization, GQA, paging — was, underneath, about moving fewer bytes. We have reached the bottom. Time to climb back up and watch the whole machine run at once.

5

Part 5 · The whole descent

Now show me the whole thing.

You have followed a single token from the prompt box, through the transformer, down into the numbers its weights are made of, across the serving stack, and onto the silicon. None of those layers is an island — they are one continuous machine. Here is the whole thing, in one pass.

Replay the descent

We started with text and ended with a token, and in between the same value moved through every layer of this descent: split into tokens, looked up as a vector, mixed with its neighbors by attention across dozens of layers, transformed by the feed-forward experts, and projected back into a distribution to sample from. Those weights were billions of numbers in low precision; the engine fed your prompt in one parallel pass and then emitted answers one token at a time — that slow, one-at-a-time decode — reusing its KV cache; and all of it ran as tiled matmuls on tensor cores, paced by memory bandwidth.

Step through it. The whole machine, from the word you typed to the next word it chose.

THE WHOLE DESCENTone token, top to bottom
stage 0 / 10
FollowingThedown through every layer you just learned.
  1. TokenizePart 1 · The transformer
  2. EmbedPart 1 · The transformer
  3. Attention × N layersPart 1 · The transformer
  4. Feed-forward / MoEPart 1 · The transformer
  5. LogitsPart 1 · The transformer
  6. SamplePart 1 · The transformer
  7. Quantized weightsPart 2 · Weights as numbers
  8. Prefill & decodePart 3 · Inference: software
  9. Tensor cores & bandwidthPart 4 · Inference: hardware
  10. The next tokenPart 5 · The whole descent
…follow it all the way down to see the next token.
Replay one token down the entire stack — every stage now legible.

Size it yourself

Everything you have learned about memory and bandwidth now fits in one back-of-the-envelope tool. A model’s weights are one number per parameter, so they take params × bytes-per-weight; its KV cache grows with context and batch (how many requests share the card at once); together they must fit in the GPU’s VRAM — the on-card HBM from Part 4. And for decode, tokens per second ≈ memory bandwidth ÷ bytes read per token — where bytes read per token is essentially the whole model, dragged off memory once for every token it emits. So lower precision runs faster (fewer bytes per weight), a bigger context costs memory (a fatter KV cache), and a Mixture-of-Experts model — those feed-forward experts from the replay above — flies because only its active experts are read each step.

Pick a model, a precision, and a card. See whether it fits, and roughly how fast it runs.

Config sandbox — will it run, and how fast?

Model × precision × GPU → the VRAM it needs and the tokens/sec it decodes. Decode is memory-bound: tok/s ≈ bandwidth ÷ bytes/token.

Model
Precision
GPU
192 GBover by 49.0 GB
weightsKV cacheover capacity
Weights
240 GB
KV cache
1.01 GB
Over capacity
+49.0 GB
Decode ceiling
if it fit — 800 tok/s
per stream
Decode tape — one Token per stepreads 5B × 2 bytes per token
abcdefghijklmnop

MoE: 120B on disk, but only 5B fire per token — the tape flies. But it won't fit on B200 — paused; the ceiling is what you'd get if it did.

Illustrative, ~2026 — not benchmarks.

Pick a model, a precision, a GPU. Watch the memory math and the bandwidth law from Parts 2 and 4 collide.

Price it out

You know whether it fits and how fast it decodes; the last question is what it costs. Self-hosting, the answer is just the GPU’s price per hour divided by the tokens it produces — and the lever is the batch. A single decode stream is memory-bound: the weights are dragged off memory once per step whether one sequence rides along or a hundred, so serving just one wastes the GPU and every token is dear. Pack the batch full and that same weight read is amortized across every sequence — cost per token falls almost in step with the batch size, until the batch is large enough to saturate the cores at the roofline’s compute roof, where it plateaus (and where, in practice, the KV cache runs out of room). It is the whole descent in one number: memory bandwidth, the roofline, and the MoE trick, priced in dollars.

∇ Why output tokens cost more

Look at any API price sheet and output tokens cost roughly three to five times what input tokens do — that ratio is prefill versus decode, priced. Your prompt is handled by prefill, one parallel compute-bound pass where thousands of tokens share a single read of the weights, so the marginal cost of an input token is tiny; every generated token is one decode step that drags the model’s active weights off memory, so output tokens each pay the full bandwidth bill. Cached-input discounts are the same logic one step further: a prefix-cache hit (Part 3) skips prefill compute entirely, which is why providers sell cached input tokens for another order of magnitude less. Read a pricing page now and every line on it is a hardware fact you already know.

∇ Joules per token

Same arithmetic as dollars, different numerator: energy per token = board power ÷ tokens per second. An H100 drawing ~700 W while decoding 12,000 batched tokens a second spends roughly 0.06 joules per token — a 500-token answer is about 30 joules, a few seconds of a laptop’s draw, nowhere near the kettle-boiling figures headlines quote. Run the same model single-stream on a local GPU and the watts barely drop while tokens per second collapses, so energy per token can be 50–100× worse — batching saves joules for exactly the reason it saves dollars: the shared weight read. Datacenter overhead (cooling and power delivery, PUE ~1.1–1.3) and prefill add tens of percent, not orders of magnitude — and at fleet scale this is the number that binds the buildout, because moving a byte always costs more energy than doing arithmetic on it: every moving-fewer-bytes trick in Parts 2–4 was an energy trick too.

What a token costs

Cost per token = GPU price per hour ÷ tokens produced. The lever is batching — pack more sequences in and each weight read is shared, so cost collapses.

GPU (illustrative $/hr)
Model (weights at FP8)
Cost per million tokens, by batchbar length: log scale
  • batch 1
    $15/Mtok48 tok/s
  • batch 8
    $1.81/Mtok383 tok/s
  • batch 64
    $0.227/Mtok3.1k tok/s
  • batch 256
    $0.057/Mtok12.3k tok/s
  • batch 1024
    $0.049/Mtok14.3k tok/s

Batching 70B dense on H100 takes it from $15 per million tokens at batch 1 to $0.049 batched — ~299× cheaper — until the compute roof, where the last bars stop shrinking. (Real batch is also capped by KV-cache memory.) Best-case arithmetic at FP8; real utilization is lower, so real prices run higher. Illustrative, ~2026.

Cost per million tokens as the batch grows — it collapses until the roofline's compute roof, then flattens. Best-case, illustrative.

The same token that felt like magic at the top is, by the bottom, just bandwidth and arithmetic — a list of numbers multiplied by other numbers, fast enough to feel like thought. Somehow that makes it more impressive, not less. You have reached the bottom of the descent; turn around, and you can see the whole climb you just made.

Everything on this page took the weights as given. How they were learned — pretraining, RLHF, fine-tuning — how a model is aligned and evaluated, how it takes in images or audio, and how it reaches for retrieval or external tools are all upstream or sideways of the one question asked here: what happens when a frozen model answers you.

Where to go deeper — one canonical source behind each part:

All numbers illustrative; hardware figures as of 2026.