Amit Choubey

2018 · decoder-only

GPT (124M)

How a piece of text becomes a guess at the next token

~8 minGPTdecoderattentionBPE

How to read this page

  • Start at the bottom: input text → tokens → embeddings.
  • Climb the main column through one transformer block (it repeats ×12).
  • Red residual arrows are the shortcuts that keep gradients alive.
  • After the sketch, read the desk notes — especially why the MLP opens to 4×.
Ink— structure & boxes
Blue pencil— explanations
Red pencil— what to remember

Whole sketch in view — zoom in to read labels

GPT (124M) — How a piece of text becomes a guess at the next token GPT how a piece of text becomes a guess at the next token Input text “the cat sat on” Tokenizer (BPE) → token IDs [ batch = 2 , tokens = 4 ] GPT model Token embedding 50,257 × 768 Positional embedding 1,024 × 768 what the token is where it sits [2, 4, 768] Dropout 0.1 training only Transformer block × 12 Layer norm Masked multi-head attention 12 heads × 64 dims Dropout Layer norm Feed forward (same MLP on every token) Dropout residual shortcut residual shortcut Final layer norm Linear output head 768 → 50,257 Logits [2, 4, 50257] ↓ softmax next-token probabilities pick one → stick it back on the end → run the whole thing again zoom · feed forward Linear 768 → 3,072 GELU Linear 3,072 → 768 widen bend shrink zoom · layer norm per token vector x: x̂ = (x − mean) / √(var + ε) → mean 0, var 1 out = γ · x̂ + β γ scale, β shift (learned) ε keeps the divide safe. Normalising before each sublayer keeps activations in a sane range, so gradients neither explode nor die out. zoom · masked attention 1. project x → Q, K, V 2. split into 12 heads of 64 dims 3. scores = Q Kᵀ / √64 4. mask out everything to the right 5. softmax · V → concat → output proj token 4 may look at 1–4, never at 5. the numbers (124M config) vocab 50,257 context 1,024 d_model 768 heads 12 layers 12 FF hidden 3,072 = 4 × 768 dropout 0.1 norm before each sublayer output head can reuse the token-embedding weights legend add, element-wise (the residual) repeated 12 times, same shape in & out [b, t, d] tensor shape at that point the shape stays [2, 4, 768] the whole way up. my own notes · redrawn from scratch

Desk notes

same pad on every sketch

Two jobs inside one block

Attention

Look sideways. Which other tokens should colour this one?

Feed-forward

Look inward. Rethink this token alone — on a wider desk — then write back.

The 4× scratchpad

Same token. Bigger desk for a moment. Then back to the shared width so the residual + still lines up.

arrive (768)

768-d

open wide (3072)

3072-d

return (768)

768-d

GELU does not change the length — it only turns features up or down inside the wide vector. The first Linear creates the 4× room; the last Linear folds useful work back into 768.

Stream rule

The residual stream stays 768-wide on purpose. Widen only inside the MLP; shrink before you add. That is why twelve blocks can stack without reshaping the story.

Choice, not law

4× is a GPT / classic-Transformer habit — more room to compute, paid for in FLOPs. Other families pick different ratios (LLaMA-style FFNs often sit nearer ~2.7×). The pattern that matters is widen → nonlinear → shrink, not the exact number.

Attention mixes the room; the MLP rethinks one seat. Both hand the same 768-d shape back so the red + can keep working.