Exploring LLM Architectures
25 min readreadingLLM Foundations for Engineers
3 of 17LLM Engineering: APIs, Inference & Structured Output
Up next · Setting Up Your LLM Development Environment
At 2 a.m. an inference server falls over. The model it hosts ran fine all week — until a customer pasted a 300-page contract into the chat box, and memory usage, which everyone assumed grew with the model, turned out to grow with the conversation. The postmortem lands on a single line item nobody on the team could define: KV-cache. The fix — switching to a checkpoint with grouped-query attention — cuts memory several-fold without changing a line of application code. Somebody asks the obvious question: what else is in this black box that we are betting the product on?
This lesson opens the box at survey depth — enough to reason about bills, latency graphs, and model choices. The full mathematics of attention lives in Transformer Architecture & LLM Internals; here the agenda is the handful of architectural facts that show up in production: the decoder-only stack and its cache, mixture-of-experts and why the frontier runs on it, why an advertised "1M context" is not one million usable tokens, what reasoning models and thinking budgets do to your invoice, and how to read model cards and benchmarks without being fooled.
Every model you will call in this course — closed or open — is a decoder-only Transformer: three stages in a line. An embedding layer turns each token of the context into a vector of a few thousand numbers. A stack of identical Transformer blocks repeatedly refines those vectors; each block combines attention (tokens exchanging information — the pronoun "it" pulling meaning from "the animal" earlier in the sentence, under a causal mask so no token ever sees its future) with a feed-forward network (each token transformed in place — where two-thirds of the parameters and, per interpretability work, most of the stored facts live). Finally an unembedding projects the last token's vector onto the vocabulary, producing the probability distribution from Lesson 1. Residual connections and normalization are the plumbing that lets the stack run dozens of blocks deep. Real numbers, from Meta's Llama 3.1 family:
| Model | Blocks | Hidden size | Query heads | KV heads |
|---|---|---|---|---|
| Llama 3.1 8B | 32 | 4,096 | 32 | 8 |
| Llama 3.1 70B | 80 | 8,192 | 64 | 8 |
| Llama 3.1 405B | 126 | 16,384 | 128 | 8 |
Notice the last two columns. Attention runs as many parallel heads, each specializing in a different relationship. Every head's keys and values must be remembered for every token of context during generation — and that memory is the villain of the 2 a.m. story. Grouped-query attention (GQA) lets groups of query heads share one set of key/value heads: 64 query heads over 8 KV heads is an 8x cache reduction at almost no quality cost, which is why essentially every modern checkpoint ships with it.
Generation has two phases with opposite characters, and both show up in your latency graphs. Prefill processes your whole prompt at once — massively parallel, compute-bound, and the reason time-to-first-token grows with prompt length. Decode then produces one token at a time — each step touches every weight once, so it is memory-bandwidth-bound and stubbornly serial. That is why APIs quote two numbers worth watching separately: time to first token, and tokens per second after that.
Decode avoids re-processing the whole context at every step by caching each past token's keys and values — the KV-cache. Its size is pure arithmetic:
For Llama 3.1 8B at 16-bit precision (32 blocks, 8 KV heads, 128-dim heads), that is 128 KB per token of context. A 2,000-token chat costs a quarter of a gigabyte; a single 128K-token conversation costs about 16 GB — as much memory as the entire model. The server in the opening did not run out of model; it ran out of conversation. Halve the KV heads or the precision and you halve that line; this is why GQA exists, why long-context serving is expensive, and why providers bill long prompts the way they do.
Scaling laws reward parameters; your latency budget punishes them. Mixture-of-experts (MoE) models split the difference by making the expensive feed-forward layer conditional: each block carries many parallel expert FFNs, and a tiny router picks the top- for each token:
| Model | Total params | Active per token | Experts (chosen) |
|---|---|---|---|
| Mixtral 8×7B (2023) | 47B | 13B | 8 (2) |
| DeepSeek-V3 / R1 | 671B | 37B | 256 (8) + shared |
| Llama 4 Maverick | 400B | 17B | 128 (1) + shared |
| Qwen3-235B | 235B | 22B | 128 (8) |
Read the second and third columns together and the appeal is obvious: DeepSeek-V3 has the knowledge capacity of a 671B model but the per-token compute of a 37B one. That is why frontier labs are widely understood to run MoE behind their APIs, and why the strongest open models are MoE. The catch is operational: every expert must sit in memory even though few fire per token, and routing must be load-balanced. So the industry pattern is MoE at the frontier and on big serving clusters, dense at the edge — which is why single-GPU families like Gemma stay dense, and why "total parameters" on a model card tells you about memory while "active parameters" tells you about speed and compute cost. Confusing the two is a classic capacity-planning mistake.
Context windows grew five orders of magnitude in eight years: 512 tokens (GPT-1, 2018), 2K (GPT-3), 8–32K (GPT-4), 100K (Claude 2), 1M (Gemini 1.5, 2024, and mainstream frontier models since), with Llama 4 Scout advertising 10M. Position-encoding tricks (rotary embeddings and their extensions) plus attention-efficiency work made it possible. It is a genuine capability — and one of the most misleading numbers on any model card.
Three gaps separate the advertised window from what you can rely on. First, recall is uneven: the well-documented "lost in the middle" effect means information buried mid-context is recalled worse than information near the edges, and long-context stress tests like RULER show many models' effective length — where they still reason reliably rather than merely fit tokens — is a fraction of the advertised one. Passing a needle-in-a-haystack retrieval test is table stakes, not proof of comprehension. Second, every context token costs money, whether or not it helped: at $3 per million input tokens, one fully-loaded 1M-token request is $3 — per call, before any output. Third, every context token costs latency: prefill over hundreds of thousands of tokens means many seconds before the first output token.
The reasoning models from Lesson 2 add one architectural-adjacent behavior you must plan for: before the visible answer, the model generates a chain of thought — sometimes shown summarized, sometimes hidden entirely. Three operational facts follow. Thinking tokens are output tokens on your bill, even when you never see them: a ten-line answer preceded by five thousand tokens of hidden deliberation costs like a long essay. Thinking is latency: seconds to minutes before the first useful token, which changes UX design for anything interactive. And thinking is a knob: providers expose control over how much deliberation the model may spend — OpenAI's reasoning-effort levels, Anthropic's adaptive thinking with effort levels (which replaced fixed token budgets), Gemini's token-denominated thinking budgets, and explicit thinking-mode toggles on open models like Qwen 3 and DeepSeek-R1.
The decision rule is economic. Reasoning modes buy real accuracy on math, non-trivial code, planning, and multi-step analysis; they buy approximately nothing on extraction, classification, summarization, and formatting — tasks where the first plausible answer is already right. Route accordingly: default the boring 80% of traffic to a fast model with minimal thinking, and reserve deliberation budgets for requests that need them. Teams that leave "maximum reasoning" on for everything routinely discover the habit in their invoice before their metrics.
You will choose models for the rest of your career, mostly from documentation. Read a model card the way you would read a used-car listing: the facts are probably true, the emphasis is chosen by the seller. The load-bearing fields are unglamorous — knowledge cutoff; context window and the separate maximum output length; tool-calling and structured-output support; modality support; license terms (open weights range from Apache/MIT to usage-restricted community licenses — legal reads these before production does); pricing per million input and output tokens; rate limits; and the deprecation policy, because APIs retire models on their schedule, not yours.
Benchmarks deserve harder skepticism, for one structural reason: contamination. Benchmarks are published text; pretraining hoovers up published text; therefore models have often seen the test. Scores on old public suites (MMLU and friends) are saturated and gameable — treat them as a floor check, not a ranking. Weight instead the evaluations built to resist gaming: held-out and verified sets like SWE-bench Verified for real software tasks and GPQA Diamond for expert questions, agentic evaluations like τ-bench and GAIA, and rolling benchmarks like LiveCodeBench that only use problems published after a model's cutoff. Preference leaderboards (LMArena-style) measure what humans like, which rewards style and confidence as much as correctness — and vendors optimize for them: in one 2025 episode, a major lab's arena entry turned out to be an experimental chat-tuned variant, not the released weights. Read the footnotes.
Architecture literacy cashes out in one recurring decision: which model does this task deserve? The honest answer is empirical — benchmark on your own data — but the starting points are stable:
| Situation | Reach for | Because |
|---|---|---|
| Prototyping, general work | A frontier mid-tier (Sonnet-class, Gemini Flash-class) | Near-flagship quality at a fraction of the price |
| Hard reasoning, agentic coding | Frontier top tier, reasoning mode on | Capability thresholds and thinking budgets earn their cost here |
| Classification, extraction at volume | Small fast tiers (Haiku-class) or 4–12B open models | Cheap, low-latency; the quality gap barely shows on narrow tasks |
| Privacy-critical, on-prem | Open weights on vLLM or SGLang | Data never leaves; no per-token vendor bill |
| Learning, tests, CI | A local model via Ollama | Zero marginal cost — set up in the next lesson |