How LLMs Are Trained: Pretraining to RLHF
35 min readreadingLLM Foundations for Engineers
2 of 17LLM Engineering: APIs, Inference & Structured Output
Up next · Exploring LLM Architectures
Open two terminals. In the first, load a base language model — the raw artifact that comes out of pretraining — and type: "What causes tides?" The model replies: "What causes earthquakes? What causes rainbows? Quiz 3, page 47." It is not broken, and it is not being difficult. It has correctly recognized the statistical signature of a workbook full of questions, and it is continuing the pattern — because continuing patterns is the only thing it was ever trained to do. In the second terminal, load the same architecture after two more stages of training and ask again. This one explains gravitational gradients, mentions the moon's dominance over the sun's contribution, and asks whether you want the mathematics. Same parameter count, same tokenizer, same pretraining data. Different creature.
The distance between those two terminals is the subject of this lesson. Every capable assistant model — Claude, the GPT-5 family, Gemini, Llama's instruct variants — is built in three acts: pretraining, which spends millions of dollars teaching a network the structure of language and the world; supervised fine-tuning, which teaches it to behave like an assistant instead of an autocomplete; and preference optimization — RLHF and its modern successors — which teaches it judgment: what a good answer looks like, not just a plausible one. A fourth act arrived with the reasoning models of 2025: reinforcement learning against verifiable rewards, which taught models to think before answering.
You will never run the first act yourself, but you will spend your career living with its consequences — and, more practically, with the consequences of the later acts, because they are where the behaviors you rely on daily come from: tool calling, JSON discipline, refusals, and the eagerness to agree with you. This lesson covers the pipeline at working depth; the mathematics of the architecture those weights live in is treated properly in Transformer Architecture & LLM Internals.
Pretraining is conceptually the simplest stage and materially among the most expensive artifacts in the history of software. Take a Transformer with randomly initialized weights. Feed it text — trillions of tokens of it. At every position, ask it to predict the next token, and nudge every weight in the direction that would have made the right answer more likely. The objective is the cross-entropy loss over the corpus:
That is the whole recipe. No labels, no task definitions, no curriculum of skills — the supervision is the text itself, which is why the approach scales: the internet is the labeled dataset. What makes the result remarkable is what minimizing this loss forces the network to learn. Early in training, the cheapest loss reductions come from spelling and grammar. Those saturate. Then word meanings, then facts, then style, then — deep into training, when every easy regularity has been exhausted — the model starts winning loss on text that can only be predicted by modeling the process that wrote it: tracking characters through a story, carrying intermediate results through a derivation, keeping types consistent through a function body.
"Trained on the internet" is a simplification that hides the most strategically guarded asset in the field: the data mix. A modern mix is curated, filtered, deduplicated, and weighted — and its composition shapes the model's personality as surely as architecture does:
| Source | Typical share | What it teaches |
|---|---|---|
| Filtered web crawl | 50–70% | Breadth: world knowledge, registers, formats |
| Code repositories | 10–25% | Programming — and, evidence suggests, sharper general reasoning |
| Books | 5–10% | Long-range coherence, narrative, argument structure |
| Reference works | 3–5% | Dense, reliable factual scaffolding |
| Scientific papers | 3–5% | Technical vocabulary, formal reasoning |
| Multilingual text | 5–15% | Cross-lingual transfer |
| Math corpora | 2–5% | Symbolic manipulation, proof patterns |
| Model | Hardware | Duration | Order of magnitude |
|---|---|---|---|
| GPT-3 (175B, 2020) | ~1,000 V100s | ~1 month | FLOPs, ~$5M |
| Llama 3.1 (405B, 2024) | 16,384 H100s | ~2 months | FLOPs, $100M class |
| DeepSeek-V3 (671B MoE, 2024) | 2,048 H800s | ~2 months | 2.79M GPU-hours, ~$5.6M — efficiency as headline |
The output of Act I is a base model: a vast, amoral, encyclopedic autocomplete. It contains most of what the final assistant will ever know — the later acts add astonishingly little knowledge. What it lacks is a job description.
The base model's failure mode from the opening of this lesson is worth dwelling on, because it defines what the remaining acts must accomplish. Ask a base model to summarize an email and it may append three more emails. Prompt it with a question and it may answer — if the surrounding text happens to resemble contexts where answers followed questions — or it may write a forum thread of people arguing about the question. It is a mirror of the training distribution, and the training distribution is not an assistant.
Practitioners of the GPT-3 era coaxed base models into usefulness with elaborate prompt scaffolding ("The following is a transcript of a helpful expert answering questions. Q: … A:"). The insight that ended that era: instead of engineering prompts that make assistant-like text statistically likely, change the distribution itself. Show the model what assistant transcripts look like until assistant behavior becomes its default. That is Act II.
Supervised fine-tuning (SFT) continues the exact same next-token training — same loss, same optimizer — on a radically different corpus: tens of thousands of curated conversations in which an assistant responds the way the model's builders want it to respond. The examples are formatted with a chat template that marks who is speaking, so the model learns the turn structure itself:
# One SFT training example (messages format)
example = {
"messages": [
{"role": "user",
"content": "My Flask app returns 404 for /api/users but the "
"route is defined. What should I check first?"},
{"role": "assistant",
"content": "Three checks, in order of how often they are the "
"culprit:\n"
"1. Blueprint registration — is the blueprint that "
"defines /api/users actually registered on the app?\n"
"2. URL prefix — a blueprint registered with "
"url_prefix='/api' plus a route of '/api/users' "
"yields /api/api/users.\n"
"3. Trailing slash — Flask treats /api/users and "
"/api/users/ differently unless strict_slashes is "
"disabled.\n"
"Run `flask routes` to see what was actually "
"registered rather than what you intended."},
]
}
Training on such transcripts only ever computes loss on the assistant's tokens — the user's words are context, not targets. A few properties of good SFT data have outsized effects:
| Property | Typical value | Why it matters |
|---|---|---|
| Volume | 10K–100K dialogues | Small next to pretraining; this stage sets form, not knowledge |
| Quality bar | Expert-written or expert-filtered | The model imitates everything — including sloppy reasoning |
| Diversity | Thousands of task types | Generalization to instructions nobody wrote an example for |
| Tool-use episodes | Deliberately included | Structured tool calls are demonstrated, not hoped for |
| Refusal examples | Deliberately included | The model must also learn what not to do |
After SFT the model answers questions, follows instructions, and stays in its lane. But it has only ever imitated single demonstrations. It has no notion that some correct answers are better than others — clearer, safer, better calibrated, more honest about uncertainty. Teaching better requires comparisons, and comparisons require Act III.
Reinforcement learning from human feedback exploits an asymmetry you know from your own life: it is far easier to judge which of two essays is better than to write the best possible essay. RLHF turns that cheap judgment signal into a training objective, in two steps.
Step 1 — distill human preferences into a reward model. Sample two responses from the SFT model for the same prompt. A human annotator marks which one is better. Repeat a few hundred thousand times. Then train a separate network — the reward model — to predict those judgments:
where and are the preferred and rejected responses. The reward model becomes a scalable, tireless proxy for human taste: point it at any (prompt, response) pair and it returns a score.
Step 2 — optimize the policy against the reward. Now treat the language model as a reinforcement-learning policy. Generate responses, score them with the reward model, and update the weights — classically with PPO — to make high-scoring responses more likely, under one crucial constraint:
The KL-divergence term tethers the policy to the SFT model it started from. Remove it and the optimizer will find the reward model's blind spots rather than genuine quality — the policy drifts into degenerate text that happens to score well. This failure mode has a name you will meet again whenever models are optimized against proxies: reward hacking.
Classic RLHF is powerful and operationally miserable: four models in memory (policy, reference, reward, value), an unstable RL loop, and a small army of hyperparameters. In 2023, Direct Preference Optimization showed that for this setting the reinforcement-learning machinery is unnecessary. The preference data can train the policy directly, with a single supervised loss:
Read it as: increase the probability of preferred responses and decrease the rejected ones, relative to a frozen reference model, with controlling how far you may drift. No reward model, no rollouts, no PPO. The implicit reward and the KL tether are baked into the loss itself.
| RLHF (PPO) | DPO | |
|---|---|---|
| Separate reward model | Yes | No |
| Models in memory | 4 | 2 |
| Training character | RL loop, delicate | Supervised, stable |
| Online exploration | Yes — can discover novel behavior | No — bound to the preference dataset |
| Typical users | Frontier labs (with GRPO-family variants) | Open-weights community, most fine-tuners |
Preference optimization has a ceiling: a reward model can be fooled, and human preferences are noisy on hard technical questions. The 2025 generation of reasoning models sidestepped both problems for domains where correctness is checkable. A math answer either matches; code either passes its tests. Replace the learned reward model with a rule-based verifier and you get reinforcement learning from verifiable rewards (RLVR) — a signal that cannot be sweet-talked and needs no annotators.
DeepSeek-R1 made the lineage public. Its precursor, R1-Zero, was the striking experiment: take a strong base model, skip SFT entirely, and run large-scale RL against verifiable rewards. Long chains of thought emerged — the model learned, unprompted, to work step by step, check itself, backtrack, and try alternative approaches, because those behaviors won reward. (The raw result rambled and mixed languages, so the released R1 added a small cold-start SFT stage and further polish.) The algorithm behind it, GRPO (Group Relative Policy Optimization), is a PPO variant that drops the separate value network: sample a group of responses per prompt, score each, and use the group-normalized score as the advantage —
— which halves the memory bill and stabilizes training. Two more facts complete the overview. First, reasoning distills: R1's chains of thought, used as plain SFT data, transferred much of the capability into small dense models — reasoning turned out to be teachable by demonstration once someone had learned it by RL. Second, every frontier lab now runs some version of this act: the o-series and GPT-5's thinking modes, Claude's extended and adaptive thinking, Gemini's thinking budgets. For you as an engineer, the visible consequences are a new class of "thinking" tokens on your bill and a new knob — how long the model may think — that Lesson 3 examines from the cost side.
It is tempting to file this lesson under background theory. Resist that. The behaviors you will lean on every working day are post-training products, and knowing where they come from changes how you engineer around them:
The acts, on one page:
| Act | Data | Objective | Contributes |
|---|---|---|---|
| Pretraining | Trillions of tokens, unlabeled | Next-token cross-entropy | Knowledge, language, latent reasoning |
| SFT | 10K–100K curated dialogues | Same loss, assistant tokens only | Format, instruction-following, tool-call shape, persona |
| RLHF / DPO | 100K+ preference pairs | Reward + KL tether, or DPO loss | Judgment, calibration, safety — and sycophancy |
| Reasoning RL | Verifiable problems (math, code) | GRPO-family RL on checkable rewards | Deliberate multi-step thinking — and thinking-token costs |