Introduction to Large Language Models
30 min readreadingLLM Foundations for Engineers
1 of 17LLM Engineering: APIs, Inference & Structured Output
Up next · How LLMs Are Trained: Pretraining to RLHF
It is a few minutes before midnight and a developer is staring at a stack trace she has never seen before. The service went down an hour ago, the on-call channel is quiet, and the error message references a library she did not know her codebase used. She pastes the trace into a chat window. Ninety seconds later she has an explanation of the failure, the exact configuration flag that triggers it, and a patch — with a caveat that the patch changes retry semantics and a suggestion to check one specific test. The thing on the other side of that chat window has never seen her codebase. It has no debugger attached, no access to her logs, and no idea what her company does.
What it has is a statistical model of language so large and so thoroughly trained that explaining stack traces, writing patches, and hedging about retry semantics all fall out of a single, almost embarrassingly simple objective: predict the next token. That is the entire trick. There is no database of answers inside, no rule engine, no lookup table of stack traces. There is a neural network that, given a sequence of text, outputs a probability for every possible next piece of text — and everything you will build in this course sits on top of that one capability.
This course — LLM Engineering: APIs, Inference & Structured Output — teaches the daily craft of driving these models through their APIs: making calls that come back reliably, streaming tokens to users, forcing output into shapes code can parse, and keeping the bill under control. Its sibling course, Transformer Architecture & LLM Internals, opens the model itself; here the model is a component you integrate. This first lesson builds the mental model that everything else rests on: what an LLM actually computes, why that produces something that looks like understanding, what the 2026 model landscape offers you, where the hard limits are, and what "LLM engineering" means as a discipline distinct from prompting folklore.
Strip away the chat interface, the system prompts, and the product polish, and a large language model is a function. It takes a sequence of tokens — chunks of text, typically a few characters each — and returns a probability distribution over its entire vocabulary for what token comes next:
Feed it "The capital of France is" and the distribution puts most of its mass on " Paris", a little on " a", almost none on " photosynthesis". Sample one token from that distribution, append it to the sequence, and ask again. Repeat a few hundred times and you have a paragraph. Every essay, every function, every apology an LLM has ever produced was generated one token at a time by this loop — each step conditioned on everything that came before it.
Two details of this loop matter more than anything else in the course, because every technique you will learn — prompting, structured output, tool calling, agents — is a way of manipulating one of them:
On first contact, "next-token prediction" sounds like it should produce autocomplete — and until about 2019, it did. The surprise of the decade is that the same objective, pushed hard enough, produces translation, arithmetic, code synthesis, and multi-step reasoning. Nobody programmed those capabilities in. They emerged.
The cleanest way to see why is to notice that predicting text well eventually requires modeling whatever process produced the text. To predict the next token of "237 + 486 =" you must do addition. To predict the next line of a Python function you must track variable types and intent. To predict the last sentence of a murder mystery you must have followed the plot, the clues, and the misdirection across a hundred pages. A model that merely memorized surface patterns would plateau quickly; loss keeps falling only if the model internalizes grammar, facts, style, and — at sufficient scale — something that behaves a lot like reasoning.
This is also the honest frame for the model's limits: the model learned the distribution of text, not a verified database of facts. When the distribution says a confident-sounding citation is likely, the model produces one — whether or not the paper exists. Fluency and truth are correlated in the training data, not guaranteed by the machinery. Hold on to that asymmetry; it explains most production incidents involving LLMs.
One more distinction before the history: the raw artifact of next-token training is a base model — a mirror of internet text that continues patterns rather than answering questions. The assistant behavior you actually meet at an API — answering, following instructions, calling tools, refusing some requests — is layered on afterward by post-training. Knowledge comes from pretraining; behavior comes from post-training. Lesson 2 walks that pipeline end to end, because half the model behaviors you will engineer around are post-training fingerprints.
Language modeling is old — statistical n-gram models shipped in the 1990s, and recurrent networks predicted tokens for a decade before anyone cared. What changed in 2017 was the Transformer ("Attention Is All You Need"): by letting every token attend to every other token in parallel, it removed the training bottleneck of recurrence, and better language models became something you could simply buy with compute and data. The years since are the story of that purchase:
| Year | Milestone | Why it mattered |
|---|---|---|
| 2017 | Transformer | Attention replaces recurrence; scale becomes purchasable |
| 2018–19 | GPT-1 / GPT-2 | Pretrain on raw text, then adapt — the recipe that stuck |
| 2020 | GPT-3 (175B) | In-context learning emerges; the API era begins |
| 2022 | ChatGPT | RLHF meets a chat box; 100M users in two months |
| 2023 | GPT-4, Llama 2 | Frontier multimodal reasoning; open weights go mainstream |
| 2024 | Claude 3, Gemini 1.5, DeepSeek-V3 | 200K–1M contexts; an open MoE nears closed quality at a fraction of the training cost |
| 2025 | Reasoning generation | o-series, DeepSeek-R1, GPT-5, Gemini 3: models that spend inference compute thinking before answering |
| 2026 | You are here | Model quality is a moving target; the durable skill is the engineering around it |
The most consequential empirical discovery behind that table is that model quality improves predictably with scale. Loss falls as a power law in parameters and training tokens , smoothly, across seven orders of magnitude:
The 2022 Chinchilla paper sharpened this into a budgeting rule: for a fixed compute budget there is an optimal balance between model size and data volume — roughly 20 training tokens per parameter — and the field had been training models that were too big on too little data. A 70B model trained on 1.4T tokens beats a 280B model trained on 300B tokens at the same compute cost. That single result redirected hundreds of millions of dollars of training budgets toward data.
Smooth loss curves hide a stranger phenomenon. Individual capabilities do not improve smoothly — they snap into existence. Three-digit arithmetic is absent at 1B parameters, unreliable at 10B, and suddenly dependable at much larger scale (or at smaller scale with better data and training, as modern distilled models show). Researchers call these emergent abilities, and their practical consequence is humility: benchmark a specific model for your specific task, because capability at one scale tells you surprisingly little about the next scale down.
The landscape has settled into a stable two-tier shape: a handful of frontier labs shipping closed models behind APIs at the capability edge, and a vigorous open-weights ecosystem within arm's reach behind them. You will move between both worlds professionally, so know the map:
| Family | Developer | Weights | Known for |
|---|---|---|---|
| Claude (Haiku / Sonnet / Opus) | Anthropic | Closed | Coding and agentic work; 1M-token contexts; adaptive thinking |
| GPT-5 family | OpenAI | Closed | Integrated reasoning; the largest product ecosystem; Responses API (the older Assistants API sunsets in 2026) |
| Gemini 3 (Flash / Pro) | Google DeepMind | Closed | 1M-token context; native multimodality; deep Google Cloud integration |
| Llama 4 (Scout / Maverick) | Meta | Open | Mixture-of-experts; very long contexts; the largest open ecosystem |
| Qwen 3 | Alibaba | Open | Sizes from sub-1B to 235B MoE; multilingual strength; hybrid thinking modes |
| DeepSeek V3 / R1 | DeepSeek | Open | 671B MoE (37B active); frontier reasoning at a fraction of the training cost |
| Mistral | Mistral AI | Mixed | Efficient models and MoE pioneers; strong European presence |
| Gemma 3 | Google DeepMind | Open | 1B–27B dense, distilled from larger teachers; runs on one GPU |
Two facts about this table shape your working life. First, the frontier APIs price in the same currency — dollars per million tokens — but at very different rates: within one provider's lineup the small tier can be 5–25x cheaper than the large one, which is why "which model does this task deserve" is a recurring engineering decision, not a one-time choice. Second, the open-weight column is a genuine alternative, not a toy: models you can download and serve yourself (with vLLM or SGLang in production, Ollama on a laptop) now sit a year or less behind the frontier, which changes negotiations about privacy, cost, and vendor lock-in. This course teaches against both: frontier APIs for capability, a local open-weight model as your zero-cost sandbox from Lesson 4 onward.
You will get further in this field by knowing the failure modes cold than by memorizing benchmark scores. Four limits are structural — they follow from how the models are built, and no prompt phrasing makes them disappear:
| Limit | Why it exists | What engineers do about it |
|---|---|---|
| Hallucination | The model outputs plausible text, and plausibility is not truth | Validate outputs in code; ground answers in supplied context; verify claims with tools |
| Knowledge cutoff | Weights are frozen at training time | Put fresh information into the context window; give the model search and retrieval tools |
| No persistent memory | Each API call starts from nothing but the prompt | Manage conversation state client-side (this course); build agent memory systems (Agent Architectures: Loops, Planning & Memory) |
| Token-level blindness | The model sees subword chunks, not characters or digits | Route arithmetic and exact string work to code (Tool Use, Code Execution & Sandboxing) |
Here is the job in one sentence: driving models you did not train, through APIs you do not control, to product-grade reliability at a cost your business survives. The demo version of an LLM feature is one API call and takes an afternoon. The production version has to survive a provider outage at 2 a.m., a user pasting a 400-page contract, a response that is almost valid JSON, a rate limit hit during a traffic spike, and a finance team asking why the API bill doubled. The distance between those two versions is this course.
Concretely, the craft decomposes into surfaces you will practice
one by one: authentication and secrets hygiene; the request and
response anatomy of the major APIs; streaming (a 500-token answer
takes seconds to finish but the first token can be on screen in a
few hundred milliseconds — users experience the difference as
speed); conversation state, which lives entirely on your side of
the wire; structured output — coercing the model
into schemas that survive json.loads — which is what
makes LLMs composable with ordinary software; tool and function
calling; retries, timeouts, idempotency, and fallbacks; and cost
and latency engineering, where model routing and caching routinely
cut bills by 5–10x. None of this is glamorous. All of it is what
separates AI engineers who ship from people with impressive demos.
The plan is simple. This first section builds foundations: how the models are trained and structured, and a working development environment with cost tracking from day one. From there the course moves through the API craft itself — requests, streaming, and conversation state — then structured output and tool calling, and finally the inference and reliability engineering (sampling, caching, batching, retries, evaluation) that makes it all production-grade. By the end you will have shipped small systems that call two providers, degrade gracefully, and report their own costs.
One habit to carry from here: whenever a model's behavior surprises you in the coming lessons, come back to the loop in Section 1. Somewhere, a distribution over tokens was shaped by training or by context, and a sampler made a choice. There is no other machinery. The next lesson opens the training side of that story: how a random-weight network becomes an assistant, in three acts — pretraining, supervised fine-tuning, and preference optimization.