The Decision Framework: Prompt, RAG or Weights
35 min readreadingWhen to Specialize
2 of 15Model Specialization: Fine-Tuning & Agentic RL
Up next · Preparing Datasets for Fine-Tuning
The finance business partner has one question and she asks it in the first minute: "what does a resolved ticket cost us?" The answer, once someone divides the model bill by the resolution count, is 4.1 cents, and 4.1 cents times 200,000 tickets a month is a line item large enough to have its own meeting. Somebody says "we should fine-tune a small model." Somebody else says "we should trim the system prompt." A third person says the retrieval step is where the tokens actually go. All three are proposing the same thing — move work out of the expensive per-call path — and none of them has the numbers that decide which move wins.
Lesson 1 gave you a ladder. This lesson gives you the arithmetic and the traps, because the ladder is only useful if you can price each rung. There are four instruments for changing what an agent does — prompt and context engineering, retrieval, fine-tuning, and distillation — and they differ less in what they can achieve than in their cost curves: what you pay up front, what you pay per call, how fast you can iterate, and what you pay forever afterwards.
By the end you will be able to do three things: price a fine-tuning project honestly before starting it, recognize the four situations where fine-tuning is a trap regardless of how appealing it sounds, and see why the instruments are not rivals at all — the expensive one generates the data that trains the cheap one, which is the flywheel that makes this entire course pay for itself.
| Instrument | Changes | Setup | Iteration loop | Marginal cost per call | Maintenance |
|---|---|---|---|---|---|
| Prompt / context engineering | What the model is told | Hours | Seconds | High — you pay for every instruction token, every call | Trivial: edit a string, redeploy |
| Retrieval | What the model knows right now | Days to weeks | Minutes (re-index) | Moderate — retrieval latency plus retrieved tokens | Ongoing: index freshness, chunking, recall drift |
| Fine-tuning | What the model does by default | Weeks (mostly data work) | Hours to days | Low — behavior is free once it is in the weights | Heavy: dataset, pipeline, evals, base-model upgrades |
| Distillation | Which model can do the job | Weeks (teacher inference + fine-tuning) | Days | Lowest — a small model serving a large model's behavior | Heavy, plus a teacher dependency |
Read the table as two columns that trade against each other. Prompting has near-zero fixed cost and the highest marginal cost: every token of tool guidance, every few-shot trajectory, every formatting rule is re-transmitted and re-processed on every single request, forever. Fine-tuning inverts it: a large fixed cost that buys a permanently cheaper per-call path. That is the whole economic argument, and it means the decision is a function of volume at least as much as it is a function of quality.
Concrete numbers, at 2026 prices, for the support agent from the opening scene. The current design calls a frontier hosted model with roughly 6,000 prompt tokens (system prompt, tool schemas, few-shot trajectories, retrieved context) and produces about 400 completion tokens per ticket. At frontier pricing of roughly $3 per million input tokens and $15 per million output tokens, one ticket costs about $0.024 in model spend, and 200,000 tickets a month costs about $4,800.
The alternative is a specialist 8B, fine-tuned on your traces, served on your own GPU with vLLM. The system prompt drops to 800 tokens because the tool discipline and output format now live in the weights. But the cost structure changes shape entirely: you no longer pay per token, you pay for provisioned capacity. One H100-class GPU at roughly $2.50 per hour is about $1,800 a month whether it serves ten requests or ten million. Self-hosting is a fixed cost pretending to be a variable one, and the single most common mistake in these reviews is comparing a per-token price to a per-hour price without dividing by utilization.
Run it and the shape of the answer is unmistakable. At 20,000 requests a month the hosted frontier model is cheaper and the specialist is a vanity project. Break-even lands around 75,000 requests a month. At 200,000 you save roughly $3,000 a month against an $18,000 project, so payback is about six months — a real but unspectacular business case that lives or dies on whether the quality holds. At five million requests a month the frontier bill is $120,000 and the specialist is a rounding error, and nobody needs a spreadsheet to see it.
Two adjustments make the model more honest. First, utilization: if your traffic is bursty and you must provision for peak, divide your effective throughput accordingly, or use a serverless fine-tuned endpoint that charges per token for LoRA adapters, which is the right answer for the awkward middle band between 50,000 and 500,000 requests a month. Second, the latency term: a tuned 8B answering in 400 ms where a frontier model took 2.5 s is often worth more to the product than the dollars, and should be priced as a product improvement rather than folded into the cost line.
The cost table has a column that engineers underweight and it is not the money one. Prompt iteration takes seconds: change a sentence, re-run the eval suite, read the delta, change it again. You can run forty experiments in an afternoon. Retrieval iteration takes minutes because re-indexing and re-scoring sit in the loop. Fine-tuning iteration takes hours in the best case and days in the realistic one, because every experiment implies a dataset change, a training run, an eval sweep, and a forgetting probe.
This matters more than the dollar figures early in a project, because early in a project you do not yet know what good behavior looks like. The discipline that follows is simple and almost universally violated: specify the behavior with prompts until it stops changing, then bake the stabilized behavior into weights. A prompt is a fast, editable, human-readable specification. Fine-tuning is a compiler that turns a finished specification into a cheaper artifact. Compiling a specification that is still being argued about is how teams end up with four adapters, none of which anyone trusts.
The four instruments are usually presented as alternatives. In a mature agent system they are stages of one pipeline, and seeing that is what turns fine-tuning from a project into a capability.
You ship version one on a strong hosted model with a long, carefully engineered prompt. It is expensive per call and it is also, every single day, generating exactly the artifact you need: thousands of trajectories showing a competent model choosing tools, formatting arguments, recovering from errors, and completing tasks — each one already labelled by production reality with whether the ticket was resolved, whether the user thumbed it up, whether the refund went through, whether a human had to intervene. Your traces are a training set that is being written for you while you sleep. The observability stack from Observability, Reliability & Cost Engineering is, viewed from this course, a data collection pipeline.
The flywheel turns like this. Traffic produces traces. Traces filtered by success signals and quality judgments produce a training set. The training set produces a specialist model that serves the common path at a fraction of the cost and latency. The savings fund more traffic and more ambitious features, which produce more traces. Meanwhile the frontier model stays in the architecture for the hard tail — the 10% of requests the specialist should hand off rather than fumble — and continues to generate the next dataset.
Four situations where the training run will complete successfully, the loss curve will look beautiful, and the project will still fail.
Moving targets. Your tool catalogue changes every sprint, the output schema is still under negotiation, or the product's core flow is being redesigned. Weights freeze at the end of the run; a specification that changes monthly means retraining monthly, and each retrain drags the full dataset-plus-eval tail behind it. Wait for the target to stop moving. If it never stops moving, the answer is a prompt.
Tiny data. You have 150 examples and a strong belief that they are excellent. Narrow format tasks genuinely respond at a few hundred examples, but agent behavior — multi-turn, multi-tool, error-recovering — does not. With too little data you memorize rather than generalize, and the model's confident performance on inputs resembling those 150 is not evidence of anything. Either invest in data generation (Lesson 4) or stay on the prompt.
Capability-versus-behavior confusion. This is the expensive one. If the base model cannot do the task at all with a perfect prompt and unlimited context, fine-tuning rarely conjures the missing capability. Low-rank adaptation steers machinery that pretraining built; it does not build new machinery. The diagnostic is cheap: give the base model the ideal prompt, the ideal retrieved context, and as many tries as it wants. If it still cannot do the task, you have a capability gap, and the honest options are a stronger base, a decomposition into easier subtasks, or — genuinely, sometimes — reinforcement learning on a verifiable reward, which is Lesson 10's subject precisely because it can push past what demonstrations teach.
Fine-tuning to compensate for a broken component. The retriever misses the right chunk 30% of the time, so someone proposes training the generator to guess better from bad context. The tool returns ambiguous errors, so someone proposes training the model to interpret them. Both work slightly, both entrench the defect, and both are far more expensive than fixing the component. Fine-tuning is not a patch layer for the rest of the system.
Your team measures the base model with a hand-tuned ideal prompt, full retrieved context, and five attempts per case. It solves 34% of the eval suite. The proposal on the table is a 12,000-example SFT run to reach 85%. The most accurate objection is:
Distillation deserves separate treatment because it is the pattern that most reliably produces a specialist agent worth shipping. The setup: a strong teacher model — a frontier hosted model or a large open-weights one — produces trajectories for your task, and a small student learns to reproduce them. In the strict sense distillation matches the teacher's output distribution over tokens; in practice, for closed teachers, you distil from sampled outputs, which is ordinary SFT on teacher-generated data and is what almost everyone means by the word.
It works because the student is not being asked to become generally as smart as the teacher. It is being asked to reproduce the teacher's behavior on one narrow distribution — your tickets, your tools, your schemas — and narrow distributions are where small models close the gap. An 8B that has seen 20,000 of a frontier model's trajectories on your task routinely matches it on that task while failing badly on anything else, which is precisely the trade you wanted.
Three constraints keep distillation honest. The teacher's mistakes become the student's mistakes with full confidence, so filtering is mandatory and Lesson 4 builds the filters. The student inherits the teacher's reasoning style but not its knowledge, so anything factual still belongs in retrieval. And the terms of service on many hosted models restrict using their outputs to train competing models — read them, because the legal exposure lands on the company, not on the notebook.
Scenario A — the internal DevOps agent. Forty engineers use it, roughly 900 requests a day. It reads runbooks, queries Grafana, and drafts incident timelines. Complaints: it sometimes uses last quarter's runbook, and its timelines are inconsistently formatted. Verdict: no fine-tuning. The volume is 27,000 requests a month, an order of magnitude below the break-even for a dedicated GPU, so the economics motive is absent. The runbook complaint is knowledge-shaped and belongs to retrieval freshness. The formatting complaint is real behavior, but at this volume a 600-token format specification in the prompt costs about $50 a month, which is less than one afternoon of the engineering it would take to remove it.
Scenario B — the ticket triage agent. 200,000 tickets a month, three tools, a fixed output schema that has not changed in five months, and eleven months of traces with a resolution flag on every one. The eval suite says the frontier model scores 0.91 and the best-prompted 8B scores 0.68, with failures concentrated in tool selection and schema violations. Verdict: fine-tune, and specifically distil. Every precondition is present: stable target, behavioral gap, abundant labelled data, and volume above break-even. The design is a distillation run from the frontier traces into the 8B, with the frontier model retained as an escalation path for the cases the specialist's confidence signal flags. Expected outcome from this shape of project is the specialist landing in the high 0.80s at roughly a third of the cost.
Scenario C — the compliance research agent. Low volume, 600 requests a month, but each one must cite regulation text exactly, the regulations are amended quarterly, and a wrong citation is a reportable incident. Someone proposes fine-tuning on the regulatory corpus so the model "knows the rules." Verdict: no fine-tuning, and the proposal is actively dangerous. The requirement is verbatim citation of a changing corpus, which is the canonical retrieval problem; weights produce paraphrase without provenance, and a fine-tuned model that hallucinates regulation text in the house style is strictly worse than a base model that hedges. If anything is trained here, it is a re-ranker or a citation-verification classifier — narrow, checkable components — not the agent's judgment.
Six questions. If any answer is no, you are not ready to train, and the answer tells you what to do instead.