Preparing Datasets for Fine-Tuning

Two teams set out to specialize the same 8B base for the same job: turning a messy support ticket into a resolved case, using three tools. Team one exports eleven thousand historical agent runs from the trace store, converts them wholesale to JSONL, and starts a training job on Tuesday morning. Team two spends the same week differently — they hand-write forty trajectories, argue for two days about eleven of them, generate three thousand more with a large teacher model, filter those down to nine hundred that pass a rubric, and train on that.

Team one's model reproduces every bad habit in eleven thousand runs: the six-tool-call flailing that eventually worked, the abandoned trajectories where the user gave up, the two months when the retriever was broken and the agent compensated by guessing. Team two's model behaves like their best engineer on their best day, because that is literally what it studied. Both teams ran the same trainer with the same hyperparameters. The dataset was the entire difference, and the dataset is almost always the entire difference.

This is the lesson where fine-tuning projects are actually won. You will build the canonical example format including tool calls, learn the four quality properties that decide a run before it starts, write a cleaning and deduplication pipeline, split without leaking, and version the result so that Lesson 6's training run is reproducible and Lesson 8's evaluation is honest. Lesson 4 then takes this machinery to production scale against your real trace store.

1. The Shape of a Training Example

Modern instruction tuning uses the chat format: a list of role-tagged messages, one JSON object per line in a JSONL file. This is what TRL, Axolotl, Unsloth, and LLaMA-Factory all consume natively, and it is a superset of every older format you will meet. A simple single-turn example looks like this:

json

{"messages": [
  {"role": "system", "content": "You triage support tickets."},
  {"role": "user", "content": "Ticket #8841: checkout hangs on Pay since 14:00, EU customers, card never charged."},
  {"role": "assistant", "content": "SEVERITY: high\nCOMPONENT: payments/checkout\nSUMMARY: Checkout hangs at payment submission for EU traffic since 14:00; no charges processed.\nACTION: escalate to payments on-call"}
]}

An agent example is the same structure with two more roles in play. The assistant emits tool_calls instead of (or alongside) content, and the environment answers with a tool message carrying the result. A two-step trajectory:

json

{"messages": [
  {"role": "system", "content": "You triage support tickets. Use tools before concluding."},
  {"role": "user", "content": "Ticket #8841: checkout hangs on Pay since 14:00, EU customers."},
  {"role": "assistant", "content": "",
   "tool_calls": [{"id": "c1", "type": "function",
                   "function": {"name": "search_incidents",
                                "arguments": "{\"component\": \"payments/checkout\", \"since\": \"2026-03-11T13:00Z\"}"}}]},
  {"role": "tool", "tool_call_id": "c1",
   "content": "{\"incidents\": [{\"id\": \"INC-4412\", \"status\": \"open\", \"title\": \"EU card auth timeouts\"}]}"},
  {"role": "assistant", "content": "SEVERITY: high\nCOMPONENT: payments/checkout\nLINKED: INC-4412\nACTION: attach ticket to INC-4412, no new escalation"}
 ],
 "tools": [{"type": "function",
            "function": {"name": "search_incidents",
                         "description": "Find open incidents for a component since a timestamp.",
                         "parameters": {"type": "object",
                                        "properties": {"component": {"type": "string"},
                                                       "since": {"type": "string"}},
                                        "required": ["component"]}}}]}

Three properties of that example are load-bearing and each one is a classic source of ruined runs.

The tool schemas travel with the example. Most chat templates render the available tools into the prompt, so the tools list must be part of the training row, not a global constant applied at training time and forgotten at inference time. If you train with five tools rendered and serve with twelve, you are off-distribution from your own tune.
One consistent system prompt. Whatever system text you train with is what you must deploy with. The pair (system prompt, behavior) is learned together — train with none and deploy with a long one and the model has never seen the condition it is being asked to respond to.
Loss on assistant tokens only. The model should learn to produce assistant turns, not to imitate user text or hallucinate tool outputs. Trainers mask the non-assistant spans for you — but only if you configure it, and a dataset flattened into one concatenated string silently trains on everything. Lesson 4 takes this apart in detail because it is the single most common silent defect in agent SFT.

2. The Chat Template Belongs to the Model

A chat template is a Jinja program shipped in the tokenizer config that turns your message list into the exact token sequence the model was post-trained on: turn delimiters, role markers, the tool-call serialization, the generation prompt. Qwen-family models use one dialect, Llama-family another, and their tool-call encodings differ in ways that are invisible in rendered text and fatal in tokens.

python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

text = tok.apply_chat_template(
    example["messages"],
    tools=example["tools"],      # rendered into the prompt by the template
    tokenize=False,
    add_generation_prompt=False, # False for training rows, True at inference
)
print(text[:600])

Print that string once, early, and read every character of it. You are looking for four things: the delimiters match what the model card documents, the tool schemas appear where the template says they appear, the tool results are wrapped in the model's tool-response markers rather than pasted as user text, and there is no stray whitespace or duplicated system block. Ten minutes here saves a training run.

3. Quality: The Four Properties That Decide the Run

Property	What it means for agent data	Failure it prevents
Correct	Every assistant turn is genuinely what you want produced — the right tool, valid arguments, an answer that actually resolved the case	Mistakes taught with authority: one wrong tool choice, learned as a reflex
Consistent	Same situation, same convention — one severity vocabulary, one argument style, one place where the agent stops and asks	The model rolling dice between the three behaviors you accidentally taught
Diverse	Coverage of trajectory lengths, tool combinations, phrasings, and the ugly inputs production actually sends	Distribution lock-in: brilliant on clean tickets, lost on real ones
Complete at the edges	Failure recovery (the tool errored and the agent handled it), refusal and escalation (this needs a human), and the ambiguity escape hatch (ask one clarifying question)

pythoneditable · runnable

# A cleaning pass over agent SFT rows. Stdlib only; the real version runs
# the same gates over your trace export.
import hashlib, json, re

RAW = [
  {"id": "t1", "user": "Checkout hangs on Pay since 14:00, EU customers",
   "tool": "search_incidents", "answer": "SEVERITY: high\nACTION: link INC-4412"},
  {"id": "t2", "user": "checkout   hangs on PAY since 14:00, EU customers!",
   "tool": "search_incidents", "answer": "SEVERITY: high\nACTION: link INC-4412"},
  {"id": "t3", "user": "Checkout hangs on Pay since 14:00, EU customers",
   "tool": "list_customers", "answer": "SEVERITY: low\nACTION: ask user to retry"},
  {"id": "t4", "user": "Exports time out for accounts over 10k rows",
   "tool": "search_incidents", "answer": "SEVERITY: high\nACTION: page data-platform"},
  {"id": "t5", "user": "Password reset email never arrives for one user",
   "tool": "check_email_log", "answer": "sev: HIGH; action: escalate"},
  {"id": "t6", "user": "Exports time out for accounts with more than 10k rows",
   "tool": "search_incidents", "answer": "SEVERITY: high\nACTION: page data-platform"},
]

SEVERITIES = ("critical", "high", "normal", "low")

def normalize(text):
    return re.sub(r"[^a-z0-9 ]", "", re.sub(r"\s+", " ", text.strip().lower()))

def token_set(text):
    # Unigram bag for a readable demo; production uses MinHash over
    # character shingles, which survives word insertions far better.
    return set(normalize(text).split())

def jaccard(a, b):
    return len(a & b) / len(a | b) if (a | b) else 0.0

def schema_ok(answer):
    first = answer.split("\n")[0]
    return first.startswith("SEVERITY: ") and first.split(": ", 1)[1] in SEVERITIES

kept, dropped = [], []
seen_exact, seen_input, token_index = set(), {}, []

for row in RAW:
    # Gate 1 — mechanical validity of the TARGET. Cheapest, run first.
    if not schema_ok(row["answer"]):
        dropped.append((row["id"], "schema")); continue
    # Gate 2 — exact duplicate of an already-kept (input, output) pair.
    h = hashlib.sha256((normalize(row["user"]) + normalize(row["answer"])).encode()).hexdigest()
    if h in seen_exact:
        dropped.append((row["id"], "exact-dup")); continue
    # Gate 3 — same input, DIFFERENT behavior. Worse than duplication:
    # it teaches contradiction. Needs a human to adjudicate, so quarantine.
    hi = hashlib.sha256(normalize(row["user"]).encode()).hexdigest()
    if hi in seen_input and seen_input[hi] != (row["tool"], row["answer"]):
        dropped.append((row["id"], "contradiction")); continue
    # Gate 4 — near-duplicate by token Jaccard over the input.
    ts = token_set(row["user"])
    if any(jaccard(ts, prev) >= 0.6 for prev in token_index):
        dropped.append((row["id"], "near-dup")); continue

    seen_exact.add(h); seen_input[hi] = (row["tool"], row["answer"])
    token_index.append(ts); kept.append(row)

print("kept   :", [r["id"] for r in kept])
print("dropped:", dropped)
print("retention: {:.0%}".format(len(kept) / len(RAW)))

# A cleaning pass over agent SFT rows. Stdlib only; the real version runs
# the same gates over your trace export.
import hashlib, json, re

RAW = [
  {"id": "t1", "user": "Checkout hangs on Pay since 14:00, EU customers",
   "tool": "search_incidents", "answer": "SEVERITY: high\nACTION: link INC-4412"},
  {"id": "t2", "user": "checkout   hangs on PAY since 14:00, EU customers!",
   "tool": "search_incidents", "answer": "SEVERITY: high\nACTION: link INC-4412"},
  {"id": "t3", "user": "Checkout hangs on Pay since 14:00, EU customers",
   "tool": "list_customers", "answer": "SEVERITY: low\nACTION: ask user to retry"},
  {"id": "t4", "user": "Exports time out for accounts over 10k rows",
   "tool": "search_incidents", "answer": "SEVERITY: high\nACTION: page data-platform"},
  {"id": "t5", "user": "Password reset email never arrives for one user",
   "tool": "check_email_log", "answer": "sev: HIGH; action: escalate"},
  {"id": "t6", "user": "Exports time out for accounts with more than 10k rows",
   "tool": "search_incidents", "answer": "SEVERITY: high\nACTION: page data-platform"},
]

SEVERITIES = ("critical", "high", "normal", "low")

def normalize(text):
    return re.sub(r"[^a-z0-9 ]", "", re.sub(r"\s+", " ", text.strip().lower()))

def token_set(text):
    # Unigram bag for a readable demo; production uses MinHash over
    # character shingles, which survives word insertions far better.
    return set(normalize(text).split())

def jaccard(a, b):
    return len(a & b) / len(a | b) if (a | b) else 0.0

def schema_ok(answer):
    first = answer.split("\n")[0]
    return first.startswith("SEVERITY: ") and first.split(": ", 1)[1] in SEVERITIES

kept, dropped = [], []
seen_exact, seen_input, token_index = set(), {}, []

for row in RAW:
    # Gate 1 — mechanical validity of the TARGET. Cheapest, run first.
    if not schema_ok(row["answer"]):
        dropped.append((row["id"], "schema")); continue
    # Gate 2 — exact duplicate of an already-kept (input, output) pair.
    h = hashlib.sha256((normalize(row["user"]) + normalize(row["answer"])).encode()).hexdigest()
    if h in seen_exact:
        dropped.append((row["id"], "exact-dup")); continue
    # Gate 3 — same input, DIFFERENT behavior. Worse than duplication:
    # it teaches contradiction. Needs a human to adjudicate, so quarantine.
    hi = hashlib.sha256(normalize(row["user"]).encode()).hexdigest()
    if hi in seen_input and seen_input[hi] != (row["tool"], row["answer"]):
        dropped.append((row["id"], "contradiction")); continue
    # Gate 4 — near-duplicate by token Jaccard over the input.
    ts = token_set(row["user"])
    if any(jaccard(ts, prev) >= 0.6 for prev in token_index):
        dropped.append((row["id"], "near-dup")); continue

    seen_exact.add(h); seen_input[hi] = (row["tool"], row["answer"])
    token_index.append(ts); kept.append(row)

print("kept   :", [r["id"] for r in kept])
print("dropped:", dropped)
print("retention: {:.0%}".format(len(kept) / len(RAW)))

Running it keeps t1, t4, and nothing else: t2 collapses onto t1 exactly once whitespace, case, and punctuation are normalized, t3 is the same ticket with a contradictory tool choice and severity, t5 fails the schema gate on a lowercase field name, and t6 is t4 reworded. Retention of 33% on a hand-built six-row sample is unrepresentative, but the direction is real: expect to drop 20-50% of a raw trace export, and treat a pipeline that drops almost nothing as evidence that a gate is broken rather than that your data is pristine.

6. Splits Without Leakage

Three-way split, made before you look too closely at anything: train (about 90%), validation (about 5%, which steers checkpoint selection during Lesson 6's run), and test (about 5%, touched once by Lesson 8's evaluation, after every decision is frozen).

Two rules carry all the weight. Deduplicate before splitting, or your test set is a memorization quiz. And split by natural group whenever one exists — the same customer, the same incident, the same document, the same seed scenario in a synthetic batch must land entirely on one side of the line. Random splitting on grouped data is the most common way teams convince themselves a mediocre fine-tune worked.

pythoneditable · runnable

# Why grouped splitting matters: same incident, many tickets.
import random

random.seed(7)
# 240 tickets belonging to 40 incidents; tickets in one incident are near-copies.
rows = [{"ticket": i, "incident": i % 40} for i in range(240)]

def contamination(train, test):
    train_groups = {r["incident"] for r in train}
    leaked = sum(1 for r in test if r["incident"] in train_groups)
    return leaked / len(test)

# Strategy A: random row split.
shuffled = rows[:]; random.shuffle(shuffled)
cut = int(0.9 * len(shuffled))
print("random split      -> test rows whose incident is also in train: {:.0%}".format(
    contamination(shuffled[:cut], shuffled[cut:])))

# Strategy B: split by group, then take rows.
groups = list({r["incident"] for r in rows}); random.shuffle(groups)
gcut = int(0.9 * len(groups))
train_g, test_g = set(groups[:gcut]), set(groups[gcut:])
train = [r for r in rows if r["incident"] in train_g]
test = [r for r in rows if r["incident"] in test_g]
print("grouped split     -> test rows whose incident is also in train: {:.0%}".format(
    contamination(train, test)))
print("split sizes: train={} test={}".format(len(train), len(test)))

# Why grouped splitting matters: same incident, many tickets.
import random

random.seed(7)
# 240 tickets belonging to 40 incidents; tickets in one incident are near-copies.
rows = [{"ticket": i, "incident": i % 40} for i in range(240)]

def contamination(train, test):
    train_groups = {r["incident"] for r in train}
    leaked = sum(1 for r in test if r["incident"] in train_groups)
    return leaked / len(test)

# Strategy A: random row split.
shuffled = rows[:]; random.shuffle(shuffled)
cut = int(0.9 * len(shuffled))
print("random split      -> test rows whose incident is also in train: {:.0%}".format(
    contamination(shuffled[:cut], shuffled[cut:])))

# Strategy B: split by group, then take rows.
groups = list({r["incident"] for r in rows}); random.shuffle(groups)
gcut = int(0.9 * len(groups))
train_g, test_g = set(groups[:gcut]), set(groups[gcut:])
train = [r for r in rows if r["incident"] in train_g]
test = [r for r in rows if r["incident"] in test_g]
print("grouped split     -> test rows whose incident is also in train: {:.0%}".format(
    contamination(train, test)))
print("split sizes: train={} test={}".format(len(train), len(test)))

The random split leaks essentially 100% — every test ticket has a sibling from the same incident sitting in training, so the test score measures recall of near-copies. The grouped split leaks 0% and reports a number you can act on. The cost is that split sizes drift from your target ratio whenever groups differ in size, which is a price worth paying every time.

7. Versioning and the Datasheet

A dataset is a production artifact, not a file someone had on a laptop. The minimum viable discipline is a content hash, an immutable copy in object storage, and a short datasheet committed next to it. When Lesson 8 says the tuned model regressed on multi-tool cases, the first question is which dataset produced it, and the second is what changed since the last version — both of which are unanswerable without this.

python

import hashlib, json, pathlib

def write_split(rows, path):
    body = "\n".join(json.dumps(r, sort_keys=True, ensure_ascii=False) for r in rows)
    pathlib.Path(path).write_text(body + "\n", encoding="utf-8")
    return hashlib.sha256(body.encode()).hexdigest()[:12]

manifest = {
    "name": "ticket-triage-agent-sft",
    "version": "v3",
    "base_model": "Qwen/Qwen3-8B",
    "splits": {k: write_split(v, "data/{}.jsonl".format(k)) for k, v in dataset.items()},
    "sources": {"production_traces": 8412, "synthetic_teacher": 3100, "hand_written": 62},
    "filters": ["schema", "exact-dup", "contradiction", "[email protected]", "judge>=4", "eval-decontam"],
    "eval_suites_decontaminated_against": ["triage-v4", "tau-bench-retail"],
    "notes": "v3 adds 900 error-recovery trajectories; v2 had none.",
}
print(json.dumps(manifest, indent=2))

The two fields people skip are the ones that matter most six months later. filters tells the next engineer what was already removed, so they do not spend a week rediscovering it. eval_suites_decontaminated_against is the record that your headline number is not measuring memorization — and if it is empty, no score derived from this dataset should be quoted anywhere.

8. The Pre-Training Checklist

Run this before every training run. Each line is one of this lesson's failure stories, compressed:

Canonical chat format throughout, converted at ingestion; the template applied by the tokenizer, never by hand.
Tool schemas carried on the row and identical to what production will send; one system prompt, identical to production's.
The rendered training string printed and read once, end to end.
Every assistant turn reviewed, judged, or outcome-filtered — no raw scrapes, no unfiltered teacher output.
Error-recovery, escalation, and clarifying-question trajectories present at 10-15% of the set.
Exact and near duplicates removed; same-input contradictions quarantined and adjudicated.
Test split sealed, grouped correctly, and decontaminated against every eval suite you intend to report.
Dataset hashed, versioned, and described in a datasheet that names its sources and its filters.