End-to-End LLM and Fine-Tuning Pipeline

Plan, run and gate a hashed stage DAG. Head swap, SFT loss mask and DPO loss from the capstones, with defaults.

Commands#

Command	What it does
`python code/main.py plan`	Validates the manifest, prints DAG and gates
`python code/main.py run`	Runs stages, prints the manifest JSON
`python code/main.py gate`	Applies eval gates. Prints `SHIP` or `HOLD`
`python code/main.py`	Demo: plan, run, a passing and a failing gate
`./pipeline.sh config/llama3.3-8b-domainX.yaml`	Capstone 07: data to model card
`run_demo()`	Capstones 38 to 40. Exits zero on success

pipeline_version: 1.2.3

seed: 42

git_commit: a1b2c3d4

stages:

01_tokenizer:

input_hash: sha256:...output hash of the stage before

output_hash: sha256:...input hash of the next stage

gates:

mmlu: >= baseline + 0.5no regression

humaneval: >= baseline + 1.0

safety_refusal_rate: <= 0.05

kl_from_reference: <= 25.0KL budget gate

cost_total_usd: <= 50000

diff_w = logp_w_pol - logp_w_refchosen log-ratio

diff_l = logp_l_pol - logp_l_refrejected log-ratio

margin = diff_w - diff_limplicit reward difference

z = beta * margin

loss = -F.logsigmoid(z)numerically stable

for p in reference.parameters():

p.requires_grad = Falsereference never trains

reference.eval()

Factor	38 Classifier head swap	39 Instruction SFT	40 DPO
Builds	Body + mean pool + 2-class head	`TinyGPT` trained on response tokens	Policy against a frozen reference
Data	800 SMS, 640 train, 160 test	200 pairs, 160 train, 40 test	12 (prompt, chosen, rejected) triples
Model	hidden 64, 4 heads, 2 blocks, max 32	hidden 96, 2 blocks	hidden 64, 2 blocks
Train	20 epochs head-only, 20 full	20 epochs, lr 5e-4	30 epochs, beta 0.2, lr 1e-3
Metric	Precision, recall, F1, confusion	Exact-match per category	Chosen minus rejected margin
Check	Head-only and full both report	Exact-match rises over epochs	Loss falls, margin grows

Task	Code
Freeze the body for head-only training	`freeze_body(model)`
Pool hidden states with the mask	`mean_pool(h, mask)`
Pad a batch and mask labels with -100	`sft_collate(batch)`
Causal loss on response tokens	`shifted_loss(logits, labels)`
Score a completion under a model	`sequence_log_prob(...)`
DPO loss and implicit reward margin	`dpo_loss(lp_w_pol, lp_l_pol, lp_w_ref, lp_l_ref, beta=cfg.beta)`
Start the policy from the reference	`policy.load_state_dict(reference.state_dict())`

Symptom	Cause	Fix
`HOLD: missing metric <metric>`	Eval report has no value	Add the metric to the eval stage
`input hash missing for stage`	Upstream artifact not in store	Re-run the upstream stage
`halted_over_budget`	Total cost above `budget_usd`	Fix the cost estimate at plan time
DPO model sent through the SFT path	Untyped artifacts	Typed artifacts and stage signatures
Model learns to predict the instruction	No loss mask	`-100` on instruction, pad, boundary
Full fine-tuning overfits 800 examples	Small data washes out the body	Train the head only
MMLU-Pro accuracy jumps	Contamination in training data	MinHash check, reject any overlap
DPO prefers short completions	Length bias	Divide log-prob by completion length

Which two gates catch most disasters?: A regression gate and a KL budget gate.
Reproducible or deterministic?: Reproducible. Floats differ at the 1e-5 level.
Why does log Z(x) drop out of DPO?: It is the same for chosen and rejected.
Why greedy for exact-match?: Temperature makes the metric stochastic.