Fine-Tuning with LoRA

Add LoRA adapters, load a 4-bit base for QLoRA, mask SFT labels, train, merge and debug. PyTorch, PEFT and bitsandbytes.

Concepts and terms
Tasks and commands
Annotated code
Diagrams
Defaults and numbers
Troubleshooting
Questions and tips

Source

Tasks#

Task	Code
Add LoRA to a model	`model = get_peft_model(model, lora_config)`
Show trainable share	`model.print_trainable_parameters()`
Load the base in 4-bit	`BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")`
Count trainable params	`count_parameters(model)`
Save the adapter only	`model.save_pretrained("./lora-adapter")`
Merge the adapter into W	`merge_lora_weights(model)`
Skip masked labels in the loss	`F.cross_entropy(..., ignore_index=ignore_index)`
Head-only training	`freeze_body(model)`

From Lesson 11.08, 19.39 code/main.py, 19.38 code/main.py

LoRA path in one layer#

From Lesson 11.08

LoRA from scratch#

class LoRALayer(nn.Module):

def __init__(self, in_features, out_features, rank=8, alpha=16):

super().__init__()

self.scaling = alpha / rankalpha / r

self.A = nn.Parameter(torch.randn(in_features, rank) * (1 / math.sqrt(rank)))

self.B = nn.Parameter(torch.zeros(rank, out_features))BA starts at zero

def forward(self, x):

return (x @ self.A @ self.B) * self.scalingscaled low-rank update

class LinearWithLoRA(nn.Module):

__init__ wraps linear and builds self.lora

for param in self.linear.parameters():

param.requires_grad = Falsefreeze W

def forward(self, x):

return self.linear(x) + self.lora(x)Wx + scaled BAx

From Lesson 11.08, 11.08 code/lora.py

QLoRA with PEFT#

bnb_config = BitsAndBytesConfig(load_in_4bit=True,4-bit base

bnb_4bit_quant_type="nf4",NF4 levels

bnb_4bit_compute_dtype=torch.bfloat16,

bnb_4bit_use_double_quant=True)quantize the scales too

model = AutoModelForCausalLM.from_pretrained("<model>",

quantization_config=bnb_config, device_map="auto")

lora_config = LoraConfig(task_type=TaskType.CAUSAL_LM, r=16,

lora_alpha=32, lora_dropout=0.05,

target_modules=["q_proj", "v_proj"])query and value

model = get_peft_model(model, lora_config)adds A and B

model.save_pretrained("./lora-adapter")10-100MB adapter

From Lesson 11.08

One SFT training step#

From Lesson 19.39, 19.39 code/main.py

Key formulas#

LoRA forward: $inline$ with $inline$ frozen.
Scaled update: $inline$
Parameter count: rank $inline$ trains $inline$ parameters instead of $inline$ .
Example: $inline$ on 4096x4096 trains 131,072 of 16,777,216. That is 0.78%.
Merge: $inline$ . Same size, no inference overhead.
Full fine-tune memory: 8B in fp16 is about 56GB with gradients and Adam states.

From Lesson 11.08, Lesson 10.06

Tips#

Try prompt engineering first, then RAG, then fine-tuning.
Evaluate every 100 steps on a held-out set. Keep the best checkpoint.
Compare base plus prompt against base plus LoRA on 200+ held-out examples.
On 800 examples, train the head only with a frozen body.
Track raw text loss. A rise above 10-15% means SFT is too aggressive.

From Lesson 11.08, 11.08 outputs/skill-fine-tuning-guide.md, Lesson 19.38, Lesson 10.06

Defaults#

Setting	Default	Note
r	16	safe start for most tasks
lora_alpha	32	2 x rank, alpha = rank is conservative
lora_dropout	0.05	0.1 below 5K examples
target_modules	q_proj, v_proj	add more only if quality is low
QLoRA learning rate	2e-4	higher than full fine-tuning
LoRA fp16 learning rate	5e-5
Full SFT learning rate	2e-5	Llama 2 Chat, 15x below pre-training
Epochs	1-3	more risks overfitting

From 11.08 outputs/skill-fine-tuning-guide.md, 11.08 outputs/prompt-lora-advisor.md, Lesson 11.08, Lesson 10.06

Troubleshooting#

Symptom	Cause	Fix
NaN losses with Llama models	pad token not set	set the pad token
Loss does not decrease	learning rate too low for LoRA	try 2e-4
Loss falls, eval quality bad	overfitting	fewer epochs, more dropout, more data
Weak on complex tasks	rank too low or wrong targets	try r=32 or add MLP layers
General skills drop after SFT	learning rate too high	use 1e-5 to 5e-5 and 1-3 epochs
Model predicts the instruction	no loss mask	set `-100` on instruction and padding
OOM on long sequences	optimizer states exceed GPU memory	use `optim="paged_adamw_8bit"`

From 11.08 outputs/skill-fine-tuning-guide.md, Lesson 11.08, Lesson 10.06, Lesson 19.39

Abbreviations#

LoRA: Low-Rank Adaptation. Freeze W, train A and B.
QLoRA: Quantized LoRA. 4-bit NF4 base, fp16 adapters.
NF4: Normal Float 4. 16 levels at normal quantiles.
PEFT: Hugging Face library for LoRA, QLoRA, DoRA and IA3.
TRL: Hugging Face trainers for SFT, DPO, GRPO, PPO and ORPO.
SFT: Supervised Fine-Tuning on instruction and response pairs.
ChatML: role format with <|im_start|> and <|im_end|> tokens.
Double quantization: quantizes the per-block scales to fp8.

From Lesson 11.08, Lesson 10.06

Questions#

Why is B zero at init?: BA starts at zero. Training starts from the original behavior.
What does alpha do?: It scales the update by alpha / r, apart from the base learning rate.
Merge or keep separate?: Keep separate to serve many tasks. Merge for one model.
Why mask the instruction?: Unmasked, the model learns to predict the instruction and the response signal gets smaller.
When not to fine-tune?: When a prompt works, when you need facts (use RAG) or below 500 examples.

From Lesson 11.08, Lesson 19.39, 11.08 outputs/skill-fine-tuning-guide.md

Choose a method#

Choice	Use when
Full fine-tune, about 56GB for 7B	VRAM >= 2x model size in fp16
LoRA on fp16 base, about 18GB	VRAM >= model size in fp16
QLoRA on 4-bit base, about 6GB	VRAM >= model size / 4
`r=16`	multi-domain tasks, instruction following, chat
`r=32`	code generation, complex reasoning, math

From 11.08 outputs/prompt-lora-advisor.md, Lesson 11.08