Models
GPU memory, quantization and hardware
Where GPU memory goes, how quantization frees it, and how work splits across GPUs.
10 of 15 planned
L1 1L2 3L3 5L4 1
RolesInference Engineering
- The model is 35 GB now, so it fits
- Count the KV bytes per token from the config
- Pick a 4-bit format for a 14B model on 24 GB
- Why an 8B model fails to start on a 24 GB card
- Pick the memory fraction on unified memory
- Split the model or copy it across two cards
- The FP8 KV scales that never loaded
- The pool was 95 percent empty and the server died
- Why FP8 grew the pool 4.77 times and not 2
- Eight copies of the same KV cache
No item matches this level.
Taught in
The answers on this card come from these lessons and manuals.
- Scaling: Distributed Training, FSDP, DeepSpeed
- Quantization: Making Models Fit
- DualPipe Parallelism
- Gradient Checkpointing and Activation Recomputation
- Fine-Tuning with LoRA & QLoRA
- Hardware-Specialized Inference Compilation
- Production Quantization
- Gradient Clipping and Mixed Precision
- Gradient Accumulation
- Checkpoint Save and Resume
- Sharded Checkpoint and Atomic Resume
- End-to-End Distributed Training