Math & training · Glossary term
What is Loss Function?
An objective that maps predictions and targets, sometimes with regularization terms, to a value optimization tries to reduce. The loss determines which errors training directly rewards or penalizes.
“A number that measures training error.”
What is the common confusion about Loss Function?
A low training loss does not guarantee useful, safe, or generalizable behavior on production tasks.
Learn Loss Function in the course
Lessons that name Loss Function in a title or section
- Loss Functions
Your network makes a prediction. The ground truth says otherwise. How wrong is it? That number is the loss. Pick the wrong loss function and your model optimizes for the wrong thing entirely.
- Information Theory
Information theory measures surprise. Loss functions are built on it. Language: Python Compute entropy, cross-entropy, and KL divergence from scratch and explain their relationship.
- Norms and Distances
Your distance function defines what "similar" means. Choose wrong and everything downstream breaks. Language: Python Implement L1, L2, cosine, Mahalanobis, Jaccard, and edit distance functions from…
- Support Vector Machines
Find the widest street between two classes. That is the entire idea. Language: Python Implement a linear SVM from scratch using hinge loss and gradient descent on the primal formulation.
- Build Your Own Mini Framework
You have built neurons, layers, networks, backprop, activations, loss functions, optimizers, regularization, initialization, and LR schedules. All as separate pieces.
- Introduction to PyTorch
You built the engine from pistons and crankshafts. Now learn the one everyone actually drives. Build and train neural networks using PyTorch's nn.Module, nn.Sequential, and autograd.
Covered in Phase 01: Math Foundations, Phase 02: ML Fundamentals and Phase 03: Deep Learning Core.
Related terms
- Cross-EntropyA loss based on the negative log probability assigned to the target outcome. In next-token training, it penalizes the model when it…
- GradientA vector of partial derivatives pointing in the direction of steepest increase. In ML, you go opposite to the gradient (gradient descent)…
- Evaluation (Eval)A defined process for measuring model or system behavior on representative tasks using explicit success criteria, data, scorers, and…
- Contrastive LearningTraining by pulling similar pairs closer and pushing dissimilar pairs apart in embedding space.
- GAN (Generative Adversarial Network)A generator network tries to create realistic data while a discriminator network tries to tell real from fake.
- Knowledge DistillationTraining a student model to reproduce selected behavior or output distributions from a more capable teacher, often alongside ordinary…
- UnderfittingA model or training setup has insufficient effective capacity, optimization, features, or training signal to capture useful patterns in…
More terms in Math & training
- Activation Checkpointing
- Activation Function
- Adam (Optimizer)
- AdamW
- Autograd
- Backpropagation
- Batch Size
- Contrastive Learning
- Cross-Entropy
- Data Augmentation
- DPO (Direct Preference Optimization)
- Dropout
- Eigenvalue
- Epoch
- Fine-tuning
- Gradient
- Gradient Accumulation
- Gradient Clipping
- Gradient Descent
- Hyperparameter
- JAX
- Knowledge Distillation
- Learning Rate
- Learning Rate Schedule
- LoRA (Low-Rank Adaptation)
- Mixed Precision
- NaN (Not a Number)
- Normalization
- Optimizer
- Overfitting
- QLoRA
- ReLU
- RLHF (Reinforcement Learning from Human Feedback)
- SFT (Supervised Fine-Tuning)
- Softmax
- Stochastic Gradient Descent (SGD)
- Transfer Learning
- Underfitting
- Warmup
- Weight
- Weight Decay
This entry comes from glossary/terms.md on GitHub. Browse all 250 glossary terms.