What is NaN (Not a Number)?

NaN (Not a Number) in practice

Find the first non-finite tensor, inspect its inputs, and add assertions or anomaly detection near that operation.

Learn NaN (Not a Number) in the course

Lessons that name NaN (Not a Number) in a title or section

• Numerical Stability

  Floating point is a leaky abstraction. It will bite you during training, and you will not see it coming. Language: Python Implement numerically stable softmax and log-sum-exp using the…

  Phase 01: Math Foundations

• Debugging Neural Networks

  Your network compiled. It ran. It produced a number. The number is wrong and nothing crashed. Welcome to the hardest kind of debugging -- the kind where there is no error message.

  Phase 03: Deep Learning Core

• Gradient Clipping and Mixed Precision

  The optimizer and schedule from the previous lesson assume gradients are sane. They usually are not. A single bad batch can spike the gradient norm by three orders of magnitude.

  Phase 19: Capstone Projects

Covered in Phase 01: Math Foundations, Phase 03: Deep Learning Core and Phase 19: Capstone Projects.

Related terms

• Mixed PrecisionA numerical strategy that uses different data types for different operations, often lower precision for many matrix operations and higher…
• Learning RateA scale factor used by an optimizer to control parameter-update magnitude. Values that are too large can destabilize training; values that…
• GradientA vector of partial derivatives pointing in the direction of steepest increase. In ML, you go opposite to the gradient (gradient descent)…
• Gradient ClippingLimiting gradient values or their combined norm before an optimizer update when they exceed a chosen threshold.

More terms in Math & training

• Activation Checkpointing
• Activation Function
• Adam (Optimizer)
• AdamW
• Autograd
• Backpropagation
• Batch Size
• Contrastive Learning
• Cross-Entropy
• Data Augmentation
• DPO (Direct Preference Optimization)
• Dropout
• Eigenvalue
• Epoch
• Fine-tuning
• Gradient
• Gradient Accumulation
• Gradient Clipping
• Gradient Descent
• Hyperparameter
• JAX
• Knowledge Distillation
• Learning Rate
• Learning Rate Schedule
• LoRA (Low-Rank Adaptation)
• Loss Function
• Mixed Precision
• Normalization
• Optimizer
• Overfitting
• QLoRA
• ReLU
• RLHF (Reinforcement Learning from Human Feedback)
• SFT (Supervised Fine-Tuning)
• Softmax
• Stochastic Gradient Descent (SGD)
• Transfer Learning
• Underfitting
• Warmup
• Weight
• Weight Decay

Open the Math & training list in the glossary