Probability and Distributions
Probability is the language AI uses to express uncertainty. Language: Python Implement PMFs and PDFs from scratch for Bernoulli, categorical, Poisson, uniform, and normal distributions. Compute expected value, variance, and use the Central Limit Theorem to explain why Gaussians dominate. Build softmax and log-softmax functions with the numerical stability trick (subtract max logit). Calculate cross-entropy loss from logits and connect it to negative log-likelihood. A classifier outputs [0.03, 0.91, 0.06]. A language model picks the next word from 50,000 candidates. A diffusion model generates images by sampling from learned distributions. All of these are probability in action. Every prediction a model makes is a probability distribution. Every loss function measures how far the predicted distribution is from the true one. Every training step adjusts parameters to make one distribution look more like another. Without probability, you cannot read a single ML paper, debug a single model, or understand why your training loss is NaN. The sample space S is the set of all possible outcomes. An event is a subset of the sample space. Probability maps events to numbers between 0 and 1. Three axioms define all of probability: P(A) >= 0 for any event A. P(S) = 1 (something always happens). P(A or B) = P(A) + P(B) when A and B cannot both occur. Everything else (Bayes' theorem,…
Probability and Distributions: Probability is the language AI uses to express uncertainty. Language: Python Implement PMFs and PDFs from scratch for…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.