Information Theory
Information theory measures surprise. Loss functions are built on it. Language: Python Compute entropy, cross-entropy, and KL divergence from scratch and explain their relationship. Derive why minimizing cross-entropy loss is equivalent to maximizing log-likelihood. Calculate mutual information between features and a target to rank feature importance. Explain perplexity as the effective vocabulary size a language model chooses from. You call CrossEntropyLoss() in every classification model you train. You see "perplexity" in every language model paper. You read about KL divergence in VAEs, distillation, and RLHF. These are not disconnected concepts. They are all the same idea wearing different hats. Information theory gives you the language to reason about uncertainty, compression, and prediction. Claude Shannon invented it in 1948 to solve communication problems. Turns out, training a neural network is a communication problem: the model is trying to transmit the correct label through a noisy channel of learned weights. This lesson builds every formula from scratch so you see where they come from and why they work. When something unlikely happens, it carries more information. A coin landing heads? Not surprising. A lottery win? Very surprising. The information content of an event with probability p is: Using log base 2 gives you bits. Using natural log gives you nats. Same idea, different units. Certain events carry zero information. You already knew…
Information theory measures surprise. Loss functions are built on it. Language: Python Compute entropy, cross-entropy, and KL divergence from scratch and…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.