K-Nearest Neighbors and Distances
Store everything. Predict by looking at your neighbors. The simplest algorithm that actually works. Language: Python Implement KNN classification and regression from scratch with configurable K and distance-weighted voting. Compare L1, L2, cosine, and Minkowski distance metrics and select the appropriate one for a given data type. Explain the curse of dimensionality and demonstrate why KNN degrades in high-dimensional spaces. Build a KD-tree for efficient nearest neighbor search and analyze when it outperforms brute-force. You have a dataset. A new data point arrives. You need to classify it or predict its value. Instead of learning parameters from the data (like linear regression or SVMs), you just find the K training points closest to the new point and let them vote. This is K-nearest neighbors. There is no training phase. No parameters to learn. No loss function to minimize. You store the entire training set and compute distances at prediction time. It sounds too simple to work. But KNN is surprisingly competitive for many problems, especially with small to medium datasets, and understanding it deeply reveals fundamental concepts: the choice of distance metric (connecting to Phase 1 Lesson 14), the curse of dimensionality, and the difference between lazy and eager learning. KNN also shows up everywhere in modern AI, just under different names. Vector databases do KNN search over embeddings. Retrieval-augmented…
K-Nearest Neighbors and Distances: Store everything. Predict by looking at your neighbors. The simplest algorithm that actually works.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.