Multi-Object Tracking & Video Memory
Tracking is detection plus association. Detect every frame. Match this frame's detections to last frame's tracks by ID. Distinguish tracking-by-detection from query-based tracking and name the algorithm families (SORT, DeepSORT, ByteTrack, BoT-SORT, SAM 2 memory tracker, SAM 3.1 Object Multiplex). Implement IoU + Hungarian assignment from scratch for classic tracking-by-detection. Explain SAM 2's memory bank and why it handles occlusion better than IoU-based association. Read the three tracking metrics (MOTA, IDF1, HOTA) and pick which one matters for a given use case. A detector tells you where the objects are in a single frame. A tracker tells you which detection in frame t is the same object as a detection in frame t-1. Without that, you cannot count objects crossing a line, follow a ball through an occlusion, or know "car #4 has been in the lane for 8 seconds." Tracking is essential to every video-facing product: sports analytics, surveillance, autonomous driving, medical video analysis, wildlife monitoring, wordmark counting. The core building blocks are shared: a per-frame detector, a motion model (Kalman filter or something richer), an association step (Hungarian algorithm on IoU / cosine / learned features), and a track lifecycle (birth, update, death). 2026 brought two new patterns: SAM 2 memory-based tracking (feature-memory instead of motion-model association) and SAM 3.1 Object Multiplex (shared memory for many instances of…
Multi-Object Tracking & Video Memory: Tracking is detection plus association. Detect every frame. Match this frame's detections to last frame's tracks by ID.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.