SAM 3 & Open-Vocabulary Segmentation
Give a model a text prompt and an image and get masks for every matching object. SAM 3 made that a single forward pass. Distinguish SAM (visual prompts only), Grounded SAM / SAM 2 (detector + SAM), and SAM 3 (native text prompts via Promptable Concept Segmentation). Explain the SAM 3 architecture: shared backbone + image detector + memory-based video tracker + presence head + decoupled detector-tracker design. Use Hugging Face transformers SAM 3 integration for text-prompted detection, segmentation, and video tracking. Pick between SAM 3, Grounded SAM 2, YOLO-World, and SAM-MI based on latency, concept complexity, and deployment target. The 2023 SAM was a visual-prompt-only model: you click a point or draw a box and it returns a mask. For "give me all the oranges in this photo" you needed a detector (Grounding DINO) to produce boxes, then SAM to segment each. Grounded SAM turned this into a pipeline, but it was a cascade of two frozen models with inevitable error accumulation. SAM 3 (Meta, Nov 2025, ICLR 2026) collapsed the cascade. It accepts a short noun phrase or an image exemplar as prompt and returns all matching masks and instance IDs in a single forward pass. That is Promptable Concept Segmentation (PCS). Combined with the March 2026 Object Multiplex update (SAM 3.1), it tracks multiple instances of the…
SAM 3 & Open-Vocabulary Segmentation: Give a model a text prompt and an image and get masks for every matching object. SAM 3 made that a single forward pass.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.