Self-Supervised Vision — SimCLR, DINO, MAE
Labels are the bottleneck of supervised vision. Self-supervised pretraining removes them: learn visual features from 100M unlabelled images, fine-tune on 10k labelled ones. Trace the three major self-supervised families — contrastive (SimCLR), teacher-student (DINO), masked reconstruction (MAE) — and state what each one optimises. Implement an InfoNCE loss from scratch and explain why a batch of 512 works but a batch of 32 fails. Explain why MAE's 75% masking ratio is not arbitrary and how it differs from BERT's 15% for text. Use DINOv2 or MAE ImageNet checkpoints for linear probing and zero-shot retrieval. Supervised ImageNet has 1.3M labelled images, which cost an estimated $10M to annotate. Medical and industrial datasets are smaller and even more expensive to label. Every vision team asks: can we pretrain on cheap unlabelled data — YouTube frames, web crawls, webcam footage, satellite sweeps — and then fine-tune on a small labelled set? Self-supervised learning is the answer. A modern self-supervised ViT trained on LAION or JFT reaches or beats supervised ImageNet accuracy when fine-tuned. It also transfers better to downstream tasks (detection, segmentation, depth) than supervised pretraining. DINOv2 (Meta, 2023) and MAE (Meta, 2022) are the current production defaults for transferable vision features. The conceptual shift is that the pretext task — the thing the model is trained to do — does not have…
Self-Supervised Vision — SimCLR, DINO, MAE: Labels are the bottleneck of supervised vision. Self-supervised pretraining removes them: learn visual features…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.