Vision Transformers (ViT) - Computer Vision
Cut the image into patches, treat each patch as a word, run a standard transformer. Don't look back. Implement patch embedding, learned positional embedding, class token, and transformer encoder blocks from scratch to build a minimal ViT. Explain why ViT was thought to need massive pretraining data until DeiT and MAE proved otherwise. Compare ViT, Swin, and ConvNeXt on their architectural priors (none, local window attention, conv backbone). Fine-tune a pretrained ViT on a small dataset using timm and the standard linear-probe / fine-tune recipe. For a decade, convolution was synonymous with computer vision. CNNs had strong inductive biases — locality, translation equivariance — that nobody thought you could replace. Then Dosovitskiy et al. (2020) showed that a plain transformer applied to flattened image patches, with no convolutional machinery at all, could match or beat the best CNNs at scale. The catch was "at scale." ViT on ImageNet-1k lost to ResNet. ViT pretrained on ImageNet-21k or JFT-300M then fine-tuned on ImageNet-1k beat it. The conclusion was that transformers lacked useful priors but could learn them from enough data. Subsequent work (DeiT, MAE, DINO) showed that with the right training recipes — strong augmentation, self-supervised pretraining, distillation — ViTs train fine on small data too. By 2026, pure CNNs are still competitive on edge devices (ConvNeXt is the strongest), but transformers…
Vision Transformers (ViT): Cut the image into patches, treat each patch as a word, run a standard transformer. Don't look back.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.