Cheat sheets

Self-Supervised Vision, CLIP and Retrieval

Pretrain without labels, align images and text, classify zero-shot, shape embeddings and search them. PyTorch.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Families and models#

Contrastive (SimCLR, MoCo, CLIP)
pull two views together. Push the rest of the batch away.
Teacher-student (DINO, BYOL, iBOT)
the student matches an EMA teacher. No negatives.
Masked (MAE, BEiT, SimMIM)
mask 75% of patches. Reconstruct the pixels.
CLIP
image and text towers in one space. 400M web pairs, no labels.
SigLIP
per-pair sigmoid. No all-gather. Trains better at small batches.
Linear probe
frozen encoder plus one linear layer. Measures feature quality.
Probe top-1
SimCLR ~71%, DINO ~77%, MAE ~76%, DINOv2 ~86%.
Category or instance retrieval
off-the-shelf embeddings work for category. Instance needs metric learning.

From Lesson 04.17, Lesson 04.18, Lesson 04.20, Lesson 12.02

CLIP two towers#

diagonal targetsclientImagebatch of NmodelImage encoderViT or ResNetL2 normalise(N, D)clientCaptionN captionsmodelText encodertransformerL2 normalise(N, D)Similarity SN x N, over tauSymmetric CE(i2t + t2i) / 2diagonal targetsclientImagebatch of NmodelImage encoderViT or ResNetL2 normalise(N, D)clientCaptionN captionsmodelText encodertransformerL2 normalise(N, D)Similarity SN x N, over tauSymmetric CE(i2t + t2i) / 2

From Lesson 04.18, Lesson 12.02

Formulas#

InfoNCE
-\log \frac{\exp(\mathrm{sim}(z_i, z_j)/\tau)}{\sum_{k \ne i} \exp(\mathrm{sim}(z_i, z_k)/\tau)}
CLIP matrix
S_{ij} = \cos(f(x_i), g(y_j)) / \tau. Shape N x N.
CLIP loss
CE over rows plus CE over columns, divided by 2.
SigLIP
\log(1 + \exp(-y_{ij} \, \mathrm{sim}_{ij})) per pair, with y_{ij} = \pm 1.
DINO EMA
\theta_t \leftarrow m \, \theta_t + (1 - m) \, \theta_s
Triplet
\max(0, \lVert f(a) - f(p) \rVert^2 - \lVert f(a) - f(n) \rVert^2 + \mathrm{margin})
Cosine and L2
\lVert a - b \rVert^2 = 2 - 2 \cos(a, b) for unit vectors.
Recall@K
fraction of queries with one correct match in the top K.

From Lesson 04.17, Lesson 04.18, Lesson 04.20, Lesson 12.02

InfoNCE and MAE masking#

def info_nce(z1, z2, tau=0.1):z1, z2 L2-normalised
N, D = z1.shape
z = torch.cat([z1, z2], dim=0)(2N, D)
sim = z @ z.T / tau(2N, 2N)
mask = torch.eye(2 * N, dtype=torch.bool, device=z.device)
sim = sim.masked_fill(mask, float("-inf"))no self match
targets = torch.cat([torch.arange(N, 2 * N), torch.arange(0, N)]).to(z.device)
return F.cross_entropy(sim, targets)
num_patches = 196
visible, masked = random_mask_indices(num_patches, mask_ratio=0.75)keep 25%

From Lesson 04.17, 04.17 code/main.py

CLIP loss and zero-shot#

sim = logit_scale * image_emb @ text_emb.T(N, N)
targets = torch.arange(N, device=sim.device)match on the diagonal
l_i = F.cross_entropy(sim, targets)image to text
l_t = F.cross_entropy(sim.T, targets)text to image
loss = (l_i + l_t) / 2random init: near log(N)
i = F.normalize(model.image_proj(image_feats), dim=-1)
t = F.normalize(model.text_proj(class_text_feats), dim=-1)
pred = (i @ t.T).argmax(dim=-1)one class per image

From Lesson 04.18, 04.18 code/main.py

Triplet loss and recall@K#

def triplet_loss(anchor, positive, negative, margin=0.2):
d_ap = F.pairwise_distance(anchor, positive, p=2)
d_an = F.pairwise_distance(anchor, negative, p=2)
return F.relu(d_ap - d_an + margin).mean()
pos_idx, neg_idx = semi_hard_negatives(emb, y)hardest positive, semi-hard negative
loss = triplet_loss(emb, emb[pos_idx], emb[neg_idx])
sim = query_emb @ gallery_emb.Tcosine on unit vectors
_, top_k = sim.topk(k, dim=-1)
matches = (gallery_labels[top_k] == query_labels[:, None]).any(dim=-1)
matches.float().mean().item()recall@K

From Lesson 04.20, 04.20 code/main.py

Zero-shot classification#

1Promptswrite "a photo of a {class}" for each class.2Ensembleaverage many templates per class for 1-3% moretop-1.3Text towerencode the prompts. T has shape (C, d).4Image towerencode the image. I has shape (1, d).5Normalisemake both sides unit length.6ScoreI @ T.T gives shape (1, C).7Predictargmax gives the class. Warn when the top-1score is below 0.2.

From Lesson 04.18, 12.02 outputs/skill-clip-zero-shot.md

Tasks in code#

TaskCode
Contrastive loss on two viewsinfo_nce(z1, z2, tau=0.1)
DINO teacher with centringDinoHead().teacher(x, temp=0.04)
CLIP symmetric lossclip_loss(i, t, logit_scale)
Zero-shot labelszero_shot_classify(model, image_feats, class_text_feats, class_names)
Mine semi-hard negativessemi_hard_negatives(emb, labels, margin=0.2)
Recall at Krecall_at_k(query_emb, gallery_emb, query_labels, gallery_labels, k=1)
Symmetric InfoNCE in pure Pythoninfonce_loss(S)
SigLIP per-pair losssigmoid_loss(S, bias=0.0)

From 04.17 code/main.py, 04.18 code/main.py, 04.20 code/main.py, 12.02 code/main.py

Defaults and numbers#

SettingDefaultNote
SimCLR tau0.1Lower is sharper, needs more negatives
CLIP tau init0.07Learned. logit_scale starts at ln(1/0.07)
SimCLR batch512-8192Many negatives per positive
MAE mask_ratio0.75BERT masks 15% of tokens
MAE decoder8-layer, 512-dimEncoder sees visible patches only
DINO EMA m0.996Teacher is an EMA of the student
DINO teacher temp0.04Student temp 0.1
Triplet margin0.2Same margin for semi-hard mining

From Lesson 04.17, 04.17 code/main.py, Lesson 04.18, 04.20 code/main.py

Pick a FAISS index#

CommandWhat it does
IndexFlatIPExact inner product. Cosine on 100k vectors
IndexFlatL2Exact L2, brute force, no training
IndexIVFFlatK cells, searches the closest few. Needs training. 10M
IndexHNSWGraph index. Fastest for many queries. Large index
IndexIVFPQAdds product quantisation. 100M+ vectors
DINOv2 + FAISSGeneral visual retrieval, off-the-shelf
CLIP + FAISSQueries are text
Fine-tuned DINOv2 + FAISSInstance retrieval, re-ID, e-commerce

From Lesson 04.20

Troubleshooting#

SymptomCauseFix
InfoNCE fails at batch 32Too few negativesBatch 512-8192 or a momentum queue
DINO gives a constant vectorRepresentation collapseCentre and sharpen the teacher
Masked task too easyMask ratio too lowMask 75% of patches
High recall@10, low recall@1Noisy rankingLonger fine-tune or re-rank
Nearest neighbours changeCosine and L2 mixedMatch the embedding training
Instance search missesOff-the-shelf featuresFine-tune with metric learning
Triplet loss near 0Easy tripletsSemi-hard mining
Softmax loss slow on many GPUsAll-gather of embeddingsUse the sigmoid loss

From Lesson 04.17, Lesson 04.20, Lesson 12.02

Questions#

Why 75% for MAE but 15% for BERT?
Image patches have low entropy. Neighbours give the pixels away.
InfoNCE on random pairs?
About log(2N - 1). That is 3.4 for 16 pairs.
CLIP loss at random init?
Close to log(N). For N = 8 it is 2.08.
Linear probe or fine-tune?
Fine-tuning adds 2-5 points. The probe measures the features only.
DINOv2 embedding?
outputs.last_hidden_state[:, 0], the CLS token, 768-dim.
Zero-shot, probe or fine-tune?
A probe beats zero-shot in domain. Fine-tuning can hurt zero-shot transfer.

From Lesson 04.17, Lesson 04.18, Lesson 12.02