Cheat sheets

Computer Vision

Preprocess, convolve, classify, fine-tune, detect, segment and patchify images. PyTorch and NumPy from the lessons.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Mask R-CNN pipeline#

featurestop KInput imagelist of (3, H, W)ResNet backbonestrides 4, 8, 16, 32FPNtop down + lateralRPNabout 1000 proposalsRoIAlignbilinear, no roundingBox headclass + refineMask head28x28 per classNMSboxes, labels, masksfeaturestop KInput imagelist of (3, H, W)ResNet backbonestrides 4, 8, 16, 32FPNtop down + lateralRPNabout 1000 proposalsRoIAlignbilinear, noroundingBox headclass + refineMask head28x28 per classNMSboxes, labels, masks

From Lesson 04.08

Preprocess for a pretrained model#

1Decodethe file becomes uint8 HWC in [0, 255].2Colourconvert to RGB. BGR input costs about tenpoints.3Float and layoutdivide by 255, then HWC to NCHW.4Resizeshorter side to 256, bilinear.5Cropcenter crop to 224x224.6Standardizesubtract the ImageNet mean, divide by the std.7Modelthe input now matches the training input.

From Lesson 04.01

Key formulas#

Output size
H_{\text{out}} = \left\lfloor \frac{H - K + 2P}{S} \right\rfloor + 1
Same padding
P = (K - 1) / 2 for odd K and S = 1
Conv params
C_{\text{out}} \cdot C_{\text{in}} \cdot K \cdot K + C_{\text{out}}
Receptive field
1 + L \cdot (K - 1) for L stacked convs, stride 1
Residual
y = F(x) + x
IoU
\mathrm{IoU}(A, B) = \frac{\mathop{\mathrm{area}}(A \cap B)}{\mathop{\mathrm{area}}(A \cup B)}
Dice
\frac{2 \sum (p \cdot y)}{\sum p + \sum y + \epsilon}, and \text{Dice} = 2 \cdot \text{IoU} / (1 + \text{IoU})
YOLO decode
x = (\sigma(t_x) + \text{cell}_x) \cdot \text{stride}, w = \text{anchor}_w \cdot \exp(t_w)

From Lesson 04.02, Lesson 04.03, Lesson 04.06, Lesson 04.07, Lesson 04.04

Tasks in code#

TaskCode
HWC to CHWimg_hwc.permute(2, 0, 1)
Same size 3x3 convnn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1)
Loss with label smoothingnn.CrossEntropyLoss(label_smoothing=0.1)
Replace a ResNet headmodel.fc = nn.Linear(model.fc.in_features, num_classes)
Pretrained backbone, any familytimm.create_model("resnet50", pretrained=True, num_classes=10)
Deduplicate boxesnms(boxes, scores, iou_threshold=0.45)
U-Net with a pretrained encodersmp.Unet(encoder_name="resnet34", encoder_weights="imagenet", in_channels=3, classes=3)
Pretrained ViTtimm.create_model("vit_base_patch16_224", pretrained=True, num_classes=10)

From Lesson 04.01, Lesson 04.02, Lesson 04.04, Lesson 04.05, Lesson 04.06, Lesson 04.07, Lesson 04.14

ViT forward#

self.proj = nn.Conv2d(in_channels, dim, kernel_size=patch_size, stride=patch_size)
self.cls_token = nn.Parameter(torch.zeros(1, 1, dim))one learned token
self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, dim))learned
x = self.proj(x)(N, dim, 14, 14) at 224 px
x = x.flatten(2).transpose(1, 2)(N, 196, dim) tokens
cls = self.cls_token.expand(x.size(0), -1, -1)
x = torch.cat([cls, x], dim=1)(N, 197, dim)
x = x + self.pos_embedadd position
normed = self.ln1(x)pre-LN
a, _ = self.attn(normed, normed, normed, need_weights=False)
x = x + aresidual
x = x + self.mlp(self.ln2(x))D to 4D to D, GELU
return self.head(self.ln(x[:, 0]))classify from [CLS]

From 04.14 code/main.py, Lesson 04.14

Shapes#

TensorShape
Decoded image(H, W, 3), uint8
Model input(N, 3, 224, 224), float32
Conv weight(C_out, C_in, K, K)
YOLO head output(N, H, W, num_anchors, 5 + C)
Segmentation logits(N, C, H, W), target (N, H, W)
Mask R-CNN masks(N, 1, H, W) in [0, 1]
ViT tokens at 224 px(197, 768) with [CLS]

From Lesson 04.01, Lesson 04.02, Lesson 04.06, Lesson 04.07, Lesson 04.08, Lesson 04.14

Defaults and numbers#

SettingDefaultNote
ImageNet mean0.485, 0.456, 0.406on [0, 1] pixels
ImageNet std0.229, 0.224, 0.225per channel
NMS iou_threshold0.45RPN uses 0.7
conf_threshold0.25objectness times class score
lambda_noobj0.5lambda_coord is 5.0, YOLOv1 ratios
Mixup alpha0.2lam from Beta(alpha, alpha)
ViT patch size16196 tokens at 224 px
MAE mask ratio75%encoder sees 25%

From Lesson 04.01, Lesson 04.06, Lesson 04.04, Lesson 04.14, Lesson 04.08

Backbones#

ModelSize and idea
LeNet-561,706 params. Conv, pool, dense
AlexNet60M. ReLU, dropout, depth
VGG-16138M. Only 3x3 convs
ResNet-1811.7M. Identity skips, 69.8% top-1
ViT-B/1686M. 12 blocks, 12 heads, dim 768
Mask R-CNN R50 FPN46M. 91 COCO classes

From Lesson 04.03, Lesson 04.14, Lesson 04.08

Troubleshooting#

SymptomCauseFix
Garbage output, no errorraw uint8 into the modeldivide by 255, then standardize
First conv sees height as channelsHWC inputpermute(2, 0, 1) to CHW
Wrong gradient, no errorsoftmax before cross-entropypass raw logits
Unstable lossno zero_grad() each stepcall it before backward()
Fine-tuned worse than linear probeLR or BN bugdiscriminative LR, freeze_bn_stats
99% pixel accuracy, useless masksclass imbalanceadd Dice to the loss
Checkerboard artifactstransposed convbilinear upsample + 3x3 conv
Head has one class too fewbackground not countednum_classes = classes + 1

From Lesson 04.01, Lesson 04.04, Lesson 04.05, Lesson 04.07, Lesson 04.08

Abbreviations#

IoU
Intersection over Union. Overlap of two boxes or masks.
NMS
Non-maximum suppression. Keeps the top box, drops overlaps.
mAP@0.5:0.95
COCO AP. Mean over IoU 0.5 to 0.95, step 0.05.
mIoU
mean IoU over classes. The segmentation metric.
FPN
Feature Pyramid Network. Semantic features at every stride.
RPN
Region Proposal Network. Object or no object per anchor.
ViT
Vision Transformer. Patches as tokens.
MAE
Masked Autoencoder. Mask 75% of patches, reconstruct.

From Lesson 04.06, Lesson 04.07, Lesson 04.08, Lesson 04.14

Questions#

Why 3x3 everywhere?
Two 3x3 convs see 5x5 with 18C^2 params, not 25C^2.
Why residuals?
A block can learn identity, so depth stops hurting.
Why RoIAlign over RoIPool?
No rounding. It adds 3 to 4 mask AP on COCO.
Why anchors?
The head predicts small deltas from a prior box.
Why pre-LN in ViT?
It trains deep stacks without warmup.

From Lesson 04.02, Lesson 04.03, Lesson 04.08, Lesson 04.06, Lesson 04.14

Tips#

  • Compute mean and std on your own training set.
  • Use nearest resize for masks and label maps.
  • Print linear probe and fine-tuned accuracy for every transfer run.
  • Rebuild the optimizer when the set of trainable parameters changes.
  • Report IoU per class, not only mIoU.
  • Report precision, recall, AP@0.5 and mAP@0.5:0.95 together.

From Lesson 04.04, Lesson 04.05, Lesson 04.07, Lesson 04.06, Lesson 04.01