Cheat sheets

GANs from DCGAN to StyleGAN

Losses, DCGAN code, Pix2Pix, CycleGAN and StyleGAN. Defaults, failure signatures and fixes.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

StyleGAN generator#

AdaIN at every blockresolution doublesstorezN(0, I), 512 dimsmodelMapping network8-layer MLPstorew in Wroughly disentangledstoreLearned constant4×4×512gpuSynthesis blockupsample, conv, AdaIN,noiseclientImage4 up to 1024processPer-layer noiselearned per-channelscaleAdaIN at every blockresolution doublesstorezN(0, I), 512dimsmodelMappingnetwork8-layer MLPstorew in WroughlydisentangledstoreLearnedconstant4×4×512gpuSynthesisblockupsample,conv, AdaIN,noiseclientImage4 up to 1024processPer-layernoiselearnedper-channelscale

One training step#

1Sampledraw a real batch and noise z from N(0, I).2D stepscore real images against 1 and G(z).detach()against 0.3G stepmake fresh fakes. Score D(G(z)) against 1.4Repeatone step of D, then one step of G, every batch.5Probeevery 100 steps, count samples per mode. Warnbelow 50.6Inspectlook at 64 samples at the end of every epoch.7Equilibriump_G = p_data and D outputs 0.5 everywhere.

Losses and formulas#

Minimax
\min_G \max_D \; \mathbb{E}_x[\log D(x)] + \mathbb{E}_z[\log(1 - D(G(z)))]
D loss
binary cross-entropy with real = 1 and fake = 0.
G loss
-\log D(G(z)). This is the non-saturating form.
WGAN
\text{loss}_D = \mathbb{E}[D(\text{fake})] - \mathbb{E}[D(\text{real})], \text{loss}_G = -\mathbb{E}[D(\text{fake})]
Pix2Pix G
-\log D(x, G(x)) + \lambda \cdot \lVert y - G(x) \rVert_1
Cycle
F(G(x)) \approx x and G(F(y)) \approx y.
AdaIN
y_{\text{scale}} \cdot \frac{x - \mathrm{mean}(x)}{\mathrm{std}(x)} + y_{\text{bias}}
Truncation
w' = \hat{w} + \psi \cdot (w - \hat{w}). \psi < 1 trades diversity for quality.

DCGAN train_step#

opt_d.zero_grad()
d_real = D(real)logits, sigmoid only in the loss
d_fake = D(G(z).detach())no gradient flows into G
loss_d = (F.binary_cross_entropy_with_logits(d_real, torch.ones_like(d_real))
+ F.binary_cross_entropy_with_logits(d_fake, torch.zeros_like(d_fake)))
loss_d.backward()
opt_d.step()
opt_g.zero_grad()
d_fake = D(G(z))second pass keeps the graph
loss_g = F.binary_cross_entropy_with_logits(d_fake, torch.ones_like(d_fake))
loss_g.backward()non-saturating G loss
opt_g.step()

Tasks in code#

TaskCode
Build the generatorGenerator(z_dim=64, img_channels=3, feat=32)
Wrap a D conv in SNspectral_norm(c)
Make the optimizersAdam(lr=2e-4, betas=(0.5, 0.999))
Sample in eval modesample(G, n=8, z_dim=z_dim, device=device)
Count parameterssum(p.numel() for p in G.parameters())
Condition G or Dinp = z + one_hot(c, num_classes)
Map z to ww = mapping(z, mapping_net)
Truncate ww_bar[i] + psi * (w_test[i] - w_bar[i])

One pass against many steps#

FactorPix2PixSD-Img2ImgSDXL-TurboControlNet + SDXL
Steps1201-420-30
512² on one L4~30 ms~1.2 s~0.15-0.35 s~3-5 s

Use X when#

  • GAN real-time generation under 10 ms, style transfer, image-to-image translation.
  • StyleGAN3 photoreal human faces with a fixed pose.
  • Pix2Pix thousands of paired examples, a narrow task and fast inference.
  • CycleGAN image translation with no paired data.
  • Frozen mapping network domain adaptation from a few images.
  • Diffusion or flow matching anything multi-modal or open-ended.

Four GANs side by side#

FactorDCGANPix2PixCycleGANStyleGAN
Inputnoise zimage x, dropout as noiseimage from domain Xz mapped to w
Generatortransposed convs, tanh outU-Net with skipsG: X to Y and F: Y to Xconstant plus AdaIN per block
Extra lossnoneλ times L1cycle consistencypath-length reg in StyleGAN2
Dataimages onlypaired (x, y)unpairednarrow domain, faces
Use for32x32 baselinesketch to photohorses to zebras, seasonsphotoreal faces, editing

Defaults#

SettingDefaultNote
Adam lr2e-4DCGAN default for G and D
Adam betas(0.5, 0.999)Low beta1
z_dim64Noise for the 32x32 DCGAN
D LeakyReLU slope0.2All D layers
Pix2Pix λ on L1100Start here, anneal down
PatchGAN field~70×70Pixels per output cell
Truncation ψ0.7Use 1.0 for diversity
TTUR ratio2-4D learns faster than G

Troubleshooting#

SymptomCauseFix
D loss low, G loss oscillates, variety dropsMode collapseMinibatch discrimination, spectral norm, labels
D accuracy ~100%, G loss huge and staticD winsSmaller D, lower D learning rate, label smoothing
Both losses swing with no trendOscillationTTUR or Wasserstein loss
D above 95% accuracyD too strongCut D learning rate by 2-5x
One real mode never appearsG memorizes a modeNoise on D inputs, or WGAN-GP
Output ignores the conditionWeak condition signal in DCondition D early, projection D
Blurry Pix2Pix outputL1 weight too highAnneal λ down once stable
Droplet artifactsAdaIN zeroes the meanWeight demodulation of StyleGAN2

Terms#

DCGAN
strided convs, batch norm, ReLU in G, LeakyReLU in D.
WGAN-GP
Wasserstein distance plus gradient penalty.
SN
spectral norm. Bounds the Lipschitz constant of each D layer.
TTUR
two-timescale update rule. Two learning rates.
cGAN
conditional GAN. G(z, c) and D(x, c) both see the condition.
PatchGAN
D outputs an N × N grid of per-patch scores.
AdaIN
adaptive instance norm. Normalize, then scale and shift by w.
FID
Fréchet Inception Distance. Lower is better.

Questions#

Why G(z).detach() in the D step?
It stops gradients into G during the D update.
Why the non-saturating loss?
Early on D(G(z)) is near zero. The original G loss then has vanishing gradients.
Why switch to eval mode before sampling?
Batch norm then uses running stats.
Why does D see the pair (x, y)?
Without x, D cannot check consistency.
Why does StyleGAN still ship?
One forward pass. Under 10 ms for a 1024² face on a 4090.
Can a GAN give p(x)?
No. It is an implicit density model. It only samples.