Cheat sheets

GANs, Diffusion and Flow Compared

GAN, DDPM, latent diffusion, DiT with rectified flow, flow matching and VAR. Targets, steps, code and fixes.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Troubleshooting
  • Questions and tips

Source

Sampling cost side by side#

FamilyPassesNote
GANone forward passFast. Hard to train
DDPM1000Linear beta schedule
DDIM50Deterministic. Same training as DDPM
DPM-Solver++ 2M Karras20-30SD production default
Rectified flow (Euler)20-30SD3, FLUX, Z-Image
Schnell, Turbo, LCM1-4Distilled from a many-step model
VAR, K = 10 scales10 passesDiT needs 28-50 at 256x256

Training target side by side#

FamilyTargetLoss
GAN discriminatorP(real)Binary cross-entropy on real and fake
GAN generatorD(G(z))High is good. Non-saturating form
DDPMepsilon_theta(x_t, t)MSE on the true noise
Rectified flow (4.23)epsilon - x0MSE on the velocity
Flow matching (8.13)x1 - x0Here x1 is noise, x0 is data
VARCross-entropyOn the VQ codes of the next scale

Use X when#

  • GAN real-time generation under 10 ms, style transfer, Pix2Pix or CycleGAN translation.
  • SD 1.5 latent diffusion open-source community fine-tunes with LoRA.
  • DiT + rectified flow state of the art text to image. FLUX.1-schnell is the open-source default.
  • Distilled flow (LCM-LoRA, schnell) latency under 1s, 1-4 steps.
  • Flow matching audio, video and science as well as images.
  • VAR class-conditional ImageNet. Fewer passes, GPT-style scaling.

Stable Diffusion pipeline#

embeddingclientText promptmodelText encoderCLIP-L or T5storeNoise4x64x64gpuUNetcross-attention to textprocessSchedulerDPM-Solver++, EulerstoreClean latent4x64x64modelVAE decoderclientImage512x512 RGBembeddingclientText promptmodelTextencoderCLIP-L or T5storeNoise4x64x64gpuUNetcross-attentionto textprocessSchedulerDPM-Solver++,EulerstoreCleanlatent4x64x64modelVAE decoderclientImage512x512 RGB

Tasks in code#

TaskCode
GAN with spectral normbuild_sn_discriminator()
Linear beta schedulelinear_beta_schedule(T=1000)
DDPM samplingsample_ddpm(model, schedule, shape)
DDIM samplingsample_ddim(..., steps=50, eta=0.0)
Rectified flow steprectified_flow_train_step(model, x0, ...)
Euler samplingrectified_flow_sample(model, shape, steps=20)
Multi-scale VQ tokenstokenize_multiscale(img, codebooks)

Troubleshooting#

SymptomCauseFix
Sample variety dropsMode collapseSpectral norm or label conditioning
D accuracy ~100%D wins, G gradients vanishSmaller D, lower D learning rate
Both losses swing, no trendG and D trade winsTTUR: D learns 2-4x faster
Samples depend on the batchBN uses batch statsCall G.eval() before sampling
Heavy artefactsGuidance scale too highSD 7.5. Rectified flow 3.5-5

diffusers calls#

TaskCode
SD text to imageguidance_scale=7.5, num_inference_steps=25
Swap the schedulerDPMSolverMultistepScheduler.from_config(...)
Image to imageimg2img(..., image=init_image, strength=0.6)
Inpaint the white areainpaint(..., mask_image=mask)
Fuse a LoRApipe.fuse_lora(lora_scale=0.8)
FLUX.1-schnellguidance_scale=0.0, num_inference_steps=4
SD 3.5 Largeguidance_scale=3.5, num_inference_steps=28

DDPM against rectified flow#

t = torch.randint(0, T, (bs,), device=device)DDPM: integer step
x_t = q_sample(x0, t, noise, schedule)closed-form jump
loss = F.mse_loss(pred, noise)predict the noise
t = torch.rand(n, device=device)RF: t in [0, 1]
x_t = (1 - t[:, None, None, None]) * x0 + t[:, None, None, None] * epsilon
target_v = epsilon - x0velocity, data to noise
loss = F.mse_loss(pred_v, target_v)
dt = 1.0 / stepsEuler, steps=20
for _ in range(steps):
v = model(x, t)
x = x - dt * vwalk from noise to data
t = t - dt

One GAN training step#

opt_d.zero_grad()
d_real = D(real)
d_fake = D(G(z).detach())no gradient into G
loss_d = (F.binary_cross_entropy_with_logits(d_real, torch.ones_like(d_real))
+ F.binary_cross_entropy_with_logits(d_fake, torch.zeros_like(d_fake)))
loss_d.backward()
opt_d.step()
opt_g.zero_grad()
d_fake = D(G(z))
loss_g = F.binary_cross_entropy_with_logits(d_fake, torch.ones_like(d_fake))non-saturating
loss_g.backward()
opt_g.step()
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))DCGAN default

Key formulas#

GAN game
\min_G \max_D \; \mathbb{E}_x[\log D(x)] + \mathbb{E}_z[\log(1 - D(G(z)))]
DDPM jump
x_t = \sqrt{\bar{\alpha}_t} \, x_0 + \sqrt{1 - \bar{\alpha}_t} \, \epsilon, with \bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s
CFG
\epsilon = \epsilon_{\text{uncond}} + w \cdot (\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})
Rectified flow
x_t = (1 - t) \, x_0 + t \, \epsilon for t \in [0, 1]
CFG with velocity
v_{\text{cfg}} = (1+w) v_{\text{cond}} - w v_{\text{uncond}}
Latent saving
(3512512)/(46464) = 48x less compute.

VAR next-scale generation#

1Tokenizera multi-scale residual VQ. Train once, thenfreeze it.2Scale 1sample z_1. This is 1 token.3Scale 2sample 4 tokens in parallel, given z_1.4Scale 3sample 16 tokens in parallel, given z_1 and z_2.5Decodesum the upsampled embeddings of all scales. TheVAE decoder makes the image.6Toy runSCALES = (1, 2, 4, 8), CODEBOOK = 16. 4 passesin total.

Questions#

Why is diffusion easy to train?
The loss is MSE. There is no adversarial game.
Why does rectified flow need fewer steps?
The path is close to a straight line.
Why is VAR fast?
One pass makes all tokens of a scale. Length grows with log scale.
Why does FLUX.1-schnell use guidance 0.0?
It was trained without CFG.