Diffusion and Generative Models

Train and sample a VAE, a GAN, DDPM, latent diffusion and flow matching, then measure them. Toy Python from the lessons.

Latent diffusion with CFG#

Task	Code
Noise $inline$ to step $inline$	`x_t = math.sqrt(a_bar) * x0 + math.sqrt(1 - a_bar) * eps`
VAE reparameterize	`[m + s * e for m, s, e in zip(mu, sigma, eps)]`
Non-saturating G loss	`-sum(math.log(max(p, 1e-8)) for p in d_fake) / len(d_fake)`
Drop the label for CFG	`use_c = NULL_CLASS if rng.random() < 0.1 else c`
Mix with CFG	`eps_cfg = (1 + w) * eps_cond - w * eps_uncond`
Flow matching target	`target = x1 - x0`
Euler step, flow	`x -= dt * net_forward(x, t)`
Tile the VAE decode	`pipe.vae.enable_tiling()`

betas = [1e-4 + (0.02 - 1e-4) * t / (T - 1) for t in range(T)]linear

alphas = [1 - b for b in betas]

alpha_bars, cum = [], 1.0

for a in alphas:

cum *= a

alpha_bars.append(cum)cumulative product

eps = rng.gauss(0, 1)training noise

x_t = math.sqrt(a_bar) * x0 + math.sqrt(1 - a_bar) * epsone shot

x = rng.gauss(0, 1)sampling starts at noise

for t in range(T - 1, -1, -1):T-1 down to 0

eps_hat, _ = forward([x], t_emb, net)predicted noise

mean = (x - beta_t / math.sqrt(1 - alpha_bars[t]) * eps_hat[0]) / math.sqrt(alphas[t])

x = mean + math.sqrt(beta_t) * rng.gauss(0, 1)only when t > 0

Setting	Default	Note
$inline$ start	1e-4	linear schedule
$inline$ end	0.02	$inline$ is close to $inline$
Steps $inline$	1000	the toy code uses 40
CFG scale $inline$	3	3 to 7 is the sweet spot
CFG label dropout	10%	null token in training
SD 1.x VAE scaling factor	0.18215	applied after encoding
LoRA rank $inline$	4-16	attention layers, 64 to 128 for heavy fine tunes
FID sample count	10k	low N gives a falsely low FID

Symptom	Cause	Fix
$inline$ carries no information about $inline$	posterior collapse, KL wins	anneal $inline$ from 0 to 1, or free bits
GAN makes one mode only	mode collapse	noise on D inputs, minibatch discrimination or WGAN-GP
G stops improving, D over 95% accuracy	D is too strong	cut the D learning rate by 2-5x
Saturated, oily images	CFG too high	keep $inline$ at 3 to 7
Denoiser trains on wrong latent variance	VAE scaling factor missing	apply the checkpoint scaling factor
NaNs from the SD 1.x VAE at 1024²	fp16 cast	use the fp16-fix VAE or bf16
LoRA has no effect	trained on another base	use the matching base model
FID looks too good	N under 10k	use 10k samples or more and 3 seeds

VAE: Variational autoencoder. The encoder outputs a distribution, a KL term shapes the code space.
ELBO: Evidence lower bound. Reconstruction minus KL, a lower bound on $inline$ .
GAN: Generator $inline$ and discriminator $inline$ in a minimax game.
DDPM: Ho et al. 2020. Linear $inline$ , 1000 steps, U-Net.
DDIM: Non-Markov sampler. 20 to 50 steps, same training objective.
CFG: Classifier-free guidance. Mixes conditional and unconditional predictions.
DiT: Diffusion Transformer. A transformer over latent patches in place of the U-Net.
FID: Fréchet Inception Distance. Distance of Gaussian fits to real and generated features.

Why output log σ² and not σ?: The output stays unconstrained. Gradients die at σ near 0 with softplus.
Why are VAE samples blurry?: MSE reconstruction gives the mean of the plausible outputs.
Why diffuse in latent space?: The 64×64×4 latent needs about 64x fewer FLOPs for comparable quality.
Are flow matching and DDPM different?: For Gaussian paths they are algebraically equivalent.
How is FID computed?: $inline$ on Inception-v3 features. Lower is better.