Cheat sheets

Evaluating Generative Models

FID, CLIP score, CMMD, Elo, WER, MOS, SECS, EER and FAD. Code, sample sizes and the ways each number fails.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

FID and CLIP score#

two poolsFréchet distancemean over 30kstoreReal imagesheld-out setstoreGeneratedimages10k or moremodelInception-v32048-D featuresprocessGaussian fitmean andcovarianceclientFIDlower is betterclientPromptmodelCLIP encodersimage and textprocessCosineper imageclientCLIP scorehigher is bettertwo poolsFréchet distancemean over 30kstoreReal imagesheld-out setstoreGeneratedimages10k or moremodelInception-v32048-DfeaturesprocessGaussianfitmean andcovarianceclientFIDlower isbetterclientPromptmodelCLIPencodersimage andtextprocessCosineper imageclientCLIP scorehigher isbetter

Image metrics in code#

mu_r = mean_vec(real_features)
mu_g = mean_vec(gen_features)
cov_r = covariance(real_features, mu_r)
mean_sq = sum((a - b) ** 2 for a, b in zip(mu_r, mu_g))
sqrt_prod = jacobi_sqrt(prod)matrix square root
return mean_sq + trace(cov_r) + trace(cov_g) - 2 * trace(sqrt_prod)
score = clip_like(image, prompt)cosine, higher is better
expected_a = 1 / (1 + 10 ** ((r_b - r_a) / 400))
actual_a = 1.0 if winner == "a" else 0.0
delta = k * (actual_a - expected_a)k=32
return r_a + delta, r_b - delta

Audio metrics in code#

transform = Compose([ToLowerCase(), RemovePunctuation(), Strip()])
score = wer(truth="<ref>", hypothesis="<hyp>",
truth_transform=transform, hypothesis_transform=transform)
sv = EncoderClassifier.from_hparams("speechbrain/spkrec-ecapa-voxceleb")
emb_ref = sv.encode_batch(load_wav("reference.wav"))
emb_clone = sv.encode_batch(load_wav("cloned.wav"))
secs = torch.nn.functional.cosine_similarity(emb_ref, emb_clone, dim=-1).item()
fad = FrechetAudioDistance()
score = fad.get_fad_score("generated_folder/", "reference_folder/")

Image metrics side by side#

FactorFIDCLIP scoreCMMDHuman preference
Measuressample qualityprompt adherencequality, subtle differencesoverall preference
FeaturesInception-v3, 2048-DCLIP image and textCLIPhumans or an LLM judge
Minimum10k vs held-out real30k5k200 blinded pairs
Weak spotsmall N, domains far from ImageNetweak composition, short promptsnot coveredjudge variance, prompt choice

Sampling cost of unified models#

FactorChameleon, Emu3TransfusionShow-o
Decodingautoregressive tokenscontinuous diffusionmasked discrete diffusion
Steps1024-4096 passes~20~16
Each stepone tokenfull transformer passtop-K tokens commit

Troubleshooting#

SymptomCauseFix
Low FID from few samplesSmall N underestimates covarianceUse N of 10k or more
FID meaningless on faces or artInception trained on ImageNetDomain extractor, for example FD-DINO
FID shifts across resolutions299×299 resizeCompare at matched resolution
CLIP score up, images not better"high quality, 4k, masterpiece"Check binding and saturation
Elo is meaninglessModels saw the benchmark promptsUse held-out prompt sets
5% WER hides 30% on accentsAggregate WERReport by demographic slice
Bad UTMOS on cloned audioTrained on clean speechUse a domain MOS panel
FAD not comparableDifferent reference setsSame reference set for all models

Production eval report#

1Sample qualityFID on 10-30k samples against a held-out realset.2AdherenceCLIP score or CMMD on the same samples againsttheir prompts.3Preferencewin rate in a blinded arena against the previousmodel.4Failure auditflag 50 random outputs for hands, text andobject count.5Seedsrun 3 seeds minimum. Report the std.6Per PRFID and CLIP score on a 500-sample subset, about30 min.7Nightlyfull 10k FID, HPSv2 and Elo.

Formulas#

FID
\lVert \mu_r - \mu_g \rVert^2 + \mathrm{Tr}(\Sigma_r + \Sigma_g - 2 (\Sigma_r \Sigma_g)^{0.5})
CLIP score
cosine of CLIP_image(x_gen) and CLIP_text(prompt).
Elo expected
E_a = 1 / (1 + 10^{(r_b - r_a)/400})
Elo update
r_a \leftarrow r_a + k (s_a - E_a). B moves by the same amount the other way.
WER
(S + D + I) / N after normalization. CER uses characters.
DER
(FA + Miss + Confusion) / total speaker time.
EER
the threshold where FAR equals FRR.
Show-o mask ratio
\cos\left(\frac{\pi t}{2T}\right), from 1.0 at step 0 to 0 at step T.

Metric functions#

TaskCode
Image FIDfid(real_features, gen_features)
Elo from a pairelo_update(r_a, r_b, winner, k=32)
Word error ratewer(ref, hyp)
Character error ratecer(ref, hyp)
Equal error rateeer_from_scores(same, diff)
FAD-shaped distanceembedding_fad_like(real_embeds, fake_embeds)
Multiple-choice accuracymmau_accuracy(predictions, golds)
TTS round-trip WERttr_wer(tts_model, asr_model, texts)

Audio numbers to beat#

WER
below 5% is human parity on read speech.
UTMOS
correlates ~0.9 with human MOS. F5-TTS 3.95, ground truth 4.08.
SECS
above 0.75 is a recognizable clone.
CLAP
above 0.3 is reasonable text-audio alignment.
EER
ECAPA on VoxCeleb1-O gives 0.87%.
FAD
MusicGen-small on MusicCaps 4.5. Lower is better.
MMAU-Pro
random chance is 25% on 4-way.
Barge-in
target below 150 ms.

Defaults#

SettingDefaultNote
Elo k32elo_update
Elo start rating1000Both models
FID samples10kLow-N FID is unreliable
CLIP score samples30kAverage to one scalar
Inception input299×299Match resolutions
MOS listeners20+100+ samples per model
Show-o steps T8Cosine schedule
Keep ratio floor0.15keep_ratio per step

Rules#

  • Normalize text before WER. Report the normalization rule.
  • Report distributions. P50/P95/P99 for latency, per class, per category.
  • Run one public benchmark and one in-house held-out set.
  • Cache the real features once as a .npz file.
  • Batch eval generation with num_images_per_prompt=8.
  • Use three metrics that agree, plus a qualitative review.

Questions#

Why CMMD?
It uses CLIP features and MMD. It is less biased and has no Gaussian assumption.
High SECS, low MOS?
The timbre is right but the voice is unnatural.
EER or minDCF?
minDCF is more production-relevant. It weights cost at an operating point.
What does 10k SDXL samples cost?
About 11 hours on one L4 at one request at a time.
Why does Show-o inpaint for free?
Its training predicts masked tokens.