Evaluation — FID, CLIP Score, Human Preference
Every generative model leaderboard cites FID, CLIP score, and a win rate from a human-preference arena. Each number has a failure mode a determined researcher can game. If you do not know the failure modes, you cannot tell a real improvement from a gaming run. A generative model is judged on sample quality and conditioning adherence. Neither has a closed-form measure. Your model has to render 10,000 images; something has to assign them numbers; you have to trust the numbers across model families, across resolutions, across architectures. Three metrics survived the 2014-2026 gauntlet: FID (Fréchet Inception Distance). A distance between two distributions — real and generated — in an Inception network's feature space. Lower is better. CLIP score. Cosine similarity between a generated image's CLIP-image embedding and a prompt's CLIP-text embedding. Higher is better. Measures prompt adherence. Human preference. Pit two models head-to-head on the same prompt, have humans (or a GPT-4-class model) pick the better one, aggregate to an Elo score. You will also see: IS (inception score, largely retired), KID, CMMD, ImageReward, PickScore, HPSv2, MJHQ-30k. Each corrects for one failure of the previous. FID, CLIP, and preference: three axes, different failure modes Heusel et al. (2017). Steps: Extract Inception-v3 features (2048-D) for N real images and N generated. Fit a Gaussian to each pool: compute mean μr, μg…
Evaluation — FID, CLIP Score, Human Preference: Every generative model leaderboard cites FID, CLIP score, and a win rate from a human-preference arena. Each…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.