Audio Classification — From k-NN on MFCCs to AST and BEATs
Everything from "dog barking vs siren" to "which language is this" is audio classification. The features are mels. The architecture moves each decade. The evaluation stays AUC, F1, and per-class recall. You get a 10-second clip. You want to know: "what is it?" Urban sound (siren, drill, dog), speech command (yes/no/stop), language ID (en/es/ar), speaker emotion (angry/neutral), or environmental sound (indoor/outdoor, babble). All of these are audio classification, and in 2026 the baseline architecture is mature: log-mel → CNN or Transformer → softmax. The core difficulty is not the network. It is data. Audio datasets have brutal class imbalance, strong domain shift (clean vs noisy), and label noise (who decided "urban babble" vs "restaurant noise"?). The 80% of the problem is curation, augmentation, and evaluation, not swapping CNN for Transformer. Audio classification ladder: k-NN on MFCCs to AST to BEATs k-NN on MFCCs (the 1990s baseline). Flatten MFCCs per clip, compute cosine similarity to a labeled bank, return majority vote of the top K. Surprisingly strong on clean, small datasets (Speech Commands, ESC-50). Runs with no GPU. 2D CNN on log-mels (2015-2019). Treat the (T, nmels) log-mel as an image. Apply ResNet-18 or VGG-style. Global mean pool the time axis. Softmax over classes. Still the baseline in most 2026 kaggle competitions. Audio Spectrogram Transformer, AST (2021-2024). Patchify the log-mel (e.g.…
Audio Classification — From k-NN on MFCCs to AST and BEATs: Everything from "dog barking vs siren" to "which language is this" is audio classification. The…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.