Audio Features, Classification and Metrics
Sample, frame, window, mel, MFCC, classify and score audio. Stdlib Python from the lessons.
- Concepts and terms
- Tasks and commands
- Annotated code
- Diagrams
- Defaults and numbers
- Troubleshooting
- Questions and tips
Waveform to label#
Formulas#
- Time
- t = n / sr. A 10 s clip at 16 kHz is 160,000 floats.
- Nyquist
- sr / 2. Energy above it aliases to lower bins.
- Alias
- a 7 kHz tone sampled at 10 kHz folds to 10 - 7 = 3 kHz.
- Bin
- bin k is k \cdot sr / N Hz. Resolution is sr / N.
- Mel
- m = 2595 \cdot \log_{10}(1 + f / 700)
- Hann
- w(n) = 0.5 (1 - \cos(2 \pi n / (N - 1)))
- Frames
- 1 + (T - L) / hop. 10 s at 16 kHz gives 998 frames.
- Mixup
- x = \lambda x_1 + (1 - \lambda) x_2, same for y.
From Lesson 06.01, Lesson 06.02, Lesson 06.03
Feature defaults#
| Setting | Default | Note |
|---|---|---|
| Frame length | 400 | 25 ms at 16 kHz |
| Hop | 160 | 10 ms at 16 kHz |
| Window | Hann | Tapers frame edges to zero |
| Mel bins, ASR | 80 | Cover 0 to 8 kHz |
| Mel bins, classification | 128 | AST, PANNs, BEATs |
| MFCC count | 13 | Drop coefficient 0, it is energy |
Log floor eps | 1e-10 | log(mel_spec + 1e-10) |
| Keyword spotting | 40 MFCCs | For tiny devices |
From Lesson 06.02, Lesson 06.01
Tasks in code#
| Task | Code |
|---|---|
| Synthesize a tone | sine(freq_hz, |
| Find the dominant frequency | peak_freq(samples, |
| Frame a signal | frame_signal(signal, |
| STFT magnitude | stft_magnitude(signal, |
| Mel filterbank | mel_filterbank(n_mels, |
| Log-mel | log_transform(mel_spec, |
| MFCC | dct_ii(x, |
| k-NN vote | knn(q, |
From 06.01 code/main.py, 06.02 code/main.py, 06.03 code/main.py
MFCC k-NN baseline#
sr = 8000demo rate in the codemag = stft_mag(signal, frame_len, hop)Hann, then DFT per framefb = mel_filterbank(n_mels, frame_len, sr)n_mels x (n_fft // 2 + 1)mels = apply_filterbank(mag, fb)weighted sum per mel binlm = log_transform(mels)log(max(v, eps))feats = [dct_ii(f, n_mfcc) for f in lm]13 coefficients per frameemb = summarize(feats)mean + var, 26 dimspred = knn(emb, X_train, y_train, k=3)cosine top k, majority voteFrom 06.03 code/main.py
Score a model#
from jiwer import wer, Compose, ToLowerCase, RemovePunctuation, Striptransform = Compose([ToLowerCase(), RemovePunctuation(), Strip()])score = wer(truth="Please turn on the lights.", hypothesis="please turn on the light", truth_transform=transform, hypothesis_transform=transform)sv = EncoderClassifier.from_hparams("speechbrain/spkrec-ecapa-voxceleb")secs = torch.nn.functional.cosine_similarity(emb_ref, emb_clone, dim=-1).item()SECSfad = FrechetAudioDistance()score = fad.get_fad_score("generated_folder/", "reference_folder/")lower is betterFrom Lesson 06.17
Libraries#
| Command | What it does |
|---|---|
soundfile | Reads WAV, FLAC, OGG as float32 |
torchaudio.transforms.Resample | Resamples with anti-aliasing built in |
numpy.fft.rfft | Real FFT for real audio |
torch.stft | Vectorized STFT |
librosa.power_to_db | Reference-normalized dB |
ffprobe | Reports sr, channels and codec of a file |
jiwer | WER and CER with normalization |
ASTForAudioClassification.from_pretrained | Loads AST to fine-tune |
Pick a classifier#
- Under 1000 clips
- k-NN on MFCC means plus audio augmentation.
- 1K to 100K clips
- BEATs or AST fine-tune.
- Over 100K clips
- train from scratch or fine-tune Whisper-encoder.
- Real-time, edge
- 40-MFCC CNN, quantized to int8.
- Multi-label
- BEATs-iter3 with BCE loss, mixup and SpecAugment.
- Exclusive classes
- report top-1 and top-5 accuracy.
- Imbalanced classes
- report per-class recall and macro F1.
From Lesson 06.03
Metrics#
- WER
- Word Error Rate. (S + D + I) / N after normalization.
- CER
- Character Error Rate. For tone languages.
- RTFx
- audio seconds per wall-clock second. Higher is better.
- MOS
- Mean Opinion Score. 1 to 5, 20+ listeners, 100+ samples.
- UTMOS
- learned MOS predictor. Correlates ~0.9 with human MOS.
- SECS
- ECAPA cosine of reference and clone. Above 0.75 is recognizable.
- EER
- threshold where FAR equals FRR. ECAPA on VoxCeleb1-O: 0.87%.
- DER
- (FA + Miss + Confusion) / total speaker time.
- FAD
- Fréchet distance on VGGish embeddings. Lower is better.
From Lesson 06.17
Troubleshooting#
| Symptom | Cause | Fix |
|---|---|---|
| Garbage that looks like a model bug | 44.1 kHz stereo into Whisper | Resample to 16 kHz mono float32 |
| FFT peak at a false frequency | Energy above Nyquist aliased | Low-pass filter before downsampling |
| Silent failure after deploy | 80 mels in training, 128 at inference | Log the feature shape at both ends |
| Mels look different | Featurized at 22.05 kHz | Fix SR before featurization |
| Whisper input looks wrong | dB-mel, not log-mel | Feed log-mel. Custom code does not autodetect |
| WER doubles in production | Per-utterance norm in training, global at inference | Use the same normalization |
| Flat trailing frames | Zero-padding at the clip end | Pad symmetrically or replicate |
| 5% WER hides 30% on accents | Aggregate WER | Report by demographic slice |
From Lesson 06.01, Lesson 06.02, Lesson 06.17
Questions#
- Why mel, not Hz?
- Humans hear pitch on a log scale.
- Why a window?
- It removes spectral leakage at the frame edges.
- Why log-magnitude?
- Raw magnitudes span 5-6 orders of magnitude.
- Bigger FFT?
- Better frequency resolution. Worse time resolution.
- Why FFT, not DFT?
- The DFT is O(N²). The FFT is O(N log N).
- Why minDCF?
- It weights cost at a chosen operating point.
From Lesson 06.02, Lesson 06.01, Lesson 06.17
Tips#
- Match sample rate before you match anything else.
- If you are not working on music, start with 80 log-mels.
- Start with a frozen backbone, not a fresh model.
- Use balanced sampling in training, not in evaluation.
- Normalize text before you score WER. Report the rule.
- Report P50/P95/P99 for latency.
- Keep an in-house held-out set. Public benchmarks saturate.