Speech Recognition and Whisper

Decode CTC, run and tune Whisper, gate with VAD, and ship a WAV to captions pipeline.

Concepts and terms
Tasks and commands
Annotated code
Diagrams
Defaults and numbers
Troubleshooting
Questions and tips

Tasks in code#

Task	Code
Greedy CTC decode	`ctc_greedy(frame_probs)`
Beam CTC decode	`ctc_beam(frame_probs, beam_width=8)`
Word error rate	`wer(ref, hyp)`
Plan chunks for a 10 min clip	`chunk_schedule(600.0, chunk_s=30.0, stride_s=5.0)`
Turn-end state machine	`TurnDetector(silence_hangover_ms=500, min_speech_ms=250)`
LoRA on attention	`LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])`

From 06.04 code/main.py, Lesson 06.04, 06.05 code/main.py, Lesson 06.05, 07.10 code/main.py, 06.14 code/main.py

Whisper inside#

From Lesson 07.10, Lesson 06.05

Transcribe with Whisper#

import whisper

model = whisper.load_model("large-v3-turbo")

result = model.transcribe("clip.wav",

language="en",stops LID drift

task="transcribe",or <|translate|> to English

temperature=0.0,no sampling fallback

condition_on_previous_text=False)prevents runaway repetition

print(result["text"])

for seg in result["segments"]:start, end, text per segment

print(seg["start"], seg["end"], seg["text"])

From Lesson 06.05

Faster runtimes and VAD#

from faster_whisper import WhisperModel

model = WhisperModel("large-v3-turbo", compute_type="int8_float16")4x faster

segments, info = model.transcribe("meeting.wav", vad_filter=True)VAD gate

from silero_vad import load_silero_vad, get_speech_timestamps

vad = load_silero_vad()

segments = get_speech_timestamps(audio, vad, sampling_rate=16000, threshold=0.5,

min_speech_duration_ms=250, min_silence_duration_ms=500, speech_pad_ms=300)

From Lesson 07.10, Lesson 06.05, Lesson 06.14

ASR formulations#

CTC: per-frame tokens plus a blank. No internal LM. Fast.
Greedy CTC: collapse repeats, drop blanks. a a _ _ a b b _ c gives a a b c.
RNN-T: predictor plus joiner. Streamable, with an internal LM.
Attention enc-dec: best offline quality. Cannot stream without engineering.
WER: $inline$ . Above 20% is unusable. Below 5% is human parity.
Turbo: 4-layer decoder (from 32). 8× faster, <1% WER hit.
Prompt: <|startoftranscript|> <|en|> <|transcribe|> or <|translate|>, then <|notimestamps|>.

From Lesson 06.04, Lesson 06.05, Lesson 07.10

Defaults#

Setting	Default	Note
`temperature`	0.0	Override the fallback chain
`condition_on_previous_text`	False	Stops the hallucination cascade
`no_speech_threshold`	0.6	Silence detection
Silero `threshold`	0.5	0.3 is sensitive
`min_speech_duration_ms`	250	Rejects coughs and chair noise
Silence hangover	500-800 ms	Shorter interrupts, longer feels sluggish
Pre-roll buffer	300-500 ms	Keeps the first word

From Lesson 06.05, Lesson 06.14, solution/pipeline.py

Project flags#

Command	What it does
`--input <wav>`	Mono 16-bit PCM WAV, 20 MB at most
`--out <dir>`	Output folder, default `voice-output`
`--transcript-file <json>`	Offline reference text bound to the audio SHA256
`--endpoint <url>`	Uploads the WAV to an audio/transcriptions endpoint
`--segment`	RMS activity split before recognition
`--review-file <json>`	Rerenders an edited transcript.json

From solution/pipeline.py, voice-note-transcriber-pipeline/README.md

Run the project#

python3 scripts/project_test.py voice-note-transcriber-pipeline --init my-voice-pipeline

python3 scripts/project_test.py voice-note-transcriber-pipeline --all --solution --strict

cd projects/voice-note-transcriber-pipeline/solution

python3 pipeline.py --input fixtures/repair-note.wav --transcript-file fixtures/reference.json --out voice-output

ffmpeg -i YOUR_NOTE.m4a -ac 1 -ar 16000 -c:a pcm_s16le YOUR_NOTE.wavmono 16-bit PCM

python3 pipeline.py --input YOUR_NOTE.wav --endpoint <url> --model YOUR_MODEL --out voice-output

From voice-note-transcriber-pipeline/README.md

Project pipeline#

From solution/pcm.py, solution/activity.py, solution/transcribe.py, solution/provider.py, solution/captions.py, solution/pipeline.py

Troubleshooting#

Symptom	Cause	Fix
"Thanks for watching!" on silence	No VAD	VAD-gate before you call Whisper
Noisy clip comes out in Japanese	Auto LID drift	Force `language="en"`
One error repeats in later windows	Previous text cascade	`condition_on_previous_text=False`
Text in the padding of a 2 s clip	Pad to 30 s	`pad=False` or VAD-gate
Near-random output	librosa mels, wrong stats	`whisper.audio.log_mel_spectrogram`
Agent cuts the user off	Hangover too short	500-800 ms hangover, semantic endpoint
"16-bit mono PCM required"	Stereo or compressed input	Convert with `ffmpeg -ac 1 -ar 16000`
"Remote endpoints require HTTPS"	HTTP to a remote host	Use HTTPS. HTTP is for loopback only

From Lesson 06.04, Lesson 06.05, Lesson 06.14, solution/pcm.py, solution/provider.py, voice-note-transcriber-pipeline/README.md

Questions#

When to freeze the encoder?: With less than 10 hours of data. Tune only the decoder.
How much data to fine-tune?: 10 to 100 hours of target-domain audio with transcripts.
Why CTC beam over greedy?: Beam search allows LM fusion. CTC has no internal LM.
What is the flush trick?: On end of speech, flush the STT. A 500 ms buffer ends in ~125 ms.
Energy VAD in production?: No. It fires on any noise above -40 dBFS.

From Lesson 06.04, Lesson 06.05, Lesson 06.14