Cheat sheets

Real-Time Voice Assistants

VAD, turn detection, barge-in, latency budgets and full-duplex speech for a voice agent that answers in time.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Cascaded voice pipeline#

framesspeechpartialscommitcalltokensaudiobarge-in cancelclientMic16 kHz mono, 20 msprocessVADSilero, speech gatemodelStreaming STTpartial transcriptsmodelTurn detectorscore above 0.6modelLLM with toolsstreams tokensprocessTool sidechannelfiller after 300msmodelStreaming TTSfirst chunk in200msclientSpeakerWebRTC jitterbufferframesspeechpartialscommitcalltokensaudiobarge-in cancelclientMic16 kHz mono, 20msprocessVADSilero, speechgatemodelStreaming STTpartialtranscriptsmodelTurn detectorscore above 0.6modelLLM with toolsstreams tokensprocessTool sidechannelfiller after300msmodelStreaming TTSfirst chunk in200msclientSpeakerWebRTC jitterbuffer

From Lesson 06.12, Lesson 19.03

Latency budget per turn#

SettingDefaultNote
Mic to buffer20 msOne frame
VAD10 msSilero runs under 1 ms per frame
ASR (streaming)150 msPartial transcripts
LLM (first token)100 msStream the tokens
TTS (first chunk)100 msStart before the full reply
Render to speaker20 msOne frame
Total~400 msAbove 500 ms feels robotic
First TTS audio, laptop CPU800 msPhase 6 capstone target

From Lesson 06.11, Lesson 06.12

Silero VAD segments#

from silero_vad import load_silero_vad, get_speech_timestamps
vad = load_silero_vad()
audio = torch.tensor(waveform_16k, dtype=torch.float32)
segments = get_speech_timestamps(
audio, vad, sampling_rate=16000,
threshold=0.5,0.3 is sensitive
min_speech_duration_ms=250,drops coughs
min_silence_duration_ms=500,silence hangover
speech_pad_ms=300,pre-roll pad
)
for s in segments:
print(f"{s['start']/16000:.2f}s - {s['end']/16000:.2f}s")

From Lesson 06.14

Barge-in and the flush trick#

tts_task = asyncio.create_task(tts_loop())
while True:
chunk = await mic.get()
if vad(chunk):user speaks during TTS
tts_task.cancel()stop the reply
await speaker.stop()
await new_turn()restart STT
break
def flush_on_end(stt_client, audio_buffer):
stt_client.send_audio(audio_buffer)
stt_client.send_flush()forces output now
return stt_client.recv_transcript(timeout_ms=150)

From Lesson 06.12, Lesson 06.14

Troubleshooting#

SymptomCauseFix
First word is missingVAD starts a beat too lateThreshold 0.3 and a pre-roll
Bot talks over the userLLM keeps going after interruptWire VAD to cancel the LLM
"Thanks for watching" on silenceWhisper on silent framesAlways VAD-gate
ASR hears the bot voiceNo echo cancellationWebRTC AEC3
Agent cuts user off mid-sentenceHangover too short500-800 ms and a turn detector
Slow first replyTTS warm-up on first requestWarm it with a dummy run
Audible vocoder artifactsSub-200 ms TTS chunks320 ms chunks
Glitches under loadGIL starves the audio threadC-callback audio library

From Lesson 06.12, Lesson 06.11, Lesson 06.14

Glossary#

AEC
echo cancellation. Subtracts the speaker to mic path.
TPR @ 5% FPR
VAD benchmark point. Silero 87.7%, WebRTC VAD 50.0%.
MultimodalAgent
LiveKit class. Audio in, audio out, no text in the middle.
VoicePipelineAgent
LiveKit class. STT, LLM, TTS with text-level control.

From Lesson 06.11, Lesson 06.14, Lesson 14.22

Capstone scheduler, one turn#

1IDLEthe first speech frame moves the state toLISTENING.2LISTENINGeach silent frame adds 20 ms to silenceRunMs.3LISTENINGat 500 ms of silence, turnCompletionScore readsthe last partial.4WAITINGa score of 0.6 or more commits the turn and setsturnCompleteMs.5THINKINGthe tool call fires. At 300 ms the agent says"one second, let me check".6THINKINGthe tool returns at 420 ms. The first LLM tokencomes 140 ms later.7SPEAKINGthe first audio out comes 180 ms after the firstLLM token.8Barge-inspeech in THINKING or SPEAKING cancels TTS andre-arms ASR.

From 19.03 code/ts/src/orchestrator.ts, 19.03 code/main.py

VAD and turn settings#

SettingDefaultNote
Silero threshold0.50.3 is sensitive. Fewer first-word clips
Minimum speech250 msRejects coughs and chair noise
Silence hangover500-800 msToo short cuts the user off
Pre-roll buffer300-500 msKeeps the first word
Energy gate-40 dBFSFires on any noise above it
Frame size20 ms320 samples at 16 kHz
Ring buffer32,000 samples2 s at 16 kHz
Turn commit (capstone)0.6Plus 500ms of VAD silence

From Lesson 06.14, Lesson 06.11, Lesson 19.03

Run the lessons#

CommandWhat it does
python3 code/main.pyRuns the simulator of a lesson
python main.pyCapstone scheduler, clean and barge-in calls
pip install silero-vadReal VAD in place of the stub
npm installInstalls the TypeScript client
npm startTwo offline sessions and a ws probe, exits 0
npm start -- --serveKeeps the ws server up
npm testnode --test runner via tsx
npm run typecheckRuns tsc --noEmit

From 06.11 code/main.py, Lesson 06.12, 19.03 code/main.py, 19.03 code/ts/README.md

Core terms#

Ring buffer
fixed circular buffer. No allocations in the hot path.
Jitter buffer
reorders late packets. Too small gives gaps. Too large adds latency.
Pre-roll
audio kept before VAD fires. It prevents the first-word clip.
Silence hangover
the wait after VAD returns to 0, before end of turn.
Semantic endpoint
a classifier that reads the words, not only the silence.
Flush trick
flush the STT at end of speech. 500 ms of look-ahead ends in ~125 ms.
Filler
a short phrase when a tool takes more than 300ms.
DOWNSTREAM
Pipecat frames from source to sink, audio in and speech out.
UPSTREAM
Pipecat control frames, such as cancel, metrics and barge-in.

From Lesson 06.11, Lesson 06.14, Lesson 19.03, Lesson 14.22

Full-duplex Moshi, one 80 ms step#

1Mimiencodes user audio at 12.5 Hz with 8 codebooks.2Temporal Transformer7B parameters. Reads the user tokens and its ownlast tokens.3Temporal Transformeremits the next text token, the inner monologue.4Depth Transformer2 layers. Predicts the 8 codebooks one after theother.5Clientdecode_audio_mimi turns the tokens into audioand plays it.6Budget80 ms frame plus 80 ms acoustic delay is 160 ms.200 ms on one L4 GPU.

From Lesson 06.15, 06.15 code/main.py

Q&A#

How fast do humans take turns?
About 230 ms. Above 1500 ms feels broken.
Pipeline or Moshi?
Use a pipeline for tools. Moshi is not trained for tool calling.
Why no flush with Whisper streaming?
It is block-based and always waits for chunks.
What is end-to-end premium?
450 to 600 ms. 800 to 1200 ms is common.
What is the capstone bar?
First-audio-out p50 under 800ms, false-cutoff under 3%.
What does barge-in need in Pipecat?
UPSTREAM cancel frames back to TTS and LLM.

From Lesson 06.11, Lesson 06.14, Lesson 06.15, Lesson 14.22, Lesson 19.03