Build a Voice Assistant Pipeline — The Phase 6 Capstone
Everything from lessons 01-11, stitched together. Build a voice assistant that listens, reasons, and talks back. In 2026 that is a solved engineering problem, not a research problem — but the integration details decide whether it ships. Build an end-to-end assistant: Captures mic input (16 kHz mono). Detects start/end of user speech. Transcribes streaming. Passes transcript to an LLM that can call tools (timer, weather, calendar). Streams LLM text to a TTS. Plays audio back to the user. Stops if the user interrupts mid-response. Latency target: first TTS audio byte within 800 ms of the user finishing their utterance on a laptop CPU. Quality target: no missed words, no hallucinated subtitles on silence, no voice cloning leakage, no prompt injection success. Voice assistant pipeline: mic → VAD → STT → LLM+tools → TTS → speaker Audio capture. Mic → 16 kHz mono → 20 ms chunks. Usually sounddevice in Python or native AudioUnit/ALSA/WASAPI in production. VAD (Lesson 11). Silero VAD @ threshold 0.5, min speech 250 ms, silence hang-over 500 ms. Signals "start" and "end.". Streaming STT (Lesson 4-5). Whisper-streaming, Parakeet-TDT, or Deepgram Nova-3 (API). Partial + final transcripts. LLM with tool calling. GPT-4o / Claude 3.5 / Gemini 2.5 Flash. JSON schema for tools. Stream tokens. Streaming TTS (Lesson 7). Kokoro-82M (fastest open) or Cartesia Sonic (commercial). Start TTS…
Build a Voice Assistant Pipeline — The Phase 6 Capstone: Everything from lessons 01-11, stitched together. Build a voice assistant that listens, reasons, and…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.