Streaming Speech-to-Speech — Moshi, Hibiki, and Full-Duplex Dialogue
2024-2026 redefined voice AI. Moshi ships a single model that listens and speaks simultaneously at 200 ms latency. Hibiki does speech-to-speech translation chunk-by-chunk. Both abandon the ASR → LLM → TTS pipeline for a unified full-duplex architecture over Mimi codec tokens. This is the new reference design. Every voice agent built from Lessons 11 + 12 has a fundamental latency floor around 300-500 ms: VAD fires, STT processes, LLM reasons, TTS generates. Each stage has its own minimum latency. You can tune and parallelize, but the pipeline shape caps you. Moshi (Kyutai, 2024-2026) asks a different question: what if there is no pipeline? What if one model takes audio in and emits audio out directly, continuously, with text as an intermediate "inner monologue" instead of a required stage? The answer is full-duplex speech-to-speech. Theoretical latency 160 ms (80 ms Mimi frame + 80 ms acoustic delay). Practical latency 200 ms on a single L4 GPU. That's half what a best-in-class pipelined voice agent achieves. Moshi architecture: two parallel Mimi streams + inner-monologue text Inputs. Two Mimi codec streams, both at 12.5 Hz × 8 codebooks: Stream 1: user audio (Mimi-encoded, constantly arriving). Stream 2: Moshi's own audio (generated by Moshi). The transformer. A 7B-parameter Temporal Transformer processes both streams and a text "inner monologue" stream. At each 80 ms step,…
Streaming Speech-to-Speech — Moshi, Hibiki, and Full-Duplex Dialogue: 2024-2026 redefined voice AI. Moshi ships a single model that listens and speaks…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.