Omni Models: Qwen2.5-Omni and the Thinker-Talker Split
GPT-4o's product demo in May 2024 was disruptive not because of the underlying model but because of the product shape — a voice interface where you talk, the model sees what the camera sees, and it talks back in under 250ms. The open ecosystem spent the rest of 2024 and 2025 racing to reach that product surface. Qwen2.5-Omni (March 2025) is the reference open design: a Thinker (large text-generating transformer) plus a Talker (parallel speech-generating transformer), linked by streaming speech tokens. Mini-Omni simplified it, Moshi matched its latency, GLM-4-Voice extended it to Chinese. This lesson reads the Thinker-Talker architecture and the latency budget that makes streaming real-time dialogue work. Split the inference pipeline into Thinker (text reasoning) and Talker (speech synthesis) and explain why parallel streaming works. Compute the time-to-first-audio-byte (TTFAB) budget for a conversational interaction, component by component. Describe TMRoPE's time-aligned position encoding across vision, audio, and text within the Thinker. Name the three real-time conversational patterns: half-duplex, turn-taking, full-duplex. A real-time voice assistant has to do a lot, fast: Hear the user. Real-time speech tokenization, voice activity detection (VAD) to know when they're done speaking. Optionally see. Camera input at 2-4 FPS, streamed into the Thinker alongside audio. Think. Compose a response conditioned on the conversation history. Speak. Synthesize audio tokens, decode to waveform, stream to the user's…
Omni Models: Qwen2.5-Omni and the Thinker-Talker Split: GPT-4o's product demo in May 2024 was disruptive not because of the underlying model but because of…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.