Voice Cloning & Voice Conversion
Voice cloning reads your text in someone else's voice. Voice conversion rewrites your voice into someone else's while preserving what you said. Both hang on the same decomposition: separate speaker identity from content. In 2026, a 5-second audio clip is enough to produce a high-quality clone of anyone's voice with a consumer GPU. ElevenLabs, F5-TTS, OpenVoice v2, VoiceBox all ship zero-shot or few-shot cloning. The technology is a blessing (accessibility TTS, dubbing, assistive voices) and a weapon (scam calls, political deepfakes, IP theft). Two closely-related tasks: Voice cloning (TTS-side): text + 5-second reference voice → audio in that voice. Voice conversion (speech-side): source audio (person A saying X) + reference voice of person B → audio of B saying X. Both factor a waveform into (content, speaker, prosody) and recombine content from one source with speaker from another. Key constraint you now ship under in 2026: watermarking and consent gates are legally required in the EU (AI Act, enforceable August 2026) and in California (AB 2905, effective 2025). Your pipeline must emit an inaudible watermark and refuse non-consensual clones. Voice cloning vs conversion: factorize, swap speaker, recombine Zero-shot cloning. Pass a 5-second clip to a model that has been trained on thousands of speakers. The speaker encoder maps the clip to a speaker embedding; the TTS decoder conditions on that…
Voice Cloning & Voice Conversion: Voice cloning reads your text in someone else's voice. Voice conversion rewrites your voice into someone else's while…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.