Cheat sheets

Text-to-Speech, Voice Cloning and Codecs

Text to phonemes to mel to waveform, clone with consent and a watermark, and tokenize audio with RVQ codecs.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

TTS and cloning pipeline#

phonemesmelembeddingwaveformclientTextnormalize firstprocessPhonemizerespeak, G2PfallbackmodelAcoustic modeltext to mel, 80channelsmodelVocoderHiFi-GAN, mel towaveformstoreReference clip5 s plus itstranscriptmodelSpeaker encoderspeaker embeddingprocessWatermark~32 bits of payloadphonemesmelembeddingwaveformclientTextnormalize firstprocessPhonemizerespeak, G2P fallbackmodelAcoustic modeltext to mel, 80channelsmodelVocoderHiFi-GAN, mel towaveformstoreReference clip5 s plus itstranscriptmodelSpeaker encoderspeaker embeddingprocessWatermark~32 bits of payload

From Lesson 06.07, Lesson 06.08

Tasks in code#

TaskCode
Phonemizephonemize("Hello world", language="en-us", backend="espeak")
Kokoro on CPUKPipeline(lang_code="a"), then tts(text, voice="af_bella")
Zero-shot cloneF5TTS().infer(ref_file=<wav>, ref_text=<text>, gen_text=<text>)
Voice conversionvc.convert(source=<wav>, target_pool=[<wav>, <wav>])
Embed a watermarksc.embed(wav, sr=24000, message=payload)
EnCodec at 6 kbpsmodel.set_target_bandwidth(6.0)
Mimi semantic splitsemantic = codes[:, 0], acoustic = codes[:, 1:]
Toy RVQrvq_encode(values, codebook_size=8, n_codebooks=4)

From Lesson 06.07, Lesson 06.08, Lesson 06.13, 06.13 code/main.py

Clone and watermark#

from f5_tts.api import F5TTS
tts = F5TTS()
wav = tts.infer(ref_file="my_voice_5s.wav",5 s reference
ref_text="The quick brown fox jumps over the lazy dog.",must match exactly
gen_text="Please remind me to water the plants.")
sc = SilentCipher(model="2024-06-01")
payload = b"consent_id:abc123;ts:1745353200"
watermarked = sc.embed(wav, sr=24000, message=payload)~32 bits
detected = sc.detect(watermarked, sr=24000)payload bytes

From Lesson 06.08, Lesson 06.07

Encode with a codec#

model = EncodecModel.encodec_model_24khz()
model.set_target_bandwidth(6.0)kbps, 8 codebooks
with torch.no_grad():
encoded = model.encode(wav)
codes, scale = encoded[0](1, n_codebooks, n_frames)
wav_recon = model.decode([(codes, scale)])
mimi = loaders.get_mimi()from moshi.models
codes = mimi.encode(wav)(1, 8, frames@12.5Hz)
semantic = codes[:, 0]content, WavLM-aligned
acoustic = codes[:, 1:]timbre, prosody, noise

From Lesson 06.13

Formulas and numbers#

RVQ
codebook 1 quantizes the encoder output. Each next one quantizes the residual.
Decode
the decoder sums the chosen codes per frame.
Vocabulary
8 codebooks of 1024 codes give 1024^8 = 10^{24}.
LM tokens
10 \times 12.5 \times 8 = 1000 tokens for 10 s of Mimi.
Toy RVQ bits
8 codes are 3 bits. Bits per frame is 3 times the codebooks.
Mel hop
12.5 ms at 24 kHz is 300 samples.
HiFi-GAN
upsample rates 8, 8, 2, 2 give 256x, mel rate to audio rate.

From Lesson 06.13, 06.13 code/main.py, Lesson 06.07, 06.07 code/main.py

Codec numbers#

SettingDefaultNote
EnCodec-24k frame rate75 HzDefault 4 codebooks @ 1.5 kbps
DAC-44.1k frame rate86 HzHighest fidelity, full-band
SNAC-24k coarse rate~12 HzDetail at 50 Hz
Mimi frame rate12.5 Hz8 codebooks @ 4.4 kbps
Codes per codebook1024Each code is 0-1023
Codebook count8-12More adds fidelity and sequence length
Kokoro output24 kHzResample for 16 kHz consumers

From Lesson 06.13, Lesson 06.07

Models#

Tacotron 2
autoregressive seq2seq. Slow, wobbly on long text.
FastSpeech 2
duration predictor, 1-pass, 10× faster than Tacotron.
VITS
encoder, flow duration and HiFi-GAN, trained end-to-end. 25M.
F5-TTS
flow-matching diffusion transformer. Clones from 5 s. 335M.
Kokoro
82M, CPU-runnable, English only.
KNN-VC
swaps each WavLM source frame for its nearest target-pool frame.
Mimi
codebook 0 is distilled from WavLM. Codebooks 1-7 are acoustic.

From Lesson 06.07, Lesson 06.08, Lesson 06.13

Abbreviations#

TTS
Text-to-Speech. Text frontend, acoustic model, vocoder.
VC
Voice conversion. Speech of A rewritten in the voice of B.
PPG
Phonetic posteriorgram. Per-frame ASR posteriors as content.
RVQ
Residual Vector Quantization. A cascade of small codebooks.
SECS
speaker embedding cosine. Above 0.70 is indistinguishable for most listeners.
UTMOS
reference-free MOS predictor. Ground truth scores 4.08.
CMOS
Comparative MOS. An A-vs-B preference.
PESQ / ViSQOL
objective metrics that correlate with MOS.

From Lesson 06.07, Lesson 06.08, Lesson 06.13

Troubleshooting#

SymptomCauseFix
"Dr." reads as "Drive"No text normalizerNormalize before the phonemizer
Proper noun mispronouncedOOV tokenShip a fallback G2P model
Output overshoots ±1.0Mel scaling mismatchnp.clip(wav, -1, 1)
Aliasing downstream24 kHz Kokoro into a 16 kHz pipelineResample
Clone alignment breaksReference text differs from audioMatch it exactly, punctuation included
Echo in the cloneReverberant referenceRecord dry, close-mic
French clone keeps the accentLanguage leakageXTTS or VALL-E X
LM fails silently12.5 Hz Mimi, then 50 Hz EnCodecKeep one frame rate

From Lesson 06.07, Lesson 06.08, Lesson 06.13

Use X when#

  • Real-time English on CPU: Kokoro.
  • Clone from a 5 s reference: F5-TTS or OpenVoice v2.
  • Rewrite speech into a target voice: KNN-VC or Diff-HierVC.
  • Generative model over audio: start with Mimi or SNAC.
  • Compression pipeline: use Opus. It wins per bit.
  • Highest-fidelity reconstruction: DAC-44.1k.
  • Every clone: a watermark and a consent record.

From Lesson 06.07, Lesson 06.08, Lesson 06.13

Questions#

Why split semantic and acoustic tokens?
Pure reconstruction tokens give blurry speech from text.
Why RVQ, not one codebook?
One big codebook needs millions of codes.
Which Mimi codebook matters?
Codebook 0. Losing it destroys intelligibility.
Cloning or conversion?
Cloning takes text. Conversion takes source speech.
Few-shot fine-tune?
5-30 minutes of the target voice, LoRA for an hour.

From Lesson 06.13, Lesson 06.08