Cheat sheets

Music Generation and Audio-Language Models

Codec tokens, MusicGen, audio encoders and projectors, Thinker and Talker, TTFAB budgets and audio pitfalls.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Audio becomes codec tokens#

vectorsindicesnew tokensclientWaveform16-48 kHzprocessEncoder convper-timestep vectorstoreRVQ8 codebooks at 75 HzmodelGeneratortoken-AR or diffusionprocessCodec decodertokens to audioAudio outvectorsindicesnew tokensclientWaveform16-48 kHzprocessEncoder convper-timestep vectorstoreRVQ8 codebooks at 75 HzmodelGeneratortoken-AR or diffusionprocessCodec decodertokens to audioAudio out

From Lesson 08.11

Numbers that matter#

Codec rate
8 RVQ codebooks at 75 Hz give 600 tokens/sec.
Stream floor
at 75 tokens/second, generate ≥75 tokens/sec per user.
Talker floor
50 Hz speech tokens need ≥50 tok/s from the Talker.
Codebook
1024 or 2048 entries is typical.
MusicGen codec
EnCodec at 32 kHz with 4 codebooks.
Log-Mel shape
30 s at 100 Hz frame rate gives (3000, 80).
Round trip
below 500ms, the user stops noticing the lag.

From Lesson 08.11, Lesson 06.09, Lesson 12.19, Lesson 12.20

Two generator families#

Token-autoregressive
a decoder-only transformer over flattened RVQ tokens.
Delayed parallel
MusicGen emits K codebook streams with per-stream offsets.
VALL-E
speech tokens from a text prompt and a 3-second voice sample.
Latent diffusion
Stable Audio Open works on compressed audio. Good for loops.
Flow matching
Stable Audio 2.5 uses it on continuous audio latents.
Trend
flow matching wins for music. Token-AR dominates speech.

From Lesson 08.11, Lesson 06.09

MusicGen, melody and FAD#

from audiocraft.models import MusicGen
model = MusicGen.get_pretrained("facebook/musicgen-small")300M, fast
model.set_generation_params(duration=10)seconds of audio
wav = model.generate(["upbeat synthwave with driving drums, 128 BPM"])
torchaudio.save("out.wav", wav[0].cpu(), 32000)32 kHz output
melody, sr = torchaudio.load("humming.wav")
wav = model.generate_with_chroma(keeps the tune, swaps timbre
["jazz piano cover"], melody.squeeze(), sr)
from frechet_audio_distance import FrechetAudioDistance
fad = FrechetAudioDistance()VGGish embedding distance
fad.get_fad_score("generated_folder/", "reference_folder/")lower is better

From Lesson 06.09

Audio-language model template#

1Audio encoderWhisper encoder, BEATs, CLAP or WavLM makesfeatures.2Projectora linear layer or MLP maps features into the LLMembedding space.3LLMa decoder reads interleaved text and audiotokens and writes text.4Stage 1freeze encoder and LLM. Train the projector onASR or captioning data.5Stage 2full or LoRA fine-tune on instruction audiotasks.6Stage 3optional. Add a speech decoder for voice-in andvoice-out.

From Lesson 06.10

Projector and audio Q-former#

class AudioProjector(nn.Module):
def __init__(self, audio_dim=1280, llm_dim=4096):
super().__init__()
self.down = nn.Linear(audio_dim, llm_dim)audio to LLM width
self.act = nn.GELU()
self.up = nn.Linear(llm_dim, llm_dim)
def forward(self, audio_features):
return self.up(self.act(self.down(audio_features)))
qf = QFormer(n_queries=8, hidden=20)N=8 queries
tokens = qf.forward(frames)99 frames in, 8 tokens out

From Lesson 06.10, 12.19 code/main.py

Q&A#

When is a cascade enough?
Transcription is the goal, with no music and no emotion.
When do I need end-to-end?
Music, emotion, multi-speaker or complex audio reasoning.
What does a cascade lose?
Mood, speaker identity, timing and deepfake cues.
Why a small Talker?
A 7B Talker falls behind the 50 tok/s floor.
Why TMRoPE?
It gives each token an absolute timestamp. Vision, audio and text align.
Half or full duplex?
Qwen2.5-Omni is half-duplex by default. Moshi supports full-duplex.

From Lesson 12.19, Lesson 12.20

Log-Mel and encoder settings#

SettingDefaultNote
Sample rate16 kHzResample first
Window25mswin_ms: int = 25
Hop10mshop_ms: int = 10
Mel filters800-8000 Hz
Log compressmath.log(1 + x)Dynamic range
Q-former queries32 or 64AF3 uses 64
Whisper encoder12-layerFrozen in the ALM pattern

From Lesson 12.19, 12.19 code/main.py

Thinker and Talker#

audio tokenstext tokensRVQ indicesstreamclientMic and VADspeech tokensmodelThinker7B-80B, TMRoPEmodelTalker200M-1BprocessSpeech decoderSNAC, MoVQGANclientSpeakersaudio samplesaudio tokenstext tokensRVQ indicesstreamclientMic and VADspeech tokensmodelThinker7B-80B, TMRoPEmodelTalker200M-1BprocessSpeech decoderSNAC, MoVQGANclientSpeakersaudio samples

From Lesson 12.20

TTFAB budget at 7B#

SettingDefaultNote
Mic to audio tokens40-80msStreams
Prefill100-200msMuch more at 70B
First Thinker text token40ms
Talker on first text token20ms
First speech tokens commit40ms
Residual-VQ decode30msrvq_decode = 30
Speech waveform decode50-80mswave_decode = 70
Total TTFAB320-510ms600-900ms at 70B

From Lesson 12.20, 12.20 code/main.py

Troubleshooting#

SymptomCauseFix
Music loops past 30 sAR models loopCrossfade, or use ACE-Step
Tempo leaves the BPMModels wanderBPM tags, librosa beat_track
Mushy sung wordsOpen models are weak on lyricsCommercial API or fine-tune
Hallucinations on silenceWhisper encoder issueVAD-gate
Speakers mixed past 10 minutesLong-audio degradationDiarize first, then summarize
Multi-audio near 25%Random-chance levelDo not trust "which clip" tasks
Errors from RVQ layer 1 spreadRVQ error accumulationTemperature 0 on higher layers
Artifacts at clip joinsCrossfade boundariesCareful overlap-add

From Lesson 06.09, Lesson 06.10, Lesson 08.11

Tasks in code#

TaskCode
Load the ALM processorAutoProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B")
Generate an answermodel.generate(**inputs, max_new_tokens=200)
Load MMAU-Proload_dataset("gamma-lab-umd/MMAU-Pro", split="test")
Score MMAU-Pro officiallyevaluate_mmau_pro_comprehensive.py
Cut audio into frameswindow_frames(wave, 16000, 25, 10)
List TTFAB partsttfab(cfg)
Sample codec tokensgenerate(counts, style, start, length, rng, temperature=1.0)

From Lesson 06.10, 06.10 code/main.py, 12.19 code/main.py, 12.20 code/main.py, 08.11 code/main.py, Lesson 08.11

Tips#

  • To ship safely, generate instrumental only.
  • Tag each generation with watermarks and metadata.
  • Report each MMAU-Pro category separately.
  • Check the codec first. It caps output quality. DAC is the current open best.
  • For voice cloning, add abuse detection and opt-out lists.

From Lesson 06.09, Lesson 06.10, Lesson 08.11