Cheat sheets

Multi-Agent RL and Sim-to-Real

CTDE, value decomposition, self-play, domain randomization and VLA action formats. Formulas, numbers and fixes.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

MADDPG, QMIX, MAPPO#

FactorMADDPGQMIXMAPPO
Year201720182022
IdeaPer-agent critic sees all obs and actionsMonotonic mix of per-agent QsPPO with a centralized value function
SettingCooperative, competitive, mixedCooperative onlyCooperative
TestbedParticle WorldSMACParticle-world, SMAC, Football, Hanabi, MPE
LearningOff-policy, DDPGValue decompositionOn-policy, PPO
Key traitCritic grows with N, ~10 agentsMonotonic constraint is strictStable, minimal tuning
StatusIntroduced CTDEDominant on SMACDefault cooperative baseline

Domain randomization in code#

slip_ep = rng.uniform(slip_low, slip_high)new sim each episode
s_next, r, done = step(s, a, slip_ep, rng)slip moves perpendicular
Q_fixed = train_fixed(0.0, rng=rng)policy A, no DR
Q_dr = train_dr(0.0, 0.3, rng=rng)policy B, DR
for slip in (0.0, 0.1, 0.2, 0.3, 0.5, 0.7):0.5 and 0.7 are OOD
r_fixed = evaluate(Q_fixed, slip)
r_dr = evaluate(Q_dr, slip)

Four VLAs side by side#

FactorRT-2OpenVLAπ0GR00T N1
Action formatText tokens, 256 bins256-bin tokensFlow-matching action expertSystem 1 action head
BackbonePaLM-X VLM7B Llama, DINOv2 + SigLIPVLM plus small action transformerLarge VLM plus small action head
Speed3-5 Hz4-5 Hz on an A100~5 denoising steps, about 50 HzSystem 1 at 50-100 Hz, System 2 ~1 Hz
Detail10 tokens per step for 10-DOFLoRA fine-tuning for new robots50-step action sequenceJoint commands, >30 DOF
DataWeb plus robot, ~1:1Open X-Embodiment, ~0.5:1 web-to-robotOpen X-Embodiment, similar ratioSim-to-real data from Omniverse

Defaults in the lesson code#

SettingDefaultNote
alpha0.1MARL and DR toys
gamma0.95MARL and DR toys
epsilon, eps0.15ε-greedy exploration
MARL episodes15003000 in the DR toy
DR slip rangeUniform[0.0, 0.3]Policy B
max_steps50CTDE pattern demo, 500 trials
bins256Bin width 2/255
keep_coeff4Low-frequency DCT terms per dim

Troubleshooting#

SymptomCauseFix
Agents chase each otherEach best response movesCentralized critic, slow LR, freeze one
Replay makes it worseObsolete opponents in old dataRelabel, or weight by recency
No credit for shared rewardCredit assignmentCOMA baseline or per-agent shaping
Self-play stuck in a cycleDominance cycleLeague play with diverse opponents
DR policy too cautiousToo much randomizationMatch the expected real distribution
DR does not helpWrong parameter randomizedProfile the real robot first
Student cannot match teacherPrivileged info leakageKeep the teacher realizable from obs
Policy breaks hardwareNo safety envelopeRate, torque and joint limits outside

Core ideas and formulas#

Markov game
(S, A_1, \ldots, A_n, P, R_1, \ldots, R_n). Each agent has its own reward.
Reward types
identical is cooperative, zero-sum is adversarial, mixed is general-sum.
Non-stationarity
P(s' \mid s, a_i) depends on \pi_{-i}, and \pi_{-i} changes.
CTDE
critic Q(s, a_1, \ldots, a_n) in training. Policy \pi_i reads only o_i.
QMIX
Q_{\text{tot}} = f(Q_1, \ldots, Q_n) with \partial f / \partial Q_i \ge 0.
COMA
advantage is my Q minus the expected Q over my own action.
Reality gap
the shift between sim and real physics or sensing.
Control budget
a 30 Hz loop gives 33ms for each action.

CTDE: train central, act local#

actionactionrewardgradientgradientclientAgent 1 obslocal onlyclientAgent 2 obslocal onlymodelActor 1maps own obs to actionmodelActor 2maps own obs to actionEnvironmentjoint action, shared rewardprocessCentralized criticall obs and actions, trainingonlyactionactionrewardgradientgradientclientAgent 1 obslocal onlyclientAgent 2 obslocal onlymodelActor 1maps own obs toactionmodelActor 2maps own obs toactionEnvironmentjoint action, sharedrewardprocessCentralized criticall obs and actions,training only

Sim-to-real recipe (quadruped)#

1Simmassively parallel sim, with randomized gravity,friction, motor gains and payload.2Teachertrain with privileged info, terrain map and truebody velocity.3Studentdistill from the teacher with proprioceptiononly.4Adaptoptional observation adaptation with anautoencoder on real IMU.5Deployzero-shot on 10+ environments.6Fallbackminutes of real fine-tuning withsafety-constrained PPO.

Independent against joint Q#

a1 = epsilon_greedy(Q1, s, rng, epsilon)each agent picks alone
a2 = epsilon_greedy(Q2, s, rng, epsilon)
s_next, r, done = step(s, (a1, a2))joint step, shared reward
target1 = r + gamma * max(Q1[s_next].values())
Q1[s][a1] += alpha * (target1 - Q1[s][a1])own Q table
reward = 10.0 if done else -1.0+10 when both arrive
joint_actions = [(a, b) for a in ACTIONS for b in ACTIONS]16 pairs
ja = max(joint_actions, key=lambda a: Q[s][a]) # one greedy joint pick

Tasks in code#

TaskCode
Independent Q for two agentsindependent_q(episodes=1500)
One Q over 16 joint actionsjoint_q_learning(episodes=1500)
Train with DRtrain_dr(0.0, 0.3, rng=rng)
Zero-shot eval at one slipevaluate(Q_dr, slip)
Benchmark a CTDE patternbench(label, run_mappo_style)
Run the MARL pattern demopython3 code/main.py
Action to 256 binsdiscretize(action, bins=256)
FAST-style compressionfast_compress(trajectory, keep_coeff=4)

Abbreviations#

CTDE
Centralized training, decentralized execution.
IPPO
Independent PPO. Each agent runs PPO alone.
COMA
counterfactual multi-agent. Isolates my contribution.
DR
domain randomization. Random sim parameters in training.
SI
system identification. Measure the real robot, fit the sim.
ADR
Automatic Domain Randomization. Widens DR as the policy improves.
VLA
vision-language-action. Image and instruction in, actions out.
FAST
Frequency-domain Action Sequence Tokenizer. DCT, then quantize.

Rules of thumb#

  • Start with MAPPO for cooperative MARL.
  • Use independent Q only for loosely-coupled tasks.
  • Use self-play for zero-sum games. Use league play for strategy cycles.
  • Log the observation and action stream of every agent.
  • Test on a held-out sim variant before you deploy.
  • Flag a DR range wider than 3× the measured real variability.
  • Close a VLA domain gap with LoRA on 100-1000 task demos.

Questions#

Where does independent Q fail?
On tightly-coupled tasks, where one agent must wait.
Why does QMIX allow local argmax?
The mixer is monotonic in each per-agent Q.
What does narrow training do?
Slip 0.0 only drops hard as soon as real slip is above 0.
Why co-fine-tune a VLA?
Robot-only models fail on new instructions.
Why the GR00T split?
Slow planning does not block fast control.