Phase 16: Multi-Agent & Swarms

MARL — MADDPG, QMIX, MAPPO

Type: Learn

Languages: Python (stdlib, small NumPy-free implementations)

Prerequisites: Phase 09 (Reinforcement Learning), Phase 16 · 09 (Parallel Swarm Networks)

Time: ~90 minutes

Problem

LLM-agent systems increasingly train policies for inter-agent coordination: when to defer, when to act, which peer to call. The literature that tells you how to train such policies is Multi-Agent Reinforcement Learning (MARL), which predates the LLM wave and has a small set of dominant algorithms.

Reading MARL papers without the pattern vocabulary is painful. Centralized training with decentralized execution (CTDE), value decomposition, and centralized critics are not buzzwords — they are specific answers to specific problems:

• Independent RL (each agent learns alone) is non-stationary from each agent's perspective. Bad.
• Centralized RL (one agent controls all) does not scale and violates execution constraints.
• CTDE gets the best of both: train with global information, deploy with local policies.

Concept

Three environments the papers use

• Particle World (multi-agent particle env). Simple 2D physics with cooperative/competitive tasks. MADDPG's original testbed.
• StarCraft Multi-Agent Challenge (SMAC). Cooperative micro-management, partial observation. QMIX's testbed. Discrete actions, continuous states.
• Google Research Football, Hanabi, MPE. MAPPO baselines.

Different envs have different action/observation types. The algorithms pick accordingly.

MADDPG (2017) — the CTDE pattern

Each agent i has an actor mu_i(o_i) that maps its own observation to action. Each agent also has a critic Q_i(x, a_1, ..., a_n) that sees all observations and all actions during training. The actor is updated by policy gradient against the critic's evaluation.

Copyactor update:    grad_theta_i J = E[grad_theta mu_i(o_i) * grad_a_i Q_i(x, a_1..n) at a_i=mu_i(o_i)]
critic update:   TD on Q_i(x, a_1..n) given next-state joint estimate

Why CTDE: at training time, we know everyone's actions; we use that to reduce variance in each critic. At deploy time, each agent only sees o_i and calls mu_i(o_i).

Failure mode: critics grow with N agents (input includes all actions). Does not scale past ~10 agents without approximations.

QMIX (2018) — value decomposition

Cooperative only. Global reward is the sum of a monotone function of per-agent Q-values:

The monotonicity guarantees argmax_a Q_tot can be computed by each agent choosing argmax_{a_i} Q_i independently. That is exactly the decentralized execution property you need. At training time, a mixing network produces Q_tot from the per-agent Qs.

Why QMIX wins on SMAC: cooperative StarCraft micro-management has homogeneous agents, local obs, global reward — perfect fit for value decomposition.

Failure mode: the monotonicity constraint is restrictive; some tasks have reward structures that are not monotone decomposable (one agent sacrificing for the team). Extensions (QTRAN, QPLEX) relax this.

MAPPO (2022) — the overlooked default

Multi-Agent PPO: PPO with a centralized value function. Each agent has its own policy; all agents share (or have per-agent) value functions that see the full state. Yu et al. 2022 benchmarked MAPPO against MADDPG, QMIX, and their extensions on five benchmarks and found:

• MAPPO matches or beats off-policy MARL methods on particle-world, SMAC, Google Research Football, Hanabi, MPE.
• Minimal hyperparameter tuning required.
• Stable training; reproducible across seeds.

The community underrated on-policy MARL until this paper. In 2026, MAPPO is the default baseline for cooperative MARL; any new method must beat it.

Why LLM-agent engineers should care

Three direct uses:

1. Router training. A meta-agent chooses which sub-agent handles a task. This is a MARL problem with N decentralized sub-agents and one centralized router. MAPPO fits.
2. Role emergence. In generative-agent simulations, training agents to adopt complementary roles over time is a MARL problem in disguise. QMIX-style value decomposition forces complementarity by construction.
3. Multi-agent tool use. When agents share tools and compete for budget, training them via CTDE produces deployable local policies that respect resource constraints.

Practical caveat: in 2026, most production LLM-agent systems prompt their policies rather than train them. MARL comes in when you have (a) lots of interaction data, (b) a clear reward signal, and (c) willingness to invest in training infrastructure.

CTDE as a design pattern beyond RL

Even without training, CTDE is a useful architectural pattern:

• During design, assume full team visibility.
• At runtime, enforce decentralized execution: each agent sees only o_i.

The pattern forces you to keep per-agent state explicit and to think about partial observability up front. Many production multi-agent systems silently assume shared state everywhere — CTDE discipline prevents that.

The non-stationarity problem

When multiple agents learn simultaneously, each agent's environment (which includes others' policies) is non-stationary. Classical single-agent RL proofs break. The MARL algorithms in this lesson all address this:

• MADDPG: global critic sees all actions, so its value estimate is stationary.
• QMIX: value decomposition moves learning to a joint-Q space where optimality is well-defined.
• MAPPO: the centralized value function dampens variance from others' policy changes.

In LLM-agent systems, non-stationarity manifests as "my agent worked last month, now that other agent upstream changed, mine misbehaves." Training MARL with CTDE is the principled fix; prompt-level fixes are faster but less durable.

What this lesson does NOT cover

Training actual networks is a Phase 09 topic. This lesson builds scripted-policy versions that demonstrate the CTDE, value-decomposition, and centralized-value patterns without gradient updates. The goal is to internalize the patterns before you pick up a full MARL library (PyMARL, MARLlib, RLlib multi-agent).

Build It

code/main.py implements three pattern demonstrations, all on a tiny 2-agent cooperative grid-world:

• Environment: 2 agents on a 4x4 grid, one reward pellet. Reward = 1 if any agent reaches pellet; task finishes.
• IndependentAgents — each agent treats others as environment. Baseline.
• MADDPGStyle — centralized critic computes a joint value; actor policies update from it. Scripted policy improvement.
• QMIXStyle — value decomposition with a monotone mixer.
• MAPPOStyle — centralized value function; policies update against the shared baseline.

All four run the same episodes and report average steps-to-goal. The CTDE variants converge to shorter paths than the independent baseline.

Run:

Copypython3 code/main.py

Expected output: independent agents take ~6 steps on average; CTDE variants converge toward ~3.5 steps (optimal for the 4x4 grid is 3). The pattern difference shows up despite scripted policies.

Use It

outputs/skill-marl-picker.md is a skill that picks a MARL algorithm for a given multi-agent task: cooperative vs competitive, homogeneous vs heterogeneous, action-space type, scale, reward signal.

Ship It

MARL in production is rare. When you do use it:

• Start with MAPPO. The 2022 paper established this as the baseline; reproducing it first saves weeks of chasing fancier methods.
• Log every agent's observation and action stream. Debugging MARL without per-agent traces is hopeless.
• Separate training code from execution code. CTDE is a discipline; let the execution path really only see o_i.
• Reward shaping warning. MARL is exquisitely sensitive to reward design. One coordination bug in the shaping and agents learn to exploit it. Run adversarial tests.
• For LLM agents, consider prompt-level policies first. Only invest in MARL training when interaction data + reward signal + infrastructure are all present.

Exercises

1. Run code/main.py. Measure the steps-to-goal gap between independent and MAPPO-style agents. Does the gap grow or shrink on a 6x6 grid?
2. Implement a competitive variant: two agents, one pellet, only the first to reach gets reward. Which pattern handles competition cleanly? MADDPG historically.
3. Read MADDPG (arXiv:1706.02275) Section 3. Implement the exact critic update rule symbolically in pseudocode in your own words.
4. Read MAPPO (arXiv:2103.01955). Why do the authors argue centralized value + PPO beats off-policy MARL on their benchmarks? List the three strongest claims.
5. Apply CTDE as a design pattern to a hypothetical LLM-agent system (e.g., research agent + summarizer + coder). What is the joint information available at design time that is not available at runtime?

Key Terms

Further Reading

• Lowe et al. — Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments — MADDPG; NeurIPS 2017
• Rashid et al. — QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning — QMIX; ICML 2018
• Yu et al. — The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games — MAPPO; NeurIPS 2022
• BAIR blog post on MAPPO — readable framing of the MAPPO result
• SMAC repository — StarCraft Multi-Agent Challenge

This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.

Browse the complete course catalog or open this lesson on GitHub.