Swarm Optimization and MARL for Agents

PSO and ACO for prompts and routing, queue swarms, and MADDPG, QMIX, MAPPO and self-play set side by side.

Concepts and terms
Tasks and commands
Annotated code
Diagrams
Comparisons
Defaults and numbers
Troubleshooting
Questions and tips

PSO vs ACO#

Factor	PSO	ACO
Problem	Continuous search space	Routing or path selection
Memory	`p_best` and `g_best`	Pheromone per task-type × agent
Agent use	LMPSO, Model Swarms	AMRO-S routing
Reported gain	13.3% over 12 baselines	4.7x speedup
Population	10-30 particles	3 agents, 4 task types in the demo
Update rule	Inertia + cognitive + social	Decay, then gated deposit
Demo function	`run_lmpso()`	`run_amro_s()`

From Lesson 16.19, 16.19 code/main.py

MARL algorithms#

Factor	MADDPG	QMIX	MAPPO
Year, venue	NeurIPS 2017	ICML 2018	NeurIPS 2022
Setting	Coop, competitive, mixed	Cooperative only	Cooperative
Training signal	Per-agent critic sees all actions	Monotone mixer of per-agent Qs	Centralized value function
Policy type	Off-policy	Off-policy	On-policy PPO
Testbed	Particle World	SMAC	Football, Hanabi, MPE
Limit	Past ~10 agents critics grow	Reward must be monotone	Default baseline to beat
Demo	`run_maddpg_style`	`run_qmix_style`	`run_mappo_style`

From Lesson 16.20, 16.20 code/main.py

MARL regimes#

Factor	IQL / IPPO	CTDE	Self-play	League play
Idea	Others are environment	Central critic, local policy	Copy of past self	Population of past policies
Fits	Loosely-coupled tasks	Cooperative navigation	Zero-sum games	Strategy cycles
Examples	Independent Q	MADDPG, COMA, MAPPO, QMIX	AlphaZero	AlphaStar
Note	Theoretical convergence: none	Most common modern paradigm	Can stick in a dominance cycle	Adds exploiters and main exploiters

From Lesson 09.10

Queue swarm vs fixed assignment#

Factor	Sequential	Fixed assignment	Swarm
Who picks work	One worker	Pre-assigned worker id	Workers pull from a shared queue
Function	`run_sequential`	`run_fixed_assignment`	`run_swarm`
Load balance	None	Fast workers sit idle	Automatic
Fits	Baseline only	Durations known ahead	Many independent tasks
Weak point	Wall-clock time	Worker 0 gets all slow tasks	Ordering and debugging

From Lesson 16.09, 16.09 code/main.py

PSO update#

w, c1, c2 = 0.6, 1.2, 1.2inertia, cognitive, social

r1, r2 = rng.random(), rng.random()

cognitive = c1 * r1 * (p.p_best[d] - p.x[d])

social = c2 * r2 * (g_best_x[d] - p.x[d])

p.v[d] = w * p.v[d] + cognitive + social

p.x[d] += p.v[d]

p.x[d] = max(bounds[d][0], min(bounds[d][1], p.x[d]))clamp to bounds

f = fitness(p.x)only an evaluator, no gradient

if f > p.p_best_fit:then check g_best_fit

From 16.19 code/main.py

ACO router#

router = PheromoneRouter(task_types, agents)every trail starts at 1.0

aco_agent = router.choose(tt, rng)sample by pheromone weight

aq = simulate_task(aco_agent, tt, rng)

router.deposit(tt, aco_agent, aq)

self.pheromones[task_type][a] *= (1.0 - self.decay)all agents decay

if quality >= self.quality_threshold:quality gate

self.pheromones[task_type][agent] += self.reinforce * quality

From 16.19 code/main.py

Use each method when#

PSO: continuous parameters, cheap automatic fitness, population 10-30.
ACO: routing that repeats over time and needs interpretable evidence.
Neither: fitness needs human review, or latency must be strict.
Genetic algorithm: discrete combinatorial space that PSO does not cover.
Swarm: many independent tasks with variable duration.
Supervisor: the task needs one coherent global plan.
MAPPO: start here for cooperative MARL.
Self-play: zero-sum games. League play when strategies cycle.

From Lesson 16.19, Lesson 16.20, Lesson 16.09, Lesson 09.10

Run the patterns#

Task	Code
Run any lesson demo	`python3 code/main.py`
PSO over 2 prompt parameters	`history = run_lmpso()`
ACO routing vs random	`run_amro_s(n_tasks=200, seed=0)`
Print the trail	`print_pheromone_table(router)`
Steps-to-goal per MARL style	`bench(label, runner)` over 500 trials
Pull from a shared queue	`run_swarm(tasks, n_workers)`
Independent Q on the grid	`independent_q(episodes=1500)`
Joint Q over 16 actions	`joint_q_learning(episodes=1500)`

From 16.19 code/main.py, 16.20 code/main.py, 16.09 code/main.py, 09.10 code/main.py, Lesson 16.19

CTDE pattern#

From Lesson 16.20, Lesson 09.10

Defaults in the lesson code#

Setting	Default	Note
`n_particles`	20	`run_lmpso` population
`iterations`	30	PSO iterations
`decay`	0.05	Pheromone decay per deposit
`reinforce`	0.2	Deposit is reinforce times quality
`quality_threshold`	0.6	No deposit below it
`n_tasks`	200	Routed tasks in `run_amro_s`
`epsilon`	0.15	Exploration in `independent_q`
`gamma`	0.95	Discount, with `alpha` 0.1

From 16.19 code/main.py, 09.10 code/main.py

Failure modes#

Symptom	Cause	Fix
Router locks on a bad agent	Fast-but-wrong agent got pheromone	Quality-gate the deposits
Best fitness diverges	Catastrophic drift after data shift	Reset or double the decay rate
Swarm forgets good solutions	Decay too fast	Slower decay, monitor best fitness
Long tasks never run	Queue starvation	Priority queue with aging
Task output appears twice	Worker crashed mid-run	Idempotent workers, durable queue
Agents chase each other	Policy drift, non-stationarity	Centralized critic, slow learning rates
Old replay hurts learning	Transitions from obsolete opponents	Relabel or weight by recency
Auction agents bid zero	Reward hacking via coordination	Careful reward design, constraints

From Lesson 16.19, Lesson 16.09, Lesson 09.10, Lesson 16.20

Questions#

Why gradient-free for prompts?: A prompt is a discrete string. PSO and ACO need only an evaluator.
What does a 20-particle, 50-iteration PSO cost?: About $20 at ~$0.02 per call.
Why does QMIX decentralize?: Monotone mixing lets each agent take its own argmax.
Why not train MARL for LLM agents?: Most systems prompt policies. Train only with data, reward and infrastructure.
Why is the joint-Q demo slow to scale?: Its action space is |A|² = 16 for two agents.

From Lesson 16.19, Lesson 16.20, Lesson 09.10

Ship checklist#

Start with 10-20 particles and 20-50 iterations.
Log pheromones or g_best per iteration.
Emit a cost-per-iteration metric.
Give every swarm task a trace ID. Apply back-pressure when the queue grows.
Let the execution path see only o_i.
Log every agent's observation and action stream.

From Lesson 16.19, Lesson 16.20, Lesson 16.09