How this manual was made

The version

The manual pins Agent Substrate v1.0.0, released on 2026-11-05 at commit b34fbee742906494b5eb4f1f36987176bf8770a4. The integration sections pin kagent v1.0.0 and agentgateway v1.7.0. Every cluster ran Kubernetes 1.37.0 on kind v0.33.0. We recorded the runs and verified the facts on 2026-11-12.

Agent Substrate changed its installer, its API names, and its states in the weeks before 1.0.0. A statement in this manual is about v1.0.0 only. Section 8.4 lists what differs in older text.

Four kinds of evidence

Each fact carries one of four labels, and the sources page ranks them.

We measured: the number comes from one of our recorded runs. The paragraph names the machine and the number of samples. You can rerun the script and compare.

The code: the default or the mechanism comes from the source at the release tag. We read it and did not run it, unless the same paragraph says that we did.

The documentation: the statement comes from the project documentation at the tag. We use it for paths that this edition does not run yet, such as an install on GKE, and the text says so.

An issue: a user or a maintainer reported the behaviour in the public tracker. We name the issue and do not present it as our result.

Our runs rank first. Where a run and the documentation disagree, the manual reports the run and names the disagreement. The clearest case is the idle actor. The quickstart says that Substrate suspends it, and our actor read RUNNING in 60 of 60 samples over 900 s.

The four test beds

The four are different machines. Two of them have 16 vCPU, so every measured paragraph gives the memory size too. A number from one machine is never a controlled comparison with a number from another.

Three sections show numbers from two machines side by side. Section 2.1 has the two installs. Section 8.2 and section 9.4 have the two Postgres recoveries. Each one names both machines and says that the difference is indicative only.

The lab method

• One guest for each track. Each line of work had its own virtual machine, so that a destructive test in one could not change a number in another. Nothing was installed on the server itself.
• Disks in memory. Each guest kept its disk in the memory of the server. Disk writes were faster than on a cloud volume, so our suspend and upload times are lower limits for a real disk.
• One node, with one exception. Every cluster but one had one kind node. The exception had one control plane node and two worker nodes. It gave the placement and node loss results of section 4.3 and the runbook of section 8.2.
• Two experiments shared a guest. Experiments P and Q ran at the same time on one guest, each on its own cluster. CPU was shared, so read a difference of a few milliseconds in those runs as noise.
• An object store on the same machine. Snapshots went to rustfs on the same node and the same disk. In experiment T the bucket ran outside the cluster, on the same guest, with an emulated delay. In experiment X it ran on a second guest of the same server. No bucket was in a cloud, and the path between the guests is not a real network link.
• Two install paths. We installed once from source, as the quickstart documents, and once from the release images, on two different guests. The two paths gave us the buildx failure and the install times of section 2.1.
• A mock model, and then a small real one. In experiment K a small server in the cluster answered each model call with a fixed reply. Those turn times in section 7.1 hold almost no model time. In experiment S the model was qwen2.5, an open model of 1 to 2 GB. It ran on 12 CPUs with no GPU, at 7 to 14 tokens per second. Both graphics cards of the server were in use by another team. A model on a GPU or a hosted model gives other model times.
• Client-side times. Unless a section says otherwise, a time is what the client saw, through kubectl ate proxy or from a pod in the cluster. A warm call through the proxy took 0.058 to 0.068 s on the first guest, and that cost is inside every proxy number.
• Small samples. Most points have 5 or 10 samples, and some events were recorded once. Each paragraph gives its count. Treat a result from one run as an observation.

The experiments

The letters H and O are not in the table. H was a single check that a plain scratch image works as an actor, and we did not use the letter O. Experiment L closed three gaps of the earlier runs, and the sections of Part 4 report it.

Experiment V ran on the first guest, with a pool of two workers that hold 16 one-CPU actors. It took twelve sentences of the draft that said we did not test something, and it tested each one.

The first run of item 3 used wrong port numbers for 1 to 8 resumes, and we discarded it. The table of section 4.4 is the second run. The bursts of 32 are from the first run. The first try of item 9 failed because the actor asked for the host name before the policy existed, and section 5.3 reports it.

Experiments P and Q ran after V, on the second guest (16 vCPU, 23 GB). Each node-stop case of P ran once. Q took eight statements that the draft had from the code only, or that it marked as not tested. It ran each one on both dataplanes where the dataplane matters.

The first test of item n stopped each run when the CPU time inside the sandbox reached 10 s. That CPU time is not valid under a cap. We kept the file as the record of the mistake, and we ran the test again with a fixed amount of work. The clean-up of experiment P deleted the golden tags of two templates, and section 3.4 reports what that broke.

Experiment S ran after P and Q, on the guest with 31 GB. It took the five things that section 7.5 listed as not run, and it replaced the mock model of experiment K with a real one.

Experiments T and U ran after S, at the same time on two guests. T ran on the cluster with two worker nodes on the second guest (16 vCPU, 23 GB). U ran on the first guest, on new kind clusters from the release images. They took six things that this page listed as not tested. Each case, each outage, and each upgrade step ran once.

The first attempt at two cases of T sent its requests to the wrong port, and we ran both again. One pass of the bucket outage has rows with a client error, and the summary leaves those rows out. The capture kit names each file that we did not use.

Experiment X ran after a restart of the guest with 23 GB, at the same time as W. The bucket and the database ran in containers on the guest with 31 GB. The two guests reach each other through the user-mode network of the hypervisor, at about 100 MiB each second. The first attempt at the bucket cycles sent a revert after a 504, and we ran the cycles again without it.

Experiment W ran on the guest with 31 GB, after a restart of that guest. It took four things that this page and section 7.5 listed as not run. The models were the same two small models on CPU.

In W, the first two tries at the worker loss stopped before the delete, because the task was not on the pool of the test. We ran both again. The first try at the example stopped at a failed resume that we did not explain. The capture kit names each file that we left out.

Experiment Y ran after W and X, on the first guest (24 vCPU, 47 GB), on new kind clusters. It took five things that this page listed as not tested. Each outage, upgrade, and rollback ran once.

In Y, the resumes of the Kubernetes API outage are not valid, because running actors from the earlier runs had filled the pool. The bursts of 200 to 500 resumes in Y gave no data, because the kube-apiserver of that cluster stopped with 443 of 520 workers up.

Experiment Z ran last, on the first guest (24 vCPU, 47 GB), on a new kind cluster with a limit of 700 pods. It took the bursts that Y could not run, with the pool grown 50 workers at a time to 300.

In Z, the first count of workers used the wrong filter and read 0, and we grew the pool again. The upload of 300 snapshots filled the guest disk fast, so we deleted 150 actors and ran 200 and 300 at one time twice each. Before the router test we deleted one actor that we wrongly thought had stayed in ACTOR_STATE_RESUMING. Its two later requests got 404, and the capture kit marks them.

The capture kit

The capture kit holds the recordings and one script. Agent Substrate needs a Kubernetes cluster, so the recordings cannot run in the repository CI. We made them on our machines and stored the output in the lab folder.

cd manuals/agent-substrate-101
python3 capture/run.py
python3 capture/run.py --check

The first command reads the recordings and writes the summary files, which are the only input of the chart figures. The second command builds the summaries again in a temporary folder and compares them with the stored ones. It needs no network and no keys. It prints one skipped line for the recordings, because it does not start a cluster.

A large log is stored as an excerpt, in which each line is a complete line of the original. A list in the lab folder names each file that was left out or cut, with the reason. The credential values in the egress and network recordings are test values that we made up.

Record it again on your cluster

1. Create a kind cluster with the cluster script from the Agent Substrate checkout at v1.0.0.
2. Install with ate-setup, as section 2.1 shows.
3. Run the script of one experiment from the scripts folder. Each script writes to a folder with the name of its experiment.
4. Copy the output over the same folder in the lab folder, then run python3 capture/run.py from manuals/agent-substrate-101/.

Your seconds will differ from ours. The states, the error texts, and the order of events must not. If one of them differs on v1.0.0, we want to know, because it means that a claim of this manual depends on our machines.

What we measured, and what we read

Coming in the next edition, in section 8.5, holds the full list of planned runs.