Index of figures

Every figure of this manual, in order, by the one claim that its caption makes.

Each row links to its figure, and the claim is the claim: field of the figure block. A measured number in a figure comes from our recorded runs, and the chart figures read the summary files directly. Two charts are exceptions: figure 1.2b and figure 6.6 read their recordings directly. The colours follow the palette of the front matter:

blue is the Substrate control plane: ate-api-server, ate-controller, and ate-idp-server
violet is workers and sandboxes: atelet, worker pods, gVisor, and microVM
green is the actor, the workload that you run
plum is the agent harness and its model calls: kagent and the model
olive is the client and hosts outside the cluster
indigo is the dataplane: atenet-router, atenet-egress, Envoy, and agentgateway
amber is a lifecycle state or a step between two states
teal is durable state: Postgres, the object store, and snapshots
rose is a failure: a crashed actor, a 500, a 503, or lost calls
grey is outside Substrate: Kubernetes, the node, and the operator

Fig.	The claim it makes
0.1	Each arrow style marks one kind of exchange, and every label is a real name, message, or number from our recorded runs.
1.1	Substrate resumes an actor onto a worker when a request arrives, and the call that suspends it again must come from you.
1.2	The first request to a suspended actor makes the router call ResumeActor, and ate-api-server restores the actor before the request reaches it.
1.3	Only the first request paid for a resume, and the request after 900 idle seconds was as fast as the others because the actor never stopped.
2.1	From the release images deploy ate-system took 162 s on one guest, and from source on a larger guest it failed, then took 266 s.
2.2	kubectl ate reads atespaces, templates, actors, and workers from Postgres through ate-api-server, and kubectl reads only three custom resources from etcd.
2.3	With kube-apiserver down for 86 s the proxy served no request, and a pod that called the router served every one.
2.4	The install put the control plane, the dataplane, the node agent, Postgres, and the object store in one namespace on one node.
2.5	In ate-system only ate-api-server and ate-idp-server run two pods, and every other workload, the router and Postgres among them, runs one.
3.1	Substrate builds one golden snapshot for each template, and an actor created after that starts suspended from it.
3.2	At 1,000 MiB the first call of a new actor took 3.5 to 12.7 seconds, and a later resume took 2.2 seconds.
3.3	An actor leaves ACTOR_STATE_CRASHED only through RevertActor or DeleteActor, and no request can resume it.
3.4	At 1,000 MiB a synchronous suspend took 6.4 seconds, and the default asynchronous suspend took 2.0 seconds.
3.5	A resume from the node took 2.2 seconds at 1,000 MiB, and a resume from the object store took 4.7 seconds.
3.6	Only SNAPSHOT_FIDELITY_MEMORY keeps the processes, and the two lower levels start new processes on every resume.
3.7	An actor that held 500 MiB of zero bytes had a 4.6 MiB snapshot, and the same actor with random bytes had 504.1 MiB.
3.8	25 suspended actors that held 8,750 MiB of memory used 8,801 MiB on the node and 10,824 MiB in the object store.
3.9	For the first five minutes after an asynchronous suspend, the only copy of the new snapshot is on one node.
3.10	The text of the error tells you the state of the actor, and the state tells you the one call that helps.
3.11	An atespace holds templates, actors, and tags, a namespace holds worker pools, and only a label selector joins them.
4.1	A WorkerPool becomes a Deployment of worker pods, and each pod becomes a Worker record that reports the limits of the pod as capacity.
4.2	An actor used 0.25, 0.50, and 1.0 host cores at limits of 250, 500, and 1000 millicores, and it needed four threads to use 2.0 cores at 2000.
4.3	A gVisor actor shares the node kernel through a user-space kernel, and a microVM actor runs on its own guest kernel under KVM.
4.4	At 0 MiB a microVM resume took 0.33 seconds against 0.22 for gVisor, and at 500 MiB it took 0.80 against 1.07.
4.5	A worker must pass six checks before it is a candidate, and the scheduler then takes the less loaded of two random candidates.
4.6	A suspended actor with a complete snapshot stayed in RESUMING for 5 min 33 s from its first request, on a stopped node, then became CRASHED.
4.7	A parked request is served when a worker becomes free inside the 5 second budget, and it gets a 503 when none does.
4.8	Sixteen resumes at the same instant took 0.37 seconds each at the median, against 0.23 seconds for one resume alone.
4.9	Two workers with a 2 CPU limit ran four one-CPU actors, and one worker with no limits ran sixteen on a node with 16 vCPUs.
4.10	One node finished 32 to 39 resumes each second from 50 to 300 at one time, so the wait grew with the size of the burst.
4.11	Inside a burst about 100 restores ran at once, and each one took about ten times as long as a restore alone.
5.1	The API and the router check a client separately: the shared ate-client token owns every actor, and the router serves any identity on its one list.
5.2	A role on the global scope holds in every atespace, and a role on one atespace holds only there.
5.3	The egress gateway mints the token and replaces a header with it, so the service receives an identity that the actor did not send.
6.1	The router calls ResumeActor for every request, also when the actor already runs, and only then opens a tunnel to the worker.
6.2	Both dataplanes refuse a caller with no certificate or the wrong SPIFFE ID in the TLS layer, so the caller gets no HTTP status.
6.3	A connection from an actor passes two checks, the DNS relay for the name and atenet-egress for the request, and each one reads the same policy.
6.4	A short Service name passes the DNS relay through the search list, and the gateway then refuses it because the Host header is not in the policy.
6.5	In our polled runs a new rule reached DNS within 3 seconds, and in one unpolled run on Envoy the name stayed unresolved for 16 seconds.
6.6	With an https rule the actor holds a session with atenet-egress, and with a tlsPassthrough rule it holds a session with the real host.
6.7	The secret travels from the Kubernetes Secret to atenet-egress and on to the upstream, and it enters the actor only if the upstream sends it back.
6.8	A changed Secret reached the upstream in 1.0 to 1.6 s on Envoy and in 100.6 to 280.5 s on agentgateway.
6.9	The Envoy dataplane spreads its work over five containers and 91 MB, and agentgateway uses two containers and 38 MB with the CA key inside one.
6.10	Both dataplanes refused new requests and requests on a reused connection within 9 seconds, and neither closed an open response or pass-through connection.
7.1	Every turn resumes the Session actor through the router, and kagent calls SuspendActor as soon as the task completes.
7.2	A turn that follows the last one with no pause costs the same as a turn 4 s later, because every turn resumes the actor.
7.3	The model was about nine tenths of every clean turn, and one turn of 50 did not end.
7.4	Sixteen of 24 parallel turns ended within 5.45 s, and the other eight waited for a free CPU and ended up to 2.4 s later.
7.5	In places A and B agentgateway enforces the Substrate policy, and only in place C does it read model and tool traffic.
7.6	The installer creates cloud resources in your project and installs the same control plane in your cluster that ate-setup installs on kind.
7.7	Substrate fits only when all five answers are yes, and each no names a simpler choice that fits better.
7.8	One task became one actor and one template with a files-only snapshot, and no object opened egress or chose a pool.
7.9	A resume of a prepared task took about 0.24 s, and a resume whose clone failed took more than 10 s because the runner tried five times.
7.10	The resume after a worker loss took 0.3 s in both runs, and it kept the saved file only when the snapshot had reached the object store.
7.11	An open command ended with an error within 0.2 s of the delete, and nothing started it again.
8.1	Postgres holds the record of each actor, the bucket holds its durable snapshot, and the node disk holds a second copy for 30 minutes.
8.2	With 20 ms of added round trip, a resume of 1,000 MiB from the bucket took 10.2 s and not 4.8 s.
8.3	With the bucket on another guest, a request to an actor of 1,000 MiB got a 504 after 10 s and its first 200 after 12 s.
8.4	Requests failed without Postgres and without ate-api-server, and every request from inside the cluster succeeded without the Kubernetes API server.
8.5	Without Postgres a failed request took the full 9 s of the client timeout, and without ate-api-server it took 5 s at most.
8.6	A running actor answered again 3 s after Postgres was ready on Envoy and 21 s after it on agentgateway, in one run each.
8.7	The Kubernetes API returned after 8 s and all pods after 147 s, and the actor needed a revert and 78 more seconds.
8.8	A tag made before the stop gave a working actor in 1.2 s, and every other path waits for the workers or for the node.
8.9	A drain crashed each running actor and did not disturb any suspended one.
8.10	The delay and the body of a failed request separate six causes that share the status codes 500 and 503.
8.11	Only the worker step ends running actors, and an actor has 30 minutes there only if its process survives SIGTERM.
8.12	In both runs suspends and resumes failed in the atelet step, and a running actor that exits on SIGTERM crashed when the worker image changed.
8.13	Substrate performs the lifecycle operations that you request, and your harness or your platform team must decide when to request them.
9.1	The suspender forwards each request, keeps the time of the last one for each actor, and calls SuspendActor when an actor is quiet.
9.2	All 140 requests returned 200, and the worker held one actor for 38 of 58 samples in place of the three that had traffic.
9.3	The conversation returned under both fidelities, and the note, the file under /tmp, and the process itself returned only under memory fidelity.
9.4	A turn that resumed the actor took 0.30 seconds under memory fidelity and 4.5 to 5.1 seconds under volumes fidelity.
9.5	One download from the allowed mirror passed, three attempts to reach other hosts failed, and each revert returned an empty work directory.
9.6	Each of the six failures gave a good answer in under one second after its fix, and only one of them needed two commands.
R.1	A crashed actor returns to service only through RevertActor, and every path into RUNNING goes through ResumeActor.

Agent Substrate 1.0.0 · b34fbee · 2026-11-05 · Edition 2026.11