What Substrate does, and what it leaves to you

You run one pod for each agent session. Most sessions wait for a person or for a model, and each waiting pod still holds its CPU and its memory. If you delete the pod, the session loses its memory. If you keep the pod, you pay for a process that does nothing.

When you finish this section, you can say what Agent Substrate does about that problem and name the jobs it leaves to you. You can also find the section that proves each of our eight positions.

The mechanism: actors, workers, and snapshots

Substrate separates the session from the pod that runs it.

Actor: one stateful workload with a name, such as my-counter-1. An actor has memory, files, and a root filesystem. It is a row in Postgres, and it is not a Kubernetes object.

Worker: one pod that can run actors inside a sandbox. A WorkerPool is the Kubernetes object that creates worker pods, and the pool has a fixed number of them.

Snapshot: the saved state of one actor. Substrate writes it when the actor is suspended and reads it when the actor is resumed.

Atespace: the named group that an actor, its template, and its policies belong to. An actor is always addressed by its atespace and its name.

A suspended actor uses no worker. It is a snapshot and a row. When a request arrives for it, Substrate chooses a worker with room, restores the snapshot there, and sends the request on. When something suspends the actor again, Substrate writes a new snapshot and frees the worker. Figure 1.1 shows the three worker pods of our demo pool with one actor that runs.

So a pool of three workers can hold hundreds of sessions, as long as few of them run at one time. That is the whole offer. The cost is the time to resume, and section 3.3 measures it against the memory that holds data in an actor.

What Substrate standardizes

Substrate gives you five things with fixed names.

• One API. ate-api-server serves the Control gRPC service for atespaces, templates, actors, snapshots, workers, and policies. The kubectl ate plugin is its client. A second service, WorkerService, is for atelet only.
• One lifecycle. An actor has seven states, and five operations move it between them: create, resume, suspend, revert, and delete. Section 3.2 draws them.
• One way to reach an actor. A client sends a request to atenet-router with the header ate-target-actor and a client certificate. The router finds the worker.
• Two sandbox classes. An actor runs under gVisor or in a microVM, and the WorkerPool decides which one. We ran both on one machine, and section 4.2 compares them.
• One egress rule. An actor reaches no host until an egress policy names that host. Section 6.2 shows the denial.

The project states that it keeps this API compatible from v1.0.0. That promise covers the API. It does not cover the jobs below.

What it leaves to you

The suspend call

Substrate does not suspend an idle actor. It has no idle timer. An actor that got one request stays in ACTOR_STATE_RUNNING and holds its worker until a caller sends SuspendActor.

We measured this on an x86 server guest with 24 vCPU and 47 GB, in one run. We sent five requests to my-counter-1 and then sent nothing for 900 s. We read the state every 15 s, and all 60 samples were ACTOR_STATE_RUNNING. Raw runs

The quickstart page of the project says that an idle actor is suspended for you. The API guide of the same project says that it is not. Our run agrees with the API guide. If you plan capacity from the quickstart sentence, your pool fills with idle actors.

The program that drives the actor has this job. kagent takes it: in our kagent run, 98 of 98 turns ended with a suspend. Section 7.1 shows that run. If you write your own harness, the suspend call is part of your harness.

The pool size

Substrate has no autoscaler. A WorkerPool has a spec.replicas field and a scale subresource, so a Kubernetes autoscaler can change it. Nothing in Substrate does. When every worker is full, a request for one more actor waits and then fails.

We measured that failure on the same guest, in eight requests to four actors of one run. Each request waited 5.06 s at the median (n=8) and then returned 503 with the text no worker has room for the actor. Section 4.4 explains the wait, and section 4.5 sizes a pool. Raw runs

GPU work

Substrate v1.0.0 does not give an actor a GPU. The API guide marks devices as not supported for now, so this edition has no GPU run. Keep model inference on a serving system, and let the actor call it through an egress policy.

Recovery

Substrate marks an actor as crashed, and then it waits for you. A crashed actor returns 503 on every request until a caller sends RevertActor. In one case it is not marked at all: a process that exits leaves the actor in ACTOR_STATE_RUNNING. Section 3.5 shows both cases.

We measured the exit case on the x86 server guest (24 vCPU, 47 GB, nested KVM), with two actors. Both read ACTOR_STATE_RUNNING for 20 minutes after the exit, until we called a suspend. One of them got 20 requests in that time, and all 20 returned 503. Each dead actor kept its CPU on the worker for all that time. Raw runs

Substrate also does not move an actor off a node that stops.

We measured this on a second x86 server guest (16 vCPU, 23 GB), on a cluster with two worker nodes, one run for each case. A suspended actor had its snapshot in the object store. It stayed in ACTOR_STATE_RESUMING on the stopped node for 5 min 33 s from its first request. It was crashed 5 min 44 s after the stop command. A running actor read ACTOR_STATE_RUNNING for the 10 minutes of our probe. The other node had free workers the whole time, and section 4.3 has the four cases. Raw runs

We measured 13 more cases on the same guest (16 vCPU, 23 GB), once each, to find what an operator can do. No actor left the stopped node by itself. One revert freed an actor in 7.1 s, and only for an actor that was suspended with --mode sync and had got no request. In the other cases a revert during the outage left the actor in ACTOR_STATE_REVERTING until the node was back. A new actor from a tag that we had made before the stop served a request 1.2 s after the create command. So you own the recovery from a node loss, and section 8.2 has the runbook. Raw runs

Substrate also has no opinion about the agent. It does not choose a model, run a tool, or keep a conversation. Those belong to the harness.

Our eight positions

Each position comes from a run on our own clusters. The table is the map of this manual.

We measured every number in this table on two x86 server guests, one run each unless the section says more. The section in the last column names the machine and the count. The guests are different machines, so do not set a number from one row against a number from another row. Raw runs

Read the positions as a list of things to check on your own cluster. Your seconds will differ. The states, the error texts, and the order of events must not.

What the project site says, and what we measured

The project site, ate.dev, reached version 1.0 with the release. We read it on 2026-11-12. Its documentation pages are built from the same files as the release tag, and we found no difference in content. Its first page makes short statements, some with numbers and none with conditions. The table sets each statement beside our result.

We measured every result in this table on x86 server guests, one run each unless the section says more. The section in the last column names the machine and the count. Raw runs

Three statements match our runs as written. Four hold under a condition that the site does not state. Two did not match what we saw. One we did not reproduce on one node, and four are planned for the next edition. The two that did not match are the ones to plan for. You must send the suspend, and you must act when a node stops.

When this is the wrong tool

Substrate fits sessions that keep state and are idle most of the time. It does not fit an agent that runs all day, because that actor never frees its worker. It does not fit a workload that needs a GPU in the sandbox. Section 7.4 compares the choices, and section 8.5 lists the limits at v1.0.0.