You run one pod for each agent session. Most sessions wait for a person or for a model, and each waiting pod still holds its CPU and its memory. If you delete the pod, the session loses its memory. If you keep the pod, you pay for a process that does nothing.

When you finish this section, you can say what Agent Substrate does about that problem and name the jobs it leaves to you. You can also find the section that proves each of our eight positions.

The mechanism: actors, workers, and snapshots

Substrate separates the session from the pod that runs it.

Actor: one stateful workload with a name, such as my-counter-1. An actor has memory, files, and a root filesystem. It is a row in Postgres, and it is not a Kubernetes object.

Worker: one pod that can run actors inside a sandbox. A WorkerPool is the Kubernetes object that creates worker pods, and the pool has a fixed number of them.

Snapshot: the saved state of one actor. Substrate writes it when the actor is suspended and reads it when the actor is resumed.

Atespace: the named group that an actor, its template, and its policies belong to. An actor is always addressed by its atespace and its name.

A suspended actor uses no worker. It is a snapshot and a row. When a request arrives for it, Substrate chooses a worker with room, restores the snapshot there, and sends the request on. When something suspends the actor again, Substrate writes a new snapshot and frees the worker. Figure 1.1 shows the three worker pods of our demo pool with one actor that runs.

So a pool of three workers can hold hundreds of sessions, as long as few of them run at one time. That is the whole offer. The cost is the time to resume, and section 3.3 measures it against the memory that holds data in an actor.

What Substrate standardizes

Substrate gives you five things with fixed names.

• One API. ate-api-server serves the Control gRPC service for atespaces, templates, actors, snapshots, workers, and policies. The kubectl ate plugin is its client. A second service, WorkerService, is for atelet only.
• One lifecycle. An actor has seven states, and five operations move it between them: create, resume, suspend, revert, and delete. Section 3.2 draws them.
• One way to reach an actor. A client sends a request to atenet-router with the header ate-target-actor and a client certificate. The router finds the worker.
• Two sandbox classes. An actor runs under gVisor or in a microVM, and the WorkerPool decides which one. We ran both on one machine, and section 4.2 compares them.
• One egress rule. An actor reaches no host until an egress policy names that host. Section 6.2 shows the denial.

The project states that it keeps this API compatible from v1.0.0. That promise covers the API. It does not cover the jobs below.

What it leaves to you

The suspend call

Substrate does not suspend an idle actor. It has no idle timer. An actor that got one request stays in ACTOR_STATE_RUNNING and holds its worker until a caller sends SuspendActor.

We measured this on an x86 server guest with 24 vCPU and 47 GB, in one run. We sent five requests to my-counter-1 and then sent nothing for 900 s. We read the state every 15 s, and all 60 samples were ACTOR_STATE_RUNNING. Raw runs

The quickstart page of the project says that an idle actor is suspended for you. The API guide of the same project says that it is not. Our run agrees with the API guide. If you plan capacity from the quickstart sentence, your pool fills with idle actors.

The program that drives the actor has this job. kagent takes it: in our kagent run, 98 of 98 turns ended with a suspend. Section 7.1 shows that run. If you write your own harness, the suspend call is part of your harness.

The pool size

Substrate has no autoscaler. A WorkerPool has a spec.replicas field and a scale subresource, so a Kubernetes autoscaler can change it. Nothing in Substrate does. When every worker is full, a request for one more actor waits and then fails.

We measured that failure on the same guest, in eight requests to four actors of one run. Each request waited 5.06 s at the median (n=8) and then returned 503 with the text no worker has room for the actor. Section 4.4 explains the wait, and section 4.5 sizes a pool. Raw runs

GPU work

Substrate v1.0.0 does not give an actor a GPU. The API guide marks devices as not supported for now. We did not test GPUs. Keep model inference on a serving system, and let the actor call it through an egress policy.

Recovery

Substrate marks an actor as crashed, and then it waits for you. A crashed actor returns 503 on every request until a caller sends RevertActor. In one case it is not marked at all: a process that exits leaves the actor in ACTOR_STATE_RUNNING. Section 3.5 shows both cases.

We measured the exit case on the x86 server guest (24 vCPU, 47 GB, nested KVM), with two actors. Both read ACTOR_STATE_RUNNING for 20 minutes after the exit, until we called a suspend. One of them got 20 requests in that time, and all 20 returned 503. Each dead actor kept its CPU on the worker for all that time. Raw runs

Substrate also does not move an actor off a node that stops.

We measured this on a second x86 server guest (16 vCPU, 23 GB), on a cluster with two worker nodes, one run for each case. A suspended actor had its snapshot in the object store. It stayed in ACTOR_STATE_RESUMING on the stopped node for 5 min 33 s from its first request. It was crashed 5 min 44 s after the stop command. A running actor read ACTOR_STATE_RUNNING for the 10 minutes of our probe. The other node had free workers the whole time, and section 4.3 has the four cases. Raw runs

We measured 13 more cases on the same guest (16 vCPU, 23 GB), once each, to find what an operator can do. No actor left the stopped node by itself. One revert freed an actor in 7.1 s, and only for an actor that was suspended with --mode sync and had got no request. In the other cases a revert during the outage left the actor in ACTOR_STATE_REVERTING until the node was back. A new actor from a tag that we had made before the stop served a request 1.2 s after the create command. So you own the recovery from a node loss, and section 8.2 has the runbook. Raw runs

Substrate also has no opinion about the agent. It does not choose a model, run a tool, or keep a conversation. Those belong to the harness.

Our eight positions

Each position comes from a run on our own clusters. The table is the map of this manual.

We measured every number in this table on two x86 server guests, one run each unless the section says more. The section in the last column names the machine and the count. The guests are different machines, so do not set a number from one row against a number from another row. Raw runs

Read the positions as a list of things to check on your own cluster. Your seconds will differ. The states, the error texts, and the order of events must not.

What the project site says, and what we measured

The project site, ate.dev, reached version 1.0 with the release. We read it on 2026-11-12. Its documentation pages are built from the same files as the release tag, and we found no difference in content. Its first page makes short statements, some with numbers and none with conditions. The table sets each statement beside our result.

We measured every result in this table on x86 server guests, one run each unless the section says more. The section in the last column names the machine and the count. Raw runs

Three statements match our runs as written. Four hold under a condition that the site does not state. Two did not match what we saw. One we did not reproduce on one node, and four we did not test or could not test at our size. The two that did not match are the ones to plan for. You must send the suspend, and you must act when a node stops.

When this is the wrong tool

Substrate fits sessions that keep state and are idle most of the time. It does not fit an agent that runs all day, because that actor never frees its worker. It does not fit a workload that needs a GPU in the sandbox. Section 7.4 compares the choices, and section 8.5 lists the limits at v1.0.0.

You sent curl -X POST http://localhost:8000/ and a count came back. Six programs handled that request, and one of them wrote to a database before the actor ran. When a request fails later, the error text names one of those programs, so you need to know the order.

When you finish this section, you can name each hop of one request and say which hops a running actor skips. You can also say which parts of the cluster must be up for a request to pass.

The recording

We created my-counter-1 in the atespace ate-demo-counter and started kubectl ate proxy. Then we sent five requests, one after the other. This is the start of the recording. The columns are the time, the event, the HTTP status, the seconds at the client, and the body.

ts	event	detail
2026-11-12T10:14:21.286Z	state	ACTOR_STATE_SUSPENDED	-
2026-11-12T10:14:21.837Z	call-1	200	0.358379	preserved memory count: 1 | preserved file counter: 1 | preserved rootfs counter: 1
2026-11-12T10:14:21.944Z	call-2	200	0.090770	preserved memory count: 2 | preserved file counter: 2 | preserved rootfs counter: 2
2026-11-12T10:14:22.030Z	call-3	200	0.066903	preserved memory count: 3 | preserved file counter: 3 | preserved rootfs counter: 3
2026-11-12T10:14:22.109Z	call-4	200	0.061842	preserved memory count: 4 | preserved file counter: 4 | preserved rootfs counter: 4
2026-11-12T10:14:22.196Z	call-5	200	0.068053	preserved memory count: 5 | preserved file counter: 5 | preserved rootfs counter: 5
2026-11-12T10:14:22.203Z	state	ACTOR_STATE_RUNNING	-

We measured these requests on an x86 server guest with 24 vCPU and 47 GB, through kubectl ate proxy, in one run. The first request took 0.358 s. The next four took 0.091 s, 0.067 s, 0.062 s, and 0.068 s. Raw runs

The actor was ACTOR_STATE_SUSPENDED before the first request and ACTOR_STATE_RUNNING after the fifth. No command resumed it. The first request did.

Hop by hop

Figure 1.2 draws the first request. The numbers in this list match the rows of the figure.

1. curl sends the request to 127.0.0.1:8000, where kubectl ate proxy listens.
2. The proxy opens a port-forward through the Kubernetes API server to atenet-router. Inside that port-forward it opens an mTLS CONNECT tunnel to router port 8444, with a client certificate.
3. The router reads the target actor and calls ResumeActor on ate-api-server. It does so for every request, because it keeps no record of where an actor runs. Requests that arrive at the same time for one actor share one call.
4. ate-api-server reads the actor row from Postgres. The state is ACTOR_STATE_SUSPENDED.
5. ate-api-server takes a lease on the actor, chooses a worker with room, and writes the assignment to Postgres in one transaction.
6. ate-api-server tells atelet on that node to restore the snapshot. The worker starts the sandbox from it, and the actor state becomes ACTOR_STATE_RUNNING.
7. ate-api-server returns the actor to the router, with the IP address of the worker pod and the identity of the worker.
8. The router opens a tunnel to port 8443 of the worker pod and checks that identity. The worker forwards the request to port 80 of the actor.
9. The reply goes back the same way.

A new actor has never run, so the snapshot in step 6 is not its own. It is the golden snapshot of its template, and section 3.1 explains it. After step 7 the actor object carries the assignment:

    "state": "ACTOR_STATE_RUNNING",
    "workerAssignment": {
      "nodeName": "kind-control-plane",
…
      "workerEpoch": "1",
      "workerNamespace": "ate-demo-counter",
      "workerPod": "counter-7cc7dcd596-ffxwt",
      "workerPodIps": [
        "10.244.0.23"
      ],
…
      "workerPool": "counter",
      "workerSpiffeId": "spiffe://cluster.local/ns/ate-demo-counter/sa/ateworker"

Where the 0.36 s goes

The first request paid for the restore. The four after it did not. The difference is about 0.29 s on this guest, for an actor that holds almost no memory.

The worker writes one log line when a restore starts and one when it ends. We did not save that log for the first request. We saved it for a later resume of the same actor:

{"time":"2026-11-12T10:30:17.846333794Z","message":"Actor restoring",…
{"time":"2026-11-12T10:30:18.098955499Z","message":"Actor restored",…

We measured that resume on the same guest, once. The two log lines are 0.253 s apart, and the request that caused the restore took 0.382 s at the client. So about two thirds of a first request is the restore itself. Raw runs

That share grows with the actor. A restore reads the memory that the actor held, and section 3.3 shows 2.16 s for an actor with 1000 MiB.

A request to a running actor

A request to a running actor still does steps 1 to 4. ate-api-server reads the row, finds ACTOR_STATE_RUNNING, and returns at once. Steps 5 and 6 do not happen. So a request costs one call to ate-api-server and one read from Postgres, also when nothing is resumed. Requests that arrive together for one actor share that call.

Most of the 0.06 s in our recording is the proxy, not Substrate.

We measured both paths on the second x86 guest, with 16 vCPU and 23 GB, in the 30 s before a test. A pod that sent its requests to the router answered in 0.027 s at the median, with 30 requests. The same actor through kubectl ate proxy answered in 0.067 s at the median, with 28 requests. Raw runs

Use the number from the pod when you plan latency. The proxy is a tool for your own machine, and section 2.3 shows why it must not carry real traffic.

What must be up

Three things follow from the path.

• Postgres and ate-api-server are on every request path. A running actor that needs no resume still fails when either one is down. On our second guest, requests to a running actor failed within seconds of each outage. Section 8.2 has the timeline.
• The Kubernetes API server is on no request path. Only row 2 of the figure passes it, and that row belongs to the proxy. In the same test, kube-apiserver was down for 86 s. In that time 103 of 103 requests from a pod were served: 86 to a running actor and 17 resumes.
• The object store is on the path only when a snapshot is not on the node. Section 3.4 says when that is.

The install runs one atenet-router pod and one Postgres pod. Every request to every actor passes both. Section 8.1 says what to do about Postgres.

The actor stays up

After the fifth request we sent nothing for 900 s. Figure 1.2b shows all six requests of the recording, and every state sample between the last two.

The sixth request took 0.080 s and returned count 6. A resume would have cost about 0.36 s. The actor was never suspended, and it held its worker for the whole 15 minutes. Section 1.1 names this as the first job that Substrate leaves to you.

You follow a quickstart on a clean machine. The third command runs for almost three minutes and then stops with an error from Docker about a flag you never typed. We hit that error on our first install, and you do not need to.

When you finish this section, you can install Agent Substrate v1.0.0 on a kind cluster from the release images. You can also say what the build from source needs, and fix the install errors we hit.

What you need

• Linux or macOS with Docker. On Apple silicon we used Docker Desktop with 10 CPU and 8 GB.
• Go 1.27. The checkout asks for toolchain go1.27.3, and Go downloads kind for you.
• git and kubectl.
• About 13 GB of disk. Our guest used 13 GB in all after this install, with the operating system.

Installer: ate-setup is the Go program in the checkout that installs and removes Agent Substrate. The release has no ate-setup binary, so you run it with go run from a checkout of the tag. The manifests also come from that checkout, so check out the tag that you install.

On Linux we raised two inotify limits before the install, because many pods on one node use many watches:

sudo sysctl -w fs.inotify.max_user_instances=8192 fs.inotify.max_user_watches=1048576

Create the kind cluster

Clone the tag, then run the cluster script from the repository root:

git clone --branch v1.0.0 https://github.com/agent-substrate/substrate.git
cd substrate
hack/create-kind-cluster.sh

The script does more than kind create cluster. Substrate needs three settings that a plain kind cluster does not have:

• It starts a local image registry named kind-registry on 127.0.0.1:5001.
• It turns on three Kubernetes preview features: ClusterTrustBundle, ClusterTrustBundleProjection, and PodCertificateRequest. Substrate gives every pod its certificate through them.
• It looks for /dev/kvm. With KVM, the node can run the microVM sandbox class. Without KVM, only gVisor works.

The script deletes an existing kind cluster with the same name before it creates the new one. The default name is kind. Set KIND_CLUSTER_NAME if you already have a cluster with that name.

Install the control plane from the release images

Tell ate-setup that the target is kind, then deploy:

export ATE_INSTALL_KIND=true
go run ./cmd/ate-setup deploy ate-system \
  --credential-provider='{"name":"k8s.io"}' \
  --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client \
  --image-repo ghcr.io/agent-substrate/substrate --image-tag v1.0.0

Each flag has a reason:

• ATE_INSTALL_KIND=true selects the kind profile: the kind overlays, the local registry, and the object store in the cluster. Export it in every shell that runs ate-setup.
• --credential-provider is required. The value {"name":"k8s.io"} deploys the bundled provider that reads Kubernetes Secrets. Section 6.5 uses it.
• --ingress-allowed-spiffe-ids lists the clients that the router lets through. With no value, no client reaches any actor. The identity in this command is the one that kubectl ate proxy presents.
• --image-repo and --image-tag install published images. Each of the two needs the other.

The installer prints every setting first, and then one line for each step. These lines tell you that the images came from the registry and that the install is complete:

>> Downloading kata-static 4.3.1 (amd64)...
>> Downloading cloud-hypervisor v53.0 (cloud-hypervisor-static)...
…
[step]: images: pre-built from ghcr.io/agent-substrate/substrate, tag v1.0.0
…
[step]: Waiting for ATE system components to be ready...
  (rollout status deployment/ate-api-server took 0.005s)
…
  (rollout status daemonset/atelet took 0.003s)
  (rollout status deployment/k8s-credential-provider took 0.003s)

The first two lines show a download that the release images do not remove. The installer fetches the guest files of the microVM class from GitHub on the first install from a directory, and it keeps them under bin/microvm-assets. A later install from the same directory skips the download when the files and their version stamp are present. The installer documents --sandbox-classes=gvisor to install gVisor only. We ran that flag for one install, on a cluster with three kind nodes on the second x86 server guest (16 vCPU, 23 GB). The system step took 114 s.

Deploy the counter demo

The demo gives you a WorkerPool, a template, and an atespace to test with:

go run ./cmd/ate-setup deploy demo counter

This step builds two images with ko and pushes them to kind-registry, also on the release path. The release images leave out the demos. The step ends when the golden snapshot of the counter template is ready.

What we measured

Figure 2.1 shows the steps of both x86 installs.

We measured the release-image install once on a second x86 server guest (16 vCPU, 23 GB). The kind cluster took 55 s, deploy ate-system took 162 s, and deploy demo counter took 94 s. The disk held 13 GB afterwards. Raw runs

We measured the build from source once on the x86 server guest (24 vCPU, 47 GB, nested KVM). The first deploy ate-system failed after 163 s. After we installed one package, the same command took 266 s, and the demo took 47 s. The disk held 18 GB afterwards. Raw runs

We measured the release-image install once on an Apple silicon laptop under Docker Desktop. The kind cluster took 119 s, and deploy ate-system took 702 s. The longest wait in its log was 178 s for the object store pod. The kind node held 2.7 GiB of memory after the install. Raw runs

The two x86 installs ran on different guests, so the difference of 104 s between 162 s and 266 s is indicative only. It is not a controlled comparison. What we can state without a number is what each path does. The release-image path skips the image builds and the Rust compile, and the next heading shows both in the source log. The laptop is a third machine on another network, so keep its seconds apart from the seconds of the guests.

What the build from source costs

Leave out --image-repo and --image-tag, and ate-setup builds every image from the checkout. The upstream quickstart shows that form. It lists Go, kubectl, and Docker as the things you need. On Ubuntu 24.04 with the stock Docker package, that list is one item short:

unknown flag: --platform
…
Error: while building the envoy-dataplane image: exit status 125

The Envoy dataplane image is built from a Dockerfile with docker buildx. The other images are built with ko. On Ubuntu the fix is one package:

sudo apt-get install -y docker-buildx

The second cost is a Rust compile. The Envoy image contains a Rust module for the egress policy. The build clones the Envoy repository and compiles that module:

#11 0.323     Updating git repository `https://github.com/envoyproxy/envoy`
…
#11 53.04    Compiling proc-macro2 v1.0.107
…
#11 91.89     Finished `release` profile [optimized] target(s) in 1m 31s
…
#11 DONE 92.2s

That one build step took 92 s of the 266 s. You pay it again when the build cache is empty. Build from source when you change Substrate itself. For every other install, use the release images.

Change the dataplane later

The same command switches a running cluster from Envoy to agentgateway. Add one flag and run it again:

go run ./cmd/ate-setup deploy ate-system \
  --credential-provider='{"name":"k8s.io"}' \
  --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client \
  --image-repo ghcr.io/agent-substrate/substrate --image-tag v1.0.0 \
  --atenet-dataplane=agentgateway

We measured this switch once on the second x86 server guest (16 vCPU, 23 GB). It took 32 s and needed no new cluster. Section 6.6 compares the two dataplanes. Raw runs

If the install fails

The last row happened to us on the laptop. After a restart of Docker Desktop, kubectl pointed at the cluster that Docker Desktop includes, and our first test ran there.

To remove everything, delete the cluster and the registry:

./hack/delete-kind-cluster.sh

Old guides for v0.x use install scripts that no longer exist. Section 8.4 lists what changed.

You type kubectl get actors on a cluster that runs Agent Substrate, and Kubernetes answers that it has no such resource. The actor exists. You asked the wrong server. Actors, templates, and workers are rows in the Substrate database, and one plugin reads them.

When you finish this section, you can install kubectl ate and say how each command reaches the control plane. You can also explain the warning that it prints on every call.

Install the plugin

Plugin: kubectl-ate is one binary. When it is on your PATH, kubectl runs it for every command that starts with kubectl ate.

The release publishes the binary for Linux and macOS, on amd64 and arm64. There is no Windows build. Download the file and its checksum, check it, and install it:

BASE=https://github.com/agent-substrate/substrate/releases/download/v1.0.0
curl -LO $BASE/kubectl-ate-linux-amd64
curl -LO $BASE/kubectl-ate-linux-amd64.sha256
shasum -a 256 -c kubectl-ate-linux-amd64.sha256
mkdir -p ~/.local/bin
install -m 0755 kubectl-ate-linux-amd64 ~/.local/bin/kubectl-ate

On a Mac with Apple silicon, the file is kubectl-ate-darwin-arm64. If you have the checkout and Go, you can build the plugin from the tag:

go install ./cmd/kubectl-ate

kubectl finds the plugin only on your PATH. Check that ~/.local/bin is on it, or the directory $(go env GOPATH)/bin when you use go install.

We measured the build once on an x86 server guest with 16 vCPU and 23 GB. go install took 5 s after the install of the control plane, because the Go modules were already on disk. In our runs we always built the plugin from the tag. We checked that both release files exist, and we did not time the download. Raw runs

Check the version before you go on:

kubectl ate --version

How a command reaches the control plane

kubectl ate does not ask the Kubernetes API for actors. It uses your kubeconfig to open a port-forward to the Service api in the namespace ate-system. Without --token-file, it also uses the kubeconfig to request a token for the shared ate-client ServiceAccount. Then it sends a gRPC request through that port-forward to ate-api-server, and ate-api-server answers from Postgres. Figure 2.2 shows that path beside the path of a plain kubectl command.

Two facts follow from the figure.

First, each command opens its own port-forward. So kubectl ate needs a working Kubernetes API server, and your account needs the right to port-forward into ate-system. With kube-apiserver down, a command fails like this:

Error: failed to connect to ate-api-server: getting api service: Get "https://127.0.0.1:44015/api/v1/namespaces/ate-system/services/api": EOF

Actors kept serving requests during that outage. Only the CLI and the proxy failed. Section 2.3 has the numbers.

Second, the flag --endpoint skips the port-forward. A program inside the cluster passes the address of ate-api-server and dials it directly. Its pod must be in a namespace with the label ate.dev/api-access=true. We did not set that label ourselves, and we did see the network rule behind it in the kagent run. A pod in default could not reach the API, and a pod in kagent could. Section 5.1 has that probe.

The shared identity warning

Every command in our recordings starts with the same line:

Warning: no --token-file given; acting as the shared identity system:serviceaccount:ate-system:ate-client rather than as yourself. Pass --token-file to act as yourself.
ATESPACE           NAME           TEMPLATE                   STATE                   WORKER POD   WORKER IPS   VERSION   AGE
ate-demo-counter   my-counter-1   ate-demo-counter/counter   ACTOR_STATE_SUSPENDED   <none>                    1         0s

Shared identity: the ServiceAccount ate-client in ate-system. When you pass no token, the plugin asks Kubernetes for a token of that ServiceAccount and sends it to ate-api-server.

The warning is correct, and you must not ignore it on a shared cluster. Every person and every script that can port-forward into ate-system is the same caller to Substrate. An access policy cannot tell them apart, because it sees one name.

On a kind cluster on your own machine, the shared identity is fine. On a cluster with more than one team, give each caller a token and pass it with --token-file. Section 5.1 shows which token to use, and section 5.2 limits what a caller can do in an atespace.

Our scripts removed the line from their output with grep -v '^Warning: no --token-file'. Do that in a script that parses a table. Do not do it for people.

The commands of the first day

An actor is addressed by its atespace and its name, so most commands need -a with an atespace. No default atespace exists.

-o json and -o yaml print the full object. Use them when you debug, because the table hides the snapshots. This is the new actor as JSON:

  "status": {
    "lastAssignedGeneration": 1,
    "snapshots": [
      {
…
        "durableSnapshot": {
…
          "state": "SNAPSHOT_STORAGE_STATE_COMPLETED"
        },
        "fidelity": "SNAPSHOT_FIDELITY_MEMORY",
        "generation": 1,
        "owner": "SNAPSHOT_OWNER_TAG",
…
    "state": "ACTOR_STATE_SUSPENDED"

The actor has never run, and it already has one snapshot. The owner is a tag, and the tag belongs to the template. Section 3.1 explains that snapshot. The command reference lists every command and flag.

Three mistakes we made

-A on workers. A worker belongs to a Kubernetes namespace and not to an atespace. The flag that works for actors fails for workers:

Error: unknown flag "-A"
…
  -a, --atespace string        filter worker pods hosting actors in a specific atespace
…
  -n, --namespace string       scope output to a specific Kubernetes namespace

Run kubectl ate get workers with no flag to see all workers.

kubectl get for an actor. kubectl get actor and kubectl get actortemplate return nothing, because those objects are not custom resources. kubectl get workerpool does work, because a WorkerPool is one.

A flag from an old guide. Guides for v0.x create an actor with --template-ref. At v1.0.0 the flag is --template. Section 8.4 lists the other changes.

The install is done, and you want to see an actor answer. The quickstart gives you two commands and a curl. They work. Then you put the same proxy command in a test script. The control plane of the cluster restarts, and every request times out while the actors are fine.

When you finish this section, you can create an actor, reach it through kubectl ate proxy, and suspend it. You can also say why the proxy must not carry real traffic, and what a real client needs in its place.

Create the actor

The counter demo created an atespace named ate-demo-counter and a template named counter in it. Create an actor from that template:

kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter

The command prints the new actor. In our run the state was ACTOR_STATE_SUSPENDED, the worker pod was <none>, and the version was 1. Section 2.2 shows that output.

A new actor uses no worker. Nothing was started. The create call wrote one row to Postgres and pointed the actor at the golden snapshot of its template. In one of our tests we created 131 actors on a pool of three workers before any of them ran.

Start the proxy

Start the proxy in a terminal of its own. It stays in the foreground:

kubectl ate proxy actor my-counter-1 -a ate-demo-counter

Proxying 127.0.0.1:8000 -> ate-demo-counter/my-counter-1:80

With no port argument, the proxy forwards local port 8000 to port 80 of the actor. To choose the ports, add pairs in the form local:remote, such as 8080:80. The proxy listens on 127.0.0.1 only, unless you pass --address.

Proxy: kubectl ate proxy actor is a local listener that forwards TCP to one actor through atenet-router. It does three things for you. It opens a port-forward to the router. It gets a client certificate for the identity atenet-client. It names the target actor to the router.

The install must allow that identity. This is why the install command in section 2.1 passes --ingress-allowed-spiffe-ids with the atenet-client name. Without that flag, the router lets no client through, and the proxy cannot reach any actor.

Send requests

In a second terminal, send a request:

curl -X POST http://localhost:8000/

Send it a few times. This is our recording of the first three requests:

2026-11-12T10:14:21.837Z	call-1	200	0.358379	preserved memory count: 1 | preserved file counter: 1 | preserved rootfs counter: 1
2026-11-12T10:14:21.944Z	call-2	200	0.090770	preserved memory count: 2 | preserved file counter: 2 | preserved rootfs counter: 2
2026-11-12T10:14:22.030Z	call-3	200	0.066903	preserved memory count: 3 | preserved file counter: 3 | preserved rootfs counter: 3

The counter demo keeps three counts: one in memory, one in a file, and one in the root filesystem. All three rise together here. They matter later, when a snapshot keeps less than the memory. Section 3.4 uses them.

The first request resumed the actor. Now it holds a worker:

NAME                                   POOL      STATE                 ACTORS   CPU(MILLICORES)   MEMORY        POD                                         AGE
ec15263e-4c0f-45eb-adc1-53f5f60c2331   counter   WORKER_STATE_ACTIVE   1/1000   1000/1000         512Mi/960Mi   ate-demo-counter/counter-7cc7dcd596-ffxwt   74s

Read the CPU(MILLICORES) column. This one actor takes 1000 of the 1000 millicores of its worker. The actor limit of 1000 in the ACTORS column is not the limit that binds. The CPU is. So the demo pool of three workers runs three counter actors at one time, and a fourth must wait. Section 4.5 sizes a pool from these columns.

Suspend it yourself

Stop sending requests and watch the actor. It stays in ACTOR_STATE_RUNNING. Substrate does not suspend it.

We measured this on an x86 server guest with 24 vCPU and 47 GB, in one run. For 900 s we sent no request and read the state every 15 s. All 60 samples were ACTOR_STATE_RUNNING. The next request took 0.080 s and returned count 6. Raw runs

To free the worker, suspend the actor:

kubectl ate suspend actor my-counter-1 -a ate-demo-counter

We measured the suspend on the same guest, once. The command returned in 0.40 s, and the state was ACTOR_STATE_SUSPENDED. The next request through the proxy took 0.36 s and returned the next count, so the memory came back. Raw runs

This default suspend is asynchronous. The new snapshot is on the node and is not yet in the object store. Section 3.3 explains the two modes, and section 3.5 shows what you lose when the node copy disappears.

The proxy depends on the Kubernetes API server

The proxy reaches the router through a Kubernetes port-forward. Every byte passes kube-apiserver. A client inside the cluster does not need that hop. Figure 2.3 puts the two paths side by side.

We measured both paths on an x86 server guest with 16 vCPU and 23 GB, in one run with one request each second. We took kube-apiserver away for 86 s. Through the proxy, 0 of 81 requests in that time were served, and each one ran into the 9 s limit of the client. From the pod, 86 of 86 were served. The proxy kept failing for 21 s after kube-apiserver was ready again. Raw runs

== T3 kube-apiserver: down at +0 s, restore action at +86 s, dependency ready at +91 s
  proxy/running (VM, kubectl ate proxy)
     codes from -10 s: 200x9 000x105 200x59
     first failure at +0 s, last failure at +112 s (= +26 s from restore action, +21 s from dependency ready); failed 105 of 173
     failure latency p50 9.00 s max 9.00 s
     105x code=000 body=''
  incluster/running (pod, mTLS to router)
     codes from -10 s: 200x186
     no failures

So do not judge the health of Substrate with the proxy, and do not put the proxy between a product and its actors. The proxy tells you about your path to the cluster. A pod that calls the router tells you about Substrate.

What a real client needs

A real client is a pod. It needs three things, and the pod we used for the test shows all of them.

kind: Pod
…
metadata: {name: e-prober, namespace: ate-system, labels: {app: e-prober}}
…
  serviceAccountName: atenet-client
…
      - podCertificate: {credentialBundlePath: credential-bundle.pem, keyType: ECDSAP256, signerName: podidentity.podcert.ate.dev/identity}
…
      - clusterTrustBundle: {labelSelector: {matchLabels: {podcert.ate.dev/canarying: live}}, path: trust-bundle.pem, signerName: servicedns.podcert.ate.dev/identity}

1. An allowed identity. The pod runs as the ServiceAccount atenet-client in ate-system. That is the identity in --ingress-allowed-spiffe-ids. The install does not create this ServiceAccount, so we created it.
2. A client certificate. The podCertificate volume asks Kubernetes for a certificate with that identity. The clusterTrustBundle volume holds the CA that verifies the router.
3. The target header. Each request names its actor in the header ate-target-actor, as the atespace, a slash, and the actor name.

The request of that pod was this command:

curl --cacert /run/ca/trust-bundle.pem \
  --cert /run/id/credential-bundle.pem --key /run/id/credential-bundle.pem \
  -X POST -H 'ate-target-actor: lab-e/e-run' \
  https://atenet-router.ate-system.svc:443/

Any listed client can reach any actor through the router. The allowlist says who can enter. It does not say which actor each client can call. Section 5.1 and section 6.1 cover the identity and the router.

The install ended with no error, and kubectl get pods -A now prints a page of names you did not choose. One of them will be in an error message next week. You want to know what each pod does, which ones have a second replica, and where your actors are stored.

When you finish this section, you can name the job of each pod that the install created. You can also say which objects live in etcd and which live in Postgres, and which pods run as a single replica.

The pods

This is the namespace ate-system after our install from source. The install from the release images created the same pods.

NAMESPACE                          NAME                                         READY   STATUS      RESTARTS   AGE …
ate-system                         ate-api-server-696d95dd66-9q7f8              2/2     Running     0          117s …
ate-system                         ate-api-server-696d95dd66-t85dd              2/2     Running     0          117s …
ate-system                         ate-controller-7479ff54f-jrrtc               1/1     Running     0          114s …
ate-system                         ate-idp-server-7c46c85d94-66sf2              1/1     Running     0          116s …
ate-system                         ate-idp-server-7c46c85d94-cfbt4              1/1     Running     0          116s …
ate-system                         atelet-sqzvr                                 2/2     Running     0          113s …
ate-system                         atenet-egress-78f759fcb8-74d4m               3/3     Running     0          76s …
ate-system                         atenet-router-86454b57f5-7vtp5               2/2     Running     0          112s …
ate-system                         k8s-credential-provider-8df9859f8-jcfll      1/1     Running     0          89s …
ate-system                         postgres-0                                   2/2     Running     0          7m13s …
ate-system                         rustfs-54c86945fb-w9pnn                      1/1     Running     0          109s …
ate-system                         rustfs-bucket-init-bvq6z                     0/1     Completed   0          109s …

Figure 2.4 groups all the pods by namespace and gives each one the colour of its role.

Each workload has one job:

One more pod runs in its own namespace. podcertificate-controller signs the certificates that the pods use to prove who they are. It needs the three Kubernetes preview features that the cluster script turned on.

What has one replica

Count the pods in the listing. ate-api-server and ate-idp-server have two. Everything else has one.

That matters for three of them. Every request to every actor passes atenet-router, and ate-api-server reads Postgres for each one. Section 1.2 shows the path. So one router pod and one Postgres pod are in front of all your actors. The bundled Postgres has no second replica and no failover.

For a kind cluster on your machine, that is fine. For anything that people depend on, read section 8.1 before you go further. It covers an external database and what a Postgres outage does to requests.

Three custom resources

The install added three custom resources to Kubernetes:

NAME                       SCOPE        VERSIONS      CREATED AT
csidriverconfigs.ate.dev   Cluster      v1(storage)   2026-11-12T10:05:33Z
sandboxconfigs.ate.dev     Cluster      v1(storage)   2026-11-12T10:05:33Z
workerpools.ate.dev        Namespaced   v1(storage)   2026-11-12T10:05:33Z

WorkerPool: a namespaced object that asks for a number of worker pods of one sandbox class. It is the only Substrate object that you manage with plain kubectl day to day. Section 4.1 covers it.

SandboxConfig describes a sandbox class for the whole cluster. The install created two of them: gvisor-default and microvm. CSIDriverConfig belongs to a preview feature for volumes, and we did not use it.

Everything else is a row in Postgres: atespaces, templates, actors, tags, workers, egress policies, and access policies. So kubectl get does not list them, a backup of etcd does not hold them, and no Kubernetes RBAC rule names them. To back up your actors, back up Postgres and the object store. A snapshot from a default suspend reaches the object store about 5 minutes later, so an earlier backup does not hold it. Section 3.4 has the timers. Section 3.6 and section 5.2 cover names and access.

The demo namespace

deploy demo counter created the namespace ate-demo-counter with one WorkerPool named counter. ate-controller turned it into three worker pods. Each pod runs the image ateworker-gvisor, which holds the gVisor sandbox.

The demo also created two things that kubectl cannot show: the atespace ate-demo-counter and the template counter. Read them with the plugin:

kubectl get workerpools -A
kubectl ate get atespaces
kubectl ate get actor-templates -a ate-demo-counter

The atespace and the Kubernetes namespace have the same name here. They are two objects. An atespace is a Substrate name, and nothing ties it to a namespace.

A worker pod is not an ordinary pod. It runs as root with the SYS_ADMIN and NET_ADMIN capabilities, because it starts sandboxes. Put WorkerPools on nodes that you trust for that. Section 4.2 covers the sandbox classes.

The telemetry namespace

The kind profile also installs otel-system with three pods: an OpenTelemetry collector, Prometheus, and Jaeger. They are not part of Substrate. They receive its metrics and traces.

They are useful on day one. Add --trace to any kubectl ate command, forward the Jaeger port, and open the trace:

kubectl port-forward -n otel-system svc/jaeger 16686:16686 &
kubectl ate get actor my-counter-1 -a ate-demo-counter --trace

What it costs on one machine

We measured the size of the release-image install once on an x86 server guest with 16 vCPU and 23 GB. Docker held 2.1 GB of images and 4.8 GB of volumes. The guest used 13 GB of disk in all, with the operating system and the Go caches. Raw runs

We measured the memory once on an Apple silicon laptop under Docker Desktop. The kind node held 2.7 GiB right after the install, with no actor. Raw runs

The install also left two things outside the cluster. The first is a second container, the local registry:

kind-control-plane kindest/node:v1.37.0
kind-registry registry:3

The second is a file. ate-setup records the settings of each install in the cache directory of your user, one file for each kubeconfig context. That is ~/.cache/ate-setup/installs/ on Linux and ~/Library/Caches/ate-setup/installs/ on macOS, and the flag --record-dir names another place. Keep that file. It tells you later which flags the running cluster was installed with.

Your agent image takes several seconds to import its libraries and load its tools. With a Deployment, every new pod pays those seconds again. Substrate moves that cost to the template. It starts the image once, takes a snapshot, and creates every new actor from that snapshot. The cost does not disappear: the first request to a new actor restores the snapshot, and the restore grows with the memory in it.

When you finish this section, you can write an ActorTemplate, check that its golden snapshot exists, and predict the first-call time of a new actor.

The template we used

ActorTemplate: the description of an actor: its containers, its resource limits, its volumes, the worker pools that it can run on, and its snapshot settings. A template belongs to one atespace, and ate-api-server stores it in Postgres. It is not a Kubernetes object, so kubectl get does not list it.

This is one of the five templates of our resume tests. The actor is a small Go server that holds 200 MiB of random bytes and counts its requests.

metadata:
  atespace: lab-c
  name: ballast-200
workerSelector:
  matchLabels:
    workload: ballast
containers:
- name: ballast
  image: localhost:5001/ballast@sha256:93dba82ad560e7fc4ccb2a33a63d64713b519a7895ba3fd98130ce789eab7fd3
  command:
  - /ballast
  - --mb=200
  wakeupProbe:
    httpGet:
      path: /readyz
      port: 80
resources:
  limits:
    cpuMillicores: 1000
    memoryBytes: 478150656
snapshotConfig:
  preferredFidelity: SNAPSHOT_FIDELITY_MEMORY

Five fields decide how the actor behaves.

• image must name a digest, and a tag alone is refused. Ours is a scratch image from a plain docker build, with no change for Substrate.
• resources.limits needs both values. The scheduler books them against the capacity of a worker, which section 4.1 explains.
• workerSelector matches the labels of a WorkerPool object. Section 3.6 shows why the namespace of the pool does not count.
• wakeupProbe is a readiness check that runs on every resume. The actor becomes ACTOR_STATE_RUNNING only after the probe passes.
• preferredFidelity says how much state a snapshot keeps. Section 3.4 compares the three levels.

A template is immutable, and the API has no call that updates one. For new code you create a new template, and then you move each suspended actor with kubectl ate update actor --template.

We ran that move with two templates that differ in the image. On a running actor the update is refused.

…Error: failed to update actor: rpc error: code = FailedPrecondition desc = actor must be ACTOR_STATE_SUSPENDED to change its actor template (got: ACTOR_STATE_RUNNING) 

On the suspended actor the update was accepted, and the next call ran the new image. The count in memory started again at 1, and so did the count on the root filesystem. The count on the durableDir volume continued at 4. So this move kept the durableDir volume and dropped the old memory snapshot. We did not test a move with an external volume. Suspend first, and expect a cold start of the new image on the next call.

The golden snapshot is built once

Golden snapshot: the snapshot of one temporary actor that Substrate starts from a new template. Substrate stores it under a tag, the golden tag, and every new actor of that template starts from that tag.

ate-api-server builds the golden snapshot in four steps.

1. It creates a temporary golden actor in the atespace of the template, and it resumes that actor.
2. It waits until the actor is ready. If a container has no wakeupProbe, it waits 20 seconds more.
3. It suspends the actor at the preferredFidelity of the template, and it copies the snapshot to a published tag.
4. It deletes the golden actor, and it writes the tag into the status of the template.

The log of ate-api-server on our kagent cluster shows all four steps for one template.

2026-11-12T10:16:44.165402326Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	create	suspended
2026-11-12T10:16:44.178659651Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	resume	resuming
2026-11-12T10:16:54.017864285Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	resume	running
2026-11-12T10:16:54.03004179Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	suspend	suspending
2026-11-12T10:16:54.180722259Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	suspend	suspended
2026-11-12T10:16:54.238242923Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	delete	deleting
2026-11-12T10:16:54.328177186Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	delete	deleted

The timestamps show a resume of 9.84 seconds, because the golden actor is the one actor that boots the image from zero. The suspend took 0.15 seconds. That template had a files-only fidelity, so the snapshot held no memory.

The build pays the full start of your application, once.

We measured one template for a Python server on the x86 server guest (24 vCPU, 47 GB, nested KVM), with n=1. The server builds a dict of one million values at start. The golden tag was ready 39.3 seconds after create actor-template. The process was 32.9 seconds old at its first request, so most of that time was the application start under gVisor. A new actor of that template then answered its first call in a median of 0.63 seconds, with five new actors. Section 3.3 shows the server. Raw runs

The golden tag does not exist at the moment the template does. In our resume tests we listed the five templates 21 seconds after we created them.

ATESPACE   NAME           GOLDEN TAG                             ERROR   AGE
lab-c      ballast-0      e32029d7-3d9b-4212-8953-e88d7cede5b4           21s
lab-c      ballast-1000                                                  20s
lab-c      ballast-200    b9f97b82-b187-419d-8789-6c8c7b8a6c8c           20s
lab-c      ballast-50     0382e2fc-99db-461a-8845-a90d93484443           21s
lab-c      ballast-500    2fa7aaf9-1fe8-4076-9350-af27ba338a6b           20s

All five had a tag when we looked again at 118 seconds. Wait for the GOLDEN TAG column before you create actors. An actor that you create earlier records no snapshot. It boots from the image on its first call, and it keeps that cold start until its own first suspend.

We ran that race. We created an actor directly after its template, and create actor succeeded.

2026-11-12T12:49:47.365Z	rc1-state	ACTOR_STATE_SUSPENDED	-	
2026-11-12T12:49:48.542Z	rc1-first-call	200	0.983729	tag=v3 count=1 mb=200 fill=random sum=26936 rootfs=-1 data=-1 pid=1 process_age_s=0.8 child[none]
…
2026-11-12T12:50:00.900Z	golden-ready	waited 11.137 s more
2026-11-12T12:50:01.888Z	rc2-state-new	ACTOR_STATE_SUSPENDED	-	SNAPSHOT_OWNER_TAG:SNAPSHOT_FIDELITY_MEMORY:SNAPSHOT_STORAGE_STATE_COMPLETED:gen1

We measured this once on the x86 server guest (24 vCPU, 47 GB, nested KVM), with a 200 MiB actor. The early actor was ACTOR_STATE_SUSPENDED with an empty snapshot list. Its first call was a cold start and took 0.98 seconds. The golden tag was ready 11 seconds later, and the first call of an actor created after that took 0.93 seconds. Five more early actors of another template still had no snapshot 25 seconds later. Raw runs

So an early actor never adopts the golden snapshot. It gets its own snapshot at its first suspend. Our actor starts in under a second, so the two first calls cost the same. With an application that needs 30 seconds to start, the early actor pays 30 seconds on its first call.

You can turn the golden snapshot off with snapshotConfig.goldenSnapshotConfig.mode: GOLDEN_SNAPSHOT_MODE_DISABLED. Every new actor then boots from the image.

A new actor starts suspended

kubectl ate create actor writes one row in Postgres and touches no worker, so its time does not depend on the template. The new actor is suspended. It already has a snapshot when the template had its golden tag at that moment.

{
  "actorTemplate": {
    "atespace": "ate-demo-counter",
    "name": "counter"
  },
  "metadata": {
    "atespace": "ate-demo-counter",
    "createTime": "2026-11-12T10:13:40.987568669Z",
    "name": "my-counter-1",
    "uid": "6f80f612-7547-4f5b-b3ab-831595621806",
    "updateTime": "2026-11-12T10:13:40.987568669Z",
    "version": "1"
  },
  "status": {
    "lastAssignedGeneration": 1,
    "snapshots": [
      {
        "actorTemplateUid": "71a3a39c-5f37-4d6e-937f-9c314fea7668",
        "durableSnapshot": {
          "object": {
            "snapshotUri": "gs://ate-snapshots/atespaces/ate-demo-counter/tags/ad19ee28-1519-432a-bcc5-40cf99cce3ca"
          },
          "state": "SNAPSHOT_STORAGE_STATE_COMPLETED"
        },
        "fidelity": "SNAPSHOT_FIDELITY_MEMORY",
        "generation": 1,
        "owner": "SNAPSHOT_OWNER_TAG",
        …
    "state": "ACTOR_STATE_SUSPENDED"
  }
}

Read three fields. state is ACTOR_STATE_SUSPENDED, so no sandbox exists yet. owner is SNAPSHOT_OWNER_TAG, so the actor borrows the snapshot of the golden tag and owns none. The URI has a tags segment, and it does not hold the uid of the actor. The actor writes its own snapshot at its first suspend.

The first call pays for the restore

The first request to a new actor restores the golden snapshot. For a small actor you do not notice it. For a large one, it is the slowest request that the actor ever serves.

We measured these times on the x86 server guest (24 vCPU, 47 GB, nested KVM), with gVisor and five new actors for each size. Each time is one request through kubectl ate proxy, timed at the client. A request to an actor that already runs took 0.06 seconds on the same path. The later resume is the median of 10 resumes from the snapshot on the node. kubectl ate create actor returned in 0.20 to 0.26 seconds for all 25 actors. Raw runs

Up to 200 MiB the first call costs about the same as a later resume. At 500 MiB and above it costs three to five times more, and the spread is wide. We did not find the cause of that spread. We can say where the node kept data. Its snapshot-cache directory held 1,765 MiB, and the five snapshot sizes of our templates add up to 1,760 MiB.

Our position: the golden snapshot is a good place for slow start code and a bad place for large memory. If your template holds more than about 200 MiB, send one request to each new actor before a user does. Use a request that changes no state: our counter adds one on every call. Then suspend it. A later resume from the node copy follows the lower line.

A request to your actor returns 503. kubectl get pods shows every pod as Running, and the worker pod log does not name your actor. Kubernetes cannot tell you more, because an actor is not a pod. Its state is a row in Postgres, and kubectl ate get actors reads that row.

When you finish this section, you can read the state of an actor and name the call that moves it. You can also predict what a request does in each state.

The Actor object

Actor: one instance of an ActorTemplate, with a name that is unique in its atespace. An actor is the unit that Substrate suspends and resumes. It can run on a different worker after each resume.

This is the counter actor of the quickstart while it runs.

actorTemplate:
  atespace: ate-demo-counter
  name: counter
metadata:
  atespace: ate-demo-counter
  createTime: "2026-11-12T10:13:40.987568669Z"
  name: my-counter-1
  uid: 6f80f612-7547-4f5b-b3ab-831595621806
  updateTime: "2026-11-12T10:30:18.103768100Z"
  version: "16"
status:
  assignedNode: kind-control-plane
  lastAssignedGeneration: 3
  snapshots:
  …
  state: ACTOR_STATE_RUNNING
  workerAssignment:
    nodeName: kind-control-plane
    …
    workerPod: counter-7cc7dcd596-fps97
    …
    workerPodUid: c572e96d-dd9b-4500-b0e3-880a98d9f61d
    workerPool: counter

workerAssignment and assignedNode answer different questions. The first says where the actor runs now. The second also says where its last snapshot is after a suspend, and section 3.4 shows how long that lasts.

The seven states

The enum ActorState has seven values at v1.0.0. The state diagram on the architecture page of the project draws four of them, so trust the enum.

Three of these states are stable. ACTOR_STATE_SUSPENDED lasts until a request or a resume call arrives. ACTOR_STATE_CRASHED lasts until a revert or a delete, and a request does not end it. ACTOR_STATE_RUNNING lasts until a caller suspends the actor or Substrate marks it crashed. Section 3.3 proves that Substrate does not suspend it. The other four states are steps of one operation.

ACTOR_STATE_RUNNING does not prove that the process is alive. Two actors whose process had exited read ACTOR_STATE_RUNNING for the 20 minutes of our watch. Section 3.5 shows that run.

Old tutorials name a state ACTOR_STATE_PAUSED and a command kubectl ate pause actor. Both are gone at v1.0.0.

The transitions we saw

We drove my-counter-1 through a suspend, a resume, a crash, and a revert in 13 seconds. A script read the state after each step.

2026-11-12T10:30:09.378Z	state	ACTOR_STATE_SUSPENDED	kind-control-plane	9	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen3
2026-11-12T10:30:10.310Z	state	ACTOR_STATE_RUNNING	kind-control-plane	11	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen3
…
2026-11-12T10:30:11.590Z	state-after-crash	ACTOR_STATE_CRASHED	kind-control-plane	12	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen3
…
2026-11-12T10:30:17.724Z	state	ACTOR_STATE_SUSPENDED	-	14	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen3
…
2026-11-12T10:30:18.333Z	state	ACTOR_STATE_RUNNING	kind-control-plane	16	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen3

Look at the version column. A resume moved it from 9 to 11, and the revert moved it from 12 to 14. Each operation writes the actor two times: one write for the step state, and one for the stable state. Our script read the state after each command returned, so it never saw the state between.

The log of ate-api-server does show those states. The listing in section 3.1 has resuming before running, suspending before suspended, and deleting before deleted.

We measured the step states in the kagent run, on a second x86 server guest (16 vCPU, 31 GB). There a harness resumes and suspends an actor for every turn. Over 98 turns, the actor spent a median of 0.38 seconds in ACTOR_STATE_RESUMING. Over 99 suspends, it spent a median of 0.14 seconds in ACTOR_STATE_SUSPENDING. Those actors keep files only, so the times are low. Raw runs

A step state can also last. In our dependency test, one actor was still in ACTOR_STATE_RESUMING five minutes after Postgres came back. Section 8.2 has that case.

Who moves an actor

Substrate itself makes one move: it marks an actor ACTOR_STATE_CRASHED when the sandbox, the worker, or a restore is lost. Section 3.5 lists every way that we reached that state.

A call in the wrong state does not wait. ResumeActor on a crashed actor returns FailedPrecondition at once, and a request gets the same text as a 503. SuspendActor on a suspended actor returns the actor with no change.

The router calls ResumeActor for every request, also when the actor runs. On a running actor the call succeeds, changes nothing, and returns resumed=false. That is why a plain HTTP request is enough to start a suspended actor. Section 1.2 follows one such request.

Our position: read the state before you debug the network. kubectl ate get actors -a <atespace> costs one call. It separates a crashed actor from a full pool and from a dead process, and each of those needs a different fix.

You created an actor in the afternoon, sent it five requests, and went home. The quickstart told you that Substrate checkpoints an idle actor. The next morning the actor is still ACTOR_STATE_RUNNING, and it still holds a full CPU of its worker. Nothing is broken. Substrate v1.0.0 has no idle timer.

When you finish this section, you can decide who calls SuspendActor in your system and choose between its two modes. You can also predict the resume time of an actor from the memory that it holds.

Substrate does not suspend an idle actor

We sent five requests to my-counter-1 and then sent nothing for 900 seconds. A script read the state every 15 seconds.

2026-11-12T10:14:22.822Z	idle-watch-start	no requests for 900 s
2026-11-12T10:14:37.832Z	state	ACTOR_STATE_RUNNING	-
…
2026-11-12T10:29:35.957Z	state	ACTOR_STATE_RUNNING	-
2026-11-12T10:29:36.168Z	idle-watch-end
2026-11-12T10:29:36.265Z	call-after-idle	200	0.080377	preserved memory count: 6 | preserved file counter: 6 | preserved rootfs counter: 6

All 60 samples were ACTOR_STATE_RUNNING, and the sixth call continued the count in 0.08 seconds. The code agrees: no component of v1.0.0 suspends an actor because it is idle. The API reference says the same, and the quickstart page does not.

An idle running actor is not free. It keeps the CPU and the memory that its template booked on the worker. While my-counter-1 ran, its worker showed 1,000 of 1,000 millicores in use. Section 4.5 turns that into a pool size.

So the job belongs to the caller, and in practice to the agent harness. In our run, kagent suspended the actor after 98 of 98 turns. Section 7.1 shows that pattern and its cost.

SuspendActor: asynchronous or synchronous

SuspendActor: the call that checkpoints a running actor, stops its sandbox, and frees its place on the worker. It has two durability modes.

• The asynchronous mode is the default. The call returns when the snapshot is on the disk of the node. atelet uploads it to the object store later.
• The synchronous mode returns only when the snapshot is in the object store. Use kubectl ate suspend actor --mode sync.

The actor object shows the difference.

2026-11-12T10:30:08.028Z	state	ACTOR_STATE_SUSPENDED	kind-control-plane	5	SNAPSHOT_OWNER_TAG:SNAPSHOT_STORAGE_STATE_COMPLETED:gen1,SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_PENDING:gen2
…
2026-11-12T10:30:09.378Z	state	ACTOR_STATE_SUSPENDED	kind-control-plane	9	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen3

After the asynchronous suspend, generation 2 is SNAPSHOT_STORAGE_STATE_PENDING. The object store does not have it yet, so the older generation stays as the durable copy. After the synchronous suspend, generation 3 is SNAPSHOT_STORAGE_STATE_COMPLETED, and Substrate has released the older ones. Section 3.4 follows such a snapshot for 42 minutes.

We measured the time until kubectl ate suspend actor returned, on the x86 server guest (24 vCPU, 47 GB, nested KVM) with gVisor. The asynchronous column has 10 suspends for each size, and the synchronous column has 5. The actor held random bytes, which do not compress. The extra time is close to the upload itself. In a separate run, atelet logged 2.0 to 2.1 seconds for each of four uploads of a 500 MiB snapshot. Raw runs

A suspend does not wait for open requests

A suspend stops the sandbox at once, also when the actor is in the middle of a request. We gave an actor a handler that sleeps 10 seconds, sent it one request, and suspended the actor 2 seconds later.

2026-11-12T12:52:10.848Z	async	slow-request-ended	503	2.218030	upstream connect error or disconnect/reset before headers. reset reason: connection termination
2026-11-12T12:52:10.992Z	async	suspend-2s-into-request	0.376s	ATESPACE   NAME       TEMPLATE       STATE                   WORKER POD   WORKER IPS   VERSION   AGE lab-v      sl-async   lab-v/tiny-0   ACTOR_STATE_SUSPENDED…
…
2026-11-12T12:52:11.431Z	async	next-call	200	0.228121	tag=v3 count=2 mb=0 fill=zero sum=0 rootfs=-1 data=-1 pid=1 process_age_s=2.4 child[none]

We measured this once in each mode on the x86 server guest (24 vCPU, 47 GB, nested KVM). The client of the open request got a 503 after 2.2 seconds in both modes. The suspend call returned success in 0.38 seconds in the asynchronous mode and 0.48 seconds in the synchronous mode, with no warning. The next request resumed the actor in 0.23 seconds, and the count continued at 2. Raw runs

Substrate does not replay the request that was cut. The snapshot holds the actor in the middle of its handler, and the client is gone when the handler ends. So the caller of SuspendActor must know that the actor is idle. A harness that suspends after each turn knows it. A timer that suspends by clock does not.

ResumeActor: who calls it and where it runs

ResumeActor: the call that places a suspended actor on a worker and restores its newest snapshot. The router makes this call for every request, so a plain request is enough to start an actor. kubectl ate resume actor makes the same call with no request.

A resume looks at the node first. After a suspend, the actor stays assigned to its node for as long as that node holds the snapshot. ResumeActor then tries the workers on that node. If the snapshot is only in the object store, atelet downloads it first, on the node of the worker that the scheduler chose.

Resume time against the memory held

We built an actor that holds a fixed amount of random memory and reads one byte in each MiB on every request. We suspended it, waited, and timed the next request.

We measured one request to a suspended actor on the x86 server guest (24 vCPU, 47 GB, nested KVM), with gVisor and one kind node. The time is at the client, through kubectl ate proxy. A request to a running actor took 0.06 seconds on that path. For the object store column, atelet ran with an upload delay of 3 seconds and a detach delay of 10 seconds. All 55 requests of that run returned 200: 5 first calls, 25 resumes, and 25 calls to a running actor. Raw runs

The atelet log says where the second column loses its time.

{"time":"2026-11-12T11:37:01.905071369Z","level":"INFO","msg":"Restore timing breakdown",…"ate.actor.name":"o500",…"ate.snapshot.fidelity":"memory",…"ate.actor.restore.duration.download":1.171619862,…"ate.actor.restore.duration.worker_restore":0.326837467,"ate.actor.restore.duration.total":1.5049079490000001,…

The download took 1.17 seconds of the 1.50 seconds that atelet counted. The worker needed 0.33 seconds for the restore itself.

Our rule of thumb for this machine has two numbers. A resume costs a fixed 0.2 seconds, plus about 2 ms for each MiB from the node. From the object store, it costs about twice as much for each MiB. The rule has four conditions, and you must check each one against your case.

• The memory was random, so every MiB held data. The rule counts the memory that holds data, as the next subsection shows.
• The object store ran in the same cluster and on the same disk. On another guest, 20 ms of added round trip to the bucket took a resume of 1,000 MiB from 4.8 s to 10.2 s. Section 8.1 has that run.
• The sandbox was gVisor. On the microVM class the same test cost a fixed 0.3 seconds plus about 1 ms for each MiB, up to 500 MiB. Section 4.2 has the comparison.
• The machine was a nested guest on a server CPU from 2019.

The project states that a resume takes less than 500 ms, and it gives no actor size and no machine. On our guest, that holds up to about 200 MiB from the node and up to about 50 MiB from the object store.

The rule counts memory that holds data

Random bytes are the worst case for a snapshot. We repeated the node path with three kinds of memory at 200 and 500 MiB.

We measured 10 resumes from the node for each cell, on the x86 server guest (24 vCPU, 47 GB, nested KVM). Each cell is the median, and the sandbox was gVisor. All 60 resumes returned 200, and every count continued. The random column repeats the earlier table within 12 percent. Raw runs

An actor with 500 MiB of zero pages resumed as fast as an empty actor. Half the data cost about half the extra time. So put the MiB that hold data into the rule, and not the limit of the template. Section 3.4 shows that the snapshot size follows the same bytes.

An ordinary runtime: Python with 245 MiB

Ballast is not an application. We checked the rule with a Python server that builds a dict of one million random 100-byte values at start. It keeps a counter in memory and a SQLite file on a durableDir volume.

import os, resource, sqlite3, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

started = time.time()
state = {"count": 0, "ready": False}
data = {i: os.urandom(100) for i in range(1_000_000)}
os.makedirs("/data", exist_ok=True)
db = sqlite3.connect("/data/app.db", check_same_thread=False)
db.execute("create table if not exists hits (n integer)")
db.commit()
state["ready"] = True
…

We measured one actor of the image python:3.13-slim on the x86 server guest (24 vCPU, 47 GB, nested KVM). The sandbox was gVisor, the limit was 768 MiB, and the fidelity was memory. The process reported a peak resident size of 245 MiB. All 29 requests returned 200. The memory counter, the SQLite row count, and the dict continued across 13 suspends, and the process ID stayed 1. Raw runs

The rule holds. It gives 0.2 seconds plus 215 MiB at 2 ms, which is 0.63 seconds, and the measured median was 0.41 seconds. An interpreter heap is not all random data, so the rule is an upper estimate for it.

One detail matters for code that reads a clock. The process reported an age of 32.9 seconds at its first request and 65.1 seconds at its last, 13 suspends later. Under gVisor the clock of the actor does not count the time that it was suspended. Section 4.2 compares the two classes on this point.

The same test on Apple silicon

We measured the node path on an Apple silicon laptop under Docker Desktop, with 10 CPU and 8 GB for Docker. Each size had 10 resumes. The median resume was 0.09 seconds at 0 MiB, 0.09 seconds at 50 MiB, and 0.16 seconds at 200 MiB. The median asynchronous suspend was 0.10, 0.15, and 0.24 seconds. All 66 requests returned 200, and every count continued. We did not test more than 200 MiB or the object store path on the laptop. Raw runs

The laptop and the server guest are different machines. Read each table by itself, and do not take a ratio across them.

Our position

Call SuspendActor yourself, from the component that knows when a turn ends. Use the asynchronous mode for the normal case, because the caller waits a third as long at 1,000 MiB. Use the synchronous mode for any state that you cannot lose, because it makes the snapshot durable before the call returns.

A durable snapshot does not move the actor. In our two-node run a sync-suspended actor resumed on its own node while that node was cordoned. When the node stopped, the actor failed its requests and became ACTOR_STATE_CRASHED. A drain is the exception: three suspended actors resumed on the other node at their next request. Section 4.3 has the runs.

Suspend only an actor that has no open request. Budget the resume from the memory that holds data in your actor, and not from a headline number.

Your agent wrote a file to /tmp, started a background process, and ended its turn. On the next turn the file is gone and the process does not exist. Nothing failed. The template asked for a snapshot of files only, and /tmp is not one of those files. A second surprise waits on the node: the snapshot that you took a minute ago is not in the object store yet.

When you finish this section, you can choose a fidelity for a template and estimate the disk that suspended actors need. You can also say where a snapshot is at each minute after a suspend.

Three fidelities

Snapshot fidelity: the set of state layers that a snapshot holds. A template names the highest level that it wants in snapshotConfig.preferredFidelity, and each level includes the levels below it.

kagent uses the first level. Its template mounts one durableDir volume at /data and sets SNAPSHOT_FIDELITY_VOLUMES. In the first turn of a session, we told the agent to write to three places and to start a process.

RUN: mkdir -p /data/workspace && echo hello-from-turn1 > /data/workspace/note.txt; echo scratch-turn1 > /tmp/scratch.txt; echo rootfs-turn1 > /root-marker.txt 2>/dev/null; (nohup sleep 100000 >/dev/null 2>&1 &) ;…

kagent suspended the actor after that turn. The second turn read the same places.

note=hello-from-turn1
scratch=cat: can't open '/tmp/scratch.txt': No such file or directory
bg=
uptime=0.74

The note under /data is there. The file in /tmp is gone, the sleep process is gone, and the sandbox is 0.74 seconds old. The write to / failed in turn 1, because the harness does not run as root.

The other end is SNAPSHOT_FIDELITY_MEMORY, which our counter and our memory test used. To compare all three levels, we ran one actor under each level. The actor holds 50 MiB of random bytes and starts a child process. It keeps one count in memory, one in a file on the root filesystem, and one in a file on a durableDir volume.

We measured these three actors on the x86 server guest (24 vCPU, 47 GB, nested KVM), with gVisor and one actor for each level. Each actor got 25 requests with 13 suspends between them, and all 75 requests returned 200. The resume column is one request to the suspended actor, timed at the client, with the snapshot on the node. An asynchronous suspend took a median of 0.32 seconds at the two lower levels and 0.54 seconds at the memory level, with 10 suspends each. Raw runs

A resume at the two lower levels is a cold start of the container. The snapshot is a few KiB, so the cost is the start time of your application. Our Go actor starts in 0.3 seconds, and its resume took 0.5 seconds, twice the resume at the memory level. The kagent run was on a second x86 server guest (16 vCPU, 31 GB). There the harness image resumed in a median of 0.38 seconds under SNAPSHOT_FIDELITY_VOLUMES.

A Python agent is slower to start. We ran one on the same guest, on the cluster of the Part 9 demos, with three turns at each level. A turn after a suspend took 4.5 to 5.1 seconds under SNAPSHOT_FIDELITY_VOLUMES. It took 0.30 to 0.32 seconds under SNAPSHOT_FIDELITY_MEMORY. A stateful agent actor shows that run.

preferredFidelity is an upper limit, and a memory snapshot can restore as less. A memory snapshot records the CPU architecture, the CPU features, and the sandbox build of the worker that took it. It restores only on a worker that matches. If no such worker has room, Substrate restores the files on another worker and boots the processes new. We did not see that downgrade in any run. Write your actor so that it survives a boot from its files, also at the memory level.

A snapshot is as large as the memory that holds data

Start with the worst case, an actor whose memory is random bytes.

One file holds almost all of it.

211574784 /var/lib/ate/snapshots/a77c2723-c524-46d0-9a7e-caa97f36ab9c/local-checkpoint/…/pages.img

We measured these sizes with du on the kind node of the x86 server guest, one actor for each size. The memory was random bytes. The snapshot folder of an actor is named by the uid of the actor. Raw runs

Random bytes are the largest snapshot that an actor of that size can have. We ran the same actor again with zero bytes, and with half zero and half random bytes.

We measured six actors on the x86 server guest (24 vCPU, 47 GB, nested KVM), one for each row. Each actor had 10 resumes and 10 asynchronous suspends. All 60 resumes returned 200, and every count continued. Each size is one du reading, on the node and in the data folder of the object store. Raw runs

So the size follows the memory that holds data, and the resume time follows the same bytes. An actor with 500 MiB of zero pages resumed in 0.23 seconds, as fast as an actor that holds nothing. Do not count on that for a real heap. A Python process with a dict of one million random values reported a peak of 245 MiB. Its snapshot on the node was 215.2 MiB. Section 3.3 has its resume time.

The object store did not make the random snapshots smaller. Substrate compresses each object before the upload, and random bytes do not compress. The copy of the Python actor was 110.1 MiB in the object store, about half of its node copy. We did not find out which step made it smaller.

We measured this footprint on the x86 server guest (24 vCPU, 47 GB, nested KVM). The node had five actors for each of the five sizes, which hold 8,750 MiB together. Each actor was 3 to 8 minutes old. The 2,064 MiB that stayed in the object store belong to golden snapshots. A template keeps its golden snapshot until you delete the template. Raw runs

An actor owns one snapshot in the object store, or two after an asynchronous suspend. A synchronous suspend deletes the earlier snapshot. An asynchronous suspend keeps it, because the new snapshot is not yet durable. The first suspend after the new snapshot is durable deletes the old one, and a delete of the actor removes both.

So plan the object store for up to two times the memory that holds data in all suspended actors. Add one golden snapshot for each template. Plan the node disk for the actors that were suspended in the last 30 minutes. Add the snapshot cache, which still held 1,765 MiB here after we deleted all 25 actors.

Two timers decide where the snapshot is

Node-local snapshot: the copy that the worker writes to the disk of its node at a suspend. atelet owns it.

Durable snapshot: the copy in the object store. Its state is SNAPSHOT_STORAGE_STATE_PENDING until the upload is complete, and then SNAPSHOT_STORAGE_STATE_COMPLETED.

After an asynchronous suspend, atelet runs two timers for each snapshot.

• --snapshot-upload-delay, 5 minutes by default: the wait before the upload to the object store.
• --snapshot-detach-delay, 30 minutes by default: the time until atelet removes the copy on the node and clears the node assignment of the actor.

We suspended five actors and read their state every 10 seconds for 42 minutes. The fifth actor, at 1,000 MiB, never got a worker, because the pool had no room. It made no snapshot of its own, so the numbers below are for four actors.

2026-11-12T10:31:10.270Z	d0	suspended-async	ACTOR_STATE_SUSPENDED	kind-control-plane	…SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_PENDING:gen2:kind-control-plane
…
2026-11-12T10:36:01.379Z	d0	state	ACTOR_STATE_SUSPENDED	kind-control-plane	…SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_PENDING:gen2:kind-control-plane
2026-11-12T10:36:12.367Z	d0	state	ACTOR_STATE_SUSPENDED	kind-control-plane	…SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:kind-control-plane

2026-11-12T11:01:02.649Z	d0	state	ACTOR_STATE_SUSPENDED	kind-control-plane	…SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:kind-control-plane
2026-11-12T11:01:13.670Z	d0	state	ACTOR_STATE_SUSPENDED	-	…SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:-

We measured four actors on the x86 server guest (24 vCPU, 47 GB, nested KVM), with the default timers. They held 0, 50, 200, and 500 MiB. The upload was complete 301 to 311 seconds after the suspend. The node copy was gone 1,801 to 1,803 seconds after the suspend. At that moment the snapshots on the node shrank from 758 MiB to 1 MiB. Each time can be up to 10 seconds late, because that was our sample interval. Raw runs

The three periods mean three different things for you.

1. In the first 5 minutes, the node holds the only copy. If that copy is lost, the actor is ACTOR_STATE_CRASHED, and the work since the last durable snapshot is gone. Section 3.5 shows that loss.
2. From 5 to 30 minutes, both copies exist. A resume uses the node and takes the lower line of section 3.3.
3. After 30 minutes, only the object store has the snapshot. A resume downloads it first and takes about twice as long for each MiB. That number is for a bucket in the cluster, and section 8.1 shows a bucket that is 20 ms away.

The documentation promises neither the start nor the end of the upload. Treat 5 minutes as the shortest exposure, not as a limit.

Our position: do not accept the first period for state that matters. A synchronous suspend closes it for one actor. A shorter --snapshot-upload-delay on atelet makes it shorter for the cluster, and does not close it. We ran atelet with 3 seconds and 10 seconds for one test, and all 55 requests of that test succeeded. A short detach delay saves node disk and makes every resume a download.

A tag is a copy that does not depend on the node

Tag: a named copy of the durable snapshot of a suspended actor. kubectl ate create tag <name> --actor <actor> makes one. If the snapshot is not uploaded yet, the call uploads it first. kubectl ate create actor <new> --tag <name> creates a new actor that starts from the copy.

A tag has its own object in the object store, under the atespace and outside the folder of the actor. So it does not need the node of the actor, and it does not change when the actor runs on. That makes a tag the one way to move the state of an actor off a node that has stopped.

ATESPACE   NAME                                   SCOPE                 STATE   SNAPSHOT                                                                       FIDELITY                   AGE
lab-n      09fee120-cd19-4121-8bae-5623fcbe70eb   TAG_SCOPE_PUBLISHED   Ready   gs://ate-snapshots/atespaces/lab-n/tags/1010ca2e-25b0-44d6-864f-a31a80da48c2   SNAPSHOT_FIDELITY_MEMORY   147m
lab-n      d8328610-2fdb-4974-a6ab-7c8671586d37   TAG_SCOPE_PUBLISHED   Ready   gs://ate-snapshots/atespaces/lab-n/tags/8d60620d-653e-469f-9dd1-9f11a89ce7e0   SNAPSHOT_FIDELITY_MEMORY   147m
lab-n      keep-t1                                TAG_SCOPE_ATESPACE    Ready   gs://ate-snapshots/atespaces/lab-n/tags/aca8c8e0-6dfd-4ee5-9e55-ced17f7c346e   SNAPSHOT_FIDELITY_MEMORY   12m
lab-n      keep-u1                                TAG_SCOPE_ATESPACE    Ready   gs://ate-snapshots/atespaces/lab-n/tags/13e4dabd-ad64-417f-95a5-ec1d5909c786   SNAPSHOT_FIDELITY_MEMORY   72s
lab-n      pend-u1                                TAG_SCOPE_ATESPACE    Ready   gs://ate-snapshots/atespaces/lab-n/tags/cfefcc57-5a75-418c-ae29-424958706266   SNAPSHOT_FIDELITY_MEMORY   69s

We measured three tags in the node-stop cases, on a second x86 server guest (16 vCPU, 23 GB), with one run for each case. A tag of an actor that was suspended with --mode sync was Ready in less than 1 s. So was a tag of an actor whose newest snapshot was still PENDING on the node, and it held that newest state. Then we stopped the node. A new actor from each tag served its first request 1.2 s after the create command, on the other node. That first request took 0.48 to 0.58 s. Raw runs

A tag has three limits. It is as old as the suspend that it copied, so later work is not in it. The actor that you create from it has a new name. And you cannot make it after the fact: create tag on an actor that was held on the stopped node failed with must be ACTOR_STATE_SUSPENDED to be tagged. Section 8.2 uses tags in its runbook for a stopped node.

Do not delete a golden tag

The golden tag of a template is in the same list. Its scope is TAG_SCOPE_PUBLISHED, and its name is a uid. kubectl ate delete tag accepts it, and our own clean-up made that mistake. It deleted every tag of the atespace, and the golden tags of two templates went with them.

## get actor-templates
ATESPACE   NAME         GOLDEN TAG                             ERROR   AGE
lab-n      ballast-0    d8328610-2fdb-4974-a6ab-7c8671586d37           170m
lab-n      ballast-50   09fee120-cd19-4121-8bae-5623fcbe70eb           170m
## get tags
ATESPACE   NAME   SCOPE   STATE   SNAPSHOT   FIDELITY   AGE
## create actor from template ballast-50
Error: failed to create actor: rpc error: code = FailedPrecondition desc = Tag not found
…

The template looks healthy and is broken. get actor-templates still lists the old golden tag id, and the ERROR column is empty. Each create actor --template fails with Tag not found. We saw this once, on the same guest. We made new templates to continue, and we did not test whether a golden tag can be restored.

A node of your cluster restarts in the night. In the morning, some sessions answer 503 with a text about AssignWorker. One answers 503 with a text about an upstream connection. kubectl get pods shows every pod as Running. Substrate will not repair any of these sessions by itself, and a new request will not either.

When you finish this section, you can tell from an error text which state an actor is in. You can also bring the actor back with RevertActor and say which calls it lost.

Every way we reached CRASHED

ACTOR_STATE_CRASHED: the state of an actor whose sandbox, worker, or restore was lost. The actor has no worker. Its status.crash field holds a time and a message.

The first row is the plain case. The actor object says what happened.

status:
  assignedNode: kind-control-plane
  crash:
    crashTime: "2026-11-12T10:30:11.170436971Z"
    message: worker pod went away while hosting the actor
  lastAssignedGeneration: 3
  …
  state: ACTOR_STATE_CRASHED

The second and third rows are the cost of an asynchronous suspend. In both, the newest snapshot was SNAPSHOT_STORAGE_STATE_PENDING, so one node held the only copy. In both, the actor still looked healthy: its state was ACTOR_STATE_SUSPENDED. The request that tried to resume it found the loss and crashed it.

2026-11-12T11:30:18.606Z	x1	removed-node-local-snapshot	uid 085b45fb-edb9-4523-ad30-3da55b3f34f0
2026-11-12T11:30:18.708Z	x1	call-after-loss	500	0.082402	error resuming actor lab-c/x1
2026-11-12T11:30:18.911Z	x1	state	ACTOR_STATE_CRASHED	kind-control-plane	…
2026-11-12T11:30:23.997Z	x1	call-again	503	0.063957	actor lab-c/x1 unavailable: AssignWorker prerequisite not met for Actor: lab-c/x1 (got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPENDED)
…
2026-11-12T11:30:25.241Z	x1	call-after-revert	200	0.416152	count=4 mb=50 fill=random sum=6296 process_age_s=0.8
2026-11-12T11:30:25.434Z	x1	state	ACTOR_STATE_RUNNING	kind-control-plane	…

Substrate did not use the older, completed snapshot by itself. The actor had one in the object store, and the request still failed.

The node restart has a detail that the table hides.

We measured one node restart on the x86 server guest. The Kubernetes API answered after 8 seconds, and all pods ran after 147 seconds. We sent the request less than two seconds later. The crash message does not name a lost snapshot. It says that the restore failed three times because the worker had not registered yet. So a request that arrives too early after a node restart can crash a suspended actor, and we did not test a later request. Raw runs

One event did not crash anything. We deleted the atelet pod while a snapshot was pending, and the next request returned 200 with the count intact. The snapshot is on the disk of the node, not in atelet.

What the caller sees

Two texts tell you that an actor is crashed.

• error resuming actor <atespace>/<name>, with status 500. Only the request that finds the loss gets this text.
• actor <atespace>/<name> unavailable: AssignWorker prerequisite not met for Actor: <atespace>/<name> (got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPENDED), with status 503. Every later request gets this text, and it arrives at once.

kubectl ate resume actor is refused with the same text and the gRPC code FailedPrecondition.

2026-11-12T10:30:17.301Z	resume-while-crashed	0.207s	Error: failed to resume actor: rpc error: code = FailedPrecondition desc = AssignWorker prerequisite not met for Actor: ate-demo-counter/my-counter-1 (got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPEN

No retry helps. Do not put this 503 in the same retry loop as a 503 from a full pool.

Running in the state, dead in the sandbox

One failure does not reach ACTOR_STATE_CRASHED at all. We told an actor to call os.Exit(3).

2026-11-12T11:30:45.229Z	j1	exit-request	200	0.065314	exiting with code 3
2026-11-12T11:30:47.441Z	j1	state	ACTOR_STATE_RUNNING	kind-control-plane	…
2026-11-12T11:30:47.518Z	j1	call	503	0.058459	upstream connect error or disconnect/reset before headers. reset reason: connection termination
…
2026-11-12T11:31:00.122Z	j1	suspend-attempt	Error: failed to suspend actor: rpc error: code = Internal desc = internal server error: workflow failed at step CallAteletSuspend: actor lab-c/j1 crashed…
2026-11-12T11:31:00.333Z	j1	state	ACTOR_STATE_CRASHED	kind-control-plane	…

We measured one actor for each kind of death on the x86 server guest. After the exit, the state was ACTOR_STATE_RUNNING in 6 of 6 samples over 14 seconds, and all 6 requests got the 503 from Envoy. A memory kill is different: Substrate recorded the crash 2.3 seconds after the request that started the allocation. One request in that gap got the Envoy text. Raw runs

Fourteen seconds do not prove much, so we repeated the exit and watched for 20 minutes. One actor got no request. A second actor got one request each 60 seconds.

2026-11-12T12:45:20.264Z	e1	exit-request	200	0.068987	exiting with code 3
2026-11-12T12:45:36.467Z	e1	state	ACTOR_STATE_RUNNING	kind-control-plane	…
…
2026-11-12T13:05:57.588Z	e1	call-after-watch	503	0.070948	upstream connect error or disconnect/reset before headers. reset reason: connection termination
2026-11-12T13:05:57.886Z	e1	suspend-attempt	Error: failed to suspend actor: rpc error: code = Internal desc = internal server error: workflow failed at step CallAteletSuspend: actor lab-v/e1 crashed…
2026-11-12T13:05:58.168Z	e1	state	ACTOR_STATE_CRASHED	kind-control-plane	…
2026-11-12T13:05:58.604Z	e1	after-revert	ACTOR_STATE_SUSPENDED	-	…

We measured two actors on the x86 server guest (24 vCPU, 47 GB, nested KVM), for 20 minutes 36 seconds after the exit. Both were ACTOR_STATE_RUNNING in every sample, 81 and 80 samples at 15 second intervals. All 20 requests to the second actor returned the 503 from Envoy. The first suspend after the watch failed and marked the actor crashed. A revert returned it to ACTOR_STATE_SUSPENDED, and the next call returned 200 with the count of the last durable snapshot. Raw runs

So Substrate did not find a process exit by itself in 20 minutes, with requests or without. Each dead actor also kept one CPU of its worker for all that time, which section 4.5 counts. We did not watch longer, and we did not restart the worker.

In both runs a suspend found the exit. The checkpoint failed, and the failed suspend marked the actor crashed.

Your harness must cover the exit case. If requests return upstream connect error and the state is ACTOR_STATE_RUNNING, call SuspendActor and expect it to fail. A harness that suspends after every turn does this without extra code.

An actor on a node that stops

A node that stops and stays down gives three more cases, and none of them starts as ACTOR_STATE_CRASHED. We ran them on a cluster with two worker nodes and stopped one of the two. Placement has the timeline and the scheduler side.

We measured these cases on the second x86 server guest (16 vCPU, 23 GB), with gVisor and one actor for each case. A suspended actor with a complete snapshot stayed in ACTOR_STATE_RESUMING for 5 min 33 s from its first request. It was crashed 5 min 44 s after the stop command. A suspended actor with a pending snapshot was still in ACTOR_STATE_RESUMING 10 min 42 s after the stop command. It had received no request in the last 8 min 43 s. A running actor stayed ACTOR_STATE_RUNNING for the 10 minutes and the 150 requests of our probe. Raw runs

2026-11-12T12:22:55.258Z	node-stop	docker stop multi-worker2
2026-11-12T12:23:15.930Z	first-request	504	10.054853		|	ACTOR_STATE_RESUMING	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:-,SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_PENDING:gen3:multi-worker2
2026-11-12T12:23:16.115Z	revert-attempt-1	Error: failed to revert actor: rpc error: code = Aborted desc = another operation is in progress for this actor 
…
2026-11-12T12:33:37.802Z	revert-attempt-2	Error: failed to revert actor: rpc error: code = FailedPrecondition desc = Actor lab-n/p3 is not in a revertable state (got: ACTOR_STATE_RESUMING, want one of [ACTOR_STATE_RUNNING ACTOR_STATE_CRASHED], or ACTOR_STATE_SUS
2026-11-12T12:33:38.407Z	probe-after-revert	503	0.193436	actor lab-n/p3 unavailable: AssignWorker prerequisite not met for Actor: lab-n/p3 (got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPENDED)	|	ACTOR_STATE_CRASHED	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:-,SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_PENDING:gen3:multi-worker2

Three rules follow from these runs.

• ACTOR_STATE_RESUMING that lasts is a symptom. A resume of a small actor takes less than a second, and this state lasted for minutes. In the first run the crash came one second after Kubernetes evicted the worker pods, 5 minutes after the node was NotReady. Without a request, nothing moved the actor out of the state.
• A running actor on a stopped node is not marked crashed. The caller gets the text of a process that exited. To tell the two cases apart, run kubectl ate get workers. On a stopped node the worker is WORKER_STATE_UNAVAILABLE, or it is not in the list.
• A revert did not help while the node was down. It failed or had no lasting effect in 4 of 4 attempts on three actors. The revert of the running actor left it in ACTOR_STATE_REVERTING. Each request then got 503 with (got: ACTOR_STATE_REVERTING, want ACTOR_STATE_SUSPENDED).

A second revert is the way out of ACTOR_STATE_REVERTING, and only after the node is back. RevertActor accepts an actor that is already in that state, and it continues from the step where the earlier attempt stopped. A later set of node-stop cases tested it.

We measured one actor that was suspended with --mode sync, on a second x86 server guest (16 vCPU, 23 GB), in one run with five reverts. The first revert started 10 s after the stop command, failed after 20.2 s, and left the actor in ACTOR_STATE_REVERTING. Three more failed in 0.2 s each, at 50 s, at 71 s, and at 371 s. Then we started the node. Its atelet ran 15 s later, and the actor was ACTOR_STATE_CRASHED by then. The fifth revert returned in 0.2 s, and the next request returned 200 in 0.47 s on the other node. Raw runs

2026-11-12T14:51:05.501Z	t=30.6	revert-1-right-after-stop	20.2 s | Error: failed to revert actor: rpc error: code = Unavailable desc = while calling atelet Terminate: …
2026-11-12T14:51:24.853Z	t=50.0	revert-2-after-workers-unavailable	0.2 s | Error: failed to revert actor: rpc error: code = Unavailable desc = while calling atelet Terminate: …
…
2026-11-12T14:56:45.527Z	t=370.7	revert-4-after-eviction	0.2 s | Error: failed to revert actor: rpc error: code = Unavailable desc = while calling atelet Terminate: …
…
2026-11-12T14:57:00.955Z	t=386.1	atelet-on-node-running	atelet-gsmxg:Running atelet-tfq5g:Running 
2026-11-12T14:57:01.180Z	t=386.3	state-before-revert-5	ACTOR_STATE_CRASHED	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2
2026-11-12T14:57:01.440Z	t=386.6	revert-5-after-node-return	0.2 s | ATESPACE   NAME   TEMPLATE      STATE                   WORKER POD   WORKER IPS   VERSION   AGE lab-n      i9     lab-n/nb-50   ACTOR_STATE_SUSPENDED   <none> …

So a repeated revert does not finish the job while the node is down. In the other cases, 3 to 28 more reverts of an actor in ACTOR_STATE_REVERTING all failed. Each failed revert is one more call to an atelet that is not there.

The result after the return has a caveat. In four earlier cases the actor still showed ACTOR_STATE_REVERTING 216 to 223 s after the node start, and we sent no command in that time. In those cases the pool pods had been evicted, and the node came back with no pool pod. Our clean-up then removed those actors, and we did not record each result. So the state cleared by itself after the return in one run, and it did not in four. After a node returns, look at the state, and send the revert again if it is still ACTOR_STATE_REVERTING.

The path that the first runs left open is also measured. A revert that waits until the workers show WORKER_STATE_UNAVAILABLE frees one kind of actor: suspended with --mode sync, and with no request since the stop. It took 7.1 s, and the next request ran on the other node. Section 8.2 has the runbook for every case.

Do not revert an actor that you suspended in the default mode. Its newest snapshot is still PENDING on the stopped node, and it comes back with the node. In one run the node returned after 81 s. A request with no revert then returned 200 in 0.36 s, with count 6 after 5. A second actor in the same state got a revert first. Its next reply had count 4, so the two calls after its last durable snapshot were lost.

RevertActor

RevertActor: the call that discards the current execution of an actor and returns it to ACTOR_STATE_SUSPENDED at its last completed durable snapshot. It accepts a crashed actor, a running actor, and a suspended actor whose newest snapshot is on a node. It also accepts an actor in ACTOR_STATE_REVERTING, so that you can send a revert again. In our run that second revert worked only after the node of the actor was back.

A revert stops any sandbox, removes the snapshot files on the node, and deletes an upload that was not complete. The actor comes back with no assigned node, and the next request resumes it from the object store.

The calls in the third column are gone. The actor answers as if they never happened.

We measured one revert of the counter actor on the x86 server guest: kubectl ate revert actor returned in 0.22 seconds. The first request after a revert took 0.38 to 0.42 seconds in three runs. After the node restart, that first request waited 5 seconds and got no worker has room for the actor, although all pods ran. A request 78 seconds later returned 200. Placement explains that 503: for some seconds after the control plane returns, no worker of the pool is active. Raw runs

A revert rewinds the actor only. A tool call that the actor made after the snapshot stays made, and an external volume keeps its contents. We did not test external volumes.

Our position: decide in your harness what a crash means, because Substrate will not. After our node restart, my-counter-1 stayed crashed until we reverted it by hand. A revert is safe to automate for an actor whose last calls can be sent again without a second effect. A tool call that already ran stays done, so the tool must ignore a repeat. For any other actor, suspend with --mode sync at each point that you cannot lose.

You ran kubectl get ns to find where your actors live. You found a namespace with the name of your demo, with worker pods in it and no actor. Then you created a second atespace for a new team, and its template still ran on the pool of the first team. Both results follow from one fact: Substrate keeps two lists of names, and it connects them with labels only.

When you finish this section, you can create an atespace and name an actor and a template in it. You can also select a worker pool from a template, and find a snapshot in the object store from its URI.

An atespace is not a namespace

Atespace: the group that an actor, an ActorTemplate, and a tag belong to in the Substrate API. An atespace is a row in Postgres. It must exist before you create anything in it, and you can delete it only when it is empty.

Our test setup made both kinds of object with one name. That is our choice, and Substrate did not ask for it.

kind: Namespace
metadata:
  name: lab-c
…
kind: WorkerPool
metadata:
  name: ballast
  namespace: lab-c
  labels:
    workload: ballast
…
kubectl apply -f pool.yaml
kubectl ate create atespace lab-c 2>&1 | tail -1

kubectl apply wrote the namespace and the pool to Kubernetes. kubectl ate create atespace wrote one row through ate-api-server. The quickstart demo does the same with the name ate-demo-counter, so a new user sees one name and assumes one object.

The atespace is the boundary for access. A role in an atespace covers every actor and every template in it, and no grant exists for one actor. Do not use the name of a deleted atespace again. The documentation warns that a request authorized for the old atespace can reach the new one.

Pool selection crosses namespaces

A template selects its workers with workerSelector.matchLabels. The scheduler compares those labels with the labels on the WorkerPool object. It does not read the labels of the pods, and it does not read the namespace of the pool. Each worker carries a copy of the labels of its WorkerPool object, and the scheduler matches against that copy.

Three rules follow from the code and the documentation.

• Only equal labels match. No set expressions exist.
• An actor can add its own selector. The scheduler applies both selectors together, so the selector of the actor only narrows the choice. A change takes effect at the next resume.
• A template with no selector can run on every pool.

So the namespace of a pool is not a fence. A template in any atespace can name the labels of any pool, and Kubernetes RBAC on that namespace does not stop it. No document at v1.0.0 states this rule. It follows from the code, and our run below confirms it.

We ran it. We created an atespace xa and no Kubernetes namespace of that name. A template in xa selected workload: vpool, the label of a pool in the namespace lab-v.

ATESPACE   NAME   TEMPLATE     STATE                 WORKER POD                     …
xa         x1     xa/cross-0   ACTOR_STATE_RUNNING   lab-v/vpool-69595c64bf-p4slt   …

The template built its golden snapshot on that pool, and the actor ran on a worker pod of lab-v.

A selector that matches no pool is the opposite mistake, and Substrate does not refuse it. We created a second template in xa with the label workload: no-such-pool. The create call succeeded. The template never got a golden tag, and its ERROR column stayed empty. A request to an actor of that template failed.

2026-11-12T12:51:40.304Z	call-unmatched-selector	503	5.060180	actor xa/x2 unavailable: no worker has room for the actor

We measured both templates once on the x86 server guest (24 vCPU, 47 GB, nested KVM). The first request to the actor on the pool of lab-v returned 200 in 0.26 seconds. The request with the wrong selector waited 5.06 seconds for its 503. Raw runs

That is the text of a full pool, word for word. ate-api-server logged the same reason for the template 8 times in 40 seconds of its log, and the template showed no error. When you see no worker has room for the actor, compare the selector with the pool labels before you add workers. Section 8.3 has the row.

kagent shows the convention that fills this gap. It writes the name and the namespace of the pool into two labels, and its templates select both.

workerSelector:
  matchLabels:
    kagent.dev/worker-pool-name: kagent-default
    kagent.dev/worker-pool-namespace: kagent

The second label looks like a namespace check, and it is a label like every other. A pool in another namespace with the same two labels would match.

The worker list has one filter for each list of names.

-a, --atespace string        filter worker pods hosting actors in a specific atespace
…
-n, --namespace string       scope output to a specific Kubernetes namespace
--sandbox-class string   filter by sandbox class (e.g. gvisor, microvm)
-l, --selector string        filter by worker pool labels

Names

Every name in the Substrate API is a DNS label: lowercase letters, digits, and hyphens, at most 63 characters. A name is fixed at create time.

The actor address is the form that you will read most. Error texts use it, kubectl ate takes its two halves as -a and the name, and the router reads it from a request header. Section 6.1 covers that header.

The uid matters when you delete an actor and create it again with the same name. The new actor gets a new uid. Snapshot paths use the uid, so the new actor cannot resume from the state of the old one.

The snapshot URI layout we saw

Each snapshot in an actor object has a snapshotUri. The atespace and the owner are part of the path.

snapshotUri: gs://ate-snapshots/atespaces/ate-demo-counter/tags/ad19ee28-1519-432a-bcc5-40cf99cce3ca
…
snapshotUri: gs://ate-snapshots/atespaces/ate-demo-counter/actors/6f80f612-7547-4f5b-b3ab-831595621806/snapshots/8faccdff-15cc-493d-8474-5ebcb1c18d56

We saw three details that the layout does not tell you.

• On a kind install the location is gs://ate-snapshots/, although the store is the rustfs pod in the cluster.
• kagent sets a location for each template, and its URIs start with s3://ate-snapshots/kagent/. That field is a preview feature in the upstream project.
• The last part of a golden URI is the uid of the tag. It is not the name of the tag, which is the uid of the template.

The atespace is in the path for a reason. An object store can grant access by path prefix, so you can give one team read access to the snapshots of one atespace. We did not test such a grant.

Our position: use one name for the atespace and the namespace when one team owns both, as the demos do. Then treat the labels of your pools as placement, and not as a fence. Any atespace can name them, and we found no setting at v1.0.0 that limits a pool to one atespace. Choose the labels on purpose, and check which templates name them.

You create an actor, send it a request, and look for its pod. There is no pod with the name of the actor. The actor runs inside a worker pod that existed before the actor did, and that hosts other actors too. The WorkerPool is the object that made that pod.

When you finish this section, you can write a WorkerPool, read the output of kubectl ate get workers, and say where each capacity number comes from.

Three CRDs, and no Actor among them

WorkerPool: a namespaced Kubernetes object that asks for a number of worker pods of one sandbox class.

Substrate adds three CRDs to your cluster, all in the group ate.dev at version v1. A WorkerPool is the only one you write often. A SandboxConfig names the sandbox binaries for a class, and the install creates a default for each class. A CSIDriverConfig belongs to a preview feature for external volumes.

csidriverconfigs.ate.dev   Cluster      v1(storage)   2026-11-12T10:05:33Z
sandboxconfigs.ate.dev     Cluster      v1(storage)   2026-11-12T10:05:33Z
workerpools.ate.dev        Namespaced   v1(storage)   2026-11-12T10:05:33Z

Actors, ActorTemplates and atespaces are absent from that list. They are rows in Postgres that ate-api-server serves, so kubectl get cannot show them. Use kubectl for the pool and kubectl ate for all that runs on the pool.

The pool we used

This is the pool behind most measurements in this manual. It asks for two gVisor workers, each with 2 CPUs and 3 GiB.

apiVersion: ate.dev/v1
kind: WorkerPool
metadata:
  name: ballast
  namespace: lab-c
  labels:
    workload: ballast
spec:
  replicas: 2
  workerImage: localhost:5001/ateworker-gvisor@sha256:ad6957f41bccaa8bb5f3eb5b34b7812babb862f1a3187a11578c21a7d799491c
  sandboxClasses:
  - name: gvisor
  template:
    resources:
      limits:
        cpu: "2"
        memory: 3Gi
      requests:
        cpu: 500m
        memory: 3Gi

Four fields carry the meaning:

• spec.replicas is the number of worker pods. The pool has a scale subresource, so kubectl scale and an autoscaler can change it.
• spec.workerImage is the worker binary for the class. Use ateworker-gvisor for gvisor and ateworker-microvm for microvm.
• spec.sandboxClasses takes exactly one entry at v1.0.0, and you cannot change its name later. Sandbox classes compares the two.
• spec.template.resources.limits is the capacity of each worker. It is the most important field in the object.

The label workload: ballast sits on the pool object, under metadata.labels. An ActorTemplate selects pools with workerSelector.matchLabels, and the match reads the labels of the pool object. Labels under spec.template.labels go to the Deployment and its pods, and they do not affect which actors the pool accepts.

A selector is the only link between a template and a pool. The namespace of the pool and Kubernetes RBAC do not limit which templates can select it. An actor can add its own workerSelector, which narrows the selector of the template and takes effect at the next resume. Put a label on each pool that only the intended templates use.

From the pool to a registered worker

Four steps separate your YAML from a worker that can take an actor:

1. ate-controller reads the WorkerPool and writes a Deployment with the same replica count. Each pod has the label ate.dev/worker-pool.
2. ate-controller watches those pods and creates one Worker record for each in ate-api-server. The name of the record is the UID of the pod.
3. atelet on the node registers the worker and reports which sandbox runtimes it can run.
4. When the pod is Ready, the Worker moves to WORKER_STATE_ACTIVE. Only an active worker receives actors.

Worker: the record in ate-api-server for one worker pod, with its state, its capacity, and the resources that actors have booked on it.

A Worker has three states. WORKER_STATE_UNAVAILABLE means registered and not Ready. WORKER_STATE_ACTIVE means ready for actors. WORKER_STATE_DRAINING means the pod is terminating, and a worker never leaves that state.

A terminating worker pod is slow on purpose. The controller sets its termination grace period to 3,600 seconds, and the worker gives its actors up to 30 minutes to stop. A rollout of a pool with running actors can take that long.

What kubectl ate get workers shows

We ran the command when four actors with 1 CPU each were running on the pool.

NAME                                   POOL      STATE                 ACTORS   CPU(MILLICORES)   MEMORY         POD                              AGE
b17a6492-9643-42f1-bfdb-8bb88ee46772   ballast   WORKER_STATE_ACTIVE   2/1000   2000/2000         512Mi/3008Mi   lab-c/ballast-65cf95d875-bzgjd   114s
ee4ba8b4-dbfa-4555-a621-7cf7662c1091   ballast   WORKER_STATE_ACTIVE   2/1000   2000/2000         512Mi/3008Mi   lab-c/ballast-65cf95d875-f65dl   114s

Each of the three middle columns is a pair: booked, then capacity.

The worker kept 64 MiB of its 3 GiB limit, so it reported 3008Mi. The demo pools with a 1 GiB limit reported 960Mi, the same 64 MiB less.

The booked side is the sum of the limits that each running actor declares in its ActorTemplate. Each actor here declared 1000 millicores and 256 MiB. Two actors booked the whole CPU capacity and a sixth of the memory. The worker is full, and CPU is the reason.

cpuMillicores has two jobs. The worker books it for placement, and the worker also writes it into the sandbox as a CPU quota for each 100 ms period. The documentation says that the node kernel slows the actor to that quota.

The quota is a hard cap. We tested it with a program that runs a fixed number of loop steps, about 5 seconds of work on one free core. It ran on 1 thread and on 4 threads, under four templates. We read the CPU time of the actor from its cgroup on the node, before and after each run.

We measured three runs for each row on a second x86 server guest (16 vCPU, 23 GB). The pool had one worker pod with a Kubernetes limit of 6 CPU. Raw runs

The actor never used more host cores than its limit. Up to 1000 millicores, four threads did not get more CPU than one thread. The same work took 5.4 times longer at 250 millicores than at 1000. In one row the limit was not the bound: one thread at 2000 millicores used 1.12 cores, because one thread fills one core. Four threads used the whole limit there.

The cap is a cgroup for each actor, inside the cgroup of the worker pod. The worker pod had cpu.max=600000 100000, which is its Kubernetes limit of 6 CPU. Inside it we found one cgroup for each actor, named with the uid of the actor and the suffix _sandbox. Its cpu.max was the template limit, and the gVisor processes of the actor ran in it.

250 25000/100000 usage_usec 41467478 nr_periods 1672 nr_throttled 1649 throttled_usec 428931388 
500 50000/100000 usage_usec 38723256 nr_periods 793 nr_throttled 765 throttled_usec 157096786 
1000 100000/100000 usage_usec 31895780 nr_periods 345 nr_throttled 312 throttled_usec 30529410 
2000 200000/100000 usage_usec 23220151 nr_periods 204 nr_throttled 38 throttled_usec 854079 

Two facts about the inside of the sandbox follow, and both can mislead you.

• The actor does not see its limit. The program counted 2 CPUs at all four limits, and it could not read the cgroup files. A runtime that sizes a thread pool from the CPU count gets 2 at 250 millicores.
• CPU time read inside a throttled sandbox is wrong. At 250 millicores with 1 thread, the program reported 22 to 24 seconds of CPU time. The same run used 6.9 seconds on the host. Do not use getrusage or /proc inside an actor to measure CPU use. Read the cgroup on the node.

The second fact explains an earlier reading of ours. A demo command reported 2.0 seconds of CPU time in 2.38 seconds under 500 millicores, which looked like no cap. That CPU time came from inside the sandbox, so it does not count. A tool sandbox has that run and the correction.

We tested one actor at a time. We did not test memory limits, or CPU limits with several busy actors on one worker. One row is not explained. Four threads at 2000 millicores used 2.8 seconds of host CPU, and 1 thread used 4.6 to 5.1 seconds for the same work.

A suspended actor books nothing. The command counts running actors only.

Two habits from kubectl do not carry over. The command has no -A flag, and it lists the workers of every namespace by default. It has four filters: -n for one Kubernetes namespace, -l for pool labels, -a for the workers that host actors of one atespace, and --sandbox-class.

A pool without limits reports the whole node

kagent v1.0.0 installs a pool named kagent-default with no resources at all. We listed its worker on a guest with 16 vCPUs and 31 GiB.

NAME                                   POOL             STATE                 ACTORS   CPU(MILLICORES)   MEMORY         POD                                      AGE
6dfb9d1d-cce7-4e42-af53-799a4132fa5d   kagent-default   WORKER_STATE_ACTIVE   0/1000   0/16000           0/32797744Ki   kagent/kagent-default-679c58c6cb-pljdk   19m

The worker claims all 16 CPUs and all the memory of the node. The Substrate documentation says that a worker without limits reports zero and has no bound. Our run disagrees: the worker read the size of the node and reported that. A user reported the same on a larger cluster.

The result is a capacity that nothing backs. The pod has no limit, so Kubernetes reserves nothing for it. A second pool on the same node would claim the same 16 CPUs again. The control plane, the dataplane and Postgres run on that node too. kagent 1.0 on Substrate shows what that worker did under load.

Our position: always set limits on a pool, and set the memory request equal to the memory limit. Then the memory that the worker reports is memory that Kubernetes has reserved. Kubernetes reserves CPU only up to the CPU request. The example above requests 500 millicores and reports 2 CPUs. Set the CPU request equal to the limit when actors must get the CPU that they book. Sizing a pool turns that rule into numbers.

One more fact matters before you give a pool to a team. A worker pod runs as root with SYS_ADMIN and NET_ADMIN, and its seccomp profile is unconfined. The sandbox inside the pod isolates the actor. The pod does not isolate the worker from the node, so run pools on nodes that you trust.

A security reviewer asks one question about your agent platform: is each agent in a container or in a virtual machine? With Substrate the honest answer is neither one, and the choice is yours. An actor runs in a gVisor sandbox or in a small virtual machine. The choice is a field of the WorkerPool, and it is hard to change later.

When you finish this section, you can choose a class for a pool and check that a node can run it. You can also name the behaviours that differ between the two classes.

Most measurements in this manual come from the gvisor class. On the x86 server guest we also ran one test actor on both classes, at 0, 200, and 500 MiB of memory. This section reports that comparison, and it names what stays untested.

Two classes, one field

Sandbox class: the kind of isolation that the workers of a pool give to each actor. The value is gvisor or microvm.

You set the class in spec.sandboxClasses[0].name of the WorkerPool. The set is closed. The API accepts only those two names, and a plain Linux container is not an option.

What the node must provide

A gVisor worker runs on any Linux node. That is why it runs on a laptop.

A microVM worker needs the device /dev/kvm. atelet offers the device to Kubernetes as the resource ate.dev/kvm, only on nodes that have it. The controller adds a request for that resource to every microVM worker pod. So a microVM pool on a cluster without KVM does not fail loudly. Its pods stay Pending.

The script that creates the kind cluster tells you which case you are in. On the Apple silicon laptop under Docker Desktop it printed this:

Probing for /dev/kvm in the Docker environment...
/dev/kvm not available: micro-VM support disabled (gVisor still works).

On the x86 server guest, which has nested KVM, the same script printed the other answer:

Probing for /dev/kvm in the Docker environment...
/dev/kvm found: micro-VM (kata + cloud-hypervisor) support will be enabled.

On that guest the install needed no further step for microvm. The kind script had passed /dev/kvm to the node, and the installer had created the SandboxConfig.

So on a Mac under Docker Desktop you have the gvisor class only. On a cloud VM, the instance type must allow nested virtualization. Check the node type before you plan for microvm.

The installer prepares both classes by default. It creates one SandboxConfig for each class and loads the guest kernel and the other microVM files into the object store. Pass --sandbox-classes=gvisor to skip the microVM part.

How the limits apply

The ActorTemplate declares one CPU limit and one memory limit for the actor. The two classes turn them into different things.

CPU. A gVisor actor with a limit of 500 or 1000 millicores sees 2 CPUs, and the node kernel slows it to its quota. A microVM actor with 500 millicores starts with 1 virtual CPU and gets half a CPU of node time. A program that sizes its thread pool from the CPU count behaves differently on each class.

Memory. On gvisor the limit is the memory that the sandbox can use. On microvm the limit is the RAM of the guest, and the virtual machine monitor needs more memory on top. The worker counts that overhead against its capacity, so the same pool holds fewer microVM actors than the limits alone suggest. In our run the overhead was 7.6 MiB for an actor with a limit of 456 MiB.

Containers. An actor can have more than one container. On gvisor one Sentry process runs all of them, and a limit on one container has no effect. On microvm each container has its own cgroup in the guest. A container that passes its memory limit is killed alone, and the other containers of the actor continue.

That last behaviour has a cost that others reported. In issue 3615, a container in a microVM actor was killed for memory, and the actor stayed ACTOR_STATE_RUNNING with no log line. Actor and its seven states shows the same silence for a process that exits on gvisor, which we did measure.

A snapshot pins the class

The class is not only a property of the pool. The first snapshot of an actor records it, and every later restore of that actor needs a worker of the same class. This is the snapshot record of our counter actor after a synchronous suspend.

    fidelity: SNAPSHOT_FIDELITY_MEMORY
    generation: 3
    locality: kind-control-plane
    owner: SNAPSHOT_OWNER_ACTOR
    sandboxRuntime:
      sandboxClass: gvisor
      version:
        attributes:
        - key: architecture
          value: amd64
        - key: cpu_features
          value: 8c0000002e500800008003f70000000f00400004219c01ab178bfbfffff83203
        - key: sandbox_bundle_hash
          value: sha256:88e9809119d7fa2204897a60f8bb155add102018b4b4947e7876c9bc447312fa

A microVM snapshot records different attributes. This is the same part of the record for the counter actor on a microVM pool.

    fidelity: SNAPSHOT_FIDELITY_MEMORY
    generation: 5
    locality: kind-control-plane
    owner: SNAPSHOT_OWNER_ACTOR
    sandboxRuntime:
      sandboxClass: microvm
      version:
        attributes:
        - key: architecture
          value: amd64
        - key: cpu_model
          value: "23:49"
        - key: cpu_vendor
          value: AuthenticAMD
        - key: sandbox_bundle_hash
          value: sha256:1d40aaaf66896731f227d4cfe129ab724721b592ff89c73df603e8e2372a67b8

A gVisor snapshot names the CPU features as a bit string. A microVM snapshot names the CPU vendor and one CPU model. Three consequences follow from those records:

• You cannot move an actor to the other class. A template that selects pools of both classes still gives each actor one class, the class of its golden snapshot.
• A memory snapshot is tied to more than the class. It records the CPU architecture, the CPU features or the CPU model, and the exact sandbox build. A worker restores memory only when its runtime is compatible.
• The result of a mismatch is a cold start. When no compatible worker has room, Substrate takes a snapshot from the object store and restores only its files. It uses another worker of the same class. The processes start again, and the actor loses what it held in memory.

The third point is the one to plan for. A pool that mixes node types with different CPUs, or a rollout of a new sandbox build, can turn a resume into a cold start. Placement shows where that decision is made.

What others reported about parity

The two classes do not behave the same at v1.0.0, and the project tracks the gaps in issue 3226. These reports come from the issue tracker. We ran none of them.

The two classes, measured

We ran the test actor of SuspendActor and ResumeActor on a microVM pool. The machine, the node, and the actor image were the ones behind the gVisor numbers. The actor holds a fixed amount of random memory.

We measured both columns on the x86 server guest (24 vCPU, 47 GB, nested KVM), with one kind node and the Envoy dataplane. Each cell is a median. Each size had 5 first calls, 10 resumes, 10 asynchronous suspends, and 5 synchronous suspends. The first call and the resume are timed at the client, through kubectl ate proxy. All 90 requests of the microVM run returned 200. A request to a running actor took 0.065 seconds on microvm and 0.064 seconds on gvisor. Raw runs

The table has four results:

• A small actor pays a fixed cost on microvm. At 0 MiB the resume was 0.10 seconds slower and the first call was 0.22 seconds slower. The restore starts a guest with its own kernel before the actor runs.
• Memory costs less on microvm. From 0 to 500 MiB, a resume grew by about 1 ms for each MiB on microvm and by about 2 ms on gvisor. The medians were 0.02 seconds apart at 200 MiB. Above that size microvm was the faster class.
• A large new actor starts much faster on microvm. At 500 MiB the five first calls took 1.26 to 2.96 seconds on microvm and 1.86 to 6.76 seconds on gvisor.
• An asynchronous suspend costs the same. The medians differ by 0.06 seconds or less at every size. A synchronous suspend of a small actor is slower on microvm, because its snapshot is larger.

What one actor costs on the node

We measured these values on the same x86 server guest, with one sample for each cell. Resident memory is the value that ps printed on the node. The snapshot size is the disk use of the snapshot directory of one actor. The gVisor booking is from the pool of the second x86 server guest. Raw runs

A microVM snapshot holds the whole guest, with its kernel and its agent. It was 59 to 70 MiB larger than the memory that the actor held. Add that amount for each actor when you size the node disk and the bucket in sizing a pool. The file memory-ranges is sparse. Its listed size is the RAM of the guest, and its disk use is what the guest touched.

The clock inside the actor

The test actor reports the age of its process in every reply. The two classes count that age differently.

We measured three suspend and resume cycles of the 0 MiB actor on each class, on the same guest. One cycle took 1.6 to 1.8 seconds. On microvm the age grew by 1.8 or 1.9 seconds in each cycle: 28.4, 30.3, and 32.1 seconds. On gvisor it grew by 0.2 or 0.3 seconds in each cycle: 0.5, 0.8, and 1.0 seconds. Raw runs

So a microVM actor sees the time that passed while it was suspended. A gVisor actor sees only the time that it ran. A timeout, a cache lifetime, or a token check inside the actor gives a different result on each class. We read the elapsed-time clock of the process only, and we did not compare the calendar time.

What stays untested

All of these numbers come from one x86 machine with one CPU model. We did not run microvm in these cases:

• on arm64
• with the agentgateway dataplane
• with a resume from the object store
• with more than 500 MiB held
• on a pool whose nodes have different CPU models
• with the loss of a node

The reports from the issue tracker in the table above also stay reports. We ran none of them.

Our position

Choose by the memory that your actors hold. With small actors, start with gvisor. It runs on every node, it resumed a small actor 0.10 seconds faster, and its snapshot is as large as the memory and no larger. It is also the class that kagent installs.

Choose microvm in three cases. A reviewer requires hardware virtualization between tenants. You need a limit for each container. Or your actors hold more than about 200 MiB. In the third case microvm resumed as fast or faster on our machine, and a new 500 MiB actor started four times faster. You pay with a node type that has KVM and with 59 to 70 MiB of extra snapshot for each actor.

Then test your own workload. The reports above say that the two classes differ in the cases that hurt: memory pressure and many small files.

Neither class makes the worker pod a boundary. Both worker types run as root with SYS_ADMIN and NET_ADMIN, and the threat model of Substrate rates the two classes as equal.

Your actor ran on one worker pod this morning and on another one after lunch. Or a request fails with no worker has room for the actor while kubectl ate get workers shows a worker with free CPU. Both cases have the same cause. Substrate places an actor again each time it resumes, and the rules are not the rules of the Kubernetes scheduler.

When you finish this section, you can list the filters that a worker must pass and predict where an actor resumes. You can also explain a no worker has room error on a pool that is not full.

Most of our runs had one node. For this section we also built a cluster with one control plane node and two worker nodes, and we stopped one worker node four times. One result is against what the code suggested, and the section says where.

Two schedulers, two jobs

Kubernetes places worker pods on nodes, once, when the pool starts. It never sees an actor.

Substrate places actors on workers. The scheduler is a function inside ate-api-server. It runs in the ResumeActor call, and it reads the workers from memory and from Postgres, not from the Kubernetes API. That is why placement continued in our test with the Kubernetes API server down, as when a dependency is down shows.

A new actor has no worker. CreateActor writes a row with the state ACTOR_STATE_SUSPENDED, and placement waits for the first request. A suspend releases the worker again.

Placement: the choice of one worker for one actor, made at each resume.

The filters

A worker is a candidate when all of these are true:

1. It is registered, and its state is WORKER_STATE_ACTIVE.
2. Its sandbox class is the class of the snapshot that the actor resumes from.
3. It is on the node that holds the local snapshot of the actor, if a local snapshot exists.
4. Its runtime can restore the memory snapshot, if the snapshot has memory.
5. The labels of its pool match the workerSelector of the template and of the actor.
6. It has room: one free actor place, and free CPU and memory for the booking of the actor.

With no candidate, the call fails with no worker has room for the actor. The text is the same whichever filter removed the last worker. So the error does not always mean that the pool is full. It can mean that the only workers with room are of the wrong class or without the right label.

With one candidate, that worker gets the actor. With more, the scheduler takes two at random and keeps the less loaded one. The load of a worker is its fullest dimension: actor places, CPU, or memory.

The bind is one Postgres transaction. It locks the row of the worker, checks the room again, and writes the assignment. Two resumes that compete for the last place cannot both win. The loser tries again with another worker.

What our runs showed

We created four actors on a pool of two workers and sent each one a request, one after the other.

ATESPACE   NAME    TEMPLATE           STATE                   WORKER POD                       WORKER IPS    VERSION   AGE
lab-c      b0-1    lab-c/ballast-0    ACTOR_STATE_RUNNING     lab-c/ballast-65cf95d875-bzgjd   10.244.0.26   43        69s
lab-c      b0-2    lab-c/ballast-0    ACTOR_STATE_RUNNING     lab-c/ballast-65cf95d875-f65dl   10.244.0.27   3         68s
lab-c      b0-3    lab-c/ballast-0    ACTOR_STATE_RUNNING     lab-c/ballast-65cf95d875-bzgjd   10.244.0.26   3         66s
lab-c      b0-4    lab-c/ballast-0    ACTOR_STATE_RUNNING     lab-c/ballast-65cf95d875-f65dl   10.244.0.27   3         65s
lab-c      b0-5    lab-c/ballast-0    ACTOR_STATE_SUSPENDED   <none>                                         1         63s

The actors alternate between the two pods. With two candidates, the scheduler always compares both, and the emptier one wins. The fifth actor found no candidate and stayed suspended.

The parking run shows the other half of the rule. Actor p3 waited for a worker. When we suspended p1, p3 resumed on the pod that p1 had used. An actor has no home worker. It goes wherever a place is free at that moment.

Across two nodes

The scheduler compares workers, not nodes. On the cluster with two worker nodes, the pool had four workers, two on each node. We created 20 actors, sent each one a request, and left all of them running.

node multi-worker 10
node multi-worker2 10
worker lab-n/ballast-5d6cf97747-xlw7d 6
worker lab-n/ballast-5d6cf97747-dpjlc 5
worker lab-n/ballast-5d6cf97747-5cmpq 5
worker lab-n/ballast-5d6cf97747-67bkm 4

We measured this spread on the second x86 server guest (16 vCPU, 23 GB), with one run of 20 actors on gVisor. Ten actors went to each node, and the four workers got 6, 5, 5, and 4. One create and one resume through the CLI took 0.95 to 1.19 seconds, with a median of 1.00 seconds. Raw runs

The node of the last suspend

Filter 3 is the one that surprises people. After an asynchronous suspend, the snapshot is on the disk of one node, and the actor remembers that node.

  assignedNode: kind-control-plane
  lastAssignedGeneration: 2

While assignedNode is set, only workers on that node are candidates. The reason is speed: a resume from the local disk skips the download from the object store. In our runs the node kept the actor for 30 minutes after the suspend, the default of the atelet flag --snapshot-detach-delay. After that the field was empty and any node could take the actor. Snapshot fidelity, and where a snapshot lives has the measured timeline.

On the cluster with two worker nodes, the rule held in every resume.

We measured one actor on the second x86 server guest (16 vCPU, 23 GB), with five asynchronous suspends and one request after each. All 5 resumes went to the node of the suspend. The worker pod changed between the two workers of that node. Then we ran kubectl cordon on the node. The next 2 resumes still went to that node, in 0.41 and 0.29 seconds. Raw runs

2026-11-12T12:02:26.555Z	cordon	multi-worker2
2026-11-12T12:02:26.986Z	resume-call-while-cordoned	200	0.408920	count=8 mb=50 sum=6613 process_age_s=8.2
2026-11-12T12:02:27.433Z	resumed-on	multi-worker2 pod lab-n/ballast-5d6cf97747-xlw7d
2026-11-12T12:02:28.977Z	suspended-sync	ACTOR_STATE_SUSPENDED	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen8:multi-worker2
2026-11-12T12:02:29.292Z	resume-call-while-cordoned	200	0.293103	count=9 mb=50 sum=6613 process_age_s=9.0
2026-11-12T12:02:29.727Z	resumed-on	multi-worker2 pod lab-n/ballast-5d6cf97747-xlw7d

A cordon is a Kubernetes mark for new pods, and the Substrate scheduler does not read it. So a cordon does not keep a suspended actor away from its node. The fourth line shows one more fact. After a synchronous suspend the snapshot is complete, and the actor still names the node.

The rule has a cost when the node is full while another node has free workers. The code handles that case in the same resume call. ate-api-server asks atelet on that node to upload the snapshot and delete the local copy, and then it tries the other nodes.

We measured that path once, on the second x86 server guest (16 vCPU, 23 GB), with both worker nodes up. We filled all four workers, and a request to the suspended actor got 503 after 5.05 s, in 2 of 2 requests. The body was no worker has room for the actor, and the ate-api-server log names the step DetachFromFullNode. We then freed one worker on the other node, which did not hold the snapshot. The next request returned 200 in 0.61 s on that node, and the count of the actor continued. Raw runs

A drain takes the same path. After kubectl drain, the node had no worker, and its atelet still ran. Three suspended actors resumed on the other node at their first request, in 0.49 to 1.19 s. When a dependency is down has the five drain cases.

When the node stops

Our reading of the code said that an actor with a durable snapshot forgets a node that is down and resumes on another node. Our runs did not show that. We stopped one worker node four times, with the actor in a different condition each time. The other node had two active workers with room in every run.

First, what Substrate knows about the node.

We measured the detection on the second x86 server guest (16 vCPU, 23 GB), in 4 of 4 stops. The workers of the stopped node were listed WORKER_STATE_ACTIVE for 43 to 51 seconds after the stop command. Then they were marked WORKER_STATE_UNAVAILABLE, at the time that Kubernetes reported the node NotReady. Kubernetes evicted the worker pods 5 minutes later, and the Deployment created two new pods on the other node. Raw runs

{"time":"2026-11-12T12:03:19.231654833Z","level":"INFO","msg":"Syncer: marking worker unavailable (pod not Ready)","worker":"090b9723-a53d-4e0c-a139-741942971a78","pod":"lab-n/ballast-5d6cf97747-67bkm"}

The reason in that line is a pod that is not Ready. Substrate learned of the dead node from Kubernetes, and in our runs nothing in Substrate reported it sooner. For those 43 to 51 seconds, the scheduler can give a new resume to a worker on a node that is gone.

Second, what happened to the actor in each of the four runs.

2026-11-12T12:02:36.103Z	suspended-sync	ACTOR_STATE_SUSPENDED	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2
2026-11-12T12:02:36.563Z	node-stop	docker stop multi-worker2
2026-11-12T12:02:46.908Z	node-stopped	multi-worker2
2026-11-12T12:02:57.202Z	probe	504	10.055616		|	ACTOR_STATE_RESUMING	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2
2026-11-12T12:03:07.214Z	probe	503	7.777073	actor lab-n/u1 unavailable	|	ACTOR_STATE_RESUMING	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2
…
2026-11-12T12:08:14.892Z	probe	503	5.051588	actor lab-n/u1 unavailable	|	ACTOR_STATE_RESUMING	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2
2026-11-12T12:08:20.312Z	probe	503	3.190662	actor lab-n/u1 unavailable: AssignWorker prerequisite not met for Actor: lab-n/u1 (got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPENDED)	|	ACTOR_STATE_CRASHED	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2

Our client allowed 12 s for each of these requests. So the 504 at 10.06 s came from the dataplane, at the limit that section 4.4 describes. We first read it as our own timeout.

A suspended actor did not move to the other node. This is the result that is against our reading of the code. In the first run the snapshot was complete in the object store, and the other node had two free workers. The actor still stayed bound to the stopped node. It was in ACTOR_STATE_RESUMING for 5 min 33 s from its first request, and it was ACTOR_STATE_CRASHED 5 min 44 s after the stop command. Substrate v1.0.0 did not move an actor off a dead node by itself in any of our runs.

A revert did not free the actor either, in 4 of 4 attempts in three runs. In the second run it came too early. The workers of the stopped node were still listed as active, and the next request chose one of them. In the third run the resume that was in progress blocked both attempts. In the fourth run the revert needed atelet on the stopped node, and the actor stayed in ACTOR_STATE_REVERTING.

Those four runs left one question: does a revert help when you send it after the workers are marked unavailable? A second set of 13 node-stop cases answered it. Yes, for one kind of actor only.

We measured each case once, on a second x86 server guest (16 vCPU, 23 GB). The cluster had two worker nodes, the Envoy dataplane, and the default timers. In the first case, an actor was suspended with --mode sync and got no request after the stop. The workers of the node showed WORKER_STATE_UNAVAILABLE at 45 s. One revert then took 7.1 s and returned the actor with no node. The next request resumed it on the other node in 0.56 s, and the count continued from the snapshot. Raw runs

2026-11-12T12:53:34.330Z	t=0.7	node-stopped	multi-worker2
2026-11-12T12:54:18.228Z	t=44.6	workers-of-stopped-node-not-active	      2 WORKER_STATE_ACTIVE       2 WORKER_STATE_UNAVAILABLE 
2026-11-12T12:54:25.821Z	t=52.1	revert	7.1 s | ATESPACE   NAME   TEMPLATE           STATE                   WORKER POD   WORKER IPS   VERSION   AGE lab-n      a1     lab-n/ballast-50   ACTOR_STATE_SUSPENDED   <none>                    8         56s 
2026-11-12T12:54:26.821Z	t=53.1	first-request	200	0.560088	count=4 mb=50 sum=6613 process_age_s=4.9	|	ACTOR_STATE_RUNNING	multi-worker	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:-

The condition is narrow. The same kind of actor, reverted 10 s after the stop command, went to ACTOR_STATE_REVERTING and stayed there until the node was back. An actor that had got a request was refused 19 of 19 times while it was ACTOR_STATE_RESUMING. An actor with a PENDING snapshot, and a RUNNING actor, went to ACTOR_STATE_REVERTING at the first revert. In all of these cases the revert calls atelet on the stopped node, and that call fails.

One path worked in every state: a tag that exists before the node stops. A new actor from the tag served its first request on the other node 1.2 s after the create command, in 3 of 3 rescues. Section 8.2 has the runbook for each case, and figure 8.2d puts it on one page. Without a tag, treat a stopped node as an outage for every actor that got a request.

A RUNNING actor on a dead node is the worst case to see. Its state did not change in 10 minutes, so a list of actors shows nothing wrong. CRASHED and RevertActor has the error texts for each case.

When the node returned, it was Ready within seconds. It had none of the four pool pods in three of the four returns, because the new pods ran on the other node. Kubernetes does not move them back, and no new actor went to the returned node for more than 3 minutes. After we deleted two pool pods by hand, the first actor went to the returned node 23 to 35 seconds after the node start.

The second set of cases gave the same split, with more returns. After an outage shorter than the eviction time, the node kept its pool pods. The first new actor went to it 23 to 45 seconds after the start, in five returns. After a longer outage the node had no pool pod, in seven of seven returns. No new actor went to it in the 216 to 223 seconds that we watched. Substrate does not rebalance a pool.

We measured the case where the other node has no room, in two runs on the same guest. The revert was accepted, and each request got 503 after 5 s with no worker has room for the actor, in 8 of 8 requests. The actor stayed ACTOR_STATE_SUSPENDED. It answered 1.6 s after we freed a worker in one run, and 1.7 s after we scaled the pool in the other. When a dependency is down has both runs. Raw runs

We did not test a pool with spare workers on a third node, or a node loss with the microVM class. We also did not test a node that is alive and unreachable.

After a node restart

A restart of the node removes every worker at one time. We did that twice on the single node of the x86 server guest. The first run sent one request after the pods were back. The second run tried one resume every 2 to 3 seconds and listed the workers each time.

We measured the first recovery on the x86 server guest (24 vCPU, 47 GB, nested KVM), one run. The Kubernetes API answered again after 8 seconds, and all pods were running after 147 seconds. A request for a suspended actor then returned 503 with no worker has room for the actor. The same request returned 200 about 80 seconds later. Raw runs

The second run shows why.

We measured this timeline on the x86 server guest (24 vCPU, 47 GB, nested KVM), with one restart and one suspended actor. Each poll made one resume attempt. Raw runs

The mechanism has three steps. When the control plane returns, it first serves the worker list that it had before the restart. Then it marks each of those workers unavailable. Each worker becomes active again when it registers through atelet, as WorkerPool describes. Between the second and the third step, no worker of the pool is a candidate.

A resume in that window fails for lack of room or for an unavailable worker. The window was 10 to 30 seconds in this run. In the first run our one retry came 80 seconds later, so the window looked longer than it was.

The long part of the recovery is the control plane, not the workers. ate-api-server needed 160 of the 191 seconds.

Plan for this after a node restart: no call works until ate-api-server answers, and resumes can fail for about 30 seconds more. Do not use a check that all pods are Running as the signal. Kubernetes showed the old pod status at first: 0 pods not running at 10 seconds, and 21 at 18 seconds.

An actor that was running on a node that restarts is ACTOR_STATE_CRASHED, and CRASHED and RevertActor covers it. A node that stays down is different, as the table above shows. We did not repeat the restart on a cluster where the control plane and the workers are on different nodes.

Steer placement with labels

You have two tools, and both are labels:

• Pool labels and selectors. Put a label on the WorkerPool object, and select it in the template with workerSelector.matchLabels. An actor can add its own selector, and both must match.
• Node rules for the pool. spec.template.nodeSelector, tolerations and nodeAffinity of the pool decide which nodes the worker pods use. They act on Kubernetes, before any actor exists.

There is no way to pin one actor to one worker, and no priority between actors. A template can select pools with different CPU types, and that is unwise. A memory snapshot from one type cannot always restore on the other.

Twenty users open a session in the same second, and your pool has room for sixteen actors. Four requests have no worker. You need to know if those four users see an error at once, a delay, or an error after a delay. The answer is all three, and the router decides which.

When you finish this section, you can predict what a client sees when a pool is full. You can also tell the two capacity 503 texts apart and choose a client timeout.

What the router does when a pool is full

Every request to an actor passes through atenet-router, and the router calls ResumeActor for it. If the actor is suspended, ate-api-server must find a worker with room. When no worker has room, the call fails with the gRPC code ResourceExhausted and the text no worker has room for the actor.

Parked request: a request that the router holds open while it retries the resume of the actor.

The router does not return that failure at once. It parks the request and tries again. Five rules decide what happens next:

• Retry. The first retry comes after 100 ms. Each later delay is 1.1 times longer, with a small random part.
• Budget. The router stops retrying after 5 seconds, the value of --parked-request-budget. Then the client gets the 503.
• One budget for each actor. Requests for the same actor share one resume call and one budget clock. A request that joins late can fail after less than 5 seconds.
• Limit. At most 1,024 requests can be parked at one time, the value of --parked-request-max. A request takes a place only when its first resume attempt has failed and it starts to wait. When no place is free, the router sheds the request after that one attempt, with a 503. The body is actor "<atespace/name>" unavailable: router at capacity.
• No cancel. A resume that has started when the budget ends is not cancelled. The router waits for its result, so a request can be served after more than 5 seconds.

Set --parked-request-max=0 to turn parking off. Then a full pool returns the 503 at once.

The install sets none of these flags, so the defaults apply. The arguments of our router show it.

atenet-router router --mode=ingress --namespace=ate-system … --drain-delay=13s --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client --preview=

What the router parks, and what it does not

Parking is for conditions that can end without help. The router returns every other error at once.

The third row matters during a rollout of ate-api-server. A short restart of the control plane becomes a delay for the client and not an error. When a dependency is down shows the limit of that protection.

The fifth row is why a 503 after 5 seconds and a 503 after 60 ms are different problems. A slow 503 with the text no worker has room means the pool is too small. A slow 503 with only the word unavailable means the router got no answer from ate-api-server. A fast 503 with the text AssignWorker prerequisite not met means the actor is crashed, and CRASHED and RevertActor has the fix.

One request, three outcomes

The parking demo of Substrate creates a pool of two workers with room for one actor each, and four actors named p1 to p4. We sent one request each to p1 and p2, so both workers were busy. Then we sent a request to p3 two times.

2026-11-12T11:32:40.522Z	p3-no-free-worker	503	5.059628	actor ate-demo-parking/p3 unavailable: no worker has room for the actor
2026-11-12T11:32:45.600Z	p3-request-start	parked while pool is full
2026-11-12T11:32:47.970Z	suspend-p1-done	freed a worker 2 s after p3 request started
2026-11-12T11:32:45.603Z	p3-served-after-park	200	2.647469	preserved memory count: 1 | preserved file counter: -1 | preserved rootfs counter: 1

The first request to p3 found no worker for the whole budget. The second one started the same way. Two seconds into its wait we suspended p1, and that freed a worker. The next retry of the router found the worker, p3 resumed there, and the client got its answer.

We measured the wait on a full pool on the x86 server guest (24 vCPU, 47 GB, nested KVM, Envoy dataplane). Nine requests that found no worker each returned 503 after 5.05 to 5.06 seconds. The request that we freed a worker for returned 200 after 2.65 seconds. The suspend of p1 finished 2.37 seconds into that wait. The router then needed about 0.3 seconds to retry, resume the actor, and route the request. That last number comes from one run. Raw runs

The body of the slow 503 is always the same text, with the atespace and the name of the actor in it:

actor lab-c/b0-5 unavailable: no worker has room for the actor

Thirty seconds with four actors on two workers

The demo has a load script. It sends requests to all four actors for 30 seconds and suspends each actor after its answer. Two workers serve four actors, so half of the requests must wait for a suspend.

==> results
    total requests : 92
    200 OK         : 90
    503 unavailable: 2
    other          : 0
    200 latency    : avg 0.847s, slowest 4.269406s  <- parked requests sit here

We measured this load run on the same x86 server guest, one run of 30 seconds, 92 requests. The router served 90 requests and returned 503 for 2. The served requests took 0.85 seconds on average, and the slowest took 4.27 seconds. A request to a free worker in the same pool took about 0.35 seconds. Raw runs

Read those numbers together. Parking turned a pool that was two times too small into a service that answered 98% of requests. It did that with latency. The average request took more than two times longer than a request to a free worker. The slowest took twelve times longer. Two requests still failed, in a demo that was built to show parking at its best.

A burst of resumes

The demo sends its requests one after another. A real burst arrives in one instant, so we tested that. The pool had two workers with room for 16 one-CPU actors. We suspended n actors of 50 MiB each and sent one request to each of them at the same instant, five rounds for each n.

We measured these 155 requests on the x86 server guest (24 vCPU, 47 GB, nested KVM), with gVisor and the Envoy dataplane. Each request went through its own kubectl ate proxy, and the time is at the client. All 155 returned 200. Sixteen resumes at once cost 64 percent more for each request than one resume alone. ate-api-server logged no error line during the bursts. Raw runs

So the control plane did not queue at 16 resumes at once. A burst that fits in the pool costs tenths of a second, and no request waits for the budget.

A burst that does not fit is split by the pool, with no request in between. We sent 32 requests at the same instant to 32 suspended actors, when the pool had 14 free places.

We measured that burst once, on the same guest. Fourteen requests returned 200, at a median of 0.37 seconds. The other 18 returned 503 after 5.11 to 5.15 seconds, with the text no worker has room for the actor. In a second burst of 32 we suspended 16 actors 1.5 seconds into the wait. Then 21 of 32 requests returned 200, the slowest after 2.69 seconds, and 11 returned 503. Raw runs

The pool had 14 free places and not 16 for a reason that section 3.5 explains. Two actors whose process had exited were still ACTOR_STATE_RUNNING, and each one held a CPU.

A resume that takes more than 10 seconds

The parking budget is not the only clock. Envoy waits a fixed time for the router to decide where a request goes. The code sets that time to the parking budget plus 5 seconds. With the default budget that is 10 seconds. A resume that takes longer gets a 504 from Envoy, and the resume is not cancelled.

We measured this on a second x86 server guest (16 vCPU, 23 GB), in the bucket runs of section 8.1. In 32 of 61 cycles the resume had to read a large snapshot from a far or slow bucket. In each of the 32, the first request returned 504 after 10.05 s with an empty body. The client sent the request again each second and sent no other command. Each of the 32 then returned 200 with the right count. In the slowest cycle the first 200 came after 213 s and 19 requests. Raw runs

The limit is on the decision of the router, and not on the whole request. In an earlier run, three requests returned 200 after 10.09 to 10.87 s.

A 504 after 10 s has a second cause, and the state of the actor separates the two. During a slow resume the actor reads ACTOR_STATE_RESUMING. After a database outage, one actor that read ACTOR_STATE_RUNNING returned the same 504 on every request. Section 8.2 has that case.

Our position

Parking hides a short shortage. It does not add capacity. A pool that parks requests all day is too small, and your users pay for that with seconds on every request.

Three things follow for your client and your alerts:

• Client timeout. Allow at least 10 seconds for a request to an actor that can be suspended. The budget is 5 seconds, a resume that started late still completes, and the Envoy dataplane ends the wait 5 seconds after the budget.
• Retry a 504 too. A 504 after 10 seconds on a request to a suspended actor means that the resume is still at work. Send the request again after a pause, and send no revert. A revert of an actor in ACTOR_STATE_RESUMING is refused, as section 8.2 shows.
• Retry. Retry a 503 with the text no worker has room after a pause. Do not retry it in a tight loop, because each retry takes a place among the parked requests.
• Alert. The router counts parked requests in the metric atenet.router.parking.active and requests over the limit in atenet.router.parking.rejected. Alert when the first one stays above zero.

All of this section is the Envoy dataplane, where the router process of Substrate handles each request. We did not see a 503 from a full pool on the agentgateway dataplane. The kagent run uses agentgateway, and it did fill its one worker: 16 actors ran, and 8 of 24 parallel turns waited and then completed. Sizing a pool has that run.

We did not test more than 1,024 parked requests. Our bursts into a full pool stopped at 32 requests, and each request had its own kubectl ate proxy. Later bursts of up to 300 went through the router ingress to free workers. No request got the 503 of a full pool. Sizing a pool has those.

The project site states 500+/s suspend and resume activations. One node of ours finished 32 to 39 resumes each second with up to 300 at one time. It served 600 of 600 at 300. Sizing a pool has the runs.

Someone asks how many agents your cluster can hold. The question has two answers, and they differ by orders of magnitude. One answer is the number of actors that run at the same moment. The other is the number of actors that exist and wait in a suspended state.

When you finish this section, you can compute both numbers for a pool and choose the limits of a worker. You can also name the resource that runs out first.

Running actors: the arithmetic of declared limits

A worker reports a capacity, and each running actor books a part of it. WorkerPool showed where the capacity comes from. The booking is the limit that the ActorTemplate declares, plus a memory overhead that the sandbox runtime reports. The scheduler never looks at what an actor uses.

The number of actors that one worker runs is the smallest of three values:

• the CPU limit of the worker, divided by the CPU limit of the actor
• the memory limit of the worker less 64 MiB, divided by the memory booking of the actor
• 1,000, the default of --max-actors

We measured the left column on the x86 server guest (24 vCPU, 47 GB, nested KVM). The pool had two workers with a limit of 2 CPUs and 3 GiB. Each actor declared 1 CPU and 256 MiB. Four actors ran, two on each worker. Eight requests that needed a fifth running actor all returned 503. CPU was the limit: each worker had booked 512Mi of 3008Mi. Raw runs

We measured the right column on a second x86 server guest (16 vCPU, 31 GB), in the kagent run. The kagent pool sets no limits, and each actor declares 1 CPU and 1 GiB. We started 24 turns at the same time, one run. The worker reached 16 running actors and 16000 of 16000 millicores, and the other 8 turns waited. All 24 turns completed, in 4.3 to 7.8 seconds. With 8 turns at the same time, all 8 completed in 2.5 to 3.6 seconds. Raw runs

The parallel turns chart in section 7.1 shows the time of each turn in both runs.

The second result looks like free capacity, and it is the opposite. That worker promised every CPU of the node to actors, and the control plane runs on the same node. Set limits, as WorkerPool explains.

The CPU value of a template is a booking for placement, and the worker also writes it into the sandbox as a CPU quota. That quota is a hard cap.

We measured a fixed amount of work under four limits on a second x86 server guest (16 vCPU, 23 GB). Each of eight rows had three runs. The actor used 0.25, 0.50, and 1.01 host cores at 250, 500, and 1000 millicores, with one thread or with four. At 2000 millicores, one thread used 1.12 cores and four threads used 1.96 to 2.01. The work took 27.1 to 27.6 seconds at 250 millicores and 4.9 to 5.4 seconds at 1000. Raw runs

WorkerPool has the table and the chart.

So a low value packs more actors on a worker, and each of them is that much slower when it is busy. Size the value from the CPU that one turn needs, not from the average. The actor still sees 2 CPUs at every limit, so set thread counts in the template yourself. We did not test several busy actors on one worker at the same time.

A burst of resumes inside the capacity of the pool is cheap.

We measured 16 resumes at the same instant on the x86 server guest (24 vCPU, 47 GB, nested KVM), 80 requests in five rounds. The median was 0.37 seconds, against 0.23 seconds for one resume alone, and all 80 returned 200. Request parking has the chart. Raw runs

A dead actor counts as a running one. An actor whose process has exited stays ACTOR_STATE_RUNNING, and it keeps its booking. Two such actors each held 1,000 millicores of a worker for the 20 minutes of our watch. A burst of 32 resumes then found 14 free places in a pool of 16. CRASHED and RevertActor shows how to find and free them.

Two reports by other people help you choose the CPU limit of an actor. We did not repeat either.

A contributor measured real CPU use on workers that were fully booked, in issue 3441. Seven actors with 1 CPU each used between 0.55 and 0.8 cores together, at the median. A generous limit therefore costs you most of a worker, because Substrate at v1.0.0 has no overcommit.

A user measured resume time against the CPU limit of the actor on a release candidate of v1.0.0, in issue 3431. The median was 0.46 seconds at 50 millicores, 0.23 seconds at 100, and about 0.15 seconds from 250 up. A limit below 250 millicores saves capacity and costs resume time.

Suspended actors: disk, in three places

A suspended actor books nothing on a worker. You can keep far more suspended actors than running ones. Each of them costs disk.

The footprint chart in section 3.4 shows these sizes as bars.

We measured disk use on the x86 server guest (24 vCPU, 47 GB, nested KVM) with 25 suspended actors, one sample. The actors held 8,750 MiB of memory in total. The node held 8,801 MiB of snapshots and the object store held 10,824 MiB. After we deleted the actors, the node held 1 MiB of snapshots and the object store held 2,064 MiB. A cache of 1,765 MiB stayed on the node in both samples. Raw runs

Three rules come out of those numbers:

• A memory snapshot is as large as the memory that holds data. An actor with 200 MiB of random bytes made a snapshot of 201.9 MiB. Random bytes are the worst case. The same actor with 200 MiB of zero bytes made a snapshot of 4.0 MiB, and with half of each, 103.8 MiB. On the microVM class the snapshot also holds the guest, and it was 59 to 70 MiB larger, as sandbox classes shows.
• You pay two times for 30 minutes. With the default timers, the node keeps its copy for 30 minutes after the suspend. The object store has a second copy after 5 minutes. Snapshot fidelity, and where a snapshot lives has the timeline.
• Golden snapshots stay. The 2,064 MiB that remained belong to the five templates. Deleting actors does not free that space.

The disk that one node needs for snapshots is therefore the number of suspends in 30 minutes, times the snapshot size. A node that suspends 200 actors of 200 MiB each half hour needs 40 GiB for snapshots alone. The snapshot directory is a path on the node, and Kubernetes charges it to no pod.

On a pool that spans nodes, size each node by itself. A suspended actor resumes only on the node of its last suspend, for 30 minutes with the default timers. In our runs on two worker nodes, no actor moved off a node that stopped. When the node returned, Kubernetes left the new worker pods on the other node. Placement has those runs.

Many resumes at one time

A pool with free workers still has a limit on how fast it starts actors. We sent bursts of resumes to one node and timed each request. Figure 4.5b shows every request of two runs.

We measured the first run on the x86 server guest (24 vCPU, 47 GB, nested KVM). It had one kind node and a pool of 200 gVisor workers. The actor was a Python counter with a 128 MiB limit, and each actor had its own snapshot on the node. One client process in a pod sent each burst to the router, over one or two HTTP/2 connections. Each burst size ran three times, and all 933 requests returned 200. Raw runs

The rate stops rising at about 37 resumes each second. From there a larger burst only waits longer, and twice the burst took twice the time. A first start from the golden snapshot ran at the same rate: 34 to 38 each second for 150 new actors, in three waves.

We measured a second run on the same guest, on a new kind node with a limit of 700 pods. The pool grew from 50 to 300 workers in steps of 50, and each step was ready in 17 to 21 s. At every step the Kubernetes API answered /readyz in a median of 105 to 108 ms, so 300 workers did not slow it. The actor was the same counter, and each request opened its own connection. A single resume took a median of 0.35 s, in 20 resumes. Raw runs

1,448 of 1,450 requests in these bursts returned 200. The other two got no answer in 90 s, and the rate counts the answered requests up to the slowest answer. A first start of 300 new actors, 150 at a time, ran at 30 and 34 each second. So the rate held at 32 to 39 from 50 to 300 at one time, and fell a little at 300.

What sets that rate

Figure 4.5c shows the restore inside the worker, the step of a resume that took the most time.