Limits at v1.0.0
At v1.0.0 Substrate runs, suspends, and resumes actors on request, and it leaves the suspend decision, pool size, failover, and key rotation to you.
You plan a rollout and write down what the platform will do for you. It will stop idle sessions. It will add capacity under load. It will survive the loss of one pod. For Agent Substrate v1.0.0, each of those three lines is wrong, and the release does not say so in one place.
When you finish this section, you can list what Substrate does not do at v1.0.0. You can also say which of our claims rest on a run, and which runs come in the next edition.
What Substrate leaves to you
Figure 8.5 sets each job that a platform team expects against the party that does it at v1.0.0.
No idle suspend
Nothing in Substrate suspends an actor because it has no requests. The code has no idle timer in ate-api-server, in atelet, or in either worker. The worker API has a method for a worker to ask for a suspend, RequestActorSuspend. atelet forwards that call, and a client library for it exists in the repository. No worker at v1.0.0 uses the library, so nothing sends the call.
We measured this on an x86 server guest (24 vCPU, 47 GB, nested KVM). One actor received no request for 900 s, and it read RUNNING in 60 of 60 samples. A running actor reserves its full CPU and memory limit on a worker for all that time. Raw runs
The harness must call SuspendActor. kagent does it after every turn, in 98 of 98 turns in our run, as section 7.1 shows. If you write your own harness, the suspend call is the first thing to build.
No autoscaler, and no overcommit
A WorkerPool has a fixed replicas number. Nothing in the install changes it. The WorkerPool kind has a scale subresource, so an outside autoscaler can change the number. A run with one is planned for the next edition. CPU and memory use are poor signals for it. An actor reserves its limit while it runs, whether or not it works.
Substrate also books each running actor at its full declared limit. A proposal to book less than the limit exists in the public issues, and the maintainers moved it to after the release.
We measured the effect in the kagent run, on a second x86 server guest (16 vCPU, 31 GB). One run started 24 turns at the same time. The single worker reported 16,000 millicores, and each actor reserved 1,000. The worker stopped at 16 running actors, and the other eight turns waited. Raw runs
When a pool is full, a request waits up to 5 s and then gets a 503. Section 4.5 gives the arithmetic for a pool.
No GPU
An ActorTemplate at v1.0.0 can declare two resources: cpuMillicores and memoryBytes. The API has no field for a GPU or for any other device, so we ran no GPU test.
Single-replica components
The code sets these replica counts in the install manifests.
| Component | Replicas | What stops when the one pod is away |
|---|---|---|
ate-api-server | 2 | nothing, while one replica remains |
ate-idp-server | 2 | nothing, while one replica remains |
atenet-router | 1 | every request to every actor |
atenet-egress | 1 | every new outbound connection from an actor |
ate-controller | 1 | changes to worker pools |
podcertificate-controller | 1 | new and renewed certificates |
| bundled Postgres | 1 | every request, as section 8.2 measures |
The right column is our reading of the request path, except for the Postgres row, which we ran. The router keeps its parked requests in memory, and a graceful shutdown does not drop them. After the stop signal the router keeps the parked requests until each one gets its full budget and a normal answer. The flag --drain-timeout sets that limit, and its default is longer than the parking budget. Only a crash, a forced kill, or a drain past --drain-timeout drops a parked request.
We restarted the router once, as the fix in section 8.2. What a restart does to parked requests, and a restart of the egress gateway, are runs for the next edition.
Crash detection has a gap
Substrate marks an actor crashed when its sandbox dies or when the kernel kills it for memory. It does not notice a process that exits inside a sandbox that still runs. Section 8.3 shows that actor. A design for this detection is open in the public issues.
We measured two such actors on the x86 server guest (24 vCPU, 47 GB, nested KVM). Both read RUNNING for 20 minutes 36 s after the exit, in 81 and 80 samples. All 20 requests to one of them returned 503. Each kept 1,000 millicores of its worker for all that time. The first suspend call failed and marked the actor crashed. A longer watch and a worker restart are on the list for the next edition. Raw runs
It also does not mark a running actor whose node stops. On a cluster with two worker nodes, that actor read RUNNING for the 10 minutes of our probe. Each of 150 requests got a 503.
An actor does not leave a dead node by itself
A suspended actor is bound to the node of its last suspend, and Substrate v1.0.0 did not release it when that node stopped.
We measured this on the second x86 server guest (16 vCPU, 23 GB), with two worker nodes and one actor for each of four cases. An actor had a complete snapshot in the object store. It stayed in RESUMING on the stopped node for 5 min 33 s from its first request. It was crashed 5 min 44 s after the stop command. The other node had two free workers. A revert during the outage failed or had no lasting effect in 4 of 4 attempts on three actors. Raw runs
The workers of a stopped node were listed as active for 43 to 51 s after the stop command. A new resume can go to a dead node in that time. Kubernetes moved the worker pods to the other node after 5 minutes and did not move them back. Section 4.3 has the timeline.
We measured 13 more node-stop cases on a second x86 server guest (16 vCPU, 23 GB), once each, with the default timers. In none of them did an actor leave the stopped node by itself. One revert freed an actor in one situation only: suspended with --mode sync, no request since the stop, and the workers already marked unavailable. That revert took 7.1 s. In every other situation a revert left the actor in REVERTING until the node was back. A new actor from a tag that existed before the stop served its first request after 1.2 s, in 3 of 3 rescues. Raw runs
So recovery from a node loss is yours to build. Substrate does not move actors, and it does not move pool pods back to a node that returns. Section 8.2 has the runbook.
Access is coarse at the router
The ingress allowlist holds client identities, and each listed client can reach every actor in every atespace. Access policies for each atespace govern the API calls, as section 5.2 explains. They do not govern requests through the router at v1.0.0.
Manual key rotation
The key that signs actor JWTs lives in a pool that an operator edits. The CLI has kubectl ate admin add-jwt-key, activate-jwt-key, and remove-jwt-key, and the documentation gives the waits between them. No component rotates the key on a schedule. Section 5.3 covers the tokens. Our install had one key, and our decoded token named it. A rotation is a run for the next edition.
A suspend does not wait for requests
SuspendActor stops the sandbox at once. A request that is open at that moment gets a 503, and the suspend call still reports success. We saw this in both suspend modes, as section 3.3 shows. The caller of the suspend must know that the actor is idle.
Preview features are off
Three features exist behind a server flag and a client flag. All three are off by default.
| Preview name | What it adds |
|---|---|
ExternalVolumes | CSI volumes that an actor mounts and that no snapshot includes |
GoldenEgressPolicy | an egress policy for the actor that builds the golden snapshot |
PerTemplateSnapshotLocation | a snapshot location in each template, in place of one for the cluster |
To use one, start every component with --preview and its name, and pass --preview to kubectl ate. A request without the client flag behaves as if the feature were off. We tried all four combinations with a template that has a goldenEgressPolicy field.
Server --preview | kubectl ate --preview | Result of create actor-template |
|---|---|---|
| empty | no | InvalidArgument desc = actor_template.golden_egress_policy: Forbidden |
| empty | yes | the same error |
GoldenEgressPolicy | no | the same error |
GoldenEgressPolicy | yes | accepted, and the golden tag was ready within 30 s |
2026-11-12T13:06:36.842Z on no-flag Error: failed to create actor template: rpc error: code = InvalidArgument desc = actor_template.golden_egress_policy: Forbidden We ran each combination once on the x86 server guest. The error does not name the preview feature or the flag, so you must know that the field is a preview field. Once the template existed, kubectl ate get read it without the client flag.
Our test had two limits. We set the flag on ate-api-server only, with an edit of its Deployment, and the documented way sets it on every component through ate-setup. A test that the egress rule of the golden actor works is planned for the next edition. Every other experiment ran with --preview= empty, so this manual describes none of the three features in use.
Smaller limits that will reach you
- An open response or pass-through connection outlives its egress rule. Removing a rule stops requests, also on a reused connection, within 9 s. Only a suspend cuts the rest, as section 6.6 shows.
- A changed Secret takes up to 5 minutes to reach requests on agentgateway. We measured 100.6 to 280.5 s there, and 1.0 to 1.6 s on Envoy. Section 6.5 has the run.
- The egress gateway trusts public roots only. A host with a certificate from a private CA returned 503 until we replaced the root file in the proxy container. The installer removes that patch. Section 6.4 has it.
- On agentgateway, the stats port 15020 of the router answers
/metricswith no credentials, and the egress gateway logs atdebugafter every installer run. Section 6.1 and section 6.6 have both. kubectl ate delete tagaccepts the golden tag of a template. The template then lists a tag that is gone, andcreate actorfails withTag not found. Section 3.4 has that mistake.- An actor sees 2 CPUs at every CPU limit, and the limit is a hard cap. Section 4.1 has the measurement.
- Names are identities. The documentation tells you never to reuse the name of a deleted atespace. A request that was allowed for the old one can apply to the new one.
- Postgres is the only store. The maintainers declined a storage plugin before the release.
- A durable directory has no size limit in the API, so one actor can fill the node disk. This comes from an open issue, and a run on it is planned for the next edition.
- The gVisor class accepts a limit for each container and enforces only the limit of the actor.
Our position: none of these limits makes v1.0.0 unusable. Together they mean that Substrate is one layer of a platform. Budget engineering time for four things. They are a harness that suspends, an autoscaler that reads free worker slots, a database with failover, and a probe for dead processes. Section 7.4 weighs that cost against the alternatives.
Coming in the next edition
This edition measured Substrate on kind clusters on our own machines. The next edition adds the runs below. Until then, treat each item as not measured.
Most numbers here came from clusters with one kind node, and one cluster had two worker nodes. Each cluster ran one of the two dataplanes. The object store was in the cluster, except in two experiments: once outside the cluster on the same guest, and once on a second guest.
| Area | Planned runs |
|---|---|
| Clouds | GKE first, then other managed Kubernetes services |
| Storage | Google Cloud Storage, Amazon S3, a real link with 100 ms of delay to a bucket, more than one cycle of 1,000 MiB with 100 ms on the path between two guests, a managed Postgres, a Postgres outage with dropped packets, a connection pooler, a restore of either store. Also the cause of one running actor that stayed at 504 after a Postgres outage: we saw it once and have not found the cause yet |
| Scale | more than 10,750 actor rows in one cluster, more than 450 actors with a snapshot of their own, more than 300 resumes at one time, more than 300 workers on one node, a burst on more than one node, a burst to actors whose snapshot is only in the bucket, what inside the gVisor restores sets about 37 resumes each second, more replicas of ate-api-server with a higher connection limit of Postgres, a third run of 200 and of 300 at one time, more than 24 parallel turns, more than two worker nodes, more than 20 actors across nodes, and the site claims of millions of agents and of 10x density |
| Sandbox | the microVM class on arm64, with agentgateway, with a resume from the object store, above 500 MiB, and across CPU models. Also images with an init system, non-root images, large root filesystems |
| Agents | a hosted model, a model on a GPU, or a model large enough for real agent work behind kagent, any real model behind a model gateway, a kagent turn that runs away while a bound is set, the maxCompletionTokens field of a ModelConfig, a streamed model reply through a model gateway, client authentication at a model gateway, the codex and claude harnesses, MCP servers through the egress gateway |
| Lifecycle | an upgrade from v0.6.0 that keeps actors, an upgrade between two supported releases, a rollback across a schema change, an upgrade on more than one node or with the microVM class, a third run of an upgrade step, a drain with a PodDisruptionBudget on the pool, a drain of the node that runs ate-system, a revert in the first 20 s after a drain, the restore of a deleted golden tag, a resume at the two lower fidelities for an application with a long start on this guest |
| Network | a suspend as a cut-off for a tlsPassthrough connection, the step that takes 0.6 s in an intercepted handshake, the log level of atenet-router, a restart of atenet-egress as a way to refresh a credential, the content of every metric label on the agentgateway stats port, port 9902 of the Envoy container |
| Limits | memory limits under load, CPU limits with several busy actors on one worker, an outside autoscaler on a pool, a durable directory that fills the node disk |
| Identity | the signature check of an actor JWT with the published key, a key rotation, federation with a second issuer or a cloud provider, a second token issuer for clients |
| Snapshots | why the object store copy of the Python actor was half of its node copy, and equal for random bytes |
| Failure | a dead process that stays RUNNING for more than 20 minutes or across a worker restart, the three outages on more than one node, egress during an outage, parked requests during a router restart, an egress restart, a node that is alive and unreachable, a node that loses power or its network, a node loss on the microVM class or on agentgateway, a node that is full by CPU or by a count of actors, a second run of the Kubernetes API outage with resumes and free workers, a pool with spare workers on a third node, other node timers of Kubernetes. Also two causes we saw and have not found yet: a burst request with no answer in 90 s, and an actor left in ACTOR_STATE_RESUMING after a database outage |
| Agent Executor | a hosted or large model in a task, the example agent with a model that can do its work, a node loss under a task, a large or private clone, an MCP server that the image starts as a command, an MCP registry with listed servers, an MCP server outside the cluster, the release v0.3.1 on the Substrate commit that it names |
| Apple silicon | actors above 200 MiB, a resume from the object store |
| Preview | ExternalVolumes and PerTemplateSnapshotLocation on upstream Substrate, the flag on every component, and the behaviour of a golden egress policy |
Until then, a section that states a fact from these areas takes it from the code or the documentation at the tag. The text says so.
Agent Substrate 1.0.0 · b34fbee · 2026-11-05 · Edition 2026.11