Manuals / Agent Substrate 101 / Part 1 · Substrate on One Page
1.2

One request, from the client to the actor and back

A request to a suspended actor passes the proxy, atenet-router, ate-api-server, and Postgres before a worker restores the actor, and a request to a running actor still asks all four.

You sent curl -X POST http://localhost:8000/ and a count came back. Six programs handled that request, and one of them wrote to a database before the actor ran. When a request fails later, the error text names one of those programs, so you need to know the order.

When you finish this section, you can name each hop of one request and say which hops a running actor skips. You can also say which parts of the cluster must be up for a request to pass.

The recording

We created my-counter-1 in the atespace ate-demo-counter and started kubectl ate proxy. Then we sent five requests, one after the other. This is the start of the recording. The columns are the time, the event, the HTTP status, the seconds at the client, and the body.

outputfive requests to a new actorFull log
ts	event	detail
2026-11-12T10:14:21.286Z	state	ACTOR_STATE_SUSPENDED	-
2026-11-12T10:14:21.837Z	call-1	200	0.358379	preserved memory count: 1 | preserved file counter: 1 | preserved rootfs counter: 1
2026-11-12T10:14:21.944Z	call-2	200	0.090770	preserved memory count: 2 | preserved file counter: 2 | preserved rootfs counter: 2
2026-11-12T10:14:22.030Z	call-3	200	0.066903	preserved memory count: 3 | preserved file counter: 3 | preserved rootfs counter: 3
2026-11-12T10:14:22.109Z	call-4	200	0.061842	preserved memory count: 4 | preserved file counter: 4 | preserved rootfs counter: 4
2026-11-12T10:14:22.196Z	call-5	200	0.068053	preserved memory count: 5 | preserved file counter: 5 | preserved rootfs counter: 5
2026-11-12T10:14:22.203Z	state	ACTOR_STATE_RUNNING	-
The first eight lines of the trace. The actor state was read before the first request and after the fifth.

We measured these requests on an x86 server guest with 24 vCPU and 47 GB, through kubectl ate proxy, in one run. The first request took 0.358 s. The next four took 0.091 s, 0.067 s, 0.062 s, and 0.068 s. Raw runs

The actor was ACTOR_STATE_SUSPENDED before the first request and ACTOR_STATE_RUNNING after the fifth. No command resumed it. The first request did.

Hop by hop

Figure 1.2 draws the first request. The numbers in this list match the rows of the figure.

Fig. 1.2the first request to my-counter-1sequence
The first request to my-counter-1 Six lifelines, each under a small drawing of its party: curl as a terminal, the kubectl ate proxy actor as a small unit with the Kubernetes mark, atenet-router as a gateway with the Envoy mark, ate-api-server as a server with the Agent Substrate mark, Postgres as a cylinder with its mark, and the worker pod as a frame that holds a gVisor sandbox and the actor. Nine numbered exchanges: curl sends POST to 127.0.0.1:8000, the proxy opens an mTLS CONNECT to router port 8444 through a kube-apiserver port-forward, the router calls ResumeActor, ate-api-server reads the actor row and finds it suspended, takes the lease and binds a worker, has atelet restore the golden snapshot so the actor goes from SUSPENDED to RUNNING, and returns worker 10.244.0.23 to the router. The router opens a CONNECT to worker port 8443, and curl gets 200 with count 1 after 0.358 seconds. Steps 5 and 6 have amber numbers because they happen only when the actor is suspended. In a later resume of the same actor the restore took 0.25 seconds, and a request to the running actor took 0.067 seconds. curl client kubectl ate proxy actor atenet-router dataplane ate-api-server control plane Postgres actor rows worker pod my-counter-1 1 POST localhost:8000/ 2 mTLS CONNECT to port 8444 through a kube-apiserver port-forward 3 ResumeActor sent for every request 4 read the actor row ACTOR_STATE_SUSPENDED 5 take the lease, bind a worker 6 atelet restores the golden snapshot SUSPENDED to RUNNING, 0.25 s in a later resume 7 worker 10.244.0.23 with its SPIFFE ID 8 CONNECT to port 8443 then actor port 80 9 200, count: 1 0.358 s at the client 5 6 only when the actor is suspended. A running actor skips them and answers in 0.067 s.
The first request to a suspended actor makes the router call ResumeActor, and ate-api-server restores the actor before the request reaches it. Read the rows from top to bottom. The drawing above each lifeline is its party, with the mark of its project. Row 2 is the only one that passes kube-apiserver. Rows 5 and 6 have amber numbers because a request to a running actor skips them. The note under row 6 is the restore time of a later resume of the same actor.
  1. curl sends the request to 127.0.0.1:8000, where kubectl ate proxy listens.
  2. The proxy opens a port-forward through the Kubernetes API server to atenet-router. Inside that port-forward it opens an mTLS CONNECT tunnel to router port 8444, with a client certificate.
  3. The router reads the target actor and calls ResumeActor on ate-api-server. It does so for every request, because it keeps no record of where an actor runs. Requests that arrive at the same time for one actor share one call.
  4. ate-api-server reads the actor row from Postgres. The state is ACTOR_STATE_SUSPENDED.
  5. ate-api-server takes a lease on the actor, chooses a worker with room, and writes the assignment to Postgres in one transaction.
  6. ate-api-server tells atelet on that node to restore the snapshot. The worker starts the sandbox from it, and the actor state becomes ACTOR_STATE_RUNNING.
  7. ate-api-server returns the actor to the router, with the IP address of the worker pod and the identity of the worker.
  8. The router opens a tunnel to port 8443 of the worker pod and checks that identity. The worker forwards the request to port 80 of the actor.
  9. The reply goes back the same way.

A new actor has never run, so the snapshot in step 6 is not its own. It is the golden snapshot of its template, and section 3.1 explains it. After step 7 the actor object carries the assignment:

jsonthe actor after the first requestFull log
    "state": "ACTOR_STATE_RUNNING",
    "workerAssignment": {
      "nodeName": "kind-control-plane",
…
      "workerEpoch": "1",
      "workerNamespace": "ate-demo-counter",
      "workerPod": "counter-7cc7dcd596-ffxwt",
      "workerPodIps": [
        "10.244.0.23"
      ],
…
      "workerPool": "counter",
      "workerSpiffeId": "spiffe://cluster.local/ns/ate-demo-counter/sa/ateworker"
Cut from the actor object. The metadata, the snapshot list, and the sandbox version attributes are cut.

Where the 0.36 s goes

The first request paid for the restore. The four after it did not. The difference is about 0.29 s on this guest, for an actor that holds almost no memory.

The worker writes one log line when a restore starts and one when it ends. We did not save that log for the first request. We saved it for a later resume of the same actor:

outputone restore in the lifecycle logFull log
{"time":"2026-11-12T10:30:17.846333794Z","message":"Actor restoring",…
{"time":"2026-11-12T10:30:18.098955499Z","message":"Actor restored",…
The span and trace fields of each line are cut.

We measured that resume on the same guest, once. The two log lines are 0.253 s apart, and the request that caused the restore took 0.382 s at the client. So about two thirds of a first request is the restore itself. Raw runs

That share grows with the actor. A restore reads the memory that the actor held, and section 3.3 shows 2.16 s for an actor with 1000 MiB.

A request to a running actor

A request to a running actor still does steps 1 to 4. ate-api-server reads the row, finds ACTOR_STATE_RUNNING, and returns at once. Steps 5 and 6 do not happen. So a request costs one call to ate-api-server and one read from Postgres, also when nothing is resumed. Requests that arrive together for one actor share that call.

Most of the 0.06 s in our recording is the proxy, not Substrate.

We measured both paths on the second x86 guest, with 16 vCPU and 23 GB, in the 30 s before a test. A pod that sent its requests to the router answered in 0.027 s at the median, with 30 requests. The same actor through kubectl ate proxy answered in 0.067 s at the median, with 28 requests. Raw runs

Use the number from the pod when you plan latency. The proxy is a tool for your own machine, and section 2.3 shows why it must not carry real traffic.

What must be up

Three things follow from the path.

  • Postgres and ate-api-server are on every request path. A running actor that needs no resume still fails when either one is down. On our second guest, requests to a running actor failed within seconds of each outage. Section 8.2 has the timeline.
  • The Kubernetes API server is on no request path. Only row 2 of the figure passes it, and that row belongs to the proxy. In the same test, kube-apiserver was down for 86 s. In that time 103 of 103 requests from a pod were served: 86 to a running actor and 17 resumes.
  • The object store is on the path only when a snapshot is not on the node. Section 3.4 says when that is.

The install runs one atenet-router pod and one Postgres pod. Every request to every actor passes both. Section 8.1 says what to do about Postgres.

The actor stays up

After the fifth request we sent nothing for 900 s. Figure 1.2b shows all six requests of the recording, and every state sample between the last two.

Fig. 1.3six requests to my-counter-1, and its state for 900 idle secondschart
Six requests to my-counter-1, and its state in the 900 seconds between the last two The upper part has six horizontal bars with the client time of each request through kubectl ate proxy on the x86 server guest: request 1 0.358 s, request 2 0.091 s, request 3 0.067 s, request 4 0.062 s, request 5 0.068 s, request 6 0.080 s. The first bar is amber because it includes the resume from the golden snapshot, which adds about 0.29 s. The other bars are green because the actor was already running. The lower part is a time axis of the 913 seconds with no request between request 5 and request 6. It has one dot for each of the 60 state samples, and all 60 are in the row ACTOR_STATE_RUNNING. The row ACTOR_STATE_SUSPENDED is empty. CLIENT TIME OF EACH REQUEST request 1 0.358 s request 2 0.091 s request 3 0.067 s request 4 0.062 s request 5 0.068 s request 6 0.080 s after 900 s idle about 0.29 s more than requests 2 to 5: the resume from the golden snapshot 0.067 s, the median of requests 2 to 5 0 0.1 0.2 0.3 0.4 seconds STATE OF MY-COUNTER-1 BETWEEN REQUEST 5 AND REQUEST 6 60 of 60 samples ACTOR_STATE_RUNNING ACTOR_STATE_SUSPENDED 0 samples: no idle timer suspended the actor request 5 request 6, 0.080 s 0 150 300 450 600 750 900 seconds with no request, one state sample every 15 s
Only the first request paid for a resume, and the request after 900 idle seconds was as fast as the others because the actor never stopped. Each bar is the time of one request at the client. The amber bar includes the resume from the golden snapshot, and the dashed line is the median of requests 2 to 5. Each dot in the lower part is one state sample from the 900 seconds with no request.

The sixth request took 0.080 s and returned count 6. A resume would have cost about 0.36 s. The actor was never suspended, and it held its worker for the whole 15 minutes. Section 1.1 names this as the first job that Substrate leaves to you.

Agent Substrate 1.0.0 · b34fbee · 2026-11-05 · Edition 2026.11