Manuals / Agent Substrate 101 / Part 3 · Actors
3.1

ActorTemplate and the golden snapshot

Substrate snapshots each ActorTemplate once, and an actor created after that starts suspended from the snapshot, so its first call pays a restore.

Your agent image takes several seconds to import its libraries and load its tools. With a Deployment, every new pod pays those seconds again. Substrate moves that cost to the template. It starts the image once, takes a snapshot, and creates every new actor from that snapshot. The cost does not disappear: the first request to a new actor restores the snapshot, and the restore grows with the memory in it.

When you finish this section, you can write an ActorTemplate, check that its golden snapshot exists, and predict the first-call time of a new actor.

The template we used

ActorTemplate: the description of an actor: its containers, its resource limits, its volumes, the worker pools that it can run on, and its snapshot settings. A template belongs to one atespace, and ate-api-server stores it in Postgres. It is not a Kubernetes object, so kubectl get does not list it.

This is one of the five templates of our resume tests. The actor is a small Go server that holds 200 MiB of random bytes and counts its requests.

yamlan ActorTemplate that holds 200 MiBFull log
metadata:
  atespace: lab-c
  name: ballast-200
workerSelector:
  matchLabels:
    workload: ballast
containers:
- name: ballast
  image: localhost:5001/ballast@sha256:93dba82ad560e7fc4ccb2a33a63d64713b519a7895ba3fd98130ce789eab7fd3
  command:
  - /ballast
  - --mb=200
  wakeupProbe:
    httpGet:
      path: /readyz
      port: 80
resources:
  limits:
    cpuMillicores: 1000
    memoryBytes: 478150656
snapshotConfig:
  preferredFidelity: SNAPSHOT_FIDELITY_MEMORY
The complete file. We created it with kubectl ate create actor-template -f.

Five fields decide how the actor behaves.

  • image must name a digest, and a tag alone is refused. Ours is a scratch image from a plain docker build, with no change for Substrate.
  • resources.limits needs both values. The scheduler books them against the capacity of a worker, which section 4.1 explains.
  • workerSelector matches the labels of a WorkerPool object. Section 3.6 shows why the namespace of the pool does not count.
  • wakeupProbe is a readiness check that runs on every resume. The actor becomes ACTOR_STATE_RUNNING only after the probe passes.
  • preferredFidelity says how much state a snapshot keeps. Section 3.4 compares the three levels.

A template is immutable, and the API has no call that updates one. For new code you create a new template, and then you move each suspended actor with kubectl ate update actor --template.

We ran that move with two templates that differ in the image. On a running actor the update is refused.

outputa template change on a running actorFull log
…Error: failed to update actor: rpc error: code = FailedPrecondition desc = actor must be ACTOR_STATE_SUSPENDED to change its actor template (got: ACTOR_STATE_RUNNING)
One line of the trace. The time and the event name at the start of the line are cut.

On the suspended actor the update was accepted, and the next call ran the new image. The count in memory started again at 1, and so did the count on the root filesystem. The count on the durableDir volume continued at 4. So this move kept the durableDir volume and dropped the old memory snapshot. A move with an external volume is a run for the next edition. Suspend first, and expect a cold start of the new image on the next call.

The golden snapshot is built once

Golden snapshot: the snapshot of one temporary actor that Substrate starts from a new template. Substrate stores it under a tag, the golden tag, and every new actor of that template starts from that tag.

ate-api-server builds the golden snapshot in four steps.

  1. It creates a temporary golden actor in the atespace of the template, and it resumes that actor.
  2. It waits until the actor is ready. If a container has no wakeupProbe, it waits 20 seconds more.
  3. It suspends the actor at the preferredFidelity of the template, and it copies the snapshot to a published tag.
  4. It deletes the golden actor, and it writes the tag into the status of the template.

The log of ate-api-server on our kagent cluster shows all four steps for one template.

outputthe golden actor of one template, from create to deleteFull log
2026-11-12T10:16:44.165402326Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	create	suspended
2026-11-12T10:16:44.178659651Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	resume	resuming
2026-11-12T10:16:54.017864285Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	resume	running
2026-11-12T10:16:54.03004179Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	suspend	suspending
2026-11-12T10:16:54.180722259Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	suspend	suspended
2026-11-12T10:16:54.238242923Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	delete	deleting
2026-11-12T10:16:54.328177186Z	dd7b8625-0d96-4d7d-89da-4ec9cc3b914a	delete	deleted
The first seven lines of the file. The columns are the time, the uid of the actor, the operation, and the new state.

The timestamps show a resume of 9.84 seconds, because the golden actor is the one actor that boots the image from zero. The suspend took 0.15 seconds. That template had a files-only fidelity, so the snapshot held no memory.

The build pays the full start of your application, once.

We measured one template for a Python server on the x86 server guest (24 vCPU, 47 GB, nested KVM), with n=1. The server builds a dict of one million values at start. The golden tag was ready 39.3 seconds after create actor-template. The process was 32.9 seconds old at its first request, so most of that time was the application start under gVisor. A new actor of that template then answered its first call in a median of 0.63 seconds, with five new actors. Section 3.3 shows the server. Raw runs

The golden tag does not exist at the moment the template does. In our resume tests we listed the five templates 21 seconds after we created them.

outputfive templates, 21 seconds after createFull log
ATESPACE   NAME           GOLDEN TAG                             ERROR   AGE
lab-c      ballast-0      e32029d7-3d9b-4212-8953-e88d7cede5b4           21s
lab-c      ballast-1000                                                  20s
lab-c      ballast-200    b9f97b82-b187-419d-8789-6c8c7b8a6c8c           20s
lab-c      ballast-50     0382e2fc-99db-461a-8845-a90d93484443           21s
lab-c      ballast-500    2fa7aaf9-1fe8-4076-9350-af27ba338a6b           20s
The complete file. The 1,000 MiB template has no golden tag yet.

All five had a tag when we looked again at 118 seconds. Wait for the GOLDEN TAG column before you create actors. An actor that you create earlier records no snapshot. It boots from the image on its first call, and it keeps that cold start until its own first suspend.

We ran that race. We created an actor directly after its template, and create actor succeeded.

outputan actor created before the golden snapshot existsFull log
2026-11-12T12:49:47.365Z	rc1-state	ACTOR_STATE_SUSPENDED	-	
2026-11-12T12:49:48.542Z	rc1-first-call	200	0.983729	tag=v3 count=1 mb=200 fill=random sum=26936 rootfs=-1 data=-1 pid=1 process_age_s=0.8 child[none]
…
2026-11-12T12:50:00.900Z	golden-ready	waited 11.137 s more
2026-11-12T12:50:01.888Z	rc2-state-new	ACTOR_STATE_SUSPENDED	-	SNAPSHOT_OWNER_TAG:SNAPSHOT_FIDELITY_MEMORY:SNAPSHOT_STORAGE_STATE_COMPLETED:gen1
Four lines of the trace. The actor rc1 is early, and the actor rc2 was created after the golden tag. The lines between them are cut.

We measured this once on the x86 server guest (24 vCPU, 47 GB, nested KVM), with a 200 MiB actor. The early actor was ACTOR_STATE_SUSPENDED with an empty snapshot list. Its first call was a cold start and took 0.98 seconds. The golden tag was ready 11 seconds later, and the first call of an actor created after that took 0.93 seconds. Five more early actors of another template still had no snapshot 25 seconds later. Raw runs

So an early actor never adopts the golden snapshot. It gets its own snapshot at its first suspend. Our actor starts in under a second, so the two first calls cost the same. With an application that needs 30 seconds to start, the early actor pays 30 seconds on its first call.

You can turn the golden snapshot off with snapshotConfig.goldenSnapshotConfig.mode: GOLDEN_SNAPSHOT_MODE_DISABLED. Every new actor then boots from the image.

A new actor starts suspended

kubectl ate create actor writes one row in Postgres and touches no worker, so its time does not depend on the template. The new actor is suspended. It already has a snapshot when the template had its golden tag at that moment.

jsonthe actor my-counter-1 directly after createFull log
{
  "actorTemplate": {
    "atespace": "ate-demo-counter",
    "name": "counter"
  },
  "metadata": {
    "atespace": "ate-demo-counter",
    "createTime": "2026-11-12T10:13:40.987568669Z",
    "name": "my-counter-1",
    "uid": "6f80f612-7547-4f5b-b3ab-831595621806",
    "updateTime": "2026-11-12T10:13:40.987568669Z",
    "version": "1"
  },
  "status": {
    "lastAssignedGeneration": 1,
    "snapshots": [
      {
        "actorTemplateUid": "71a3a39c-5f37-4d6e-937f-9c314fea7668",
        "durableSnapshot": {
          "object": {
            "snapshotUri": "gs://ate-snapshots/atespaces/ate-demo-counter/tags/ad19ee28-1519-432a-bcc5-40cf99cce3ca"
          },
          "state": "SNAPSHOT_STORAGE_STATE_COMPLETED"
        },
        "fidelity": "SNAPSHOT_FIDELITY_MEMORY",
        "generation": 1,
        "owner": "SNAPSHOT_OWNER_TAG",
        …
    "state": "ACTOR_STATE_SUSPENDED"
  }
}
The sandboxRuntime block and the end of the file are cut.

Read three fields. state is ACTOR_STATE_SUSPENDED, so no sandbox exists yet. owner is SNAPSHOT_OWNER_TAG, so the actor borrows the snapshot of the golden tag and owns none. The URI has a tags segment, and it does not hold the uid of the actor. The actor writes its own snapshot at its first suspend.

Fig. 3.1from ActorTemplate to the first callflow
From ActorTemplate to the first call of a new actor Two rows. The top row happens once per template: the template lab-c/ballast-200 leads to a temporary golden actor, which is suspended and copied to a golden tag. The bottom row happens for each new actor: kubectl ate create actor b200-1 makes an actor in ACTOR_STATE_SUSPENDED that points at the tag snapshot, and the first request restores it to ACTOR_STATE_RUNNING in 0.62 s at 200 MiB, the median of 5 new actors. ONCE FOR EACH TEMPLATE tag listed 21 s after the template ActorTemplate lab-c/ballast-200 golden actor temporary, same atespace golden tag SNAPSHOT_OWNER_TAG 1 2 3 deleted after the copy FOR EACH NEW ACTOR create actor kubectl ate, b200-1 ACTOR_STATE_SUSPENDED gen 1 is the tag snapshot ACTOR_STATE_RUNNING count=1 in the reply 4 5 restore reads it 1 ate-api-server creates a temporary golden actor from the template and resumes it. 2 It suspends that actor at preferredFidelity and copies the snapshot to a published tag. 3 It deletes the golden actor. In our run the tag was listed 21 s after the template. 4 A new actor records the tag snapshot as generation 1 and starts SUSPENDED. 5 The first request restores the tag snapshot on a worker: 0.62 s at 200 MiB.
Substrate builds one golden snapshot for each template, and an actor created after that starts suspended from it. Read the top row once for each template and the bottom row once for each actor. The numbers match the key. Names and seconds are from our resume tests on the x86 server guest.

The first call pays for the restore

The first request to a new actor restores the golden snapshot. For a small actor you do not notice it. For a large one, it is the slowest request that the actor ever serves.

Memory in the templateFirst call of a new actor, medianFastest and slowest of 5Later resume of the same actor, median
0 MiB0.23 s0.22 to 0.24 s0.22 s
50 MiB0.25 s0.24 to 0.40 s0.24 s
200 MiB0.62 s0.56 to 0.86 s0.47 s
500 MiB5.76 s1.86 to 6.76 s1.07 s
1,000 MiB6.60 s3.46 to 12.75 s2.16 s

We measured these times on the x86 server guest (24 vCPU, 47 GB, nested KVM), with gVisor and five new actors for each size. Each time is one request through kubectl ate proxy, timed at the client. A request to an actor that already runs took 0.06 seconds on the same path. The later resume is the median of 10 resumes from the snapshot on the node. kubectl ate create actor returned in 0.20 to 0.26 seconds for all 25 actors. Raw runs

Fig. 3.2the first call of a new actor against its memorychart
The first call of a new actor against the memory in its template Seconds for the first request to a new actor, which restores the golden snapshot, against the MiB of memory the template holds, on the x86 server guest. Each of the 25 new actors is one dot, five for each size, and a line joins the medians: 0.23 s at 0 MiB, 0.25 s at 50 MiB, 0.62 s at 200 MiB, 5.76 s at 500 MiB, 6.60 s at 1000 MiB. The five first calls at 500 MiB took 1.86, 3.03, 5.76, 6.24, 6.76 seconds, and the five at 1000 MiB took 3.46, 3.56, 6.60, 11.85, 12.75 seconds. A second line shows the median of ten later resumes of one actor from its own snapshot on the node, which reaches 2.16 s at 1000 MiB. 0 2 4 6 8 10 12 14 0 50 200 500 1000 memory the template holds, MiB seconds from request to reply, at the client 12.75 s slowest first call 6.60 s median of 5 3.46 s fastest later resume 2.16 s median of 10 five new actors at 500 MiB: 1.86, 3.03, 5.76, 6.24, 6.76 s up to 200 MiB the first call stays under 0.9 s the first call of one new actor median of one size
At 1,000 MiB the first call of a new actor took 3.5 to 12.7 seconds, and a later resume took 2.2 seconds. Each teal dot is the first call of one new actor, five for each size, and the teal line joins the medians. The violet line is the median of ten later resumes of one actor from the node. Measured on the x86 server guest.

Up to 200 MiB the first call costs about the same as a later resume. At 500 MiB and above it costs three to five times more, and the spread is wide. We did not find the cause of that spread. We can say where the node kept data. Its snapshot-cache directory held 1,765 MiB, and the five snapshot sizes of our templates add up to 1,760 MiB.

Our position: the golden snapshot is a good place for slow start code and a bad place for large memory. If your template holds more than about 200 MiB, send one request to each new actor before a user does. Use a request that changes no state: our counter adds one on every call. Then suspend it. A later resume from the node copy follows the lower line.

Agent Substrate 1.0.0 · b34fbee · 2026-11-05 · Edition 2026.11