Manuals / Agent Substrate 101 / Part 3 · Actors
3.4

Snapshot fidelity, and where a snapshot lives

A snapshot keeps what its fidelity names, it is as large as the memory that holds data, and for five minutes it exists on one node only.

Your agent wrote a file to /tmp, started a background process, and ended its turn. On the next turn the file is gone and the process does not exist. Nothing failed. The template asked for a snapshot of files only, and /tmp is not one of those files. A second surprise waits on the node: the snapshot that you took a minute ago is not in the object store yet.

When you finish this section, you can choose a fidelity for a template and estimate the disk that suspended actors need. You can also say where a snapshot is at each minute after a suspend.

Three fidelities

Snapshot fidelity: the set of state layers that a snapshot holds. A template names the highest level that it wants in snapshotConfig.preferredFidelity, and each level includes the levels below it.

FidelityThe snapshot holdsA resume
SNAPSHOT_FIDELITY_VOLUMESThe durableDir volumesBoots the containers from the image, with the volumes back
SNAPSHOT_FIDELITY_ROOTFSThe volumes, and the writes to the root filesystem since bootBoots the containers from the image, with those writes back
SNAPSHOT_FIDELITY_MEMORYThe volumes, the root filesystem writes, and the process memoryContinues the same processes

kagent uses the first level. Its template mounts one durableDir volume at /data and sets SNAPSHOT_FIDELITY_VOLUMES. In the first turn of a session, we told the agent to write to three places and to start a process.

outputwhat turn 1 wroteFull log
RUN: mkdir -p /data/workspace && echo hello-from-turn1 > /data/workspace/note.txt; echo scratch-turn1 > /tmp/scratch.txt; echo rootfs-turn1 > /root-marker.txt 2>/dev/null; (nohup sleep 100000 >/dev/null 2>&1 &) ;…
The first line of the file. The end of the command, which only prints values, is cut.

kagent suspended the actor after that turn. The second turn read the same places.

outputwhat turn 2 foundFull log
note=hello-from-turn1
scratch=cat: can't open '/tmp/scratch.txt': No such file or directory
bg=
uptime=0.74
The first four lines of the file.

The note under /data is there. The file in /tmp is gone, the sleep process is gone, and the sandbox is 0.74 seconds old. The write to / failed in turn 1, because the harness does not run as root.

The other end is SNAPSHOT_FIDELITY_MEMORY, which our counter and our memory test used. To compare all three levels, we ran one actor under each level. The actor holds 50 MiB of random bytes and starts a child process. It keeps one count in memory, one in a file on the root filesystem, and one in a file on a durableDir volume.

FidelityMemory countRoot filesystem countVolume countProcesses after a resumeSnapshot on the nodeResume, median of 10
SNAPSHOT_FIDELITY_VOLUMESback to 1back to 1continued to 25new, 0.2 to 0.4 s old44 KiB0.52 s
SNAPSHOT_FIDELITY_ROOTFSback to 1continued to 25continued to 25new, 0.2 to 0.3 s old72 KiB0.50 s
SNAPSHOT_FIDELITY_MEMORYcontinued to 25continued to 25continued to 25the same, with the same child56.9 MiB0.25 s

We measured these three actors on the x86 server guest (24 vCPU, 47 GB, nested KVM), with gVisor and one actor for each level. Each actor got 25 requests with 13 suspends between them, and all 75 requests returned 200. The resume column is one request to the suspended actor, timed at the client, with the snapshot on the node. An asynchronous suspend took a median of 0.32 seconds at the two lower levels and 0.54 seconds at the memory level, with 10 suspends each. Raw runs

A resume at the two lower levels is a cold start of the container. The snapshot is a few KiB, so the cost is the start time of your application. Our Go actor starts in 0.3 seconds, and its resume took 0.5 seconds, twice the resume at the memory level. The kagent run was on a second x86 server guest (16 vCPU, 31 GB). There the harness image resumed in a median of 0.38 seconds under SNAPSHOT_FIDELITY_VOLUMES.

A Python agent is slower to start. We ran one on the same guest, on the cluster of the Part 9 demos, with three turns at each level. A turn after a suspend took 4.5 to 5.1 seconds under SNAPSHOT_FIDELITY_VOLUMES. It took 0.30 to 0.32 seconds under SNAPSHOT_FIDELITY_MEMORY. A stateful agent actor shows that run.

Fig. 3.6what each fidelity keepscomparison
What each snapshot fidelity keeps Three columns for the three snapshot fidelity levels. SNAPSHOT_FIDELITY_VOLUMES keeps only durable volumes and resumes with new processes. SNAPSHOT_FIDELITY_ROOTFS keeps durable volumes and root filesystem writes and also resumes with new processes. SNAPSHOT_FIDELITY_MEMORY keeps volumes, root filesystem writes and process memory, so the same processes continue. Rows under the columns give what one actor for each level showed on the x86 server guest after 25 calls with suspends between them: its three counts, its median resume from the node, and its snapshot size. VOLUMES: counts memory 1, rootfs 1, volume 25, 0.52 s median of 10, snapshot 44 KiB. ROOTFS: counts memory 1, rootfs 25, volume 25, 0.50 s median of 10, snapshot 72 KiB. MEMORY: counts memory 25, rootfs 25, volume 25, 0.25 s median of 10, snapshot 56.9 MiB. LAYER VOLUMES ROOTFS MEMORY durable volumes durableDir mounts kept kept kept root filesystem writes since boot gone kept kept process memory processes and RAM gone gone kept ON RESUME COUNTS after 25 calls new processes 0.52 s median of 10 snapshot 44 KiB memory 1, rootfs 1, volume 25 new processes 0.50 s median of 10 snapshot 72 KiB memory 1, rootfs 25, volume 25 same processes 0.25 s median of 10 snapshot 56.9 MiB memory 25, rootfs 25, volume 25 The enum names carry the prefix SNAPSHOT_FIDELITY_. Each level includes the levels to its left. A count of 1 means that the count started again at the last resume.
Only SNAPSHOT_FIDELITY_MEMORY keeps the processes, and the two lower levels start new processes on every resume. Read each column top to bottom. Teal layers are in the snapshot, and grey layers are lost at the suspend. The last rows give the median resume and the three counts of one actor for each level, measured on the x86 server guest.

preferredFidelity is an upper limit, and a memory snapshot can restore as less. A memory snapshot records the CPU architecture, the CPU features, and the sandbox build of the worker that took it. It restores only on a worker that matches. If no such worker has room, Substrate restores the files on another worker and boots the processes new. We did not see that downgrade in any run. Write your actor so that it survives a boot from its files, also at the memory level.

A snapshot is as large as the memory that holds data

Start with the worst case, an actor whose memory is random bytes.

Memory the actor holdsSnapshot on the node
0 MiB1.3 MiB
50 MiB51.7 MiB
200 MiB201.9 MiB
500 MiB502.2 MiB
1,000 MiB1,002.7 MiB

One file holds almost all of it.

outputthe largest file in the snapshot of a 200 MiB actorFull log
211574784 /var/lib/ate/snapshots/a77c2723-c524-46d0-9a7e-caa97f36ab9c/local-checkpoint/…/pages.img
The last line of the file. The first number is the size in bytes, and the name of the checkpoint folder is cut.

We measured these sizes with du on the kind node of the x86 server guest, one actor for each size. The memory was random bytes. The snapshot folder of an actor is named by the uid of the actor. Raw runs

Random bytes are the largest snapshot that an actor of that size can have. We ran the same actor again with zero bytes, and with half zero and half random bytes.

Fig. 3.7snapshot size on the node for three kinds of memorychart
Snapshot size on the node against memory held, for three kinds of memory MiB of the snapshot on the node against the MiB of memory that the actor holds, on the x86 server guest with gVisor, one actor for each of six templates. All random bytes: 203.8 MiB at 200 MiB held and 504.1 MiB at 500 MiB held. Half zero and half random: 103.8 MiB at 200 MiB held and 254.4 MiB at 500 MiB held. All zero bytes: 4.0 MiB at 200 MiB held and 4.6 MiB at 500 MiB held. A dashed line marks a snapshot as large as the memory held. The median resume from the node, of ten for each actor, is 0.42 s and 1.07 s for random, 0.28 s and 0.57 s for half and half, and 0.24 s and 0.23 s for zero. 0 100 200 300 400 500 0 200 500 memory the actor holds, MiB snapshot on the node, MiB size equals memory held all random bytes 203.8 and 504.1 MiB the size of the memory resume 0.42 and 1.07 s half zero, half random 103.8 and 254.4 MiB half of the memory resume 0.28 and 0.57 s all zero bytes 4.0 and 4.6 MiB about 4 MiB at both sizes resume 0.24 and 0.23 s one actor, one du reading resume: median of 10, from the node
An actor that held 500 MiB of zero bytes had a 4.6 MiB snapshot, and the same actor with random bytes had 504.1 MiB. Each dot is the snapshot of one actor on the node, and each line joins the two sizes of one kind of memory. The dashed line marks a snapshot as large as the memory held. The violet text is the median of ten resumes from the node. Measured on the x86 server guest.
Memory the actor holdsSnapshot on the nodeCopy in the object storeResume from the node, medianAsynchronous suspend, median
200 MiB of zero bytes4.0 MiB1.1 MiB0.24 s0.35 s
500 MiB of zero bytes4.6 MiB1.1 MiB0.23 s0.35 s
200 MiB, half zero and half random103.8 MiB101.1 MiB0.28 s0.66 s
500 MiB, half zero and half random254.4 MiB251.2 MiB0.57 s0.79 s
200 MiB of random bytes203.8 MiB201.2 MiB0.42 s0.80 s
500 MiB of random bytes504.1 MiB501.3 MiB1.07 s1.07 s

We measured six actors on the x86 server guest (24 vCPU, 47 GB, nested KVM), one for each row. Each actor had 10 resumes and 10 asynchronous suspends. All 60 resumes returned 200, and every count continued. Each size is one du reading, on the node and in the data folder of the object store. Raw runs

So the size follows the memory that holds data, and the resume time follows the same bytes. An actor with 500 MiB of zero pages resumed in 0.23 seconds, as fast as an actor that holds nothing. Do not count on that for a real heap. A Python process with a dict of one million random values reported a peak of 245 MiB. Its snapshot on the node was 215.2 MiB. Section 3.3 has its resume time.

The object store did not make the random snapshots smaller. Substrate compresses each object before the upload, and random bytes do not compress. The copy of the Python actor was 110.1 MiB in the object store, about half of its node copy. We have not found yet which step made it smaller, and that search is planned for the next edition.

WhereWith 25 suspended actorsAfter we deleted them
Snapshots on the node8,801 MiB1 MiB
Snapshot cache on the node1,765 MiB1,765 MiB
Object store10,824 MiB2,064 MiB

We measured this footprint on the x86 server guest (24 vCPU, 47 GB, nested KVM). The node had five actors for each of the five sizes, which hold 8,750 MiB together. Each actor was 3 to 8 minutes old. The 2,064 MiB that stayed in the object store belong to golden snapshots. A template keeps its golden snapshot until you delete the template. Raw runs

Fig. 3.8disk use of 25 suspended actors, and after their deletechart
Disk use of 25 suspended actors, and what a delete gives back Horizontal bars with the MiB that du reports in three places on the x86 server guest, with 25 suspended actors that hold 8,750 MiB of memory together and again after we deleted the actors. Snapshots on the node: 8,801 MiB, then 1 MiB. Snapshot cache on the node: 1,765 MiB, then 1,765 MiB. Object store: 10,824 MiB, then 2,064 MiB, which are the golden snapshots of the five templates. A dashed line marks 8,750 MiB. with 25 suspended actors after we deleted the actors 8,750 MiB: the memory the 25 actors hold snapshots on the node gone with the actors 8,801 1 snapshot cache on the node did not shrink 1,765 1,765 object store golden snapshots stay 10,824 2,064 0 2,000 4,000 6,000 8,000 10,000 12,000 MiB Five actors for each of 0, 50, 200, 500, and 1,000 MiB of random bytes, each 3 to 8 minutes old. The object store holds one snapshot for each actor and one golden snapshot for each template.
25 suspended actors that held 8,750 MiB of memory used 8,801 MiB on the node and 10,824 MiB in the object store. Each pair of bars is one place. The full bar is the size with the 25 suspended actors, and the outlined bar is the size after we deleted them. The dashed line marks the memory that the actors hold together. Measured on the x86 server guest.

An actor owns one snapshot in the object store, or two after an asynchronous suspend. A synchronous suspend deletes the earlier snapshot. An asynchronous suspend keeps it, because the new snapshot is not yet durable. The first suspend after the new snapshot is durable deletes the old one, and a delete of the actor removes both.

So plan the object store for up to two times the memory that holds data in all suspended actors. Add one golden snapshot for each template. Plan the node disk for the actors that were suspended in the last 30 minutes. Add the snapshot cache, which still held 1,765 MiB here after we deleted all 25 actors.

Two timers decide where the snapshot is

Node-local snapshot: the copy that the worker writes to the disk of its node at a suspend. atelet owns it.

Durable snapshot: the copy in the object store. Its state is SNAPSHOT_STORAGE_STATE_PENDING until the upload is complete, and then SNAPSHOT_STORAGE_STATE_COMPLETED.

After an asynchronous suspend, atelet runs two timers for each snapshot.

  • --snapshot-upload-delay, 5 minutes by default: the wait before the upload to the object store.
  • --snapshot-detach-delay, 30 minutes by default: the time until atelet removes the copy on the node and clears the node assignment of the actor.

We suspended five actors and read their state every 10 seconds for 42 minutes. The fifth actor, at 1,000 MiB, never got a worker, because the pool had no room. It made no snapshot of its own, so the numbers below are for four actors.

outputthe upload, five minutes after the suspendFull log
2026-11-12T10:31:10.270Z	d0	suspended-async	ACTOR_STATE_SUSPENDED	kind-control-plane	…SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_PENDING:gen2:kind-control-plane
…
2026-11-12T10:36:01.379Z	d0	state	ACTOR_STATE_SUSPENDED	kind-control-plane	…SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_PENDING:gen2:kind-control-plane
2026-11-12T10:36:12.367Z	d0	state	ACTOR_STATE_SUSPENDED	kind-control-plane	…SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:kind-control-plane
Three lines for the actor d0. The first snapshot in each line, the golden one, is cut.
outputthe detach, thirty minutes after the suspendFull log
2026-11-12T11:01:02.649Z	d0	state	ACTOR_STATE_SUSPENDED	kind-control-plane	…SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:kind-control-plane
2026-11-12T11:01:13.670Z	d0	state	ACTOR_STATE_SUSPENDED	-	…SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:-
Two lines for the actor d0. The node name becomes a dash in the fifth column and at the end of the snapshot.

We measured four actors on the x86 server guest (24 vCPU, 47 GB, nested KVM), with the default timers. They held 0, 50, 200, and 500 MiB. The upload was complete 301 to 311 seconds after the suspend. The node copy was gone 1,801 to 1,803 seconds after the suspend. At that moment the snapshots on the node shrank from 758 MiB to 1 MiB. Each time can be up to 10 seconds late, because that was our sample interval. Raw runs

Fig. 3.9where a snapshot lives after an asynchronous suspendtimeline
Where a snapshot lives after an asynchronous suspend A timeline of four actors, d0, d50, d200 and d500, over 43 minutes after an asynchronous suspend with default atelet timers on the x86 server guest. For the first 5 minutes each snapshot is only on the node and its state is PENDING. From about 5 minutes to about 30 minutes the snapshot is on the node and in the object store. After about 30 minutes the node copy is removed and only the object store copy remains. The first three uploads completed after 5 min 1 s to 5 min 2 s, and the 500 MiB one after 5 min 11 s. A request 42 minutes after the suspend resumed each actor from the object store in 0.25 s, 0.52 s, 1.19 s, 2.44 s. A row of dots under the bars shows the size of the snapshots on the node disk in each of 38 samples: 758 MiB until the node copies were removed and 1 MiB after that. node only, PENDING node and object store object store only d0 0 resumed in 0.25 s d50 50 resumed in 0.52 s d200 200 resumed in 1.19 s d500 500 resumed in 2.44 s MiB upload done after 5 min 1 s node copy removed after 30 min 3 s node disk 758 MiB of snapshots, one dot for each sample 1 MiB 0 5 10 15 20 25 30 35 40 MINUTES one request 42 minutes after the suspend: every actor resumed from the object store In the rose part, losing the node copy leaves the actor CRASHED. Each time is at most 10 s late.
For the first five minutes after an asynchronous suspend, the only copy of the new snapshot is on one node. Each row is one actor, with its memory in MiB. Rose is the time with one copy, violet is the time with two, and teal is the time with the object store copy only. The amber dot is the resume that ended our watch, with its time at the client. The dots under the rows are the size of the snapshots on the node disk.

The three periods mean three different things for you.

  1. In the first 5 minutes, the node holds the only copy. If that copy is lost, the actor is ACTOR_STATE_CRASHED, and the work since the last durable snapshot is gone. Section 3.5 shows that loss.
  2. From 5 to 30 minutes, both copies exist. A resume uses the node and takes the lower line of section 3.3.
  3. After 30 minutes, only the object store has the snapshot. A resume downloads it first and takes about twice as long for each MiB. That number is for a bucket in the cluster, and section 8.1 shows a bucket that is 20 ms away.

The documentation promises neither the start nor the end of the upload. Treat 5 minutes as the shortest exposure, not as a limit.

Our position: do not accept the first period for state that matters. A synchronous suspend closes it for one actor. A shorter --snapshot-upload-delay on atelet makes it shorter for the cluster, and does not close it. We ran atelet with 3 seconds and 10 seconds for one test, and all 55 requests of that test succeeded. A short detach delay saves node disk and makes every resume a download.

A tag is a copy that does not depend on the node

Tag: a named copy of the durable snapshot of a suspended actor. kubectl ate create tag <name> --actor <actor> makes one. If the snapshot is not uploaded yet, the call uploads it first. kubectl ate create actor <new> --tag <name> creates a new actor that starts from the copy.

A tag has its own object in the object store, under the atespace and outside the folder of the actor. So it does not need the node of the actor, and it does not change when the actor runs on. That makes a tag the one way to move the state of an actor off a node that has stopped.

outputthe tags of one atespaceFull log
ATESPACE   NAME                                   SCOPE                 STATE   SNAPSHOT                                                                       FIDELITY                   AGE
lab-n      09fee120-cd19-4121-8bae-5623fcbe70eb   TAG_SCOPE_PUBLISHED   Ready   gs://ate-snapshots/atespaces/lab-n/tags/1010ca2e-25b0-44d6-864f-a31a80da48c2   SNAPSHOT_FIDELITY_MEMORY   147m
lab-n      d8328610-2fdb-4974-a6ab-7c8671586d37   TAG_SCOPE_PUBLISHED   Ready   gs://ate-snapshots/atespaces/lab-n/tags/8d60620d-653e-469f-9dd1-9f11a89ce7e0   SNAPSHOT_FIDELITY_MEMORY   147m
lab-n      keep-t1                                TAG_SCOPE_ATESPACE    Ready   gs://ate-snapshots/atespaces/lab-n/tags/aca8c8e0-6dfd-4ee5-9e55-ced17f7c346e   SNAPSHOT_FIDELITY_MEMORY   12m
lab-n      keep-u1                                TAG_SCOPE_ATESPACE    Ready   gs://ate-snapshots/atespaces/lab-n/tags/13e4dabd-ad64-417f-95a5-ec1d5909c786   SNAPSHOT_FIDELITY_MEMORY   72s
lab-n      pend-u1                                TAG_SCOPE_ATESPACE    Ready   gs://ate-snapshots/atespaces/lab-n/tags/cfefcc57-5a75-418c-ae29-424958706266   SNAPSHOT_FIDELITY_MEMORY   69s
The complete file. The first two tags are the golden tags of two templates, and the other three are tags that we made of two actors.

We measured three tags in the node-stop cases, on a second x86 server guest (16 vCPU, 23 GB), with one run for each case. A tag of an actor that was suspended with --mode sync was Ready in less than 1 s. So was a tag of an actor whose newest snapshot was still PENDING on the node, and it held that newest state. Then we stopped the node. A new actor from each tag served its first request 1.2 s after the create command, on the other node. That first request took 0.48 to 0.58 s. Raw runs

A tag has three limits. It is as old as the suspend that it copied, so later work is not in it. The actor that you create from it has a new name. And you cannot make it after the fact: create tag on an actor that was held on the stopped node failed with must be ACTOR_STATE_SUSPENDED to be tagged. Section 8.2 uses tags in its runbook for a stopped node.

Do not delete a golden tag

The golden tag of a template is in the same list. Its scope is TAG_SCOPE_PUBLISHED, and its name is a uid. kubectl ate delete tag accepts it, and our own clean-up made that mistake. It deleted every tag of the atespace, and the golden tags of two templates went with them.

outputa template after its golden tag was deletedFull log
## get actor-templates
ATESPACE   NAME         GOLDEN TAG                             ERROR   AGE
lab-n      ballast-0    d8328610-2fdb-4974-a6ab-7c8671586d37           170m
lab-n      ballast-50   09fee120-cd19-4121-8bae-5623fcbe70eb           170m
## get tags
ATESPACE   NAME   SCOPE   STATE   SNAPSHOT   FIDELITY   AGE
## create actor from template ballast-50
Error: failed to create actor: rpc error: code = FailedPrecondition desc = Tag not found
…
Eight lines of the file. The lines that create two new templates are cut.

The template looks healthy and is broken. get actor-templates still lists the old golden tag id, and the ERROR column is empty. Each create actor --template fails with Tag not found. We saw this once, on the same guest. We made new templates to continue. Whether a golden tag can be restored is a test for the next edition.

Agent Substrate 1.0.0 · b34fbee · 2026-11-05 · Edition 2026.11