Manuals / Agent Substrate 101 / Reference · Reference
R.3

States

An actor has seven states, a snapshot has two storage states, and a worker has three, and each one changes what a request returns.

The names and numbers below come from the proto file at v1.0.0. Store and compare the names. The numbers changed before 1.0.0, as section 8.4 explains.

The "what a request returned" column comes from our runs, through kubectl ate proxy. Most rows are from the x86 server guest (24 vCPU, 47 GB, nested KVM). The rows for a stopped node and for the database outage are from a second x86 server guest (16 vCPU, 23 GB). Those are ACTOR_STATE_RESUMING and ACTOR_STATE_REVERTING in the first table, and the stopped-node rows of the second.

Actor states

NumberStateMeaningWhat a request returned in our runs
1ACTOR_STATE_RESUMINGA worker is bound, and the sandbox is being restored.504 at the client timeout, then 503 actor lab-n/u1 unavailable, for an actor on a stopped node. One other actor stayed in this state after a database outage.
2ACTOR_STATE_RUNNINGThe actor holds a worker, and the API counts it as running. The state does not prove that its process answers.200 in 0.06 to 0.09 s. An actor whose process had exited stayed in this state and returned 503.
3ACTOR_STATE_SUSPENDINGThe worker writes a checkpoint.A request that was open when the suspend started got a 503. A new request in this state was not tested.
4ACTOR_STATE_SUSPENDEDThe actor holds no worker. It has a snapshot, or it never ran.200 in 0.36 s, after a resume
5ACTOR_STATE_CRASHEDThe sandbox or its worker was lost, or a resume failed.503 in 0.06 s, AssignWorker prerequisite not met
6ACTOR_STATE_DELETINGThe workload is being stopped, and the record is removed next.503 in 0.06 to 0.08 s, AssignWorker prerequisite not met
7ACTOR_STATE_REVERTINGThe actor returns to its durable snapshot.503 in 0.06 s, AssignWorker prerequisite not met, after a revert that could not reach the node

Number 0 is ACTOR_STATE_UNSPECIFIED, which no stored actor has. No state means idle, paused, or created. A new actor is suspended.

Fig. R.1the seven actor states, every transition, and what a request returnsstate
The seven actor states, every transition, and what a request returns in each state A reference state figure with the enum number of each state. CreateActor makes an actor in 4 SUSPENDED. ResumeActor moves it through 1 RESUMING to 2 RUNNING, and SuspendActor moves it through 3 SUSPENDING back to SUSPENDED. A failed resume, a lost sandbox, or a failed suspend moves it to 5 CRASHED, where a request returns 503. RevertActor moves a crashed, running, or suspended actor through 7 REVERTING to SUSPENDED. DeleteActor moves an actor in every state to 6 DELETING. Each box says what a request returned in our runs. 4 SUSPENDED holds no worker request: 200 in 0.36 s after a resume 2 RUNNING holds a worker request: 200 in 0.06 to 0.09 s the process exits: still RUNNING, request: 503 3 SUSPENDING writes a checkpoint request: not recorded 1 RESUMING restores the sandbox request: not recorded 5 CRASHED until RevertActor request: 503 in 0.06 s 7 REVERTING last durable snapshot request: not recorded CreateActor SuspendActor checkpoint node or bucket ResumeActor each request ready suspend failed resume failed sandbox lost pod or OOM RevertActor reverted RevertActor RevertActor 6 DELETING stops the workload DeleteActor, in every state the record is removed amber arrow: a state change that our runs show. rose arrow: a failure that our runs show. grey dotted arrow: a transition that we read in the code and did not run. dashed border: a state that a workflow passes through. double border: the record is removed next. A request to a CRASHED actor returns 503 with the body: AssignWorker prerequisite not met. ResumeActor on a CRASHED actor returns FailedPrecondition. SuspendActor on a SUSPENDED actor changes nothing.
A crashed actor returns to service only through RevertActor, and every path into RUNNING goes through ResumeActor. Read from SUSPENDED at the left. The number in each box is the enum value. Each arrow has the call or the event that causes it. Rose arrows are failures. Grey dotted arrows come from the code only.

Transitions

FromToTriggerSeen in our runs
noneSUSPENDEDCreateActoryes, in under 1 s
SUSPENDEDRESUMING, then RUNNINGResumeActor, which the router calls for each requestyes
RESUMINGCRASHEDThe resume failed. In our runs the node-local snapshot was missing, or the worker had not registered after a node restart. The documentation adds a wakeup probe that times out.yes, with a 500 error resuming actor
RUNNINGSUSPENDING, then SUSPENDEDSuspendActoryes
SUSPENDEDSUSPENDEDSuspendActor on a suspended actor returns it unchanged.from the code
SUSPENDINGCRASHEDThe suspend could not record its snapshot.from the code
RUNNINGCRASHEDThe worker pod was deleted.yes, within 0.6 s
RUNNINGCRASHEDThe kernel killed the actor for memory.yes, between 2 s and 4 s
RUNNINGCRASHEDThe node restarted.yes
RUNNINGRUNNINGThe process exited and the sandbox stayed. No transition occurs.yes, for at least 20 min
RUNNINGRUNNINGThe node of the actor stopped. No transition occurs.yes, for at least 10 min
RESUMINGCRASHEDThe node of the actor stopped before or during the resume.yes, 5 min 44 s after the stop command, after 5 min 33 s in RESUMING with requests
RUNNING or CRASHEDREVERTING, then SUSPENDEDRevertActoryes, from CRASHED in 0.22 s
SUSPENDED with a node-local snapshotREVERTING, then SUSPENDEDRevertActor, which discards the copy on the node. The snapshot itself is lost only if its upload is not yet complete.yes, after the node of the actor was back: the next reply had count 4, and the two later calls were lost. With the node down, the actor stayed in REVERTING
REVERTINGREVERTING, then SUSPENDEDA second RevertActor, which continues a revert that stopped part-wayyes, in 0.2 s, after the node and its atelet were back. With the node down, 4 of 4 reverts failed and the state stayed REVERTING
REVERTINGCRASHEDThe node of the actor returned, and we sent no command.yes, in one run, 15 s after the node start. In four runs the actor was still REVERTING after 216 to 223 s
RUNNINGRUNNINGResumeActor on a running actor returns it unchanged, with resumed=false. The router relies on this for every request.yes, on every request to a running actor
anyDELETING, then removedDeleteActoryes

Four rules follow from the table.

  • ResumeActor accepts a suspended actor, an actor that is already resuming, and a running actor, which it returns unchanged. On a crashed actor it returns FAILED_PRECONDITION.
  • RevertActor accepts an actor that is already reverting, so a second revert is the way out of a revert that stopped.
  • A crashed actor never leaves that state by itself. Something must call RevertActor or DeleteActor.
  • A revert returns the actor to its last completed durable snapshot. Calls that came after that snapshot are lost.

Snapshot storage states

StateMeaningHow long, in our runs
SNAPSHOT_STORAGE_STATE_PENDINGThe snapshot is on one node and is not in the bucket.5 min 1 s to 5 min 11 s after an asynchronous suspend, with default timers, for four actors
SNAPSHOT_STORAGE_STATE_COMPLETEDThe snapshot is in the bucket.reached before a synchronous suspend returns

The locality field of a snapshot names the node that still has a local copy. It was set for 30 minutes after the suspend in our run, and absent after that. The owner field is SNAPSHOT_OWNER_ACTOR for a snapshot of the actor itself and SNAPSHOT_OWNER_TAG for a golden or tagged snapshot.

Snapshot fidelity

ValueWhat a resume brings back
SNAPSHOT_FIDELITY_VOLUMESthe durable directories. The processes start again.
SNAPSHOT_FIDELITY_ROOTFSthe durable directories and the changes to the root filesystem. The processes start again.
SNAPSHOT_FIDELITY_MEMORYall of that and the memory of the processes, which continue.

Worker states

StateMeaning
WORKER_STATE_UNAVAILABLERegistered, and the scheduler places no actor on it. A new worker starts here.
WORKER_STATE_ACTIVEThe scheduler places actors on it while it has room.
WORKER_STATE_DRAININGTerminating. No new actor is placed, and bound actors stay until they are suspended. One-way.

kubectl ate get worker prints the state with the actors, the CPU, and the memory that are reserved against the capacity.

Durability modes of a suspend

ValueCLIWhen the call returns
DURABILITY_MODE_ASYNCHRONOUS--mode async, the defaultwhen the snapshot is on the node
DURABILITY_MODE_SYNCHRONOUS--mode syncwhen the new snapshot is in the bucket. An actor that is already suspended is returned as it is, and its snapshot can still be pending on the node.

Agent Substrate 1.0.0 · b34fbee · 2026-11-05 · Edition 2026.11