Placement: which worker gets the actor

Your actor ran on one worker pod this morning and on another one after lunch. Or a request fails with no worker has room for the actor while kubectl ate get workers shows a worker with free CPU. Both cases have the same cause. Substrate places an actor again each time it resumes, and the rules are not the rules of the Kubernetes scheduler.

When you finish this section, you can list the filters that a worker must pass and predict where an actor resumes. You can also explain a no worker has room error on a pool that is not full.

Most of our runs had one node. For this section we also built a cluster with one control plane node and two worker nodes, and we stopped one worker node four times. One result is against what the code suggested, and the section says where.

Two schedulers, two jobs

Kubernetes places worker pods on nodes, once, when the pool starts. It never sees an actor.

Substrate places actors on workers. The scheduler is a function inside ate-api-server. It runs in the ResumeActor call, and it reads the workers from memory and from Postgres, not from the Kubernetes API. That is why placement continued in our test with the Kubernetes API server down, as when a dependency is down shows.

A new actor has no worker. CreateActor writes a row with the state ACTOR_STATE_SUSPENDED, and placement waits for the first request. A suspend releases the worker again.

Placement: the choice of one worker for one actor, made at each resume.

The filters

A worker is a candidate when all of these are true:

1. It is registered, and its state is WORKER_STATE_ACTIVE.
2. Its sandbox class is the class of the snapshot that the actor resumes from.
3. It is on the node that holds the local snapshot of the actor, if a local snapshot exists.
4. Its runtime can restore the memory snapshot, if the snapshot has memory.
5. The labels of its pool match the workerSelector of the template and of the actor.
6. It has room: one free actor place, and free CPU and memory for the booking of the actor.

With no candidate, the call fails with no worker has room for the actor. The text is the same whichever filter removed the last worker. So the error does not always mean that the pool is full. It can mean that the only workers with room are of the wrong class or without the right label.

With one candidate, that worker gets the actor. With more, the scheduler takes two at random and keeps the less loaded one. The load of a worker is its fullest dimension: actor places, CPU, or memory.

The bind is one Postgres transaction. It locks the row of the worker, checks the room again, and writes the assignment. Two resumes that compete for the last place cannot both win. The loser tries again with another worker.

What our runs showed

We created four actors on a pool of two workers and sent each one a request, one after the other.

ATESPACE   NAME    TEMPLATE           STATE                   WORKER POD                       WORKER IPS    VERSION   AGE
lab-c      b0-1    lab-c/ballast-0    ACTOR_STATE_RUNNING     lab-c/ballast-65cf95d875-bzgjd   10.244.0.26   43        69s
lab-c      b0-2    lab-c/ballast-0    ACTOR_STATE_RUNNING     lab-c/ballast-65cf95d875-f65dl   10.244.0.27   3         68s
lab-c      b0-3    lab-c/ballast-0    ACTOR_STATE_RUNNING     lab-c/ballast-65cf95d875-bzgjd   10.244.0.26   3         66s
lab-c      b0-4    lab-c/ballast-0    ACTOR_STATE_RUNNING     lab-c/ballast-65cf95d875-f65dl   10.244.0.27   3         65s
lab-c      b0-5    lab-c/ballast-0    ACTOR_STATE_SUSPENDED   <none>                                         1         63s

The actors alternate between the two pods. With two candidates, the scheduler always compares both, and the emptier one wins. The fifth actor found no candidate and stayed suspended.

The parking run shows the other half of the rule. Actor p3 waited for a worker. When we suspended p1, p3 resumed on the pod that p1 had used. An actor has no home worker. It goes wherever a place is free at that moment.

Across two nodes

The scheduler compares workers, not nodes. On the cluster with two worker nodes, the pool had four workers, two on each node. We created 20 actors, sent each one a request, and left all of them running.

node multi-worker 10
node multi-worker2 10
worker lab-n/ballast-5d6cf97747-xlw7d 6
worker lab-n/ballast-5d6cf97747-dpjlc 5
worker lab-n/ballast-5d6cf97747-5cmpq 5
worker lab-n/ballast-5d6cf97747-67bkm 4

We measured this spread on the second x86 server guest (16 vCPU, 23 GB), with one run of 20 actors on gVisor. Ten actors went to each node, and the four workers got 6, 5, 5, and 4. One create and one resume through the CLI took 0.95 to 1.19 seconds, with a median of 1.00 seconds. Raw runs

The node of the last suspend

Filter 3 is the one that surprises people. After an asynchronous suspend, the snapshot is on the disk of one node, and the actor remembers that node.

  assignedNode: kind-control-plane
  lastAssignedGeneration: 2

While assignedNode is set, only workers on that node are candidates. The reason is speed: a resume from the local disk skips the download from the object store. In our runs the node kept the actor for 30 minutes after the suspend, the default of the atelet flag --snapshot-detach-delay. After that the field was empty and any node could take the actor. Snapshot fidelity, and where a snapshot lives has the measured timeline.

On the cluster with two worker nodes, the rule held in every resume.

We measured one actor on the second x86 server guest (16 vCPU, 23 GB), with five asynchronous suspends and one request after each. All 5 resumes went to the node of the suspend. The worker pod changed between the two workers of that node. Then we ran kubectl cordon on the node. The next 2 resumes still went to that node, in 0.41 and 0.29 seconds. Raw runs

2026-11-12T12:02:26.555Z	cordon	multi-worker2
2026-11-12T12:02:26.986Z	resume-call-while-cordoned	200	0.408920	count=8 mb=50 sum=6613 process_age_s=8.2
2026-11-12T12:02:27.433Z	resumed-on	multi-worker2 pod lab-n/ballast-5d6cf97747-xlw7d
2026-11-12T12:02:28.977Z	suspended-sync	ACTOR_STATE_SUSPENDED	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen8:multi-worker2
2026-11-12T12:02:29.292Z	resume-call-while-cordoned	200	0.293103	count=9 mb=50 sum=6613 process_age_s=9.0
2026-11-12T12:02:29.727Z	resumed-on	multi-worker2 pod lab-n/ballast-5d6cf97747-xlw7d

A cordon is a Kubernetes mark for new pods, and the Substrate scheduler does not read it. So a cordon does not keep a suspended actor away from its node. The fourth line shows one more fact. After a synchronous suspend the snapshot is complete, and the actor still names the node.

The rule has a cost when the node is full while another node has free workers. The code handles that case in the same resume call. ate-api-server asks atelet on that node to upload the snapshot and delete the local copy, and then it tries the other nodes.

We measured that path once, on the second x86 server guest (16 vCPU, 23 GB), with both worker nodes up. We filled all four workers, and a request to the suspended actor got 503 after 5.05 s, in 2 of 2 requests. The body was no worker has room for the actor, and the ate-api-server log names the step DetachFromFullNode. We then freed one worker on the other node, which did not hold the snapshot. The next request returned 200 in 0.61 s on that node, and the count of the actor continued. Raw runs

A drain takes the same path. After kubectl drain, the node had no worker, and its atelet still ran. Three suspended actors resumed on the other node at their first request, in 0.49 to 1.19 s. When a dependency is down has the five drain cases.

When the node stops

Our reading of the code said that an actor with a durable snapshot forgets a node that is down and resumes on another node. Our runs did not show that. We stopped one worker node four times, with the actor in a different condition each time. The other node had two active workers with room in every run.

First, what Substrate knows about the node.

We measured the detection on the second x86 server guest (16 vCPU, 23 GB), in 4 of 4 stops. The workers of the stopped node were listed WORKER_STATE_ACTIVE for 43 to 51 seconds after the stop command. Then they were marked WORKER_STATE_UNAVAILABLE, at the time that Kubernetes reported the node NotReady. Kubernetes evicted the worker pods 5 minutes later, and the Deployment created two new pods on the other node. Raw runs

{"time":"2026-11-12T12:03:19.231654833Z","level":"INFO","msg":"Syncer: marking worker unavailable (pod not Ready)","worker":"090b9723-a53d-4e0c-a139-741942971a78","pod":"lab-n/ballast-5d6cf97747-67bkm"}

The reason in that line is a pod that is not Ready. Substrate learned of the dead node from Kubernetes, and in our runs nothing in Substrate reported it sooner. For those 43 to 51 seconds, the scheduler can give a new resume to a worker on a node that is gone.

Second, what happened to the actor in each of the four runs.

2026-11-12T12:02:36.103Z	suspended-sync	ACTOR_STATE_SUSPENDED	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2
2026-11-12T12:02:36.563Z	node-stop	docker stop multi-worker2
2026-11-12T12:02:46.908Z	node-stopped	multi-worker2
2026-11-12T12:02:57.202Z	probe	504	10.055616		|	ACTOR_STATE_RESUMING	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2
2026-11-12T12:03:07.214Z	probe	503	7.777073	actor lab-n/u1 unavailable	|	ACTOR_STATE_RESUMING	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2
…
2026-11-12T12:08:14.892Z	probe	503	5.051588	actor lab-n/u1 unavailable	|	ACTOR_STATE_RESUMING	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2
2026-11-12T12:08:20.312Z	probe	503	3.190662	actor lab-n/u1 unavailable: AssignWorker prerequisite not met for Actor: lab-n/u1 (got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPENDED)	|	ACTOR_STATE_CRASHED	multi-worker2	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:multi-worker2

Our client allowed 12 s for each of these requests. So the 504 at 10.06 s came from the dataplane, at the limit that section 4.4 describes. We first read it as our own timeout.

A suspended actor did not move to the other node. This is the result that is against our reading of the code. In the first run the snapshot was complete in the object store, and the other node had two free workers. The actor still stayed bound to the stopped node. It was in ACTOR_STATE_RESUMING for 5 min 33 s from its first request, and it was ACTOR_STATE_CRASHED 5 min 44 s after the stop command. Substrate v1.0.0 did not move an actor off a dead node by itself in any of our runs.

A revert did not free the actor either, in 4 of 4 attempts in three runs. In the second run it came too early. The workers of the stopped node were still listed as active, and the next request chose one of them. In the third run the resume that was in progress blocked both attempts. In the fourth run the revert needed atelet on the stopped node, and the actor stayed in ACTOR_STATE_REVERTING.

Those four runs left one question: does a revert help when you send it after the workers are marked unavailable? A second set of 13 node-stop cases answered it. Yes, for one kind of actor only.

We measured each case once, on a second x86 server guest (16 vCPU, 23 GB). The cluster had two worker nodes, the Envoy dataplane, and the default timers. In the first case, an actor was suspended with --mode sync and got no request after the stop. The workers of the node showed WORKER_STATE_UNAVAILABLE at 45 s. One revert then took 7.1 s and returned the actor with no node. The next request resumed it on the other node in 0.56 s, and the count continued from the snapshot. Raw runs

2026-11-12T12:53:34.330Z	t=0.7	node-stopped	multi-worker2
2026-11-12T12:54:18.228Z	t=44.6	workers-of-stopped-node-not-active	      2 WORKER_STATE_ACTIVE       2 WORKER_STATE_UNAVAILABLE 
2026-11-12T12:54:25.821Z	t=52.1	revert	7.1 s | ATESPACE   NAME   TEMPLATE           STATE                   WORKER POD   WORKER IPS   VERSION   AGE lab-n      a1     lab-n/ballast-50   ACTOR_STATE_SUSPENDED   <none>                    8         56s 
2026-11-12T12:54:26.821Z	t=53.1	first-request	200	0.560088	count=4 mb=50 sum=6613 process_age_s=4.9	|	ACTOR_STATE_RUNNING	multi-worker	SNAPSHOT_OWNER_ACTOR:SNAPSHOT_STORAGE_STATE_COMPLETED:gen2:-

The condition is narrow. The same kind of actor, reverted 10 s after the stop command, went to ACTOR_STATE_REVERTING and stayed there until the node was back. An actor that had got a request was refused 19 of 19 times while it was ACTOR_STATE_RESUMING. An actor with a PENDING snapshot, and a RUNNING actor, went to ACTOR_STATE_REVERTING at the first revert. In all of these cases the revert calls atelet on the stopped node, and that call fails.

One path worked in every state: a tag that exists before the node stops. A new actor from the tag served its first request on the other node 1.2 s after the create command, in 3 of 3 rescues. Section 8.2 has the runbook for each case, and figure 8.2d puts it on one page. Without a tag, treat a stopped node as an outage for every actor that got a request.

A RUNNING actor on a dead node is the worst case to see. Its state did not change in 10 minutes, so a list of actors shows nothing wrong. CRASHED and RevertActor has the error texts for each case.

When the node returned, it was Ready within seconds. It had none of the four pool pods in three of the four returns, because the new pods ran on the other node. Kubernetes does not move them back, and no new actor went to the returned node for more than 3 minutes. After we deleted two pool pods by hand, the first actor went to the returned node 23 to 35 seconds after the node start.

The second set of cases gave the same split, with more returns. After an outage shorter than the eviction time, the node kept its pool pods. The first new actor went to it 23 to 45 seconds after the start, in five returns. After a longer outage the node had no pool pod, in seven of seven returns. No new actor went to it in the 216 to 223 seconds that we watched. Substrate does not rebalance a pool.

We measured the case where the other node has no room, in two runs on the same guest. The revert was accepted, and each request got 503 after 5 s with no worker has room for the actor, in 8 of 8 requests. The actor stayed ACTOR_STATE_SUSPENDED. It answered 1.6 s after we freed a worker in one run, and 1.7 s after we scaled the pool in the other. When a dependency is down has both runs. Raw runs

The next edition adds a pool with spare workers on a third node and a node loss with the microVM class. It also adds a node that is alive and unreachable.

After a node restart

A restart of the node removes every worker at one time. We did that twice on the single node of the x86 server guest. The first run sent one request after the pods were back. The second run tried one resume every 2 to 3 seconds and listed the workers each time.

We measured the first recovery on the x86 server guest (24 vCPU, 47 GB, nested KVM), one run. The Kubernetes API answered again after 8 seconds, and all pods were running after 147 seconds. A request for a suspended actor then returned 503 with no worker has room for the actor. The same request returned 200 about 80 seconds later. Raw runs

The second run shows why.

We measured this timeline on the x86 server guest (24 vCPU, 47 GB, nested KVM), with one restart and one suspended actor. Each poll made one resume attempt. Raw runs

The mechanism has three steps. When the control plane returns, it first serves the worker list that it had before the restart. Then it marks each of those workers unavailable. Each worker becomes active again when it registers through atelet, as WorkerPool describes. Between the second and the third step, no worker of the pool is a candidate.

A resume in that window fails for lack of room or for an unavailable worker. The window was 10 to 30 seconds in this run. In the first run our one retry came 80 seconds later, so the window looked longer than it was.

The long part of the recovery is the control plane, not the workers. ate-api-server needed 160 of the 191 seconds.

Plan for this after a node restart: no call works until ate-api-server answers, and resumes can fail for about 30 seconds more. Do not use a check that all pods are Running as the signal. Kubernetes showed the old pod status at first: 0 pods not running at 10 seconds, and 21 at 18 seconds.

An actor that was running on a node that restarts is ACTOR_STATE_CRASHED, and CRASHED and RevertActor covers it. A node that stays down is different, as the table above shows. We did not repeat the restart on a cluster where the control plane and the workers are on different nodes.

Steer placement with labels

You have two tools, and both are labels:

• Pool labels and selectors. Put a label on the WorkerPool object, and select it in the template with workerSelector.matchLabels. An actor can add its own selector, and both must match.
• Node rules for the pool. spec.template.nodeSelector, tolerations and nodeAffinity of the pool decide which nodes the worker pods use. They act on Kubernetes, before any actor exists.

There is no way to pin one actor to one worker, and no priority between actors. A template can select pools with different CPU types, and that is unwise. A memory snapshot from one type cannot always restore on the other.