2.3

The checkpoint write path: put, put_writes, and durability

A crash right after a node returned reran it in 97 of 100 sync and 95 of 100 async trials, so key node side effects on the task namespace.

Three nodes run in parallel. One asks the model a question, one calls a weather tool, and one is still working when the process dies. You run durability="sync", because the docs call it the safe mode. You resume the thread, and the model node runs again. Its answer was back, and its side effect, had it had one, would now happen twice.

When you finish this section, you can say which saver call makes each part of a superstep durable, and what a crash repeats. You can also build an idempotency key from the task namespace.

Two writes for every superstep

put_writes: the saver call that stores the writes of one finished task, as rows of the writes table. The rows hang on the checkpoint that the superstep started from.

put: the saver call that stores one checkpoint, the state after a superstep, as one row of the checkpoints table.

When a task finishes, the loop records its writes in memory and submits put_writes to a background executor. By default nothing waits for that future. The DeltaChannel writes are the exception: _checkpointer_put_after_previous waits for the pending write futures and for the previous put. After the update phase, _put_checkpoint builds the checkpoint and submits put. With durability="sync", the loop then waits for that one future before it plans the next superstep. That is all sync waits for.

With durability="exit", put_writes is never called during the run, and the intermediate _put_checkpoint calls skip the saver. The one checkpoint is saved when the run exits.

The order of the calls, captured

We recorded every saver call of one create_agent run on SqliteSaver, with Qwen3 4B on the x86 server guest. The model asked for two tool calls and then answered. This is the sync run, from the tools step to the end:

outputsaver calls of the tools step and the last model step, durability syncFull log
  [ 21] tick step=2 run=['tools', 'tools']
  [ 22]   task done  tools      -> loop.put_writes ['messages', 'branch:to:model']
  [ 23]   task done  tools      -> loop.put_writes ['messages', 'branch:to:model']
  [ 25]   loop._put_checkpoint source=loop exiting=False
  [ 27]     SAVER.put_writes ckpt=..3a2745cf3dd2 task=9360c94c channels=['messages', 'branch:to:model']
  [ 28]     SAVER.put_writes ckpt=..3a2745cf3dd2 task=b0bb9318 channels=['messages', 'branch:to:model']
  [ 29]     SAVER.put id=..0f5fddfed5a3 parent=..3a2745cf3dd2 step=2 source=loop
  [ 31] tick step=3 run=['model']
  [ 32]   task done  model      -> loop.put_writes ['messages']
  [ 34]   loop._put_checkpoint source=loop exiting=False
  [ 36]     SAVER.put id=..930ea1dbdfc0 parent=..0f5fddfed5a3 step=3 source=loop
  [ 39]     SAVER.put_writes ckpt=..0f5fddfed5a3 task=7f67758b channels=['messages']

Read lines 34 to 39. The checkpoint of step 3 was stored at line 36, and the loop moved on. The writes of the model task of that step, 7f67758b, reached the saver at line 39, after the checkpoint that already holds them. Here the order did no harm, because the checkpoint covers the step. It shows that the loop does not wait for put_writes, in sync too.

The harm comes earlier in a superstep. A task that returned while its siblings still run has only its put_writes to keep its output. In one run on a loaded guest, each of the five tasks waited 3.0 to 4.4 ms for that call to finish. Figure 2.4 zooms into the end of each superstep.

Fig. 2.4how long each finished task waits for its put_writes, in one sync runtimeline
How long each finished task waits for its put_writes, in one sync run Four panels, one per superstep of one create_agent run with Qwen3 4B, durability sync, on SqliteSaver on the x86 server guest. Each panel spans 7 ms after the first task of the step returned. The upper lane is the loop: a dot for each task that returned and an amber line where the next superstep starts. The lower lane is the background saver: filled blocks are put calls that store a checkpoint, and outlined blocks are put_writes calls that store the writes of one task. A rose band runs from each task's return to the end of its put_writes. Those spans were 3.0 to 4.4 ms, in one run on a loaded guest. In step 3, the checkpoint put ended before the put_writes of the task 7f67758b. Between the panels the model ran for seconds. loop saver step 0 0 2 4 6 start model 3.55 s step 1 0 2 4 6 model tools 0.01 s step 2 0 2 4 6 tools, tools model 1.54 s step 3 0 2 4 6 model ms 1 2 1 step 3: the checkpoint put ends, sync stops waiting, and step 4 starts 2 the put_writes of 7f67758b ends 1.4 ms later task returned put put_writes not durable, 3.0 to 4.4 ms next superstep starts between panels: the next task and its seconds
In one loaded run, a finished task was not durable for 3.0 to 4.4 ms, and sync waited only for the checkpoint put. Each panel is the end of one superstep, zoomed to milliseconds after its last task returned. The upper lane is the loop, the lower lane is the background saver. Filled teal blocks are puts, outlined blocks are put_writes calls, and rose bands are the spans before a task is durable. The labels between panels name the next task and how long it took.

The same trace shows a gap at the start. The input checkpoint, step -1, reached the saver at line 9, after the __start__ task had returned at line 5. So sync does not wait for the input checkpoint before the first task. It waits before the first node of your graph.

A crash between finished tasks and the checkpoint

The test graph runs a, b, and c in parallel, then d, on SqliteSaver. a asks Qwen3 4B to name a fruit, and b calls the get_weather tool. c waits until both have returned, waits a set time, and calls os._exit(17). A new process then resumes the thread with invoke(None) and logs each node that runs.

We measured this on the x86 server guest, 50 trials in each cell, in two runs. Run 1 is the crash matrix. Run 2 repeats the zero and 0.2 s cells and logs the task namespaces too. The intervals are Wilson 95 percent intervals: Raw runs

ModeWait before the crashRun 1: reran aRun 2: reran aPooled, Wilson 95 percent
sync0 s49 of 5048 of 5097 of 100, 0.915 to 0.990
async0 s45 of 5050 of 5095 of 100, 0.888 to 0.978
sync0.2 s0 of 500 of 500 of 100
async0.2 s0 of 500 of 500 of 100
exit0 s and 0.2 sno checkpoint, 100 of 100not run

b never ran again in any trial. The tool returned in milliseconds, so its put_writes finished long before the crash. a returned last, a few milliseconds before c exited, so its writes were still on their way. Every exit trial left no checkpoint, and the resume raised EmptyInputError: Received no input for __start__. Figure 2.5 plots every trial.

Fig. 2.5what the resume ran after a crash in the middle of a superstepchart
What the resume ran after a crash in the middle of a superstep Dot rows for a crash test on SqliteSaver with Qwen3 4B on the x86 server guest. Nodes a, b, and c run in parallel: a asks the model for a fruit, b calls a weather tool, and c exits the process after a and b returned and a wait of 0 or 0.2 seconds. A new process resumes the thread. Each dot is one trial, 50 per cell, two runs. With no wait, sync reran the model node a in 49 and 48 trials, 97 of 100, and async in 45 and 50, 95 of 100. The tool node b never ran again. With a 0.2 second wait every trial reran only c. In exit mode no checkpoint existed in all 100 trials of run 1. Run 2 did not include exit. RUN 1 RUN 2 mode, wait sync 0.0 s 49/50 48/50 async 0.0 s 45/50 50/50 sync 0.2 s 0/50 0/50 async 0.2 s 0/50 0/50 exit 0.0 s none not run exit 0.2 s none not run reran a and c reran only c nothing to resume zero wait, pooled: sync reran a in 97 of 100, async in 95 of 100 the tool node b never ran again, and a 0.2 s wait left only c to rerun
With no wait, the model node that returned last reran in 97 of 100 sync and 95 of 100 async trials, and with 0.2 s in none. Each dot is one trial, 50 per cell, two runs. Rose dots reran the model node a and the crashed node c, teal dots reran only c, and grey dots had nothing to resume.

The durability mode made no difference at zero wait. The intervals overlap, and run 2 even had more repeats under async. Both modes wait at the end of the step, and the crash came before the end. The risk is the window of one put_writes call after each task. On SqliteSaver, that call took a mean of 4.6 ms under sync and 4.2 ms under async, on a loaded guest (section 2.6). In an agent, a model node often returns while a slower sibling still runs, and that is the window.

Why a resume finds the right tasks

Task id: the id of one task in one superstep. The loop computes it as an xxh3 hash. A PUSH task hashes six values, and a PULL task hashes five fixed values plus its triggers. The values are the checkpoint id, the namespace, the step, the node name, PULL or PUSH, and the triggers or the Send index.

The writes of a task are saved under the checkpoint that its superstep started from, with the task id. A resume loads that checkpoint, plans the same tasks, and gets the same ids. Each task with saved writes gets them back, and the runner runs only the tasks with no writes. A task that failed or stopped at an interrupt has no saved output, so it runs again.

The checkpoint id itself is a new uuid6 each time. So the task ids repeat only when the start checkpoint was saved. When it is lost, the resume starts from an older checkpoint and makes new ids. That did not happen in any of the 200 trials of run 2. In those trials, the model call in a took longer than the put of the checkpoint before it.

The task namespace as an idempotency key

A node can read its task namespace, <node>:<task id>, from its config. Run 2 of the crash matrix logged it on the first line of every node:

pythonreading the task namespace in a nodeFull log
        rec["ns"] = get_config()["configurable"]["checkpoint_ns"]
outputthe namespace in the crashed run against the resumeFull log
  durability=sync  wait=0.0s: {"n": 50, "categories": {"a and c": 48, "only c": 2}, … "ns": {"c ns same": 50, "a ns same when a rerun": 48, "a rerun": 48, "b ns same when b rerun": 0, "b rerun": 0}}
  durability=async wait=0.0s: {"n": 50, "categories": {"a and c": 50}, … "ns": {"c ns same": 50, "a ns same when a rerun": 50, "a rerun": 50, "b ns same when b rerun": 0, "b rerun": 0}}
Cut to the namespace counts of each cell.

Every node that ran again had the namespace of its first run: c in 200 of 200 trials, and a in 98 of 98 repeats. A side effect keyed on the namespace would have skipped all 98 repeats of a. That held in sync and in async.

So key the side effects of a node on its namespace. Store the key with the effect, as section 5.3 shows, and skip the effect when the key is there. A new turn plans new tasks with new ids, so the same request in a new turn is a new effect. Under exit, a crash leaves nothing to resume, so build the key from your own data there.

The namespace holds only while the start checkpoint is saved. When that checkpoint is lost, the replay gets new task ids. Then only a business key that you store, such as an order id, stops a repeated effect.

LangChain (Python): langchain, langchain-core, langgraph, deepagents, langsmith 1.5.0 · c2f4036 · 2026-11-11 · Edition 2026.11