Add a watchdog before you leave a run alone

We left a run alone for an hour. When we came back, the GPU was idle and the log had grown to 147 MB. The rollout engine had died during startup, and the run had sent requests to it ever since.

When you finish this section, you can recognize a run whose engine died. You can also start a watchdog that stops that run within a few minutes.

Our result: a run whose engine died logged 200,533 HTTP 503 errors in 63 minutes and never failed. The watchdog stopped a test of the same error in 183 s. A replay of a healthy run showed that six checks in a row are too few, so the script waits for ten.

What a dead engine looks like

The run below used a setting that this GPU cannot run. The rollout engine stopped during startup. The trainer did not stop, and it sent requests to the engine for 63 minutes:

[2026-10-17 21:03:03] Enable deterministic inference because of legacy rl_on_policy_target.
AssertionError: FlashAttention v3 Backend requires SM>=80 and SM<=90. Please use `--attention-backend flashinfer`.
[2026-10-17 21:03:24] Received sigquit from a child process. It usually means the child failed.
[2026-10-17 21:03:30.637 rollout_executor] http_utils.py:241 - Error: Server error '503 Service Unavailable' for url 'http://172.17.0.3:20035/generate'
[2026-10-17 22:06:46.137 rollout_executor] http_utils.py:241 - Error: Server error '503 Service Unavailable' for url 'http://172.17.0.3:20035/generate'

$ grep -c "503 Service Unavailable" true-on-policy-fa3.log
200533

The run wrote 200,533 error lines and never reported a failure.

Router: the small service that forwards each rollout request to an engine. Section 8.6 explains it.

503 Service Unavailable: the HTTP status that the router returns when no rollout engine can take the request.

Why the watchdog waits for ten checks

In our 17 runs that ended normally, the log held no 503 Service Unavailable line at all. Our 16-answer run of section 3.1 logged 22,357 of them. They came in bursts of up to about 60 s, with up to 2,209 in one minute. The client retried each request, and the run finished all its steps.

In the failed run, the errors never stopped. Every full minute held at least 2,669 of them, for 63 minutes.

A count alone cannot tell these two runs apart, but the time can. Every 30 s, the watchdog counts the errors of the last 60 s. It stops the run only when that count is above 100 in a set number of checks in a row.

$ python3 watchdog_replay.py out-gb10/e2-n16-503-per-second.txt
healthy run: 22357 errors from 2026-10-22 07:16:29, checks every 30 s over the last 60 s, limit 100
checks  healthy offsets stopped  most checks in a row  dead engine stopped after
     6  17 of 30                    6                     153 to 182 s
     8   0 of 30                    6                     213 to 242 s
    10   0 of 30                    6                     273 to 302 s

We measured the rule on the real error times of that healthy run. With 6 checks, it stopped the run in 17 of the 30 cases, because two bursts came 45 s apart. No case reached more than 6 checks in a row. With 10 checks, no case stopped the healthy run, and a dead engine is stopped in about five minutes. Raw runs

Start the watchdog

The script reads the last 60 seconds of the container log every 30 seconds. It stops the container when the count passes the limit in a set number of checks in a row.

If Docker itself fails, the script says so. It exits with 2 when it cannot list the containers. A failed log read does not count as a check. The script also exits with 2 for a bad limit or number of checks: a value that is not a whole number, or 0 checks.

The third value is the number of checks in a row. Its default is 10, which a healthy run did not reach in our replay:

#!/bin/sh
name=${1:-miles-plain}
limit=${2:-100}
checks=${3:-10}
for value in "$limit" "$checks"; do
  case $value in
    ''|*[!0-9]*|??????????*) echo "limit and checks must be whole numbers below one billion"; exit 2;;
  esac
done
if [ "$checks" -lt 1 ]; then
  echo "checks must be 1 or more"
  exit 2
fi
strikes=0
while :; do
  if ! running=$(docker ps -q --filter "name=^${name}$"); then
    echo "$(date -u +%FT%TZ) $name: docker ps failed, the watchdog cannot see the run and stops"
    exit 2
  fi
  [ -n "$running" ] || break
  if logs=$(docker logs --since 60s "$name" 2>&1); then
    errors=$(printf '%s\n' "$logs" | grep -c "503 Service Unavailable")
    if [ "$errors" -gt "$limit" ]; then
      strikes=$((strikes + 1))
    else
      strikes=0
    fi
  else
    echo "$(date -u +%FT%TZ) $name: docker logs failed, this check does not count"
  fi
  if [ "$strikes" -ge "$checks" ]; then
    echo "$(date -u +%FT%TZ) $name: more than $limit errors from the rollout engine in each of $checks checks, last $errors in 60 s, stopping the run"
    docker stop "$name"
    exit 1
  fi
  sleep 30
done
echo "$(date -u +%FT%TZ) $name: the run ended"

Save it as ~/miles-lab/watchdog.sh, or download the same script:

curl -fsSL -o ~/miles-lab/watchdog.sh https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/manuals/miles-101/capture/gpu/watchdog.sh

Start it in a second terminal right after the run starts, with the container name from section 1.1:

sh ~/miles-lab/watchdog.sh miles-plain 100 10

Test the watchdog first

Test it on two containers that print the error on purpose. They need no GPU. The first prints ten error lines each second and never stops, like a dead engine:

$ sh capture/gpu/watchdog.sh wd-dead 100
2026-10-22T15:41:09Z wd-dead: more than 100 errors from the rollout engine in each of 6 checks, last 578 in 60 s, stopping the run
wd-dead
$ echo $?
1

The watchdog stopped the container 183 s after it started, and exited with 1. We ran this test before we raised the default to 10, so it used 6 checks. With 10 checks, the same rule needs four more checks, so it stops about two minutes later. A script that starts runs one after another can read that exit code and skip to the next run. An exit code of 2 means that the watchdog lost Docker, and the run has no guard.

The second container prints the error for 60 s and then stays quiet for two minutes. It does this three times, like the bursts of a busy engine. The watchdog let it run to its end after 541 s, and exited with 0. Its quiet gaps of two minutes are longer than the real gaps of 45 s, so test with the replay above too.

Guard the GPU heat

Our GB10 reached 84 C again and again under hours of load. The heat guard from the front matter paused our runs 1,005 times in five days. Run it on a GB10, or on any GPU that gets hot when you leave a run alone for hours. A short run of a few steps does not need it.

Download the script that we ran, and start it in a third terminal on the host, not in the container. The rm part deletes a stop file that an earlier session can leave behind, because the script exits at once when that file exists. Delete it only when no queue of yours uses the file to stop the guard. The commands stop at the first failure, so a failed download does not start an old copy:

curl -fsSL -o ~/miles-lab/heat-guard.sh https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/manuals/miles-101/capture/gpu/heat-guard.sh \
  && rm -f ~/miles-lab/queue2.done \
  && sh ~/miles-lab/heat-guard.sh

Every 5 s, the script reads the GPU temperature with nvidia-smi. It pauses the first running container of the radixark/miles:latest image at 84 C. It resumes the container at 74 C or below after at least 45 s, or after 120 s at most. Each pause and resume goes to ~/miles-lab/runs/thermostat.log.

Stop the script with Ctrl+C. It resumes the container that it paused before it exits, so no run stays paused. In our check, Ctrl+C during a pause resumed the run within 5 s. It also stops when the file ~/miles-lab/queue2.done exists, which is how our run queue stopped it.

Read the pause log when you compare times. A step that overlapped a pause took longer for a reason that is not the work. Section 5.3 shows how to time only clean steps.

The watchdog only stops the run. To find out why the engine died, read the engine lines above the first error. Section 7.1 lists the causes we met.