Documentation menu
On this page

ADR 0060 — serve re-adopts a live worker after a restart

Status: accepted (edge sweep decision D2, round R4; ES-F130–ES-F136; Linux bubblewrap ES-F210–ES-F212). Related: ADR 0006 (supervisor re-verification), ADR 0008 (suspend and resume), ADR 0011 (network proxy), ADR 0017 (the serve daemon). Invariants touched: I1, I2, I8, I12.

Question

Restarting skep serve killed every run in flight. On startup recover_interrupted_runs killed any worker still running (ES-F5, B10) and marked the run worker_crashed; the worktree survived only when a version-2 checkpoint existed. React-style runs write one every round; plan-style runs only at approval gates, so a plan run lost its worktree and its work (field test finding L18). The operator chose (D2): serve re-adopts the running worker and supervises it to completion as if it had never restarted — with one deliberate exception (D35, section 3): a re-adopted run never auto-applies; it lands by a human approval.

Decision

1. How a worker outlives its parent

The spawner starts every worker with start_new_session=True: the worker leads its own session and process group, so the terminal's SIGHUP and a signal to serve's group do not reach it. Its stdout and stderr go to audit/<task>/worker.log, not a pipe, so serve's exit does not SIGPIPE it. The worker is reparented to init (launchd on macOS, systemd or pid 1 on Linux), which reaps it when it exits.

  • macOS (Seatbelt). sandbox-exec execs the worker in place: same pid, same start time, no signal handling of its own. A sandboxed worker survives serve's exit. Its only egress is localhost:<proxy port>, which is baked into its Seatbelt profile and its environment.

  • Linux (bubblewrap). bwrap keeps --die-with-parent (ES-F2, B50) in every invocation; what changes is who its parent is (ES-F210–ES-F212). Dropping the flag would be unsafe: bwrap runs the sandbox in a session of its own (--new-session) under a pid namespace whose init is bwrap's child, so kill_session on bwrap's group would kill only the outer bwrap and leave the sandbox running, reparented to init, with nobody to reap or bound it (measured on bwrap 0.13). With re-adoption on, the spawner starts a keeper in bwrap's place: a python -c one-liner that runs bwrap as its child, waits for it and exits with its status. The keeper is what serve spawns (start_new_session), so its pid and start time are what the run records and what a new serve verifies; pids inside the namespace are never recorded. Serve's death no longer reaches bwrap. Killing the keeper's session (a breach, kill_session) kills the keeper and bwrap together, bwrap's death still takes the sandbox down, and if the keeper alone dies bwrap dies with it — the keeper lives exactly as long as the sandbox. The keeper also bounds a sandbox no serve comes back for: it kills bwrap at the run's wall clock + grace + 60 s, a minute after the monitor would have, so a live monitor always acts first. With --no-readopt there is no keeper: bwrap is serve's child and dies with it, as before this ADR.

    The proxy bridge: bwrap used to bind the host proxy's socket file into the sandbox, which pins the inode of a listener that dies with serve; a re-adopting serve's proxy listens on a new inode at the same path. bwrap now binds the socket's private directory (it holds only that socket) read-only, and the shim connects through it on every connection, so it reaches whichever proxy listens there now. The run records the socket path; the new serve removes the stale socket file and listens on the same path and the same port. The netns is untouched by a restart (the shim inside keeps listening on the sandbox loopback), and the R5 socket masks are unchanged.

  • podman never carried --die-with-parent: the container runs under conmon, outside serve's process tree, and the podman client serve spawns leads its own session, so a podman worker already outlives serve and is adopted like a workspace-mode one. No test covers podman across a restart.

2. What serve records, and how it knows the worker is the same one

At spawn the run's running transition now reads pid <N> started <lstart> [proxy port <P>] [proxy socket <path>] (the socket only on Linux, section 1). <lstart> is the process start time from ps -o lstart= in the C locale and UTC — the same helper the CLI landing claim already uses (W27, store._process_started); no second one. A pid is only a number and is reused; the pair (pid, start time) is not. Serve also records itself as the run's supervisor (runs.supervisor_owner, the W27 {"pid", "started"} owner of the dispatching process).

On startup a run is re-adoptable only when all hold: the recorded start time exists (rows written before this change have none and are killed as before), the process with that pid has that start time (owner_alive), and its command line still names the task (the ES-F5 check, kept). A live process with the recorded pid but another start time is a stranger: it is neither killed nor adopted, and the run takes the dead-worker path.

3. Supervising a process that is not our child

There is no waitpid for a non-child. watch_worker takes an AdoptedWorker in place of the Popen: liveness is the owner check (polled at most twice a second, it runs ps), and the exit code is unknown, so the verdict carries exit_code=None, as recovery already did. Everything else is the existing loop: the event log under .events/<task>.ndjson is the heartbeat, the result envelope under results/<task>/result.json is the claim, and a breach kills the session with the existing kill_session. After the worker exits, the run takes the same ingest → G10 re-verification → approval path recovery already uses (I1, I2), and its finish reaches the dispatching chat as a live run's does: the one-line notice and the Queen's continuation turn (v105-F1).

Landing is the exception (D35). Auto-apply rules are not replayed on this path, as for every recovered run: an adopted run lands only by a human approval, even on a project whose maintain lane or unlocked mode would have auto-applied it. The reason: after a restart the chain of trust is reconstructed, not witnessed. The new serve rebuilt supervision from the record — pid and start time, the event log, the result envelope — and nobody watched the gap between the old serve's death and the adoption; G10 still re-verifies, but the only lane that lands without a human (I1) is the one place a reconstructed chain is not enough, so a human verdict lands it (I1, I13). The run's audit trail says so: an auto_apply_blocked transition names the restart and the skep review <task> --approve that lands it.

The filtering proxy lived in the old serve. Before watching, the new serve starts a proxy with the run's own allowlist and asks (from task.json in the audit dir) on the recorded port. If that port cannot be bound, the worker is killed and the run crashes honestly: its profile still points at that port, and whatever holds it now is not skep's filter (I12).

The supervision runs on a daemon thread, so serve's startup is not held for the length of a run; if serve dies again the worker survives again and the next serve adopts it again.

4. Budgets across the restart

The wall-clock deadline counts from the worker's own start time (lstart, one-second resolution), not from adoption: a restart never extends a budget. A worker already past its budget plus grace is killed on adoption. The heartbeat window starts fresh at adoption (the event file's growth is measured from then); a worker silent for 3× the heartbeat after adoption is killed as before.

5. Two serves on one home

Since ES-F274 a second skep serve on the same home refuses to start: serve holds an exclusive lock on <home>/serve.lock for its life (the OS drops it when serve dies, so no stale pid is judged) and the refusal names the holder's pid and port. What follows still guards everything else that recovers runs (skep run, skep team, two serves started on different homes over one copied store). Recovery skips any run whose recorded supervisor is still alive — a second serve no longer kills the workers of a first one that is still running (it used to). Recovery takes each run, live worker or dead, with a compare-and-set on runs.supervisor_owner in one SQLite UPDATE: of two serves starting together, exactly one wins; the loser leaves the run alone. A run is named with its supervisor when its row is created, so a serve starting while a dispatch is still setting the run up leaves it alone too.

6. Kill switch

skep serve --no-readopt (or SKEP_READOPT_WORKERS=0 in the environment or in serve.env) restores the old behaviour: a live worker is killed with its session and the run is recovered as before. On Linux it also drops the keeper (section 1), so a sandboxed worker dies with the serve that spawned it; the variable (not the flag, which is serve's) does the same for runs a skep run or skep team process spawns. The skip of runs whose supervisor is still alive (5) stays on either way: killing a live sibling's worker was never intended.

Consequences

  • A serve restart no longer costs a plan run its work, sandboxed or not, on macOS and on Linux (section 1).
  • A pid is never trusted alone; a reused pid is left alone (I8, I12).
  • A worker runs without a filtering proxy between serve's death and the next serve's start; its egress is refused for that window, as before.
  • If no serve comes back, nothing enforces the budget of a macOS worker, a workspace-mode worker or a podman one; it runs until it ends on its own. A Linux bubblewrap worker is killed by its keeper a minute past its wall clock + grace.
  • A service manager that stops a whole cgroup on restart (systemd's default KillMode=control-group) kills the workers with serve, and there is nothing left to re-adopt; skep ships no unit file.
  • Rows written before this change carry no start time and are killed, as before. No migration rewrites them.