ADR 0060 — serve re-adopts a live worker after a restart
Status: accepted (edge sweep decision D2, round R4; ES-F130–ES-F136; Linux bubblewrap ES-F210–ES-F212). Related: ADR 0006 (supervisor re-verification), ADR 0008 (suspend and resume), ADR 0011 (network proxy), ADR 0017 (the serve daemon). Invariants touched: I1, I2, I8, I12.
Question
Restarting skep serve killed every run in flight. On startup
recover_interrupted_runs killed any worker still running (ES-F5, B10) and
marked the run worker_crashed; the worktree survived only when a version-2
checkpoint existed. React-style runs write one every round; plan-style runs
only at approval gates, so a plan run lost its worktree and its work (field
test finding L18). The operator chose (D2): serve re-adopts the running
worker and supervises it to completion as if it had never restarted —
with one deliberate exception (D35, section 3): a re-adopted run never
auto-applies; it lands by a human approval.
Decision
1. How a worker outlives its parent
The spawner starts every worker with start_new_session=True: the worker
leads its own session and process group, so the terminal's SIGHUP and a
signal to serve's group do not reach it. Its stdout and stderr go to
audit/<task>/worker.log, not a pipe, so serve's exit does not SIGPIPE it.
The worker is reparented to init (launchd on macOS, systemd or pid 1 on
Linux), which reaps it when it exits.
-
macOS (Seatbelt).
sandbox-execexecs the worker in place: same pid, same start time, no signal handling of its own. A sandboxed worker survives serve's exit. Its only egress islocalhost:<proxy port>, which is baked into its Seatbelt profile and its environment. -
Linux (bubblewrap). bwrap keeps
--die-with-parent(ES-F2, B50) in every invocation; what changes is who its parent is (ES-F210–ES-F212). Dropping the flag would be unsafe: bwrap runs the sandbox in a session of its own (--new-session) under a pid namespace whose init is bwrap's child, sokill_sessionon bwrap's group would kill only the outer bwrap and leave the sandbox running, reparented to init, with nobody to reap or bound it (measured on bwrap 0.13). With re-adoption on, the spawner starts a keeper in bwrap's place: apython -cone-liner that runs bwrap as its child, waits for it and exits with its status. The keeper is what serve spawns (start_new_session), so its pid and start time are what the run records and what a new serve verifies; pids inside the namespace are never recorded. Serve's death no longer reaches bwrap. Killing the keeper's session (a breach,kill_session) kills the keeper and bwrap together, bwrap's death still takes the sandbox down, and if the keeper alone dies bwrap dies with it — the keeper lives exactly as long as the sandbox. The keeper also bounds a sandbox no serve comes back for: it kills bwrap at the run's wall clock + grace + 60 s, a minute after the monitor would have, so a live monitor always acts first. With--no-readoptthere is no keeper: bwrap is serve's child and dies with it, as before this ADR.The proxy bridge: bwrap used to bind the host proxy's socket file into the sandbox, which pins the inode of a listener that dies with serve; a re-adopting serve's proxy listens on a new inode at the same path. bwrap now binds the socket's private directory (it holds only that socket) read-only, and the shim connects through it on every connection, so it reaches whichever proxy listens there now. The run records the socket path; the new serve removes the stale socket file and listens on the same path and the same port. The netns is untouched by a restart (the shim inside keeps listening on the sandbox loopback), and the R5 socket masks are unchanged.
-
podman never carried
--die-with-parent: the container runs under conmon, outside serve's process tree, and the podman client serve spawns leads its own session, so a podman worker already outlives serve and is adopted like a workspace-mode one. No test covers podman across a restart.
2. What serve records, and how it knows the worker is the same one
At spawn the run's running transition now reads
pid <N> started <lstart> [proxy port <P>] [proxy socket <path>] (the
socket only on Linux, section 1). <lstart> is the process start
time from ps -o lstart= in the C locale and UTC — the same helper the CLI
landing claim already uses (W27, store._process_started); no second one.
A pid is only a number and is reused; the pair (pid, start time) is not.
Serve also records itself as the run's supervisor (runs.supervisor_owner,
the W27 {"pid", "started"} owner of the dispatching process).
On startup a run is re-adoptable only when all hold: the recorded start
time exists (rows written before this change have none and are killed as
before), the process with that pid has that start time (owner_alive), and
its command line still names the task (the ES-F5 check, kept). A live
process with the recorded pid but another start time is a stranger: it is
neither killed nor adopted, and the run takes the dead-worker path.
3. Supervising a process that is not our child
There is no waitpid for a non-child. watch_worker takes an
AdoptedWorker in place of the Popen: liveness is the owner check
(polled at most twice a second, it runs ps), and the exit code is
unknown, so the verdict carries exit_code=None, as recovery already did.
Everything else is the existing loop: the event log under
.events/<task>.ndjson is the heartbeat, the result envelope under
results/<task>/result.json is the claim, and a breach kills the session
with the existing kill_session. After the worker exits, the run takes the
same ingest → G10 re-verification → approval path recovery already uses
(I1, I2), and its finish reaches the dispatching chat as a live run's does:
the one-line notice and the Queen's continuation turn (v105-F1).
Landing is the exception (D35). Auto-apply rules are not replayed on
this path, as for every recovered run: an adopted run lands only by a human
approval, even on a project whose maintain lane or unlocked mode would have
auto-applied it. The reason: after a restart the chain of trust is
reconstructed, not witnessed. The new serve rebuilt supervision from the
record — pid and start time, the event log, the result envelope — and nobody
watched the gap between the old serve's death and the adoption; G10 still
re-verifies, but the only lane that lands without a human (I1) is the one
place a reconstructed chain is not enough, so a human verdict lands it
(I1, I13). The run's audit trail says so: an auto_apply_blocked transition
names the restart and the skep review <task> --approve that lands it.
The filtering proxy lived in the old serve. Before watching, the new serve
starts a proxy with the run's own allowlist and asks (from task.json in
the audit dir) on the recorded port. If that port cannot be bound, the
worker is killed and the run crashes honestly: its profile still points at
that port, and whatever holds it now is not skep's filter (I12).
The supervision runs on a daemon thread, so serve's startup is not held for the length of a run; if serve dies again the worker survives again and the next serve adopts it again.
4. Budgets across the restart
The wall-clock deadline counts from the worker's own start time (lstart,
one-second resolution), not from adoption: a restart never extends a
budget. A worker already past its budget plus grace is killed on adoption.
The heartbeat window starts fresh at adoption (the event file's growth is
measured from then); a worker silent for 3× the heartbeat after adoption is
killed as before.
5. Two serves on one home
Since ES-F274 a second skep serve on the same home refuses to start: serve
holds an exclusive lock on <home>/serve.lock for its life (the OS drops it
when serve dies, so no stale pid is judged) and the refusal names the
holder's pid and port. What follows still guards everything else that
recovers runs (skep run, skep team, two serves started on different
homes over one copied store). Recovery skips any run whose
recorded supervisor is still alive — a second serve no longer kills the
workers of a first one that is still running (it used to). Recovery takes
each run, live worker or dead, with a compare-and-set on
runs.supervisor_owner in one SQLite UPDATE: of two serves starting
together, exactly one wins; the loser leaves the run alone. A run is named
with its supervisor when its row is created, so a serve starting while a
dispatch is still setting the run up leaves it alone too.
6. Kill switch
skep serve --no-readopt (or SKEP_READOPT_WORKERS=0 in the environment
or in serve.env) restores the old behaviour: a live worker is killed with
its session and the run is recovered as before. On Linux it also drops the
keeper (section 1), so a sandboxed worker dies with the serve that spawned
it; the variable (not the flag, which is serve's) does the same for runs a
skep run or skep team process spawns. The skip of runs whose
supervisor is still alive (5) stays on either way: killing a live sibling's
worker was never intended.
Consequences
- A serve restart no longer costs a plan run its work, sandboxed or not, on macOS and on Linux (section 1).
- A pid is never trusted alone; a reused pid is left alone (I8, I12).
- A worker runs without a filtering proxy between serve's death and the next serve's start; its egress is refused for that window, as before.
- If no serve comes back, nothing enforces the budget of a macOS worker, a workspace-mode worker or a podman one; it runs until it ends on its own. A Linux bubblewrap worker is killed by its keeper a minute past its wall clock + grace.
- A service manager that stops a whole cgroup on restart (systemd's default
KillMode=control-group) kills the workers with serve, and there is nothing left to re-adopt; skep ships no unit file. - Rows written before this change carry no start time and are killed, as before. No migration rewrites them.