ADR 0062 — the provider-key relay: the worker never holds the provider key
Status: accepted (operator decision D28), delivered as a strangler behind
the provider_key_relay kill switch, default OFF.
Related: ADR 0011 (network proxy), ADR 0019 (the llm-secret exception to
G2), ADR 0047 (CLI agent engines), ADR 0050 (one verb, three faces), ADR 0051
(provider shelf, per-profile keys), ADR 0059 D3 (V6a provider_hosts; this
ADR is its deferred V6b). Invariants touched: I5, I8, I9, I11, I12.
Question
A sandboxed builtin worker that calls its model provider holds the provider
key. It reads llm-secret / llm-secret-<id> (the B125 carve-out in
sandbox.secret_paths) or the profile's api_key_env variable when the
operator put it on the env allowlist. Anything that runs in the worker —
a test the model wrote, an install script, a compromised dependency — can
read the key and send it to any host the run may reach. ADR 0059 deferred
the fix (V6b) because netproxy is a CONNECT tunnel: it cannot add a header
inside TLS.
Decision
1. Transport: a route on the run's existing filtering proxy
The relay is not a new listener. It is a second route on the per-run
FilteringProxy (netproxy.py) that dispatch already starts for every run
with a domain allowlist, and which every backend already lets the worker
reach:
- Seatbelt pins the worker's egress to
localhost:<proxy port>. - bubblewrap (
--unshare-net) reaches it throughskep.netshim, which presents the same port on the sandbox loopback and bridges to the proxy's AF_UNIX door. Ones-r5b-landlockthat door is one of the two socket-connect rules (the worker's roots and the proxy bridge), so the relay needs no new rule. - With
es-r2a-network(D6: loopback closed by default) the proxy port is still the one loopback port a run may dial. The relay rides it.
A forward-proxy request is always absolute-form (POST http://host/...) or
CONNECT. The relay takes the two shapes that are addressed to the proxy
itself: an origin-form request (POST /api/chat, the worker dialled the
proxy port directly, as Seatbelt's NO_PROXY=127.0.0.1 makes it) and an
absolute-form request to 127.0.0.1:<the proxy's own port> (bubblewrap,
where the worker's client goes through HTTP_PROXY). Before this ADR both
were refused (400 / a loop to itself). Nothing that worked before changes.
The worker's provider URL becomes http://127.0.0.1:<port> plus the
upstream base URL's own path, so every path rule the clients apply to a base
URL (openai_style_prefix's bare-host /v1, ollama's /api strip) gives
the same path through the relay as direct. The relay replaces only the
origin: it forwards to the upstream's scheme, host and port, over TLS when
the upstream is https, refusing a path outside the upstream base path.
A run whose network is allow-all has no proxy today. With the relay ON it
gets a relay-only proxy (empty allowlist); the worker reaches its port
because allow-all permits loopback. A deny-all run never needs one: a caste
that calls a provider always has provider_hosts. A podman allow-all run
is not relayed: podman gives every run its own network namespace, so the
host's loopback is out of reach (a podman domain-list run already falls
back to the native backend); it keeps today's path and its record says so.
Rejected:
- A unix socket bound into the sandbox. A second door per run, a second Landlock rule, a second path for R4 to rebuild, and Seatbelt would need its socket allowed by path. The proxy door already exists on every backend.
- A new loopback port. On
es-r2a-networkthat is a new hole in the loopback wall that has to be named in every profile. - TLS interception in the CONNECT tunnel. A CA in the worker's trust store, for every host, to rewrite one header.
2. Per-run capability: a token bound to the run and to one upstream
The worker sends a per-run token where its client already sends a key
(Authorization: Bearer <token>; x-api-key: <token> for anthropic), so the
protocol clients in serve/llm.py change in no line. The relay compares it
in constant time, strips every credential header (authorization,
x-api-key, api-key, x-goog-api-key, proxy-authorization) and adds the
real key in the protocol's header. A request without the token gets 403
and no upstream connection.
The token is HMAC-SHA256(relay-secret, task_id). relay-secret is a 0600
file the supervisor creates in the supervisor home before it writes any
sandbox profile (ES-F264: the mask list holds only files that exist, so a
secret made on first use was readable by a run already going); its name
matches *-secret, so the sandbox mask list and the secret floor already
hide it from every worker and every Queen read tool. One relay serves one
run: another run's worker holds another token, and a relay is bound to one
upstream origin and base path, so a token cannot reach another provider.
Deriving the token, rather than drawing it at random, means a restarted
supervisor recomputes it without storing it anywhere (section 7).
3. Kill switch
provider_key_relay (bool, default false), a global knob set like every
other supervisor setting: PUT /api/policy, the chat set_policy card, the
web UI, and listed in the policy registry. Dispatch reads it from the store
for each run, so skep run and serve obey the same switch. OFF is exactly
today's path. Each run's dispatched line records which path it used.
4. Golden-parity gate
tests/supervisor/test_provider_relay_parity.py sends the same model calls
(one per relayed protocol: ollama, openai-compat, anthropic,
openai-responses) direct with the key and through the relay with the token,
against the repo's fake providers, and requires the recorded requests to be
identical apart from the auth header: path, query, every other header (Host,
Connection and Content-Length included) and the JSON body, and the parsed
replies identical. The credential header itself must match too: the worker
sent the token, the provider must get the key in the same header and form.
The gate's first run caught one difference (the relay dropped the worker's
Connection: keep-alive); the relay now passes a plain keep-alive/close
through. The default may only flip to ON with this gate and the full suite
green.
5. Which providers and engines
- Builtin worker castes (coding, document, reviewer) — relayed when ON.
They resolve their provider through one function
(
llm_plan.worker_provider_from_env); the supervisor resolves it with the same code (worker_provider_from_home) and hands the worker the relay URL, the token, the protocol and the model. The worker no longer reads a key file, and the sandbox masks them (the B125 carve-out is not applied), and the profile'sapi_key_envvariable is dropped from the worker env. - Not relayed, today's path:
bedrock: SigV4 signs the host, path and body with the secret, so a relay cannot swap a header; it would have to re-sign, i.e. hold the secret key in the signer. Possible later; out of scope.- a provider with no key (local ollama): nothing to hide.
- claude_code can be pointed at a base URL (
ANTHROPIC_BASE_URL) and sendsx-api-keyorAuthorization: Bearer. codex can (OPENAI_BASE_URL/ amodel_providersentry) and speaks the Responses API. aider can, per provider, through litellm (OPENAI_API_BASE,ANTHROPIC_API_BASE, …), but which provider is in the operator's aider config, which skep does not read. pi is multi-provider by its own config. All four authenticate from their own auth env (ADR 0047, W33) and stay on that path this round.skep doctorsays so in one line. Relaying claude_code and codex is a follow-up: the same route with the engine's base-URL variable, once each engine's full request set (count-tokens, model lists, websocket upgrades) passes the parity gate.
6. Streaming, logging, accounting
The relay streams: it forwards the response status and headers (minus hop-by-hop), then the body as it arrives, closing the connection to end it. SSE and NDJSON reach the worker chunk by chunk, as through the tunnel.
It logs, per request, one counter (relayed, relay_refused) on the proxy
next to allowed / denied. Never the key, the token, a header value or a
body: the run record already keeps the prompts it keeps (I8), and the relay
adds none. Usage accounting is unchanged: the worker still counts tokens
from the stream (ProviderUsageTally) and the budget's
max_provider_calls still binds it; the relay's request count is a check
on that self-report, not a second budget (I5).
7. Without serve, and serve restarting mid-run
The relay lives in the process that runs dispatch and spawns the worker. Under
skep run that is the CLI process; under serve it is serve. Both start it
through the same code. No serve is needed.
On main, a serve restart ends the run (recovery kills the worker), and the
relay dies with the proxy. With re-adoption (es-r4-readopt, ADR 0060) the
new serve restarts the run's proxy on the recorded port from task.json;
the relay comes back with it because nothing about it is stored: the
upstream and key are resolved again from the same configuration, and the
token is recomputed from relay-secret and the task id. The window between
the old serve's death and the new proxy's start refuses provider calls, as
it refuses all egress today. When R4 lands, its proxy restart must pass the
relay too; that is one call site.
Strangler order
- Relay route and kill switch, default OFF (nothing calls it).
- The golden-parity gate.
- The builtin worker uses it when ON.
- The key is absent from the worker (env and files) when ON, proved by a real sandboxed worker reading its own environment.
- Doctor line.
The old path (key files readable to LLM castes, api_key_env passthrough)
stays until the relay has run ON as the default through a dogfood period; it
is deleted only then, which is when this door becomes one-way.
Consequences
- With the relay ON, a worker's only credential is a token that works on its own run's relay, for its own provider, while the run lives.
supervisor.sqlite3stays readable to a sandboxed worker this round (other worker code reads settings from it). With the relay ON the worker no longer needs it for its provider, so masking it is now possible, as a follow-up.~/.npmrcis unrelated to the provider key and stays readable.- Workspace mode has no wall: with the relay ON the key leaves the worker's environment, but the key files remain readable to a workspace-mode worker.
- A worker can still spend the operator's quota through the relay, at its own provider, within its budget. That was true with the key; what it can no longer do is take the key away.