An approval, ask or park can be resolved by an operator, a rule, a classifier or automatically, so the vocabulary stops naming a person: - step type human.approve -> workflow.approve - park class needs_human -> needs_decision (ParkNeedsDecision) - event kinds human_* -> operator_* - alert kind task.needs_human -> task.needs_decision - status grouping "needs you" -> "needs attention" 0061 rewrites the stored values, including the approval step type in the workflow-definitions resource and each task's pinned definition, with the digest recomputed so resumed tasks still match.
415 lines
25 KiB
Markdown
415 lines
25 KiB
Markdown
# External agent harnesses
|
|
|
|
**Status:** Approved **Scope:** the harness model and, in "Session contract",
|
|
the duplex session that the setup terminal and the run-scoped live view both
|
|
use. Code cites this file for both (`ui/src/harness/SetupTerminal.vue`,
|
|
`internal/infrastructure/postgres/queries/harness_secrets.sql`).
|
|
|
|
## Decision
|
|
|
|
A workflow stage can run on an external coding-agent CLI instead of the built-in
|
|
agent loop: Claude Code, Codex, Pi, OMP, GitHub Copilot CLI, or any other CLI
|
|
packaged as a Docker Sandbox Kit v3. Each authenticates with an API key or the
|
|
operator's subscription login, and can carry the operator's own skills, plugins,
|
|
subagents and instructions.
|
|
|
|
Two boundaries hold regardless of what the harness does:
|
|
|
|
- **The real credential never enters the container.** The container sees
|
|
sentinel values; archie's egress proxy substitutes the real credential on
|
|
outbound requests and performs OAuth refresh on the host.
|
|
- **Archie's workflow guarantees are enforced outside the harness.** Git,
|
|
protected paths, read-only stages, gates and budgets are checked by archie
|
|
around the process and against the worktree afterwards.
|
|
|
|
## Kits
|
|
|
|
A harness is packaged as a Kit v3 image: an OCI image whose
|
|
`vnd.docker.sandbox.kit.descriptor` manifest annotation declares what the
|
|
harness needs. Archie reads descriptors with the published
|
|
`github.com/docker/sandbox-kit-spec/v3/spec` package and does not maintain its
|
|
own descriptor format. Published Kits run unmodified.
|
|
|
|
Archie is a partial Kit runtime. It implements these capability types:
|
|
|
|
| Type | Archie behaviour |
|
|
| ------------------ | ------------------------------------------------------------------------------------------------------------------------- |
|
|
| `credential@1` | Proxy-managed only, described below. |
|
|
| `network-policy@1` | Enforced by the egress proxy. |
|
|
| `volume@1` | A Docker volume keyed by WorkflowExecution, deleted with it. |
|
|
| `lifecycle@1` | Install and startup hooks run on container create. |
|
|
| `agent-sessions@1` | The headless `prompt` and `resume` invocations for the stage. |
|
|
| `agent-skills@1` | Discovery paths for bundled skills and optional shared skills; host access is always read-only. |
|
|
| `agent-skill@1` | Selected image-bundled skills copied into the declared discovery paths. |
|
|
| `agent-context@1` | The resolved discovery directory, defaulting beside the workspace; preserve instructions outside Archie-managed sections. |
|
|
| `resources@1` | Container CPU and memory limits. |
|
|
|
|
Every other type fails closed: a required entry refuses the launch, an optional
|
|
one is skipped and recorded on the step. `credential@1` with `passthrough: true`
|
|
and any credential for a forge service are refused.
|
|
|
|
The operator's own setup (plugins, subagents, instructions, settings) is a mixin
|
|
Kit, published with the harness Kit as a Kit set: the set merges their layers
|
|
into one image. It is versioned image content, rewritten on every create, never
|
|
state a run can modify.
|
|
|
|
## Contract
|
|
|
|
A harness run is a second implementation of `agentexec.Runner`. It takes the
|
|
same `Request` and returns the same `Result`, so workflows, the daemon and the
|
|
State Store do not know which runner ran a stage.
|
|
|
|
`agent-sessions@1` supplies the invocation. Archie supplies, per CLI, an output
|
|
adapter that translates the CLI's JSON output stream into `ToolCallReporter`
|
|
calls, `Usage` and a session ID. A CLI with no adapter can run stages with no
|
|
capture tools, reporting no usage; its token budget falls back to wall clock.
|
|
|
|
Workflow definition validation rejects, at load time, a stage whose needs the
|
|
selected harness cannot meet: capture tools without an adapter, or a gate retry
|
|
without a `resume` verb.
|
|
|
|
## Selection
|
|
|
|
A harness is an agent profile. `docs/prds/event-automation.md` defines a profile
|
|
as the execution environment a workflow selects; a harness profile names one Kit
|
|
pinned by digest, a workload Kit or a published Kit set, in place of a container
|
|
image. Archie runs that Kit's image and composes no Kits itself; a bare mixin is
|
|
refused. Everything else about profiles holds: a workflow names its profile, a
|
|
stage may name a different one, and a profile holds no secrets.
|
|
|
|
`archie-agent` builds the stage's runner from the stage's profile: the ai-sdk
|
|
agent loop for an image profile, the harness runner for a Kit profile. Image
|
|
profiles, and every workflow already written against them, are unchanged. A Kit
|
|
profile named by no workflow is a validation error.
|
|
|
|
A stage naming a different profile than its workflow's is a separate run, not a
|
|
mid-container runner switch: a Kit task runs in one container provisioned for
|
|
one profile (Container layout), so a stage that needs a different one is its own
|
|
callee, started the way `workflow.call` already starts one
|
|
(`docs/prds/event-automation.md` "Workflows call workflows"), with its own agent
|
|
profile resolved off its own task row. This needs no mechanism beyond
|
|
`workflow.call` plus per-task profile resolution (`daemon.pinTaskProfile`), both
|
|
of which already treat every task's profile independently of any caller's -- see
|
|
`TestWorkflowCallerConformance` (staterpc) for the callee's independent
|
|
`Workflow` field and `TestWorkflowCallGivesACalleeItsOwnProfile` (daemon) for
|
|
the Kit-vs-image runner selection that independence produces.
|
|
|
|
Profiles are org resources in the State Store, so a Kit profile belongs to one
|
|
org, is granted to its workspaces like any profile, and applies without a
|
|
restart.
|
|
|
|
## Organisations
|
|
|
|
`docs/prds/orgs-and-access.md` is the boundary; a harness adds nothing that
|
|
crosses it.
|
|
|
|
- A Kit profile, its credential bindings and its OAuth tokens belong to one org.
|
|
No record is shared between orgs.
|
|
- A harness run is a run like any other: it acts as the workflow's identity,
|
|
under the run credential issued at dispatch.
|
|
- The egress proxy injects a credential only when the run credential carries the
|
|
secret the binding names. A Kit that declares a service the identity is not
|
|
granted gets no credential: a required one refuses the launch, an optional one
|
|
is skipped.
|
|
- A Kit's network policy is what the Kit asks for; the identity's network grants
|
|
are what the org allows. The proxy enforces the intersection.
|
|
- Concurrency slots are counted per binding within its org.
|
|
|
|
## Credentials
|
|
|
|
A credential binding maps a Kit's `credential@1` `service` name to an org
|
|
secret, which the org grants to identities like any secret. The secret holds one
|
|
of:
|
|
|
|
- **API key:** a value in the org's secret store.
|
|
- **OAuth:** tokens captured by the setup terminal, stored in the State Store
|
|
under the at-rest envelope of `binding-secret-encryption.md` with its own
|
|
domain separator.
|
|
|
|
The egress proxy resolves the binding for each request against the run
|
|
credential. The container gets the Kit's sentinel in the named variable or
|
|
credential file. OAuth refresh happens at the proxy, which intercepts the Kit's
|
|
`tokenEndpoint`, so a refreshed token is written to the State Store and never
|
|
returned to the container.
|
|
|
|
Runs sharing an OAuth binding share its subscription's rate limit.
|
|
`max_concurrent` on the binding (default 1) caps its simultaneous runs. A stage
|
|
waiting for a slot is `pending`, not `running`.
|
|
|
|
## Session contract
|
|
|
|
A **session** is a duplex PTY into a Kit container the daemon owns. One concept
|
|
covers both targets; only the address, the lifetime and the established egress
|
|
phase differ.
|
|
|
|
| | Setup session | Run session (deferred) |
|
|
| --------- | ------------------------------------------------------------- | ------------------------------ |
|
|
| Address | `{org, profile}` | `{org, task, attempt}` |
|
|
| Container | ephemeral, from the profile's Kit | the task's running container |
|
|
| Purpose | run the CLI's own login so the proxy captures the tokens | watch and optionally take over |
|
|
| Lifetime | bounded by the session TTL; the container is removed on close | bounded by the task |
|
|
|
|
- **Transport.** The daemon serves `harness.v1.HarnessService`, a bidi gRPC
|
|
stream, and the dashboard implements its `HarnessTerminal` interface
|
|
(`internal/webui`) over that client; the browser's WebSocket route is
|
|
unchanged. The contract lives on the daemon because the daemon owns
|
|
containers: the dashboard process must not link `internal/container`,
|
|
`internal/app/archied` or `internal/agentexec`.
|
|
- **Authorisation.** Opening a session is the `update` action on the secret,
|
|
decided by the dashboard's authorizer before the route runs
|
|
(`internal/webui/authorize.go`). The dashboard then dials the daemon with the
|
|
harness service token. The daemon does not re-authorise an operator; it trusts the
|
|
asserted `{org, profile}` exactly as the task-action RPC trusts an asserted
|
|
actor. Because that authority is a shell in a container holding an org's
|
|
credential, the harness listener requires `HARNESS_TOKEN` even on loopback,
|
|
unlike the State Store's loopback convention.
|
|
- **Egress phase.** Token interception is filtered by phase (`oauth.go`'s
|
|
`rule.runtime == atRun`), and `EnterRuntime` is one-way. A setup session
|
|
therefore follows a run's lifecycle exactly: install hooks at the install
|
|
phase, `EnterRuntime` before the shell. A credential whose token endpoint must
|
|
be captured declares `phase: runtime`; a credential capturing at install phase
|
|
is captured by the install hooks that run before the switch.
|
|
- **Credentials.** Capture is gated: the proxy intercepts a token endpoint only
|
|
for a service the session is granted (`Proxy.granted`). A setup session has no
|
|
task, so the daemon registers a session grant: its proxy token resolves to
|
|
`{org, the Kit's declared credential services the org binds}`. That is the
|
|
intersection the `update`-on-secret authorisation covers. The real token never
|
|
enters the container: the response is sentinelized and stored as the org's
|
|
harness secret.
|
|
- **Container discipline.** The setup container runs a keepalive entrypoint with
|
|
an empty `Config.Env`; the PTY is an exec as the Kit's harness user with the
|
|
harness env and nothing inherited from any worker. No worktree is mounted. It
|
|
is acquired through the container pool, so it carries the `archie-daemon`
|
|
label, the concurrency cap and the max-uptime reaper; the session TTL is the
|
|
smaller bound. Closing revokes the egress session, removes the sandbox network
|
|
and the Kit volumes, then releases the container.
|
|
- **Kit unavailable.** The daemon serves the contract only when it has a
|
|
container pool and a Kit launcher; otherwise there is no listener, and a
|
|
client's refused connection is gRPC's `Unavailable`. The dashboard's existing
|
|
failure surface is the honest degrade; `/api/harness`'s `terminal` field means
|
|
"a client is configured", not "the daemon can run a Kit".
|
|
- **Run target.** `RunTarget{task_id, attempt}` opens a read-only view of a
|
|
running task's sandbox; it ships as the peek. The daemon resolves the pair
|
|
itself, so the dashboard never learns a container name: the task store says
|
|
whether the task is `running` and which attempt is current, and the Kit
|
|
launcher names the container `kitrun.TaskContainer(task_id, attempt)` (one
|
|
container per attempt, removed when the run ends; volumes remain task-scoped
|
|
for retries). A finished or unknown task, a task that is not running, an
|
|
attempt other than the running one, or a container that is gone is `NotFound`
|
|
with the reason in the message; an unreachable task store or container engine
|
|
is `Unavailable`. The view follows the container's stdout and stderr,
|
|
including the harness output forwarded by the worker, carries no stdin and no
|
|
exec, and ends when the task's container ends or the client leaves, so it
|
|
confers no authority over the container. A read-only tail of the existing
|
|
per-attempt logs was rejected: those refresh only on task state changes and
|
|
are the daemon's account, not the sandbox's. Mid-stream failures are shown as
|
|
a reason in the terminal before it closes.
|
|
- **Authorisation.** The dashboard route `GET /api/tasks/{id}/sandbox` is
|
|
decided as `read-logs` on the task before the handler runs, so a caller sees
|
|
a sandbox exactly when they may read that task's log. The daemon checks that
|
|
the task belongs to the asserted org before following output.
|
|
- **Grants.** The view takes no grant of its own. The run's egress session
|
|
(its proxy token, install then `EnterRuntime`) already belongs to the run
|
|
and is revoked by `Launcher.Release` when the task ends; the view reads
|
|
after that phase switch and dies with the container.
|
|
- **Credentials.** The sandbox holds sentinels, never real credentials: API
|
|
keys are `egress.Sentinel` (`credentials.go`), and an OAuth token set is
|
|
sentinelised in the response and swapped back by the proxy only on a granted
|
|
service's resource host in the matching phase (`oauth.go`, `injectOAuth`).
|
|
The proxy runs in the daemon outside the container, so a shell in the
|
|
container sees only sentinels.
|
|
- **Takeover (interactive PTY into a running task) is out of scope for this
|
|
bead.** It needs the dangerous-command authority (`archie-core-sod1`) to
|
|
gate it, and that authority is not wired for it: the harness listener has no
|
|
approval step; `ToolSetOptions.Approval` in `internal/agentexec/modelloop`
|
|
is never set by a caller, so task runs never build approval-required tools;
|
|
approvals reach an operator only on the chat paths (dashboard ask, Telegram,
|
|
Gateway). Takeover would also need an audit record naming the actor, task,
|
|
attempt and open/close times, and revocation tied to task end (the exec is
|
|
closed with the container). Those follow the setup session's shape
|
|
(`harnesssession.Session`, `container.ExecPTY`) once the authority exists.
|
|
- **Audit.** Opening and closing a session and a successful capture are recorded
|
|
with the asserted org and profile. A run view's open and close are logged with
|
|
the task and attempt; it reads output only, so it has no takeover record.
|
|
|
|
Each session is the only interactive path into a harness container.
|
|
|
|
## Egress
|
|
|
|
Only Kit profiles run on the egress path. An image profile keeps the network
|
|
access its identity is granted, because workflows such as a network investigator
|
|
need raw DNS, ICMP and arbitrary TCP that no HTTP proxy carries.
|
|
|
|
- **Sandbox network.** Each Kit run gets its own internal Docker bridge on which
|
|
the host holds no address. Host services, including those listening on every
|
|
interface, are unreachable from it by any address.
|
|
- **Relay.** One relay container, `archie-agent relay`, sits on the sandbox
|
|
networks under a fixed alias and on the host-reachable network. It forwards
|
|
three fixed ports, to the proxy, NATS and the State Store, and can reach
|
|
nothing else. It holds no secrets.
|
|
- **Proxy.** The proxy runs inside `archied`, listening on the host gateway of
|
|
the host-reachable network. It terminates TLS with a per-daemon CA installed
|
|
into the container at create, identifies the container by a per-run proxy
|
|
password distinct from the worker's run grant, and applies that run's network
|
|
policy and credential inject rules. Upstream dials refuse loopback, and refuse
|
|
internal addresses unless the policy names the exact IP, so an allowed name
|
|
cannot resolve back into the host.
|
|
|
|
## Container layout
|
|
|
|
A Kit task runs in one container, from the Kit composition's image, on the
|
|
task's sandbox network. The worker and every harness invocation share it; they
|
|
are separated by user, not by container, so a run's cost scales with the run and
|
|
not with how many agents it fans out to.
|
|
|
|
- **Worker.** Archie mounts its own `archie-agent` binary read-only and runs it
|
|
as the container's entrypoint, as root. It holds the task's NATS and State
|
|
Store credentials in its own environment and in root-only files, runs the
|
|
workflow, and serves `archie-agent mcp`. The Kit image's entrypoint is not
|
|
PID 1.
|
|
- **Harness.** Every invocation runs as the Kit's user (the image's `User`,
|
|
default uid 1000) with an explicit environment: the Kit's variables, the proxy
|
|
environment and the credential sentinels, and nothing inherited from the
|
|
worker. That user cannot read the worker's environment or files, or trace it.
|
|
Sibling invocations in one run share the user and the worktree.
|
|
- **Hooks.** `lifecycle@1` install and startup hooks run as the user each
|
|
declares, before the worker accepts work.
|
|
- **Mounts.** The task worktree at the Kit's workspace directory, owned by the
|
|
Kit's user, and the read-only skills store. Nothing else from the host.
|
|
- **Network.** The relay forwards three fixed ports on the sandbox network: the
|
|
egress proxy, NATS and the State Store. The worker reaches NATS and the State
|
|
Store with its credentials; a harness can open those ports but holds no
|
|
credential either accepts.
|
|
- **Volumes.** `volume@1` paths persist for the life of the WorkflowExecution,
|
|
which covers session resume across gate iterations and retries. They are not
|
|
shared between executions.
|
|
|
|
## Archie tools inside the harness
|
|
|
|
Capture tools return a stage's structured results through `archie-agent mcp`, a
|
|
stdio MCP server. The same command forwards central-tool discovery and calls
|
|
over Streamable HTTP to `http://archie.tools/mcp` through the egress proxy.
|
|
Claude Code, Codex, Pi, OMP and GitHub Copilot CLI register it using their
|
|
native MCP configuration. Registration files are per-stage and outside the
|
|
worktree; the Kit's bundled settings remain intact. A CLI that exits without
|
|
initializing the required MCP server fails the stage.
|
|
|
|
`archied` handles that exact endpoint inside the authenticated task proxy; it is
|
|
not a listener or an upstream network destination. It is available only during
|
|
task runtime, never during install or a setup session. The harness supplies a
|
|
sentinel, and the proxy injects a separate random gateway token held only on the
|
|
host. Ending the proxy session revokes that token and cancels active calls.
|
|
|
|
The proxy reaches `archie-gateway` through its authenticated gRPC contract. The
|
|
Gateway derives the task, execution principal and org from a live run grant on
|
|
every request. Its tool list is bounded by the profile in that org and the
|
|
package's accepted tool authority, checked again at invocation. Workspace tools,
|
|
chat file delivery, conversation-only tools and tools that require an
|
|
unavailable approver are not served. Tool result limits apply without exposing
|
|
host spill paths; media is returned as MCP content.
|
|
|
|
Task MCP providers run in the Gateway with task-bound sessions and sampling.
|
|
Sampling uses the task's model and credentials, records task usage and refuses
|
|
revoked grants. It does not use the chat identity or chat model. The worker does
|
|
not start another copy of those providers for Kit tasks. The harness's own
|
|
built-in tools remain outside Archie's tool-result limits.
|
|
|
|
## Enforcement around the harness
|
|
|
|
| Property | Enforcement |
|
|
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| Model never runs git | No forge credential can be bound. Archie records `HEAD` and every ref before the run; any change fails the step (`failed`, policy class). The workflow commits the stage's changes. |
|
|
| Protected paths | A diff touching a `Protection` path fails the step. |
|
|
| Read-only stage | Any worktree diff fails the step. |
|
|
| Gate | Archie runs the stage's gate after the harness exits. On failure it resumes the session with the gate output, up to the stage's iteration budget. |
|
|
| Budget | Archie kills the process on wall-clock timeout, or when streamed usage passes the token budget. |
|
|
| Cancellation | Cancelling the step kills the process group, then the container. |
|
|
| Egress | Only the proxy is reachable; the Kit's network policy decides the rest. |
|
|
|
|
Harness settings that deny git or protected paths are added where the CLI
|
|
supports them, to save wasted turns. They are never the enforcement.
|
|
|
|
## Execution tree
|
|
|
|
A harness run is one `agent` StepExecution. Subagents the harness spawns
|
|
internally are not recorded as children. Their usage is included in the step's
|
|
`tokens_used` when the adapter reports it.
|
|
|
|
## Agents never hold a harness
|
|
|
|
An Agent reaches a harness only by invoking a Workflow whose stage names it. No
|
|
Agent tool, chat tool or capability exposes a harness, its setup terminal or its
|
|
credential. The stage boundary is what carries the container, the git and path
|
|
checks, the gate and the budget; a harness outside a stage has none of them.
|
|
|
|
## Out of scope
|
|
|
|
- Chat through a harness. The gateway chat loop stays on the built-in runtime.
|
|
- `network-policy@2` HTTP method and path rules; a Kit requiring it is refused.
|
|
- Recording the harness's internal subagents as StepExecutions.
|
|
- Cost in currency.
|
|
- Choosing a harness automatically per task.
|
|
|
|
## Plan
|
|
|
|
1. Descriptor resolution with the Kit spec package; capability support table and
|
|
fail-closed refusal.
|
|
2. Egress proxy: internal network, per-daemon CA, per-container token,
|
|
`network-policy@1`.
|
|
3. `credential@1`: API-key bindings, sentinel rendering, inject rules.
|
|
4. Harness `Runner`: container create with worktree, volumes, lifecycle hooks
|
|
and the mounted `archie-agent`; `agent-sessions@1` invocation; worktree
|
|
checks for git, protected paths and read-only; budget and cancellation.
|
|
5. Claude Code output adapter; `archie-agent mcp`; gate-and-resume loop.
|
|
6. Kit profiles, runner selection from the stage's profile, and definition
|
|
validation.
|
|
7. OAuth bindings as org secrets: token-endpoint interception, encrypted
|
|
storage, per-binding concurrency; dashboard setup terminal.
|
|
8. Output adapters for Codex, Pi, OMP and GitHub Copilot CLI.
|
|
|
|
## Verification
|
|
|
|
- Every named adapter discovers and calls a permitted central tool through the
|
|
proxy and records a local capture. A guessed tool outside the profile or
|
|
accepted package authority is refused.
|
|
- The gateway token never appears in the container, generated MCP files,
|
|
worktree, logs or events. It is not the container's proxy password.
|
|
- A permission change after discovery takes effect at the next call; ending the
|
|
proxy session cancels active calls, and a revoked grant cannot execute tools
|
|
or sample a model.
|
|
- Bundled skills remain visible with shared skills missing or disabled; shared
|
|
mounts are per-skill and read-only. Explicit context directories preserve
|
|
non-managed instructions and refuse symlinked staging paths.
|
|
- The same workflow definition completes on `builtin` and on the Claude Code Kit
|
|
with the same `Result` shape and captures.
|
|
- After `/login` in the setup terminal, the container's credential file holds
|
|
only sentinels, the State Store holds the encrypted token, and a task run
|
|
authenticates.
|
|
- A real API key or OAuth token never appears in the container environment,
|
|
filesystem, worktree, task log or event stream.
|
|
- A request to a host outside the Kit's runtime allow list fails, including a
|
|
direct connection that bypasses the proxy settings.
|
|
- A harness instructed to `git commit` fails its step, and the task branch is
|
|
unchanged.
|
|
- A harness edit to a protected test file fails the step in a TDD fix stage.
|
|
- A read-only feasibility stage that writes a file fails.
|
|
- A file a run writes into a Kit-managed path such as the CLI's settings file is
|
|
gone on the next execution.
|
|
- A Kit requiring an unsupported capability is refused at launch with the
|
|
capability named.
|
|
- A stage with a failing gate resumes the same session with the gate output,
|
|
then succeeds or exhausts its iterations.
|
|
- Two tasks sharing an OAuth binding with `max_concurrent = 1` run one at a
|
|
time; the second waits as `pending`.
|
|
- Stopping a run kills the harness process and removes the container.
|
|
- A harness invocation cannot read the worker's NATS or State Store credential
|
|
from its environment, its files or `/proc`.
|
|
- A run fanning out to many agent invocations uses one container.
|
|
- A Kit profile in org A cannot be selected, and its binding cannot be resolved,
|
|
by a workflow in org B.
|
|
- A harness run whose identity is not granted a Kit's required credential is
|
|
refused at launch; the proxy never injects a secret outside the run
|
|
credential.
|
|
- The firewall investigation example runs unchanged on an image profile with
|
|
direct network access.
|