feat(agentexec): run harness invocations as the Kit's user with no inherited environment

A Kit task runs worker and harness in one container, separated by user:
the worker keeps its credentials as root, and every harness invocation
runs as the Kit's user with only its declared environment. A nil env
would inherit the worker's, so the runner always passes an explicit one;
root is refused as a harness user, and a root worker refuses to start a
harness with no user. The PRD's container layout now says one container
per run, with the relay forwarding the proxy, NATS and the State Store.
This commit is contained in:
sam committed 2026-09-26 03:08:34 +10:00
1 parent 2a9567cdf5
commit 9937caad7c
8 files changed
+189 -16

No files matched your search

+11
View File
@@ -261,6 +261,13 @@
},
"type": "array"
},
"env": {
"description": "Env is the invocation's entire environment. Nothing is inherited from\nthe worker, whose environment carries its credentials.",
"items": {
"type": "string"
},
"type": "array"
},
"launch": {
"items": {
"type": "string"
@@ -278,6 +285,10 @@
"type": "string"
},
"type": "array"
},
"user": {
"description": "User is the Kit's user, a name or \"uid[:gid]\", that every invocation\nruns as. It is never root: the worker's credentials live in the same\ncontainer, readable only by the worker's user.",
"type": "string"
}
},
"required": [
+1 -1
View File
@@ -328,7 +328,7 @@
"url": "/dev/prds/external-agent-harness/",
"title": "External agent harnesses",
"section": "PRDs",
"body": "# External agent harnesses\n\n**Status:** Approved\n**Authority:** `docs/architecture/agent-system.md` (agent execution as a data\nboundary); settles the harness non-goal deferred by\n`docs/prds/service-decomposition.md`.\n**Compounds with:** `docs/prds/execution-tree-state-machine.md` (a harness\nrun is one `agent` StepExecution), `docs/prds/binding-secret-encryption.md`\n(the at-rest envelope reused for harness credentials).\n\n## Decision\n\nA workflow stage can run on an external coding-agent CLI instead of the\nbuilt-in agent loop: Claude Code, Codex, Pi, OMP, GitHub Copilot CLI, or any\nother CLI packaged as a Docker Sandbox Kit v3. Each authenticates with an API\nkey or the operator's subscription login, and can carry the operator's own\nskills, plugins, subagents and instructions.\n\nTwo boundaries hold regardless of what the harness does:\n\n- **The real credential never enters the container.** The container sees\n sentinel values; archie's egress proxy substitutes the real credential on\n outbound requests and performs OAuth refresh on the host.\n- **Archie's workflow guarantees are enforced outside the harness.** Git,\n protected paths, read-only stages, gates and budgets are checked by archie\n around the process and against the worktree afterwards.\n\n## Kits\n\nA harness is packaged as a Kit v3 image: an OCI image whose\n`vnd.docker.sandbox.kit.descriptor` manifest annotation declares what the\nharness needs. Archie reads descriptors with the published\n`github.com/docker/sandbox-kit-spec/v3/spec` package and does not maintain\nits own descriptor format. Published Kits run unmodified.\n\nArchie is a partial Kit runtime. It implements these capability types:\n\n| Type | Archie behaviour |\n| ------------------- | ----------------------------------------------------------------------------------------- |\n| `credential@1` | Proxy-managed only, described below. |\n| `network-policy@1` | Enforced by the egress proxy. |\n| `volume@1` | A Docker volume keyed by WorkflowExecution, deleted with it. |\n| `lifecycle@1` | Install and startup hooks run on container create. |\n| `agent-sessions@1` | The headless `prompt` and `resume` invocations for the stage. |\n| `agent-skills@1` | The operator's skills store, mounted read-only whatever the Kit asks. |\n| `agent-context@1` | Archie's stage instructions, delivered where the Kit declares. |\n| `resources@1` | Container CPU and memory limits. |\n\nEvery other type fails closed: a required entry refuses the launch, an\noptional one is skipped and recorded on the step. `credential@1` with\n`passthrough: true` and any credential for a forge service are refused.\n\nThe operator's own setup (plugins, subagents, instructions, settings) is a\nmixin Kit the operator builds `FROM` or alongside the harness Kit. It is\nversioned image content, rewritten on every create, never state a run can\nmodify.\n\n## Contract\n\nA harness run is a second implementation of `agentexec.Runner`. It takes the\nsame `Request` and returns the same `Result`, so workflows, the daemon and\nthe State Store do not know which runner ran a stage.\n\n`agent-sessions@1` supplies the invocation. Archie supplies, per CLI, an\noutput adapter that translates the CLI's JSON output stream into\n`ToolCallReporter` calls, `Usage` and a session ID. A CLI with no adapter can\nrun stages with no capture tools, reporting no usage; its token budget falls\nback to wall clock.\n\nWorkflow definition validation rejects, at load time, a stage whose needs the\nselected harness cannot meet: capture tools without an adapter, or a gate\nretry without a `resume` verb.\n\n## Selection\n\nA harness is an agent profile. `docs/prds/event-automation.md` defines a\nprofile as the execution environment a workflow selects; a harness profile\nnames a Kit composition (the workload Kit pinned by digest, and zero or more\nmixin Kits) in place of a container image. Everything else about profiles\nholds: a workflow names its profile, a stage may name a different one, and a\nprofile holds no secrets.\n\n`archie-agent` builds the stage's runner from the stage's profile: the\nai-sdk agent loop for an image profile, the harness runner for a Kit\nprofile. Image profiles, and every workflow already written against them,\nare unchanged. A Kit profile named by no workflow is a validation error.\n\nProfiles are org resources in the State Store, so a Kit profile belongs to\none org, is granted to its workspaces like any profile, and applies without\na restart.\n\n## Organisations\n\n`docs/prds/Line truncated
"body": "# External agent harnesses\n\n**Status:** Approved\n**Authority:** `docs/architecture/agent-system.md` (agent execution as a data\nboundary); settles the harness non-goal deferred by\n`docs/prds/service-decomposition.md`.\n**Compounds with:** `docs/prds/execution-tree-state-machine.md` (a harness\nrun is one `agent` StepExecution), `docs/prds/binding-secret-encryption.md`\n(the at-rest envelope reused for harness credentials).\n\n## Decision\n\nA workflow stage can run on an external coding-agent CLI instead of the\nbuilt-in agent loop: Claude Code, Codex, Pi, OMP, GitHub Copilot CLI, or any\nother CLI packaged as a Docker Sandbox Kit v3. Each authenticates with an API\nkey or the operator's subscription login, and can carry the operator's own\nskills, plugins, subagents and instructions.\n\nTwo boundaries hold regardless of what the harness does:\n\n- **The real credential never enters the container.** The container sees\n sentinel values; archie's egress proxy substitutes the real credential on\n outbound requests and performs OAuth refresh on the host.\n- **Archie's workflow guarantees are enforced outside the harness.** Git,\n protected paths, read-only stages, gates and budgets are checked by archie\n around the process and against the worktree afterwards.\n\n## Kits\n\nA harness is packaged as a Kit v3 image: an OCI image whose\n`vnd.docker.sandbox.kit.descriptor` manifest annotation declares what the\nharness needs. Archie reads descriptors with the published\n`github.com/docker/sandbox-kit-spec/v3/spec` package and does not maintain\nits own descriptor format. Published Kits run unmodified.\n\nArchie is a partial Kit runtime. It implements these capability types:\n\n| Type | Archie behaviour |\n| ------------------- | ----------------------------------------------------------------------------------------- |\n| `credential@1` | Proxy-managed only, described below. |\n| `network-policy@1` | Enforced by the egress proxy. |\n| `volume@1` | A Docker volume keyed by WorkflowExecution, deleted with it. |\n| `lifecycle@1` | Install and startup hooks run on container create. |\n| `agent-sessions@1` | The headless `prompt` and `resume` invocations for the stage. |\n| `agent-skills@1` | The operator's skills store, mounted read-only whatever the Kit asks. |\n| `agent-context@1` | Archie's stage instructions, delivered where the Kit declares. |\n| `resources@1` | Container CPU and memory limits. |\n\nEvery other type fails closed: a required entry refuses the launch, an\noptional one is skipped and recorded on the step. `credential@1` with\n`passthrough: true` and any credential for a forge service are refused.\n\nThe operator's own setup (plugins, subagents, instructions, settings) is a\nmixin Kit the operator builds `FROM` or alongside the harness Kit. It is\nversioned image content, rewritten on every create, never state a run can\nmodify.\n\n## Contract\n\nA harness run is a second implementation of `agentexec.Runner`. It takes the\nsame `Request` and returns the same `Result`, so workflows, the daemon and\nthe State Store do not know which runner ran a stage.\n\n`agent-sessions@1` supplies the invocation. Archie supplies, per CLI, an\noutput adapter that translates the CLI's JSON output stream into\n`ToolCallReporter` calls, `Usage` and a session ID. A CLI with no adapter can\nrun stages with no capture tools, reporting no usage; its token budget falls\nback to wall clock.\n\nWorkflow definition validation rejects, at load time, a stage whose needs the\nselected harness cannot meet: capture tools without an adapter, or a gate\nretry without a `resume` verb.\n\n## Selection\n\nA harness is an agent profile. `docs/prds/event-automation.md` defines a\nprofile as the execution environment a workflow selects; a harness profile\nnames a Kit composition (the workload Kit pinned by digest, and zero or more\nmixin Kits) in place of a container image. Everything else about profiles\nholds: a workflow names its profile, a stage may name a different one, and a\nprofile holds no secrets.\n\n`archie-agent` builds the stage's runner from the stage's profile: the\nai-sdk agent loop for an image profile, the harness runner for a Kit\nprofile. Image profiles, and every workflow already written against them,\nare unchanged. A Kit profile named by no workflow is a validation error.\n\nProfiles are org resources in the State Store, so a Kit profile belongs to\none org, is granted to its workspaces like any profile, and applies without\na restart.\n\n## Organisations\n\n`docs/prds/Line truncated
},
{
"url": "/dev/prds/identity-decision/",
+31 -10
View File
@@ -151,8 +151,8 @@ carries.
on every interface, are unreachable from it by any address.
- **Relay.** One relay container, `archie-agent relay`, sits on the sandbox
networks under a fixed alias and on the host-reachable network. It
forwards every connection to the proxy and can reach nothing else. It
holds no secrets.
forwards three fixed ports, to the proxy, NATS and the State Store, and can
reach nothing else. It holds no secrets.
- **Proxy.** The proxy runs inside `archied`, listening on the host gateway
of the host-reachable network. It terminates TLS with a per-daemon CA
installed into the container at create, identifies the container by a
@@ -163,14 +163,32 @@ carries.
## Container layout
- The task worktree is mounted at the Kit's workspace directory. Nothing else
from the host is mounted, except the read-only skills store.
- Archie mounts its own `archie-agent` binary read-only, so published Kits
need not contain it. `archie-agent` drives the harness as a subprocess and
serves `archie-agent mcp`.
- `volume@1` paths persist for the life of the WorkflowExecution, which
covers session resume across gate iterations and retries. They are not
shared between executions.
A Kit task runs in one container, from the Kit composition's image, on the
task's sandbox network. The worker and every harness invocation share it;
they are separated by user, not by container, so a run's cost scales with
the run and not with how many agents it fans out to.
- **Worker.** Archie mounts its own `archie-agent` binary read-only and runs
it as the container's entrypoint, as root. It holds the task's NATS and
State Store credentials in its own environment and in root-only files,
runs the workflow, and serves `archie-agent mcp`. The Kit image's
entrypoint is not PID 1.
- **Harness.** Every invocation runs as the Kit's user (the image's `User`,
default uid 1000) with an explicit environment: the Kit's variables, the
proxy environment and the credential sentinels, and nothing inherited from
the worker. That user cannot read the worker's environment or files, or
trace it. Sibling invocations in one run share the user and the worktree.
- **Hooks.** `lifecycle@1` install and startup hooks run as the user each
declares, before the worker accepts work.
- **Mounts.** The task worktree at the Kit's workspace directory, owned by the
Kit's user, and the read-only skills store. Nothing else from the host.
- **Network.** The relay forwards three fixed ports on the sandbox network:
the egress proxy, NATS and the State Store. The worker reaches NATS and the
State Store with its credentials; a harness can open those ports but holds
no credential either accepts.
- **Volumes.** `volume@1` paths persist for the life of the
WorkflowExecution, which covers session resume across gate iterations and
retries. They are not shared between executions.
## Archie tools inside the harness
@@ -262,6 +280,9 @@ of them.
- Two tasks sharing an OAuth binding with `max_concurrent = 1` run one at a
time; the second waits as `pending`.
- Stopping a run kills the harness process and removes the container.
- A harness invocation cannot read the worker's NATS or State Store
credential from its environment, its files or `/proc`.
- A run fanning out to many agent invocations uses one container.
- A Kit profile in org A cannot be selected, and its binding cannot be
resolved, by a workflow in org B.
- A harness run whose identity is not granted a Kit's required credential is
+13 -1
View File
@@ -2,8 +2,20 @@
package agentexec
import "os/exec"
import (
"errors"
"os/exec"
)
// setProcessGroup falls back to killing the harness process alone where
// process groups are unavailable.
func setProcessGroup(*exec.Cmd) {}
// runAsHarnessUser refuses a named harness user where user switching is
// unavailable, rather than running the harness as the worker.
func runAsHarnessUser(_ *exec.Cmd, name string) error {
if name != "" {
return errors.New("harness user switching is unsupported on this platform")
}
return nil
}
+60
View File
@@ -3,7 +3,13 @@
package agentexec
import (
"errors"
"fmt"
"os"
"os/exec"
"os/user"
"strconv"
"strings"
"syscall"
)
@@ -13,3 +19,57 @@ func setProcessGroup(cmd *exec.Cmd) {
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
}
// runAsHarnessUser drops the invocation to the Kit's user. A worker running
// as root refuses to start a harness with no user, because the harness would
// then share the worker's user and could read its credentials.
func runAsHarnessUser(cmd *exec.Cmd, name string) error {
if name == "" {
if os.Geteuid() == 0 {
return errors.New("harness has no user and the worker runs as root; refusing to give the harness the worker's privileges")
}
return nil
}
cred, err := harnessCredential(name)
if err != nil {
return err
}
cmd.SysProcAttr.Credential = cred
return nil
}
// harnessCredential resolves a user name or "uid[:gid]" to a credential.
// Root is refused whichever way it is named.
func harnessCredential(name string) (*syscall.Credential, error) {
uidText, gidText, hasGid := strings.Cut(name, ":")
uid, uidErr := strconv.ParseUint(uidText, 10, 32)
var cred syscall.Credential
if uidErr == nil {
cred.Uid, cred.Gid = uint32(uid), uint32(uid)
} else {
u, err := user.Lookup(uidText)
if err != nil {
return nil, fmt.Errorf("harness user %q: %w", uidText, err)
}
id, err := strconv.ParseUint(u.Uid, 10, 32)
if err != nil {
return nil, fmt.Errorf("harness user %q has non-numeric uid %q", uidText, u.Uid)
}
group, err := strconv.ParseUint(u.Gid, 10, 32)
if err != nil {
return nil, fmt.Errorf("harness user %q has non-numeric gid %q", uidText, u.Gid)
}
cred.Uid, cred.Gid = uint32(id), uint32(group)
}
if hasGid {
gid, err := strconv.ParseUint(gidText, 10, 32)
if err != nil {
return nil, fmt.Errorf("harness group %q is not numeric", gidText)
}
cred.Gid = uint32(gid)
}
if cred.Uid == 0 {
return nil, fmt.Errorf("harness user %q is root; a harness never runs with the worker's privileges", name)
}
return &cred, nil
}
+7 -2
View File
@@ -87,7 +87,7 @@ func (r *HarnessRunner) Run(ctx context.Context, workspace string, req Request,
for {
res.Iterations++
out := r.output()
exitErr := invoke(runCtx, workspace, harnessArgv(req.Harness, verb, prompt, session), out, report)
exitErr := invoke(runCtx, workspace, req.Harness, harnessArgv(req.Harness, verb, prompt, session), out, report)
if id := out.SessionID(); id != "" {
session = id
}
@@ -197,11 +197,16 @@ func resumeVerb(h *HarnessSpec, sessionID string) []string {
// invoke runs one harness invocation in its own process group, feeding
// stdout to out line by line. Cancelling ctx kills the whole group, so a
// CLI's children cannot outlive the step.
func invoke(ctx context.Context, workspace string, argv []string, out HarnessOutput, report ToolCallReporter) error {
func invoke(ctx context.Context, workspace string, h *HarnessSpec, argv []string, out HarnessOutput, report ToolCallReporter) error {
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
cmd.Dir = workspace
cmd.WaitDelay = killGrace
// A nil Env would inherit the worker's environment and its credentials.
cmd.Env = append([]string{}, h.Env...)
setProcessGroup(cmd)
if err := runAsHarnessUser(cmd, h.User); err != nil {
return err
}
stdout, err := cmd.StdoutPipe()
if err != nil {
return err
+59 -2
View File
@@ -28,6 +28,7 @@ commit) echo x > a.txt; git add a.txt; git -c user.email=a@b.c -c user.name=a co
config) git config remote.origin.url https://attacker.invalid/repo.git ;;
protected) echo broken >> widget_test.go ;;
sleep) sleep 30 & echo $! > "$FAKE_LOG.child"; wait ;;
env) env > "$FAKE_LOG.env" ;;
fix-on-resume) if [ -f .attempted ]; then echo ok > fixed.txt; else touch .attempted; fi ;;
esac
echo "session=S123"
@@ -91,13 +92,12 @@ func newFixture(t *testing.T, scenario string) *fixture {
t.Fatal(err)
}
log := filepath.Join(t.TempDir(), "invocations")
t.Setenv("FAKE_SCENARIO", scenario)
t.Setenv("FAKE_LOG", log)
return &fixture{workspace: ws, log: log, req: Request{
Version: ProtocolVersion, TaskID: 1, Attempt: 1, Stage: "implement", Model: "harness",
Mission: "Write the notes file.",
Budget: Budget{WallClock: 20 * time.Second},
Harness: &HarnessSpec{
Env: []string{"PATH=" + os.Getenv("PATH"), "FAKE_SCENARIO=" + scenario, "FAKE_LOG=" + log},
Launch: []string{"/bin/sh", bin},
Prompt: []string{"-p", "{{.Prompt}}"},
Resume: []string{"--resume", "{{.SessionID}}"},
@@ -287,3 +287,60 @@ func assertChildDead(t *testing.T, pidFile string) {
time.Sleep(50 * time.Millisecond)
}
}
func TestHarnessNeverInheritsTheWorkerEnvironment(t *testing.T) {
f := newFixture(t, "env")
// The fake's own variables are in the worker's environment too, so an
// inherited environment would run the fake fully and expose the leak.
t.Setenv("FAKE_SCENARIO", "env")
t.Setenv("FAKE_LOG", f.log)
t.Setenv("STATE_STORE_TOKEN", "worker-secret")
t.Setenv("NATS_TOKEN", "worker-secret")
f.req.Harness.Env = []string{
"PATH=" + os.Getenv("PATH"),
"FAKE_SCENARIO=env", "FAKE_LOG=" + f.log,
"HTTPS_PROXY=http://archie:token@archie-egress:3128",
}
res, _ := f.run(t, t.Context())
if res.Status != StatusPassed {
t.Fatalf("status %q (%s)", res.Status, res.Detail)
}
raw, err := os.ReadFile(f.log + ".env")
if err != nil {
t.Fatal(err)
}
env := string(raw)
if strings.Contains(env, "worker-secret") {
t.Fatalf("the harness saw the worker's credentials:\n%s", env)
}
if !strings.Contains(env, "HTTPS_PROXY=http://archie:token@archie-egress:3128") {
t.Fatalf("the harness did not get its declared environment:\n%s", env)
}
}
func TestHarnessUser(t *testing.T) {
tests := []struct {
user string
uid uint32
gid uint32
wantErr bool
}{
{"1000", 1000, 1000, false},
{"1000:1001", 1000, 1001, false},
{"0", 0, 0, true},
{"root", 0, 0, true},
{"not-a-user-anywhere", 0, 0, true},
}
for _, tt := range tests {
cred, err := harnessCredential(tt.user)
if tt.wantErr {
if err == nil {
t.Errorf("harnessCredential(%q) = %+v, want an error", tt.user, cred)
}
continue
}
if err != nil || cred.Uid != tt.uid || cred.Gid != tt.gid {
t.Errorf("harnessCredential(%q) = %+v, %v; want %d:%d", tt.user, cred, err, tt.uid, tt.gid)
}
}
}
+7
View File
@@ -98,6 +98,13 @@ type Request struct {
// (image entrypoint and cmd) and the agent-sessions@1 verb tails appended
// to it. Prompt must carry {{.Prompt}}; Resume, when set, {{.SessionID}}.
type HarnessSpec struct {
// User is the Kit's user, a name or "uid[:gid]", that every invocation
// runs as. It is never root: the worker's credentials live in the same
// container, readable only by the worker's user.
User string `json:"user,omitempty"`
// Env is the invocation's entire environment. Nothing is inherited from
// the worker, whose environment carries its credentials.
Env []string `json:"env,omitempty"`
Launch []string `json:"launch"`
Prompt []string `json:"prompt"`
Resume []string `json:"resume,omitempty"`