feat(harness): serve a daemon-side setup session to the dashboard

The dashboard's setup terminal was a 503: it had a WebSocket route, an
authorizer and an xterm.js surface, but nothing owned a container. The
daemon owns containers, so the session contract lands there and the
dashboard consumes it over a new gRPC contract.

One session concept, two targets. A setup session is addressed by Kit
profile and is ephemeral; a run session (archie-core-qi91) is addressed by
task and attempt and is bounded by the task. This slice serves setup and
answers Unimplemented for run.

The contract, the authorisation, the lifetime and the egress phase are
settled in docs/prds/external-agent-harness.md, "Session contract". The
short version: the daemon serves harness.v1.HarnessService; the dashboard
authorizes the operator as an update on the secret and then dials the
daemon with the harness service token, required even on loopback because a
session is a shell holding an org's credential; the setup container runs
no worker, mounts no worktree, holds a keepalive, and carries an empty
container environment. Install hooks run at the install phase and the
session enters runtime before the shell, so a runtime-phase credential's
token endpoint is the one the proxy intercepts.

Capture is gated by the egress proxy on a granted service, and a setup
session has no task, so the daemon registers a session grant: the
session's proxy token resolves to the org's bindings for the Kit's
declared services. The real token never enters the container: the token
endpoint response is sentinelized and stored as the org's harness secret.

Every teardown path releases the pool slot, revokes the egress session and
the grant, removes the sandbox network, and removes the session's volumes.
The listener requires HARNESS_TOKEN and stops with a 10s grace before
Stop, so an open session cannot hold shutdown to the watchdog leash.
This commit is contained in:
sam committed 2026-10-09 01:38:41 +11:00
1 parent 27c9647a13
commit 7ef58293be
21 files changed
+1987 -12

No files matched your search

+2
View File
@@ -38,6 +38,8 @@ func run() int {
flag.StringVar(&options.Gateway.Token, "gateway-token", "", "bearer token presented to archie-gateway")
flag.StringVar(&options.State.Target, "state-target", "", "archie-state-store gRPC address (defaults to [services.state].target)")
flag.StringVar(&options.State.Token, "state-token", "", "bearer token presented to archie-state-store")
flag.StringVar(&options.Harness.Target, "harness-target", "", "archied harness session gRPC address (defaults to [services.harness].target); empty leaves the setup terminal unwired")
flag.StringVar(&options.Harness.Token, "harness-token", "", "bearer token presented to the daemon's harness contract")
flag.DurationVar(&options.DependencyTimeout, "dependency-timeout", 0, "per-dependency readiness probe timeout (default 5s)")
flag.DurationVar(&options.EventPollInterval, "event-poll-interval", 0, "how often the activity feed polls the State Store for new events (default 1s)")
// BoolFunc rather than BoolVar so an unset flag stays distinguishable
+13
View File
@@ -292,6 +292,19 @@ target = "127.0.0.1:9090" # REQUIRED. The State Store gRPC address arch
# # same remote store. See deployments/README.md for how to
# # start archie-state-store.
# The daemon's harness session contract: the dashboard's setup terminal opens
# a PTY into an ephemeral Kit container over it. Both ends are optional and
# the dashboard's setup terminal degrades to 503 without them: the daemon
# serves the contract only when target_token (or HARNESS_TOKEN) is set, and
# archie-ui dials it only when a target is set.
# [services.harness]
# target = "127.0.0.1:8587" # The address archie-ui dials. Empty leaves the
# # setup terminal unwired.
# target_token = "..." # REQUIRED, even on loopback: a session is a
# # shell in a container that holds an org credential.
# # The daemon reads the same value to authenticate its
# # clients. Leave unset to disable the listener.
# NATS is the sole autonomous workflow handoff. mode changes only where the
# broker runs; every workflow still runs in a managed task container:
# "embedded" – in-process nats-server (the default; no external server)
+354
View File
@@ -0,0 +1,354 @@
# External agent harnesses
**Status:** Approved
**Scope:** the harness model and, in "Session contract", the duplex session
that the setup terminal and the run-scoped live view both use. Code cites
this file for both (`ui/src/harness/SetupTerminal.vue`,
`internal/infrastructure/postgres/queries/harness_secrets.sql`).
## Decision
A workflow stage can run on an external coding-agent CLI instead of the
built-in agent loop: Claude Code, Codex, Pi, OMP, GitHub Copilot CLI, or any
other CLI packaged as a Docker Sandbox Kit v3. Each authenticates with an API
key or the operator's subscription login, and can carry the operator's own
skills, plugins, subagents and instructions.
Two boundaries hold regardless of what the harness does:
- **The real credential never enters the container.** The container sees
sentinel values; archie's egress proxy substitutes the real credential on
outbound requests and performs OAuth refresh on the host.
- **Archie's workflow guarantees are enforced outside the harness.** Git,
protected paths, read-only stages, gates and budgets are checked by archie
around the process and against the worktree afterwards.
## Kits
A harness is packaged as a Kit v3 image: an OCI image whose
`vnd.docker.sandbox.kit.descriptor` manifest annotation declares what the
harness needs. Archie reads descriptors with the published
`github.com/docker/sandbox-kit-spec/v3/spec` package and does not maintain
its own descriptor format. Published Kits run unmodified.
Archie is a partial Kit runtime. It implements these capability types:
| Type | Archie behaviour |
| ------------------- | ----------------------------------------------------------------------------------------- |
| `credential@1` | Proxy-managed only, described below. |
| `network-policy@1` | Enforced by the egress proxy. |
| `volume@1` | A Docker volume keyed by WorkflowExecution, deleted with it. |
| `lifecycle@1` | Install and startup hooks run on container create. |
| `agent-sessions@1` | The headless `prompt` and `resume` invocations for the stage. |
| `agent-skills@1` | The operator's skills store (`skills_dir`), mounted read-only whatever the Kit asks. |
| `agent-context@1` | The workload's profile file beside the workspace, indexing each Kit's guidance. |
| `resources@1` | Container CPU and memory limits. |
Every other type fails closed: a required entry refuses the launch, an
optional one is skipped and recorded on the step. `credential@1` with
`passthrough: true` and any credential for a forge service are refused.
The operator's own setup (plugins, subagents, instructions, settings) is a
mixin Kit, published with the harness Kit as a Kit set: the set merges their
layers into one image. It is versioned image content, rewritten on every
create, never state a run can modify.
## Contract
A harness run is a second implementation of `agentexec.Runner`. It takes the
same `Request` and returns the same `Result`, so workflows, the daemon and
the State Store do not know which runner ran a stage.
`agent-sessions@1` supplies the invocation. Archie supplies, per CLI, an
output adapter that translates the CLI's JSON output stream into
`ToolCallReporter` calls, `Usage` and a session ID. A CLI with no adapter can
run stages with no capture tools, reporting no usage; its token budget falls
back to wall clock.
Workflow definition validation rejects, at load time, a stage whose needs the
selected harness cannot meet: capture tools without an adapter, or a gate
retry without a `resume` verb.
## Selection
A harness is an agent profile. `docs/prds/event-automation.md` defines a
profile as the execution environment a workflow selects; a harness profile
names one Kit pinned by digest, a workload Kit or a published Kit set, in
place of a container image. Archie runs that Kit's image and composes no Kits
itself; a bare mixin is refused. Everything else about profiles
holds: a workflow names its profile, a stage may name a different one, and a
profile holds no secrets.
`archie-agent` builds the stage's runner from the stage's profile: the
ai-sdk agent loop for an image profile, the harness runner for a Kit
profile. Image profiles, and every workflow already written against them,
are unchanged. A Kit profile named by no workflow is a validation error.
A stage naming a different profile than its workflow's is a separate run,
not a mid-container runner switch: a Kit task runs in one container
provisioned for one profile (Container layout), so a stage that needs a
different one is its own callee, started the way `workflow.call` already
starts one (`docs/prds/event-automation.md` "Workflows call workflows"),
with its own agent profile resolved off its own task row. This needs no
mechanism beyond `workflow.call` plus per-task profile resolution
(`daemon.pinTaskProfile`), both of which already treat every task's
profile independently of any caller's -- see
`TestWorkflowCallerConformance` (staterpc) for the callee's independent
`Workflow` field and `TestWorkflowCallGivesACalleeItsOwnProfile` (daemon)
for the Kit-vs-image runner selection that independence produces.
Profiles are org resources in the State Store, so a Kit profile belongs to
one org, is granted to its workspaces like any profile, and applies without
a restart.
## Organisations
`docs/prds/orgs-and-access.md` is the boundary; a harness adds nothing that
crosses it.
- A Kit profile, its credential bindings and its OAuth tokens belong to one
org. No record is shared between orgs.
- A harness run is a run like any other: it acts as the workflow's identity,
under the run credential issued at dispatch.
- The egress proxy injects a credential only when the run credential carries
the secret the binding names. A Kit that declares a service the identity is
not granted gets no credential: a required one refuses the launch, an
optional one is skipped.
- A Kit's network policy is what the Kit asks for; the identity's network
grants are what the org allows. The proxy enforces the intersection.
- Concurrency slots are counted per binding within its org.
## Credentials
A credential binding maps a Kit's `credential@1` `service` name to an org
secret, which the org grants to identities like any secret. The secret holds
one of:
- **API key:** a value in the org's secret store.
- **OAuth:** tokens captured by the setup terminal, stored in the State Store
under the at-rest envelope of `binding-secret-encryption.md` with its own
domain separator.
The egress proxy resolves the binding for each request against the run
credential. The container gets the Kit's sentinel in the named variable or
credential file. OAuth refresh happens at the proxy, which intercepts the
Kit's `tokenEndpoint`, so a refreshed token is written to the State Store and
never returned to the container.
Runs sharing an OAuth binding share its subscription's rate limit.
`max_concurrent` on the binding (default 1) caps its simultaneous runs. A
stage waiting for a slot is `pending`, not `running`.
## Session contract
A **session** is a duplex PTY into a Kit container the daemon owns. One
concept covers both targets; only the address, the lifetime and the
established egress phase differ.
| | Setup session | Run session (deferred) |
| --- | --- | --- |
| Address | `{org, profile}` | `{org, task, attempt}` |
| Container | ephemeral, from the profile's Kit | the task's running container |
| Purpose | run the CLI's own login so the proxy captures the tokens | watch and optionally take over |
| Lifetime | bounded by the session TTL; the container is removed on close | bounded by the task |
- **Transport.** The daemon serves `harness.v1.HarnessService`, a bidi gRPC
stream, and the dashboard implements its `HarnessTerminal` interface
(`internal/webui`) over that client; the browser's WebSocket route is
unchanged. The contract lives on the daemon because the daemon owns
containers: the dashboard process must not link `internal/container`,
`internal/app/archied` or `internal/agentexec`.
- **Authorisation.** Opening a session is the `update` action on the secret,
decided by the dashboard's authorizer before the route runs
(`internal/webui/authorize.go`). The dashboard then dials the daemon with
the harness service token. The daemon does not re-authorise a human; it
trusts the asserted `{org, profile}` exactly as the task-action RPC trusts
an asserted actor. Because that authority is a shell in a container holding
an org's credential, the harness listener requires `HARNESS_TOKEN` even on
loopback, unlike the State Store's loopback convention.
- **Egress phase.** Token interception is filtered by phase
(`oauth.go`'s `rule.runtime == atRun`), and `EnterRuntime` is one-way. A
setup session therefore follows a run's lifecycle exactly: install hooks at
the install phase, `EnterRuntime` before the shell. A credential whose
token endpoint must be captured declares `phase: runtime`; a credential
capturing at install phase is captured by the install hooks that run before
the switch.
- **Credentials.** Capture is gated: the proxy intercepts a token endpoint
only for a service the session is granted (`Proxy.granted`). A setup session
has no task, so the daemon registers a session grant: its proxy token
resolves to `{org, the Kit's declared credential services the org binds}`.
That is the intersection the `update`-on-secret authorisation covers. The
real token never enters the container: the response is sentinelized and
stored as the org's harness secret.
- **Container discipline.** The setup container runs a keepalive entrypoint
with an empty `Config.Env`; the PTY is an exec as the Kit's harness user
with the harness env and nothing inherited from any worker. No worktree is
mounted. It is acquired through the container pool, so it carries the
`archie-daemon` label, the concurrency cap and the max-uptime reaper; the
session TTL is the smaller bound. Closing revokes the egress session,
removes the sandbox network and the Kit volumes, then releases the
container.
- **Kit unavailable.** The daemon serves the contract only when it has a
container pool and a Kit launcher; otherwise there is no listener, and a
client's refused connection is gRPC's `Unavailable`. The dashboard's
existing failure surface is the honest degrade; `/api/harness`'s `terminal`
field means "a client is configured", not "the daemon can run a Kit".
- **Run target.** Deferred (`archie-core-qi91`). `Open` answers `Unimplemented`
for it in the setup slice. A finished or absent attempt will be `NotFound`,
not `Unavailable`.
- **Audit.** Opening and closing a session and a successful capture are
recorded with the asserted org and profile.
Each session is the only interactive path into a harness container.
## Egress
Only Kit profiles run on the egress path. An image profile keeps the network
access its identity is granted, because workflows such as a network
investigator need raw DNS, ICMP and arbitrary TCP that no HTTP proxy
carries.
- **Sandbox network.** Each Kit run gets its own internal Docker bridge on
which the host holds no address. Host services, including those listening
on every interface, are unreachable from it by any address.
- **Relay.** One relay container, `archie-agent relay`, sits on the sandbox
networks under a fixed alias and on the host-reachable network. It
forwards three fixed ports, to the proxy, NATS and the State Store, and can
reach nothing else. It holds no secrets.
- **Proxy.** The proxy runs inside `archied`, listening on the host gateway
of the host-reachable network. It terminates TLS with a per-daemon CA
installed into the container at create, identifies the container by a
per-run proxy token, and applies that run's network policy and credential
inject rules. Upstream dials refuse loopback, and refuse internal addresses
unless the policy names the exact IP, so an allowed name cannot resolve
back into the host.
## Container layout
A Kit task runs in one container, from the Kit composition's image, on the
task's sandbox network. The worker and every harness invocation share it;
they are separated by user, not by container, so a run's cost scales with
the run and not with how many agents it fans out to.
- **Worker.** Archie mounts its own `archie-agent` binary read-only and runs
it as the container's entrypoint, as root. It holds the task's NATS and
State Store credentials in its own environment and in root-only files,
runs the workflow, and serves `archie-agent mcp`. The Kit image's
entrypoint is not PID 1.
- **Harness.** Every invocation runs as the Kit's user (the image's `User`,
default uid 1000) with an explicit environment: the Kit's variables, the
proxy environment and the credential sentinels, and nothing inherited from
the worker. That user cannot read the worker's environment or files, or
trace it. Sibling invocations in one run share the user and the worktree.
- **Hooks.** `lifecycle@1` install and startup hooks run as the user each
declares, before the worker accepts work.
- **Mounts.** The task worktree at the Kit's workspace directory, owned by the
Kit's user, and the read-only skills store. Nothing else from the host.
- **Network.** The relay forwards three fixed ports on the sandbox network:
the egress proxy, NATS and the State Store. The worker reaches NATS and the
State Store with its credentials; a harness can open those ports but holds
no credential either accepts.
- **Volumes.** `volume@1` paths persist for the life of the
WorkflowExecution, which covers session resume across gate iterations and
retries. They are not shared between executions.
## Archie tools inside the harness
Capture tools, which are how a stage returns structured results, are served
by `archie-agent mcp`, a stdio MCP server registered through the Kit's MCP
configuration or the adapter's flags. It serves the stage's `CaptureTools`
and the central tools the agent profile's `AllowTools` permits, never the
`workspace` toolset. The runner's `ToolLimits` bound its results; the
harness's own built-in tools are outside archie's limits.
## Enforcement around the harness
| Property | Enforcement |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Model never runs git | No forge credential can be bound. Archie records `HEAD` and every ref before the run; any change fails the step (`failed`, policy class). The workflow commits the stage's changes. |
| Protected paths | A diff touching a `Protection` path fails the step. |
| Read-only stage | Any worktree diff fails the step. |
| Gate | Archie runs the stage's gate after the harness exits. On failure it resumes the session with the gate output, up to the stage's iteration budget. |
| Budget | Archie kills the process on wall-clock timeout, or when streamed usage passes the token budget. |
| Cancellation | Cancelling the step kills the process group, then the container. |
| Egress | Only the proxy is reachable; the Kit's network policy decides the rest. |
Harness settings that deny git or protected paths are added where the CLI
supports them, to save wasted turns. They are never the enforcement.
## Execution tree
A harness run is one `agent` StepExecution. Subagents the harness spawns
internally are not recorded as children. Their usage is included in the
step's `tokens_used` when the adapter reports it.
## Agents never hold a harness
An Agent reaches a harness only by invoking a Workflow whose stage names it.
No Agent tool, chat tool or capability exposes a harness, its setup terminal
or its credential. The stage boundary is what carries the container, the git
and path checks, the gate and the budget; a harness outside a stage has none
of them.
## Out of scope
- Chat through a harness. The gateway chat loop stays on the built-in
runtime.
- `network-policy@2` HTTP method and path rules; a Kit requiring it is
refused.
- Recording the harness's internal subagents as StepExecutions.
- Cost in currency.
- Choosing a harness automatically per task.
## Plan
1. Descriptor resolution with the Kit spec package; capability support
table and fail-closed refusal.
2. Egress proxy: internal network, per-daemon CA, per-container token,
`network-policy@1`.
3. `credential@1`: API-key bindings, sentinel rendering, inject rules.
4. Harness `Runner`: container create with worktree, volumes, lifecycle
hooks and the mounted `archie-agent`; `agent-sessions@1` invocation;
worktree checks for git, protected paths and read-only; budget and
cancellation.
5. Claude Code output adapter; `archie-agent mcp`; gate-and-resume loop.
6. Kit profiles, runner selection from the stage's profile, and definition
validation.
7. OAuth bindings as org secrets: token-endpoint interception, encrypted
storage, per-binding concurrency; dashboard setup terminal.
8. Output adapters for Codex, Pi, OMP and GitHub Copilot CLI.
## Verification
- The same workflow definition completes on `builtin` and on the Claude Code
Kit with the same `Result` shape and captures.
- After `/login` in the setup terminal, the container's credential file holds
only sentinels, the State Store holds the encrypted token, and a task run
authenticates.
- A real API key or OAuth token never appears in the container environment,
filesystem, worktree, task log or event stream.
- A request to a host outside the Kit's runtime allow list fails, including a
direct connection that bypasses the proxy settings.
- A harness instructed to `git commit` fails its step, and the task branch is
unchanged.
- A harness edit to a protected test file fails the step in a TDD fix stage.
- A read-only feasibility stage that writes a file fails.
- A file a run writes into a Kit-managed path such as the CLI's settings file
is gone on the next execution.
- A Kit requiring an unsupported capability is refused at launch with the
capability named.
- A stage with a failing gate resumes the same session with the gate output,
then succeeds or exhausts its iterations.
- Two tasks sharing an OAuth binding with `max_concurrent = 1` run one at a
time; the second waits as `pending`.
- Stopping a run kills the harness process and removes the container.
- A harness invocation cannot read the worker's NATS or State Store
credential from its environment, its files or `/proc`.
- A run fanning out to many agent invocations uses one container.
- A Kit profile in org A cannot be selected, and its binding cannot be
resolved, by a workflow in org B.
- A harness run whose identity is not granted a Kit's required credential is
refused at launch; the proxy never injects a secret outside the run
credential.
- The firewall investigation example runs unchanged on an image profile with
direct network access.
+10 -4
View File
@@ -48,6 +48,7 @@ import (
"github.com/samcharles93/archie-core/internal/infrastructure/configuration"
"github.com/samcharles93/archie-core/internal/infrastructure/eventbus/nats"
"github.com/samcharles93/archie-core/internal/infrastructure/extension"
"github.com/samcharles93/archie-core/internal/infrastructure/kitrun"
"github.com/samcharles93/archie-core/internal/infrastructure/messagingrpc"
"github.com/samcharles93/archie-core/internal/infrastructure/modelcatalog"
"github.com/samcharles93/archie-core/internal/infrastructure/staterpc"
@@ -158,8 +159,11 @@ type boot struct {
containerPool *container.Pool
// kitLauncher starts Kit profile tasks; nil when the egress path cannot
// be built on this host.
kitLauncher daemon.KitLauncher
storeBackend storage.Backend
kitLauncher *kitrun.Launcher
// sessionGrants records live setup sessions' granted credential services
// for the egress proxy's resolver.
sessionGrants *sessionGrants
storeBackend storage.Backend
// runtimeVersions records the version of every control-plane kind boot's
// layering applied. Set once at boot before the runtime-resource watches
@@ -805,12 +809,14 @@ func (b *boot) startServices(ctx context.Context) error {
func (b *boot) setupBackends(ctx context.Context) error {
closeContainers := b.setupContainers(ctx)
// Registered before the Kit launcher so its cleanups (the harness
// listener and the egress relay) run first: cleanups unwind LIFO, and a
// session teardown needs the Docker client the pool still owns.
b.addCleanup(closeContainers)
err := b.connectNATS(ctx)
if err == nil {
b.setupKitLauncher(ctx)
}
b.addCleanup(closeContainers)
return err
}
+224
View File
@@ -0,0 +1,224 @@
package archied
import (
"context"
"errors"
"io"
"log/slog"
"net"
"os"
"slices"
"sync"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"github.com/samcharles93/archie-core/internal/app/servicekit"
"github.com/samcharles93/archie-core/internal/config"
pb "github.com/samcharles93/archie-core/internal/contracts/harness/v1"
"github.com/samcharles93/archie-core/internal/infrastructure/harnessrpc"
"github.com/samcharles93/archie-core/internal/infrastructure/harnesssession"
"github.com/samcharles93/archie-core/internal/infrastructure/kitrun"
)
// setupSessionTTL bounds a setup container's life independently of the pool's
// max-uptime reaper: a login that never finishes must not hold a container
// until the reaper's (longer) cap.
const setupSessionTTL = 30 * time.Minute
// sessionGrants records the credential services each live setup session is
// bound for. A session's proxy token names no task, so the egress proxy's run
// resolver consults this registry: without it, the token endpoint is never
// intercepted and no OAuth token is ever captured.
type sessionGrants struct {
mu sync.RWMutex
m map[string]sessionGrant
}
type sessionGrant struct {
org string
services []string
}
func (g *sessionGrants) Grant(token, org string, services []string) {
g.mu.Lock()
defer g.mu.Unlock()
if g.m == nil {
g.m = map[string]sessionGrant{}
}
g.m[token] = sessionGrant{org: org, services: slices.Clone(services)}
}
func (g *sessionGrants) Revoke(token string) {
g.mu.Lock()
defer g.mu.Unlock()
delete(g.m, token)
}
// GrantedFor reports the org and granted services a session token carries.
func (g *sessionGrants) GrantedFor(token string) (string, []string, bool) {
g.mu.RLock()
defer g.mu.RUnlock()
grant, ok := g.m[token]
return grant.org, grant.services, ok
}
// harnessServer serves the daemon's harness session contract.
type harnessServer struct {
pb.UnimplementedHarnessServiceServer
sessions *harnesssession.Manager
log *slog.Logger
}
// Open dispatches on the first frame's target. Setup sessions are served; the
// run-scoped target is reserved for archie-core-qi91.
func (h *harnessServer) Open(stream pb.HarnessService_OpenServer) error {
first, err := stream.Recv()
if err != nil {
return err
}
target := first.GetTarget()
if target == nil || target.GetOrg() == "" {
return status.Error(codes.InvalidArgument, "the first frame must name a session target with an org")
}
switch t := target.GetTarget().(type) {
case *pb.OpenTarget_Setup:
if t.Setup.GetProfile() == "" {
return status.Error(codes.InvalidArgument, "the setup target names no profile")
}
return h.openSetup(stream, target.GetOrg(), t.Setup.GetProfile())
case *pb.OpenTarget_Run:
return status.Error(codes.Unimplemented, "run-scoped sessions are not served yet")
default:
return status.Error(codes.InvalidArgument, "the target names no session")
}
}
// openSetup opens one setup session and pumps frames until the shell exits or
// either side closes. Teardown runs on every return.
func (h *harnessServer) openSetup(stream pb.HarnessService_OpenServer, org, profile string) error {
session, err := h.sessions.Open(stream.Context(), org, profile)
if err != nil {
h.log.Warn("setup session failed to open", "org", org, "profile", profile, "err", err)
// The failure rides the stream as a frame, not only as a status: the
// dashboard's terminal shows the frame's text, which is the operator's
// only honest account of why the session did not open.
return stream.Send(&pb.OpenResponse{Frame: &pb.OpenResponse_Error{Error: err.Error()}})
}
defer func() {
_ = session.Close()
h.log.Info("setup session closed", "org", org, "profile", profile)
}()
h.log.Info("setup session opened", "org", org, "profile", profile)
output := make(chan error, 1)
go func() { output <- h.pumpOutput(stream, session) }()
input := make(chan error, 1)
go func() { input <- h.pumpInput(stream, session) }()
select {
case err := <-output:
if err != nil {
return status.Error(codes.Internal, err.Error())
}
code, codeErr := session.ExitCode(context.WithoutCancel(stream.Context()))
if codeErr != nil {
return nil
}
return stream.Send(&pb.OpenResponse{Frame: &pb.OpenResponse_ExitCode{ExitCode: int32(code)}})
case err := <-input:
if errors.Is(err, io.EOF) {
return nil
}
return err
}
}
// pumpOutput copies terminal output to the stream until the shell ends.
func (h *harnessServer) pumpOutput(stream pb.HarnessService_OpenServer, session *harnesssession.Session) error {
buf := make([]byte, 32<<10)
for {
n, err := session.Read(buf)
if n > 0 {
frame := &pb.OpenResponse{Frame: &pb.OpenResponse_Stdout{Stdout: slices.Clone(buf[:n])}}
if sendErr := stream.Send(frame); sendErr != nil {
return sendErr
}
}
if err != nil {
if errors.Is(err, io.EOF) {
return nil
}
return err
}
}
}
// pumpInput copies client frames to the session until the client closes.
func (h *harnessServer) pumpInput(stream pb.HarnessService_OpenServer, session *harnesssession.Session) error {
for {
req, err := stream.Recv()
if err != nil {
return err
}
if stdin, ok := req.GetFrame().(*pb.OpenRequest_Stdin); ok {
if _, err := session.Write(stdin.Stdin); err != nil {
return err
}
}
}
}
// setupHarness serves the harness session contract on the configured listen
// address. It is skipped, with the reason logged, when no service token is
// configured: a session is a shell in a container that holds an org's
// credential, so the token is required even on loopback.
func (b *boot) setupHarness(ctx context.Context) {
conn := b.cfg.Services.Get(config.ServiceNameHarness)
listen, err := servicekit.ResolveListen(config.ServiceNameHarness, "", conn.Listen)
if err != nil {
b.log.Warn("harness sessions disabled", "err", err)
return
}
token := b.cfg.Services.ResolvedToken(config.ServiceNameHarness, os.Getenv)
if token == "" {
b.log.Warn("harness sessions disabled: [services.harness].target_token or HARNESS_TOKEN is required, even on loopback")
return
}
ln, err := (&net.ListenConfig{}).Listen(ctx, "tcp", listen)
if err != nil {
b.log.Warn("harness sessions disabled: listen", "err", err)
return
}
manager := &harnesssession.Manager{
Launcher: b.kitLauncher,
Client: b.containerPool.Client(),
Profile: func(name string) (config.AgentProfile, error) { return b.cfgHolder.Get().Containers.Profile(name) },
TTL: setupSessionTTL,
}
srv := grpc.NewServer(
//nolint:contextcheck // grpc.StreamServerInterceptor has no context.Context parameter; the interceptor derives its context from stream.Context()
grpc.StreamInterceptor(harnessrpc.StreamServerInterceptor(token)),
)
pb.RegisterHarnessServiceServer(srv, &harnessServer{sessions: manager, log: b.log})
go func() {
if err := srv.Serve(ln); err != nil && !errors.Is(err, grpc.ErrServerStopped) {
b.log.Error("harness listener stopped", "err", err)
}
}()
b.addCleanup(func() {
// An open session stream would hold GracefulStop until its TTL; give
// in-flight frames a moment, then stop. Mirrors the State Store's
// shutdown so a session cannot hold shutdown to the watchdog leash.
timer := time.AfterFunc(10*time.Second, srv.Stop)
srv.GracefulStop()
timer.Stop()
_ = ln.Close()
})
b.log.Info("harness sessions enabled", "addr", ln.Addr().String())
}
var _ kitrun.GrantRegistry = (*sessionGrants)(nil)
+8 -3
View File
@@ -60,8 +60,10 @@ func (b *boot) setupKitLauncher(ctx context.Context) {
return
}
// The proxy resolves each credential from the run credential the
// container logs in with, against the live grants and bindings.
resolver := runCredentialResolver{runs: b.stateStoreGrants.Client, config: b.cfgHolder, secrets: b.secrets}
// container logs in with, against the live grants and bindings; a setup
// session's token resolves through the session grants instead.
b.sessionGrants = &sessionGrants{}
resolver := runCredentialResolver{runs: b.stateStoreGrants.Client, sessions: b.sessionGrants, config: b.cfgHolder, secrets: b.secrets}
// Without a harness secret store, Register refuses a Kit with a
// required OAuth credential.
oauthStore, _ := b.stateStore.(egress.OAuthStore)
@@ -77,11 +79,14 @@ func (b *boot) setupKitLauncher(ctx context.Context) {
Entrypoint: []string{"archie-agent"}, Cmd: cmd, Network: pool.NetworkName(),
})
b.addCleanup(shutdownEgress(networks, srv, log))
b.kitLauncher = &kitrun.Launcher{
kitLauncher := &kitrun.Launcher{
Pool: pool, Fetch: fetcher, Proxy: proxy, Networks: networks,
AgentBinary: agentBinary, CAFile: egress.CACertPath(caDir),
Config: b.cfgHolder, Secrets: b.secrets, OAuth: oauthStore,
Grants: b.sessionGrants,
}
b.kitLauncher = kitLauncher
b.setupHarness(ctx)
log.Info("kit harness runs enabled", "proxy", ln.Addr().String())
}
@@ -12,10 +12,18 @@ import (
// credential for the run a run credential names, when the request is made,
// from the live grants and bindings. A revoked credential, a grant removed
// mid-run or a binding in another org stops resolving at once.
//
// A setup session's token names no task, so the resolver consults the session
// grants first: the session is bound for the org's bindings of the Kit's
// declared services, which is the intersection the operator's update-on-secret
// authorisation covers.
type runCredentialResolver struct {
runs interface {
TaskForCredential(ctx context.Context, token string) (*workflow.Task, error)
}
sessions interface {
GrantedFor(token string) (org string, services []string, ok bool)
}
config *config.Holder
secrets interface {
Resolve(config.SecretRef) (string, error)
@@ -23,19 +31,34 @@ type runCredentialResolver struct {
}
func (r runCredentialResolver) Resolve(ctx context.Context, credential, service string) (string, error) {
if r.sessions != nil {
if org, services, ok := r.sessions.GrantedFor(credential); ok {
return r.resolveBinding(org, services, service)
}
}
task, err := r.runs.TaskForCredential(ctx, credential)
if err != nil {
return "", egress.ErrUnbound
}
cfg := r.config.Get()
org, granted := cfg.CredentialAccess(task.Identity)
return r.resolveBinding(org, granted, service)
}
// resolveBinding resolves service from the org's binding when the run or
// session is granted it. An OAuth binding resolves to the empty string: the
// proxy supplies the org's token set itself.
func (r runCredentialResolver) resolveBinding(org string, granted []string, service string) (string, error) {
cfg := r.config.Get()
binding, ok := cfg.Containers.BoundCredentials(org, granted, []string{service})[service]
if !ok {
return "", egress.ErrUnbound
}
if binding.Secret == (config.SecretRef{}) {
// An OAuth binding: the proxy supplies the org's token set itself.
return "", nil
}
if r.secrets == nil {
return "", egress.ErrUnbound
}
return r.secrets.Resolve(binding.Secret)
}
@@ -65,3 +65,47 @@ func TestEgressResolvesOnlyGrantedCredentials(t *testing.T) {
})
}
}
// TestEgressResolvesASessionGrant pins that a setup session's token, which
// names no task, resolves only the granted services of its own org. A grant
// that widened to another org's binding would hand the container a credential
// the operator's update-on-secret authorisation never covered.
func TestEgressResolvesASessionGrant(t *testing.T) {
cfg := config.Config{Containers: config.ContainerConfig{Credentials: []config.CredentialBinding{
{Service: "github", Org: "acme", Secret: config.SecretRef{Engine: "env", Key: "ACME_GH"}},
{Service: "linear", Org: "acme"},
}}}
sessions := &sessionGrants{}
sessions.Grant("acme-session", "acme", []string{"github", "linear"})
sessions.Grant("other-session", "other", []string{"github"})
r := runCredentialResolver{
runs: runsByToken{"acme-run": {ID: 1, Identity: "acme-bot"}},
sessions: sessions,
config: config.NewHolder(cfg),
secrets: secretValues{"ACME_GH": "acme-gh"},
}
tests := []struct {
name, credential, service, want string
bound bool
}{
{name: "a bound api key resolves", credential: "acme-session", service: "github", want: "acme-gh", bound: true},
{name: "a bound oauth service resolves to the proxy's own token set", credential: "acme-session", service: "linear", bound: true},
{name: "a service outside the grant is unbound", credential: "acme-session", service: "nope"},
{name: "another org's binding is unbound", credential: "other-session", service: "github"},
{name: "an unknown token falls through to the task path", credential: "acme-run", service: "github"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := r.Resolve(context.Background(), tt.credential, tt.service)
if !tt.bound {
if !errors.Is(err, egress.ErrUnbound) {
t.Fatalf("got %q, %v; want ErrUnbound", got, err)
}
return
}
if err != nil || got != tt.want {
t.Fatalf("got %q, %v; want %q", got, err, tt.want)
}
})
}
}
+11
View File
@@ -58,6 +58,7 @@ type projection struct {
trustForwardedHeaders bool
gateway ServiceTarget
state ServiceTarget
harness ServiceTarget
capture CaptureOptions
}
@@ -67,6 +68,7 @@ func project(cfg config.Config) projection {
trustForwardedHeaders: cfg.Web.TrustForwardedHeaders,
gateway: ServiceTarget{Target: cfg.Services.Get(config.ServiceNameGateway).Target, Token: cfg.Services.Get(config.ServiceNameGateway).TargetToken},
state: ServiceTarget{Target: cfg.Services.Get(config.ServiceNameState).Target, Token: cfg.Services.Get(config.ServiceNameState).TargetToken},
harness: ServiceTarget{Target: cfg.Services.Get(config.ServiceNameHarness).Target, Token: cfg.Services.Get(config.ServiceNameHarness).TargetToken},
capture: CaptureOptions{
Retention: cfg.Capture.Retention.Std(),
MaxEvents: cfg.Capture.MaxEvents,
@@ -85,6 +87,9 @@ func withEnvTokens(o Options) Options {
if o.State.Token == "" {
o.State.Token = os.Getenv("STATE_STORE_TOKEN")
}
if o.Harness.Token == "" {
o.Harness.Token = os.Getenv("HARNESS_TOKEN")
}
return o
}
@@ -109,6 +114,12 @@ func merge(o Options, p projection) Options {
if o.State.Token == "" {
o.State.Token = p.state.Token
}
if o.Harness.Target == "" {
o.Harness.Target = p.harness.Target
}
if o.Harness.Token == "" {
o.Harness.Token = p.harness.Token
}
if o.Capture.Retention <= 0 {
o.Capture.Retention = p.capture.Retention
}
+4
View File
@@ -69,6 +69,10 @@ type Options struct {
// Gateway and State are the two remote contracts the dashboard consumes.
Gateway ServiceTarget
State ServiceTarget
// Harness is the daemon's session contract. Optional: empty leaves the
// setup terminal unwired, and /api/harness reports terminal:false rather
// than offering a control that would fail.
Harness ServiceTarget
// DependencyTimeout bounds each readiness probe's call to a dependency.
DependencyTimeout time.Duration
+19 -1
View File
@@ -22,6 +22,7 @@ import (
"github.com/samcharles93/archie-core/internal/infrastructure/configuration"
"github.com/samcharles93/archie-core/internal/infrastructure/controlplanerpc"
"github.com/samcharles93/archie-core/internal/infrastructure/gatewayrpc"
"github.com/samcharles93/archie-core/internal/infrastructure/harnessrpc"
"github.com/samcharles93/archie-core/internal/infrastructure/staterpc"
"github.com/samcharles93/archie-core/internal/webui"
)
@@ -58,7 +59,6 @@ func Run(ctx context.Context, options Options) error {
cleanup()
}
}()
tasks, closeState, err := staterpc.Dial(opts.State.Target, presence.UI, opts.State.Token)
if err != nil {
return err
@@ -100,6 +100,9 @@ func Run(ctx context.Context, options Options) error {
Principals: tasks,
Denials: tasks,
})
if err := wireHarness(opts, srv, &cleanups); err != nil {
return err
}
// Live activity has no in-process bus in this process: the pump reads
// events back out of the State Store over the same cursor SSE catch-up
@@ -132,6 +135,21 @@ func Run(ctx context.Context, options Options) error {
return serve(ctx, listener, srv.Handler(), opts)
}
// wireHarness attaches the setup terminal to the daemon's session contract
// when the operator configured one, and registers its cleanup.
func wireHarness(opts Options, srv *webui.Server, cleanups *[]func()) error {
if opts.Harness.Target == "" {
return nil
}
harness, closeHarness, err := harnessrpc.Dial(opts.Harness.Target, opts.Harness.Token)
if err != nil {
return err
}
srv.HarnessTerminal = harness
*cleanups = append(*cleanups, closeHarness)
return nil
}
// serve runs handler until ctx ends, then shuts down within
// ShutdownTimeout.
func serve(ctx context.Context, listener net.Listener, handler http.Handler, opts Options) error {
+6
View File
@@ -47,6 +47,7 @@ const (
ServiceNameGateway = "gateway"
ServiceNameState = "state"
ServiceNameMessaging = "messaging"
ServiceNameHarness = "harness"
)
var (
@@ -115,4 +116,9 @@ func init() {
RegisterService(ServiceBoth, ServiceNameGateway, "127.0.0.1:8585", "127.0.0.1:8585", "GATEWAY_TOKEN")
RegisterService(ServiceClient, ServiceNameState, "", "127.0.0.1:9090", "STATE_STORE_TOKEN")
RegisterService(ServiceBoth, ServiceNameMessaging, "127.0.0.1:8586", "127.0.0.1:8586", "MESSAGING_TOKEN")
// The daemon hosts the harness session contract in this repository; the
// dashboard dials it, and its target is the operator's to supply, so a
// deployment with no [services.harness].target leaves the setup terminal
// unwired rather than dialling a default nothing owns.
RegisterService(ServiceBoth, ServiceNameHarness, "", "127.0.0.1:8587", "HARNESS_TOKEN")
}
+61
View File
@@ -0,0 +1,61 @@
package container
import (
"bufio"
"context"
"net"
"github.com/moby/moby/client"
"github.com/samcharles93/archie-core/internal/infrastructure/kit"
)
// PTY is an interactive exec attached to a container's TTY: writes are
// terminal input, reads are terminal output, and Close closes the hijacked
// socket so a blocked Read unblocks. Docker cannot reattach to a disconnected
// exec, so the caller holds this for the life of the session.
type PTY struct {
execID string
conn net.Conn
reader *bufio.Reader
cli *client.Client
}
// ExecPTY starts argv in a container as user with exactly env, on a TTY. It
// mirrors ExecIn's explicit user and env: the exec must never inherit the
// container's worker environment.
func ExecPTY(ctx context.Context, cli *client.Client, id, user string, env, argv []string, rows, cols uint) (*PTY, error) {
exec, err := cli.ExecCreate(ctx, id, client.ExecCreateOptions{
User: user, Env: env, Cmd: argv, WorkingDir: kit.WorkspaceDir,
TTY: true, AttachStdin: true, AttachStdout: true, AttachStderr: true,
ConsoleSize: client.ConsoleSize{Height: rows, Width: cols},
})
if err != nil {
return nil, err
}
attached, err := cli.ExecAttach(ctx, exec.ID, client.ExecAttachOptions{TTY: true})
if err != nil {
return nil, err
}
return &PTY{execID: exec.ID, conn: attached.Conn, reader: attached.Reader, cli: cli}, nil
}
// Read reads terminal output. A TTY stream is raw, so there is no
// stdout/stderr to demultiplex.
func (p *PTY) Read(b []byte) (int, error) { return p.reader.Read(b) }
// Write writes terminal input.
func (p *PTY) Write(b []byte) (int, error) { return p.conn.Write(b) }
// ExitCode reports the exec's exit status. It is meaningful only once the
// output stream has ended.
func (p *PTY) ExitCode(ctx context.Context) (int, error) {
inspected, err := p.cli.ExecInspect(ctx, p.execID, client.ExecInspectOptions{})
if err != nil {
return 0, err
}
return inspected.ExitCode, nil
}
// Close closes the hijacked socket, which unblocks a blocked Read.
func (p *PTY) Close() error { return p.conn.Close() }
+505
View File
@@ -0,0 +1,505 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.36.11
// protoc (unknown)
// source: harness/v1/harness.proto
package harnessv1
import (
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
unsafe "unsafe"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
// OpenRequest is one client frame. The first frame must be target; a frame
// with no field set is a keepalive the server ignores.
type OpenRequest struct {
state protoimpl.MessageState `protogen:"open.v1"`
// Types that are valid to be assigned to Frame:
//
// *OpenRequest_Target
// *OpenRequest_Stdin
Frame isOpenRequest_Frame `protobuf_oneof:"frame"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *OpenRequest) Reset() {
*x = OpenRequest{}
mi := &file_harness_v1_harness_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *OpenRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*OpenRequest) ProtoMessage() {}
func (x *OpenRequest) ProtoReflect() protoreflect.Message {
mi := &file_harness_v1_harness_proto_msgTypes[0]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use OpenRequest.ProtoReflect.Descriptor instead.
func (*OpenRequest) Descriptor() ([]byte, []int) {
return file_harness_v1_harness_proto_rawDescGZIP(), []int{0}
}
func (x *OpenRequest) GetFrame() isOpenRequest_Frame {
if x != nil {
return x.Frame
}
return nil
}
func (x *OpenRequest) GetTarget() *OpenTarget {
if x != nil {
if x, ok := x.Frame.(*OpenRequest_Target); ok {
return x.Target
}
}
return nil
}
func (x *OpenRequest) GetStdin() []byte {
if x != nil {
if x, ok := x.Frame.(*OpenRequest_Stdin); ok {
return x.Stdin
}
}
return nil
}
type isOpenRequest_Frame interface {
isOpenRequest_Frame()
}
type OpenRequest_Target struct {
// target names the session and must be the first frame.
Target *OpenTarget `protobuf:"bytes,1,opt,name=target,proto3,oneof"`
}
type OpenRequest_Stdin struct {
// stdin is terminal input bytes.
Stdin []byte `protobuf:"bytes,2,opt,name=stdin,proto3,oneof"`
}
func (*OpenRequest_Target) isOpenRequest_Frame() {}
func (*OpenRequest_Stdin) isOpenRequest_Frame() {}
// OpenResponse is one server frame.
type OpenResponse struct {
state protoimpl.MessageState `protogen:"open.v1"`
// Types that are valid to be assigned to Frame:
//
// *OpenResponse_Stdout
// *OpenResponse_ExitCode
// *OpenResponse_Error
Frame isOpenResponse_Frame `protobuf_oneof:"frame"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *OpenResponse) Reset() {
*x = OpenResponse{}
mi := &file_harness_v1_harness_proto_msgTypes[1]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *OpenResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*OpenResponse) ProtoMessage() {}
func (x *OpenResponse) ProtoReflect() protoreflect.Message {
mi := &file_harness_v1_harness_proto_msgTypes[1]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use OpenResponse.ProtoReflect.Descriptor instead.
func (*OpenResponse) Descriptor() ([]byte, []int) {
return file_harness_v1_harness_proto_rawDescGZIP(), []int{1}
}
func (x *OpenResponse) GetFrame() isOpenResponse_Frame {
if x != nil {
return x.Frame
}
return nil
}
func (x *OpenResponse) GetStdout() []byte {
if x != nil {
if x, ok := x.Frame.(*OpenResponse_Stdout); ok {
return x.Stdout
}
}
return nil
}
func (x *OpenResponse) GetExitCode() int32 {
if x != nil {
if x, ok := x.Frame.(*OpenResponse_ExitCode); ok {
return x.ExitCode
}
}
return 0
}
func (x *OpenResponse) GetError() string {
if x != nil {
if x, ok := x.Frame.(*OpenResponse_Error); ok {
return x.Error
}
}
return ""
}
type isOpenResponse_Frame interface {
isOpenResponse_Frame()
}
type OpenResponse_Stdout struct {
// stdout is terminal output bytes.
Stdout []byte `protobuf:"bytes,1,opt,name=stdout,proto3,oneof"`
}
type OpenResponse_ExitCode struct {
// exit_code ends the session: the command's exit status.
ExitCode int32 `protobuf:"varint,2,opt,name=exit_code,json=exitCode,proto3,oneof"`
}
type OpenResponse_Error struct {
// error ends the session when it could not be opened or was aborted.
Error string `protobuf:"bytes,3,opt,name=error,proto3,oneof"`
}
func (*OpenResponse_Stdout) isOpenResponse_Frame() {}
func (*OpenResponse_ExitCode) isOpenResponse_Frame() {}
func (*OpenResponse_Error) isOpenResponse_Frame() {}
// OpenTarget addresses one session. A setup session is addressed by Kit
// profile and is ephemeral; a run session is addressed by task and attempt and
// is bounded by the task.
type OpenTarget struct {
state protoimpl.MessageState `protogen:"open.v1"`
Org string `protobuf:"bytes,1,opt,name=org,proto3" json:"org,omitempty"`
// Types that are valid to be assigned to Target:
//
// *OpenTarget_Setup
// *OpenTarget_Run
Target isOpenTarget_Target `protobuf_oneof:"target"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *OpenTarget) Reset() {
*x = OpenTarget{}
mi := &file_harness_v1_harness_proto_msgTypes[2]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *OpenTarget) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*OpenTarget) ProtoMessage() {}
func (x *OpenTarget) ProtoReflect() protoreflect.Message {
mi := &file_harness_v1_harness_proto_msgTypes[2]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use OpenTarget.ProtoReflect.Descriptor instead.
func (*OpenTarget) Descriptor() ([]byte, []int) {
return file_harness_v1_harness_proto_rawDescGZIP(), []int{2}
}
func (x *OpenTarget) GetOrg() string {
if x != nil {
return x.Org
}
return ""
}
func (x *OpenTarget) GetTarget() isOpenTarget_Target {
if x != nil {
return x.Target
}
return nil
}
func (x *OpenTarget) GetSetup() *SetupTarget {
if x != nil {
if x, ok := x.Target.(*OpenTarget_Setup); ok {
return x.Setup
}
}
return nil
}
func (x *OpenTarget) GetRun() *RunTarget {
if x != nil {
if x, ok := x.Target.(*OpenTarget_Run); ok {
return x.Run
}
}
return nil
}
type isOpenTarget_Target interface {
isOpenTarget_Target()
}
type OpenTarget_Setup struct {
Setup *SetupTarget `protobuf:"bytes,2,opt,name=setup,proto3,oneof"`
}
type OpenTarget_Run struct {
Run *RunTarget `protobuf:"bytes,3,opt,name=run,proto3,oneof"`
}
func (*OpenTarget_Setup) isOpenTarget_Target() {}
func (*OpenTarget_Run) isOpenTarget_Target() {}
// SetupTarget runs the CLI's own login in an ephemeral container built from
// the named agent profile's Kit.
type SetupTarget struct {
state protoimpl.MessageState `protogen:"open.v1"`
Profile string `protobuf:"bytes,1,opt,name=profile,proto3" json:"profile,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *SetupTarget) Reset() {
*x = SetupTarget{}
mi := &file_harness_v1_harness_proto_msgTypes[3]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *SetupTarget) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*SetupTarget) ProtoMessage() {}
func (x *SetupTarget) ProtoReflect() protoreflect.Message {
mi := &file_harness_v1_harness_proto_msgTypes[3]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use SetupTarget.ProtoReflect.Descriptor instead.
func (*SetupTarget) Descriptor() ([]byte, []int) {
return file_harness_v1_harness_proto_rawDescGZIP(), []int{3}
}
func (x *SetupTarget) GetProfile() string {
if x != nil {
return x.Profile
}
return ""
}
// RunTarget is a running task's container. It is reserved for the run-scoped
// live view; this version answers Unimplemented for it.
type RunTarget struct {
state protoimpl.MessageState `protogen:"open.v1"`
TaskId int64 `protobuf:"varint,1,opt,name=task_id,json=taskId,proto3" json:"task_id,omitempty"`
Attempt int64 `protobuf:"varint,2,opt,name=attempt,proto3" json:"attempt,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *RunTarget) Reset() {
*x = RunTarget{}
mi := &file_harness_v1_harness_proto_msgTypes[4]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *RunTarget) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*RunTarget) ProtoMessage() {}
func (x *RunTarget) ProtoReflect() protoreflect.Message {
mi := &file_harness_v1_harness_proto_msgTypes[4]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use RunTarget.ProtoReflect.Descriptor instead.
func (*RunTarget) Descriptor() ([]byte, []int) {
return file_harness_v1_harness_proto_rawDescGZIP(), []int{4}
}
func (x *RunTarget) GetTaskId() int64 {
if x != nil {
return x.TaskId
}
return 0
}
func (x *RunTarget) GetAttempt() int64 {
if x != nil {
return x.Attempt
}
return 0
}
var File_harness_v1_harness_proto protoreflect.FileDescriptor
const file_harness_v1_harness_proto_rawDesc = "" +
"\n" +
"\x18harness/v1/harness.proto\x12\n" +
"harness.v1\"`\n" +
"\vOpenRequest\x120\n" +
"\x06target\x18\x01 \x01(\v2\x16.harness.v1.OpenTargetH\x00R\x06target\x12\x16\n" +
"\x05stdin\x18\x02 \x01(\fH\x00R\x05stdinB\a\n" +
"\x05frame\"h\n" +
"\fOpenResponse\x12\x18\n" +
"\x06stdout\x18\x01 \x01(\fH\x00R\x06stdout\x12\x1d\n" +
"\texit_code\x18\x02 \x01(\x05H\x00R\bexitCode\x12\x16\n" +
"\x05error\x18\x03 \x01(\tH\x00R\x05errorB\a\n" +
"\x05frame\"\x84\x01\n" +
"\n" +
"OpenTarget\x12\x10\n" +
"\x03org\x18\x01 \x01(\tR\x03org\x12/\n" +
"\x05setup\x18\x02 \x01(\v2\x17.harness.v1.SetupTargetH\x00R\x05setup\x12)\n" +
"\x03run\x18\x03 \x01(\v2\x15.harness.v1.RunTargetH\x00R\x03runB\b\n" +
"\x06target\"'\n" +
"\vSetupTarget\x12\x18\n" +
"\aprofile\x18\x01 \x01(\tR\aprofile\">\n" +
"\tRunTarget\x12\x17\n" +
"\atask_id\x18\x01 \x01(\x03R\x06taskId\x12\x18\n" +
"\aattempt\x18\x02 \x01(\x03R\aattempt2O\n" +
"\x0eHarnessService\x12=\n" +
"\x04Open\x12\x17.harness.v1.OpenRequest\x1a\x18.harness.v1.OpenResponse(\x010\x01B\xb4\x01\n" +
"\x0ecom.harness.v1B\fHarnessProtoP\x01ZKgithub.com/samcharles93/archie-core/internal/contracts/harness/v1;harnessv1\xa2\x02\x03HXX\xaa\x02\n" +
"Harness.V1\xca\x02\n" +
"Harness\\V1\xe2\x02\x16Harness\\V1\\GPBMetadata\xea\x02\vHarness::V1b\x06proto3"
var (
file_harness_v1_harness_proto_rawDescOnce sync.Once
file_harness_v1_harness_proto_rawDescData []byte
)
func file_harness_v1_harness_proto_rawDescGZIP() []byte {
file_harness_v1_harness_proto_rawDescOnce.Do(func() {
file_harness_v1_harness_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_harness_v1_harness_proto_rawDesc), len(file_harness_v1_harness_proto_rawDesc)))
})
return file_harness_v1_harness_proto_rawDescData
}
var file_harness_v1_harness_proto_msgTypes = make([]protoimpl.MessageInfo, 5)
var file_harness_v1_harness_proto_goTypes = []any{
(*OpenRequest)(nil), // 0: harness.v1.OpenRequest
(*OpenResponse)(nil), // 1: harness.v1.OpenResponse
(*OpenTarget)(nil), // 2: harness.v1.OpenTarget
(*SetupTarget)(nil), // 3: harness.v1.SetupTarget
(*RunTarget)(nil), // 4: harness.v1.RunTarget
}
var file_harness_v1_harness_proto_depIdxs = []int32{
2, // 0: harness.v1.OpenRequest.target:type_name -> harness.v1.OpenTarget
3, // 1: harness.v1.OpenTarget.setup:type_name -> harness.v1.SetupTarget
4, // 2: harness.v1.OpenTarget.run:type_name -> harness.v1.RunTarget
0, // 3: harness.v1.HarnessService.Open:input_type -> harness.v1.OpenRequest
1, // 4: harness.v1.HarnessService.Open:output_type -> harness.v1.OpenResponse
4, // [4:5] is the sub-list for method output_type
3, // [3:4] is the sub-list for method input_type
3, // [3:3] is the sub-list for extension type_name
3, // [3:3] is the sub-list for extension extendee
0, // [0:3] is the sub-list for field type_name
}
func init() { file_harness_v1_harness_proto_init() }
func file_harness_v1_harness_proto_init() {
if File_harness_v1_harness_proto != nil {
return
}
file_harness_v1_harness_proto_msgTypes[0].OneofWrappers = []any{
(*OpenRequest_Target)(nil),
(*OpenRequest_Stdin)(nil),
}
file_harness_v1_harness_proto_msgTypes[1].OneofWrappers = []any{
(*OpenResponse_Stdout)(nil),
(*OpenResponse_ExitCode)(nil),
(*OpenResponse_Error)(nil),
}
file_harness_v1_harness_proto_msgTypes[2].OneofWrappers = []any{
(*OpenTarget_Setup)(nil),
(*OpenTarget_Run)(nil),
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_harness_v1_harness_proto_rawDesc), len(file_harness_v1_harness_proto_rawDesc)),
NumEnums: 0,
NumMessages: 5,
NumExtensions: 0,
NumServices: 1,
},
GoTypes: file_harness_v1_harness_proto_goTypes,
DependencyIndexes: file_harness_v1_harness_proto_depIdxs,
MessageInfos: file_harness_v1_harness_proto_msgTypes,
}.Build()
File_harness_v1_harness_proto = out.File
file_harness_v1_harness_proto_goTypes = nil
file_harness_v1_harness_proto_depIdxs = nil
}
@@ -0,0 +1,133 @@
// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
// versions:
// - protoc-gen-go-grpc v1.6.2
// - protoc (unknown)
// source: harness/v1/harness.proto
package harnessv1
import (
context "context"
grpc "google.golang.org/grpc"
codes "google.golang.org/grpc/codes"
status "google.golang.org/grpc/status"
)
// This is a compile-time assertion to ensure that this generated file
// is compatible with the grpc package it is being compiled against.
// Requires gRPC-Go v1.64.0 or later.
const _ = grpc.SupportPackageIsVersion9
const (
HarnessService_Open_FullMethodName = "/harness.v1.HarnessService/Open"
)
// HarnessServiceClient is the client API for HarnessService service.
//
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
//
// HarnessService opens one duplex PTY session into a Kit container the daemon
// owns. The dashboard's setup terminal consumes the setup target; the
// run-scoped live view consumes the run target (archie-core-qi91), which this
// version does not serve. See docs/prds/external-agent-harness.md,
// "Session contract".
type HarnessServiceClient interface {
// Open carries a session: the first request frame names the target, later
// frames carry terminal input and sizing, and the response stream carries
// terminal output until the command exits or the session ends.
Open(ctx context.Context, opts ...grpc.CallOption) (grpc.BidiStreamingClient[OpenRequest, OpenResponse], error)
}
type harnessServiceClient struct {
cc grpc.ClientConnInterface
}
func NewHarnessServiceClient(cc grpc.ClientConnInterface) HarnessServiceClient {
return &harnessServiceClient{cc}
}
func (c *harnessServiceClient) Open(ctx context.Context, opts ...grpc.CallOption) (grpc.BidiStreamingClient[OpenRequest, OpenResponse], error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
stream, err := c.cc.NewStream(ctx, &HarnessService_ServiceDesc.Streams[0], HarnessService_Open_FullMethodName, cOpts...)
if err != nil {
return nil, err
}
x := &grpc.GenericClientStream[OpenRequest, OpenResponse]{ClientStream: stream}
return x, nil
}
// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
type HarnessService_OpenClient = grpc.BidiStreamingClient[OpenRequest, OpenResponse]
// HarnessServiceServer is the server API for HarnessService service.
// All implementations must embed UnimplementedHarnessServiceServer
// for forward compatibility.
//
// HarnessService opens one duplex PTY session into a Kit container the daemon
// owns. The dashboard's setup terminal consumes the setup target; the
// run-scoped live view consumes the run target (archie-core-qi91), which this
// version does not serve. See docs/prds/external-agent-harness.md,
// "Session contract".
type HarnessServiceServer interface {
// Open carries a session: the first request frame names the target, later
// frames carry terminal input and sizing, and the response stream carries
// terminal output until the command exits or the session ends.
Open(grpc.BidiStreamingServer[OpenRequest, OpenResponse]) error
mustEmbedUnimplementedHarnessServiceServer()
}
// UnimplementedHarnessServiceServer must be embedded to have
// forward compatible implementations.
//
// NOTE: this should be embedded by value instead of pointer to avoid a nil
// pointer dereference when methods are called.
type UnimplementedHarnessServiceServer struct{}
func (UnimplementedHarnessServiceServer) Open(grpc.BidiStreamingServer[OpenRequest, OpenResponse]) error {
return status.Error(codes.Unimplemented, "method Open not implemented")
}
func (UnimplementedHarnessServiceServer) mustEmbedUnimplementedHarnessServiceServer() {}
func (UnimplementedHarnessServiceServer) testEmbeddedByValue() {}
// UnsafeHarnessServiceServer may be embedded to opt out of forward compatibility for this service.
// Use of this interface is not recommended, as added methods to HarnessServiceServer will
// result in compilation errors.
type UnsafeHarnessServiceServer interface {
mustEmbedUnimplementedHarnessServiceServer()
}
func RegisterHarnessServiceServer(s grpc.ServiceRegistrar, srv HarnessServiceServer) {
// If the following call panics, it indicates UnimplementedHarnessServiceServer was
// embedded by pointer and is nil. This will cause panics if an
// unimplemented method is ever invoked, so we test this at initialization
// time to prevent it from happening at runtime later due to I/O.
if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
t.testEmbeddedByValue()
}
s.RegisterService(&HarnessService_ServiceDesc, srv)
}
func _HarnessService_Open_Handler(srv interface{}, stream grpc.ServerStream) error {
return srv.(HarnessServiceServer).Open(&grpc.GenericServerStream[OpenRequest, OpenResponse]{ServerStream: stream})
}
// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
type HarnessService_OpenServer = grpc.BidiStreamingServer[OpenRequest, OpenResponse]
// HarnessService_ServiceDesc is the grpc.ServiceDesc for HarnessService service.
// It's only intended for direct use with grpc.RegisterService,
// and not to be introspected or modified (even as a copy)
var HarnessService_ServiceDesc = grpc.ServiceDesc{
ServiceName: "harness.v1.HarnessService",
HandlerType: (*HarnessServiceServer)(nil),
Methods: []grpc.MethodDesc{},
Streams: []grpc.StreamDesc{
{
StreamName: "Open",
Handler: _HarnessService_Open_Handler,
ServerStreams: true,
ClientStreams: true,
},
},
Metadata: "harness/v1/harness.proto",
}
@@ -0,0 +1,165 @@
// Package harnessrpc adapts the daemon-served harness session contract
// (internal/contracts/harness/v1) to the dashboard's HarnessTerminal
// interface and the daemon's gRPC listener. The daemon owns the container, so
// this is the only contract a session crosses.
package harnessrpc
import (
"context"
"crypto/subtle"
"errors"
"fmt"
"io"
"strings"
"sync"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/metadata"
"google.golang.org/grpc/status"
pb "github.com/samcharles93/archie-core/internal/contracts/harness/v1"
)
// tokenMetadataKey is the gRPC metadata key the harness service token travels
// in, mirroring the Gateway's and State Store's token keys.
const tokenMetadataKey = "harness-token"
// Client opens sessions against a daemon's HarnessService.
type Client struct {
svc pb.HarnessServiceClient
}
// Dial returns a harness client for target. The token is required for every
// target, loopback included: the daemon's listener enforces it because a
// session is a shell in a container that holds an org's credential. The
// cleanup closes the connection.
func Dial(target, token string, options ...grpc.DialOption) (*Client, func(), error) {
target = strings.TrimSpace(target)
if target == "" {
return nil, nil, errors.New("harness target is required")
}
if token == "" {
return nil, nil, fmt.Errorf(
"harness target %q reads a setup session, so it needs a bearer token ([services.harness].target_token / HARNESS_TOKEN)",
target,
)
}
opts := []grpc.DialOption{
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithStreamInterceptor(streamClientToken(token)),
}
conn, err := grpc.NewClient(target, append(opts, options...)...)
if err != nil {
return nil, nil, fmt.Errorf("create harness client: %w", err)
}
return &Client{svc: pb.NewHarnessServiceClient(conn)}, func() { _ = conn.Close() }, nil
}
// Open starts a setup session for profile and returns its duplex stream. The
// returned session satisfies the dashboard's HarnessTerminal interface. A
// failure the server reports before any output arrives is returned here; a
// failure after the session is open surfaces on Read.
func (c *Client) Open(ctx context.Context, orgID, profile string) (io.ReadWriteCloser, error) {
streamCtx, cancel := context.WithCancel(ctx)
stream, err := c.svc.Open(streamCtx)
if err != nil {
cancel()
return nil, err
}
target := &pb.OpenRequest{Frame: &pb.OpenRequest_Target{Target: &pb.OpenTarget{
Org: orgID,
Target: &pb.OpenTarget_Setup{Setup: &pb.SetupTarget{Profile: profile}},
}}}
if err := stream.Send(target); err != nil {
cancel()
return nil, err
}
return &session{stream: stream, cancel: cancel}, nil
}
// session adapts the bidi stream to io.ReadWriteCloser. Bytes are written as
// stdin frames and read from stdout frames; an error frame or an exit code
// ends the stream.
type session struct {
stream pb.HarnessService_OpenClient
cancel context.CancelFunc
mu sync.Mutex
pending []byte
}
func (s *session) Write(p []byte) (int, error) {
if err := s.stream.Send(&pb.OpenRequest{Frame: &pb.OpenRequest_Stdin{Stdin: p}}); err != nil {
return 0, err
}
return len(p), nil
}
func (s *session) Read(p []byte) (int, error) {
s.mu.Lock()
defer s.mu.Unlock()
for len(s.pending) == 0 {
resp, err := s.stream.Recv()
if err != nil {
return 0, err
}
switch frame := resp.GetFrame().(type) {
case *pb.OpenResponse_Stdout:
if len(frame.Stdout) == 0 {
continue
}
s.pending = frame.Stdout
case *pb.OpenResponse_ExitCode:
return 0, io.EOF
case *pb.OpenResponse_Error:
return 0, errors.New(frame.Error)
}
}
n := copy(p, s.pending)
s.pending = s.pending[n:]
return n, nil
}
func (s *session) Close() error {
s.cancel()
return nil
}
// validToken compares candidate against the configured token in constant
// time.
func validToken(token, candidate string) bool {
if token == "" || candidate == "" {
return false
}
return subtle.ConstantTimeCompare([]byte(token), []byte(candidate)) == 1
}
// StreamServerInterceptor validates the harness token on a stream. The
// session RPC is bidi, so this is the interceptor that matters. Unlike the
// State Store, a loopback listener enforces it too: a session is a shell in a
// container that holds an org's credential.
func StreamServerInterceptor(token string) grpc.StreamServerInterceptor {
return func(srv any, stream grpc.ServerStream, _ *grpc.StreamServerInfo, handler grpc.StreamHandler) error {
if !validContextToken(stream.Context(), token) {
return status.Error(codes.Unauthenticated, "missing or invalid harness token")
}
return handler(srv, stream)
}
}
func validContextToken(ctx context.Context, token string) bool {
md, ok := metadata.FromIncomingContext(ctx)
if !ok {
return false
}
values := md.Get(tokenMetadataKey)
return len(values) > 0 && validToken(token, values[0])
}
func streamClientToken(token string) grpc.StreamClientInterceptor {
return func(ctx context.Context, desc *grpc.StreamDesc, cc *grpc.ClientConn, method string, streamer grpc.Streamer, opts ...grpc.CallOption) (grpc.ClientStream, error) {
return streamer(metadata.AppendToOutgoingContext(ctx, tokenMetadataKey, token), desc, cc, method, opts...)
}
}
@@ -0,0 +1,118 @@
// Package harnesssession runs the daemon side of a harness session: it builds
// an ephemeral Kit-profile container, opens a PTY into it as the Kit's harness
// user, and tears the container down when the session closes. It owns no
// transport; internal/app/archied serves the gRPC contract over it.
package harnesssession
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"sync"
"time"
"github.com/moby/moby/client"
"github.com/samcharles93/archie-core/internal/config"
"github.com/samcharles93/archie-core/internal/container"
"github.com/samcharles93/archie-core/internal/infrastructure/kitrun"
)
// The address errors a caller distinguishes: an unknown profile is the
// caller's mistake, not a daemon that cannot run containers.
var (
ErrProfileUnknown = errors.New("agent profile is not configured")
ErrProfileNotKit = errors.New("agent profile is not a Kit profile")
)
// Launcher starts and ends a setup session's container. *kitrun.Launcher
// satisfies it.
type Launcher interface {
LaunchSetup(ctx context.Context, req kitrun.SetupRequest) (*kitrun.Run, error)
Release(ctx context.Context, run *kitrun.Run) error
}
// Manager opens setup sessions.
type Manager struct {
Launcher Launcher
// Client is the container pool's Docker client, used for the PTY exec.
Client *client.Client
// Profile resolves an agent profile by name from the running config.
Profile func(name string) (config.AgentProfile, error)
// TTL bounds a session's life. Zero means the container pool's max-uptime
// reaper is the only bound.
TTL time.Duration
}
// Open starts a setup session and returns its duplex PTY.
func (m *Manager) Open(ctx context.Context, org, profile string) (*Session, error) {
p, err := m.Profile(profile)
if err != nil {
return nil, fmt.Errorf("%w: %s", ErrProfileUnknown, profile)
}
if !p.IsKit() {
return nil, fmt.Errorf("%w: %s", ErrProfileNotKit, profile)
}
id, err := sessionID()
if err != nil {
return nil, err
}
run, err := m.Launcher.LaunchSetup(ctx, kitrun.SetupRequest{Session: id, Kit: p.Kit, Org: org})
if err != nil {
return nil, err
}
pty, err := container.ExecPTY(ctx, m.Client, run.Container.ID, run.Harness.User, run.Harness.Env, []string{"sh", "-l"}, 24, 80)
if err != nil {
_ = m.Launcher.Release(context.WithoutCancel(ctx), run)
return nil, err
}
s := &Session{pty: pty, run: run, release: m.Launcher.Release}
if m.TTL > 0 {
//nolint:contextcheck // teardown must outlive the stream context that opened the session
s.timer = time.AfterFunc(m.TTL, func() { _ = s.Close() })
}
return s, nil
}
// Session is one open setup session. Close is idempotent.
type Session struct {
pty *container.PTY
run *kitrun.Run
release func(context.Context, *kitrun.Run) error
timer *time.Timer
once sync.Once
}
func (s *Session) Read(p []byte) (int, error) { return s.pty.Read(p) }
func (s *Session) Write(p []byte) (int, error) { return s.pty.Write(p) }
// ExitCode reports the shell's exit status once its output has ended.
func (s *Session) ExitCode(ctx context.Context) (int, error) { return s.pty.ExitCode(ctx) }
// Close ends the PTY and tears the session down: the container, its volumes,
// its network and its egress session. Teardown uses its own context because
// the session's is usually already cancelled.
func (s *Session) Close() error {
var err error
s.once.Do(func() {
if s.timer != nil {
s.timer.Stop()
}
_ = s.pty.Close()
err = s.release(context.Background(), s.run)
})
return err
}
// sessionID is the per-session key for the container, network and volumes. It
// is random so two sessions for one profile never collide on a container name.
func sessionID() (string, error) {
var b [8]byte
if _, err := rand.Read(b[:]); err != nil {
return "", fmt.Errorf("mint session id: %w", err)
}
return "setup-" + hex.EncodeToString(b[:]), nil
}
+141 -2
View File
@@ -5,6 +5,8 @@ package kitrun
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"maps"
@@ -44,12 +46,24 @@ const (
caPath = "/opt/archie/ca.pem"
)
// GrantRegistry records a live session's granted credential services, so the
// egress proxy can resolve a session token that names no task. It is optional:
// a Launcher without one serves task runs only.
type GrantRegistry interface {
Grant(token, org string, services []string)
Revoke(token string)
}
// Launcher starts Kit task containers.
type Launcher struct {
Pool *container.Pool
Fetch *fetch.Client
Proxy *egress.Proxy
Networks *egress.Networks
// Grants records a setup session's granted services while it is open.
// Nil refuses a setup session rather than opening one whose token the
// egress proxy cannot resolve.
Grants GrantRegistry
// AgentBinary and CAFile are host paths mounted read-only into every
// Kit container: the worker's own binary and the egress CA.
AgentBinary string
@@ -103,6 +117,122 @@ type Run struct {
network string
token string
execution string
grants GrantRegistry
// ephemeral marks a setup session, whose volumes are never reused by a
// later run and so are removed with it.
ephemeral bool
}
// SetupRequest is one ephemeral setup session to start: a Kit-profile
// container whose only job is to run a CLI's own login so the proxy captures
// the tokens at the token endpoint.
type SetupRequest struct {
// Session keys the container, network and volumes, and roots its
// container name.
Session string
// Kit is the profile's digest-pinned Kit or published Kit set.
Kit string
// Org is the org whose credential bindings the session may use. A setup
// session carries no task and no identity grant, so its bound services
// are the org's bindings for the Kit's declared services.
Org string
}
// LaunchSetup starts an ephemeral setup session's container. Unlike Launch it
// mounts no worktree and runs no worker: the container holds a keepalive and
// the operator's PTY runs the CLI's login inside it. The launch runs the
// Kit's install hooks at the install phase and then enters runtime, exactly
// as a task run does, so a runtime-phase credential's token endpoint is the
// one the proxy intercepts.
func (l *Launcher) LaunchSetup(ctx context.Context, req SetupRequest) (*Run, error) {
if req.Kit == "" {
return nil, errors.New("kit profile names no kit")
}
if l.Grants == nil || l.Config == nil {
return nil, errors.New("setup sessions need a session grant registry and a configuration holder")
}
if err := l.startEgress(ctx); err != nil {
return nil, err
}
k, err := l.read(ctx, req.Kit, 0)
if err != nil {
return nil, err
}
bound, services := orgCredentialKinds(l.Config.Get().Containers.Credentials, req.Org, k.creds)
token, err := newProxyToken()
if err != nil {
return nil, err
}
session, err := l.Proxy.Register(egress.SessionOptions{Token: token, Org: req.Org, Network: k.network, Credentials: k.creds, Bound: bound})
if err != nil {
return nil, err
}
run := &Run{network: "archie-kit-" + req.Session, token: session.Token(), execution: req.Session, grants: l.Grants, ephemeral: true}
// Bound is deliberately empty here: a credential file renders the stored
// scopes and expiry, and there is no stored token set yet -- that is the
// state this session exists to change. The Kit's sentinels still reach the
// harness environment.
launch, err := kit.Assemble(k.plan, k.img, kit.LaunchParams{Execution: req.Session, ProxyToken: session.Token(), CAPath: caPath})
if err != nil {
l.release(run)
return nil, err
}
run.Harness, run.Volumes = launch.Harness, launch.Volumes
l.Grants.Grant(session.Token(), req.Org, services)
if err := l.Networks.Create(ctx, run.network); err != nil {
l.release(run)
return nil, err
}
// The image's own entrypoint is the harness CLI and would exit or run a
// task; a setup container holds still until the PTY's shell ends. The
// container environment stays empty (WorkerEnv nil): the harness env is
// passed explicitly on the exec, and no worker credential reaches it.
run.Container, err = l.Pool.AcquireKit(ctx, container.KitSpec{
Name: "archie-kit-" + req.Session,
Image: req.Kit,
Network: run.network,
Worker: []string{"sleep", "infinity"},
Binds: append([]string{l.CAFile + ":" + caPath + ":ro"}, skillsBinds(k.skills, l.skillsDir())...),
Launch: launch,
InstallDone: session.EnterRuntime,
})
if err != nil {
l.release(run)
return nil, errors.Join(err, l.Networks.Remove(context.WithoutCancel(ctx), run.network))
}
return run, nil
}
// orgCredentialKinds is the kind of each of the Kit's declared services the
// org binds, with no identity grant in the intersection: a setup session acts
// on the org's secret, not on a run's grant.
func orgCredentialKinds(bindings []config.CredentialBinding, org string, creds []spec.CredentialCapability) (map[string]egress.CredentialKind, []string) {
declared := make([]string, len(creds))
for i, c := range creds {
declared[i] = c.Service
}
bound := config.ContainerConfig{Credentials: bindings}.BoundCredentials(org, declared, declared)
kinds := make(map[string]egress.CredentialKind, len(bound))
services := make([]string, 0, len(bound))
for service, b := range bound {
if b.Secret == (config.SecretRef{}) {
kinds[service] = egress.CredentialOAuth
} else {
kinds[service] = egress.CredentialAPIKey
}
services = append(services, service)
}
return kinds, services
}
// newProxyToken mints a setup session's run credential. It is unguessable
// because it is the container's proxy password.
func newProxyToken() (string, error) {
var b [32]byte
if _, err := rand.Read(b[:]); err != nil {
return "", fmt.Errorf("mint session credential: %w", err)
}
return hex.EncodeToString(b[:]), nil
}
// Launch reads the request's Kit and starts its container.
@@ -224,6 +354,9 @@ func (l *Launcher) oauthFacts(ctx context.Context, org string, creds []spec.Cred
// failure, so it takes no error to join.
func (l *Launcher) release(run *Run) {
l.Proxy.Revoke(run.token)
if run.grants != nil {
run.grants.Revoke(run.token)
}
}
// kitNeeds is what a profile's Kit asks of its run, read from its admitted
@@ -277,11 +410,17 @@ func (l *Launcher) skillsDir() string {
}
// Release stops the run's container, then removes its network, revokes its
// credential grant and ends its egress session.
// credential grant and ends its egress session. A setup session's volumes are
// removed too: they are keyed to one ephemeral session and no later run reuses
// them. A task run keeps its volumes for the life of its execution.
func (l *Launcher) Release(ctx context.Context, run *Run) error {
l.Pool.Release(ctx, run.Container)
l.release(run)
return l.Networks.Remove(context.WithoutCancel(ctx), run.network)
err := l.Networks.Remove(context.WithoutCancel(ctx), run.network)
if run.ephemeral {
err = errors.Join(err, l.RemoveVolumes(context.WithoutCancel(ctx), run.Volumes))
}
return err
}
// RemoveVolumes removes an ended execution's Kit volumes.
@@ -0,0 +1,74 @@
package kitrun
import (
"testing"
"github.com/docker/sandbox-kit-spec/v3/spec"
"github.com/samcharles93/archie-core/internal/config"
"github.com/samcharles93/archie-core/internal/infrastructure/egress"
)
// A setup session has no task and no identity grant, so it must bind only the
// org's own bindings for the Kit's declared services. Widening that
// intersection would let one org's terminal carry another org's credential,
// or a service the Kit never declared.
func TestOrgCredentialKinds(t *testing.T) {
t.Parallel()
bindings := []config.CredentialBinding{
{Service: "github", Org: "acme", Secret: config.SecretRef{Engine: "builtin", Key: "gh"}},
{Service: "linear", Org: "acme"},
{Service: "github", Org: "other", Secret: config.SecretRef{Engine: "builtin", Key: "gh"}},
}
creds := []spec.CredentialCapability{{Service: "github"}, {Service: "linear"}, {Service: "unbound"}}
cases := []struct {
name string
org string
want map[string]egress.CredentialKind
services []string
}{
{
name: "binds the org's own declared services",
org: "acme",
want: map[string]egress.CredentialKind{
"github": egress.CredentialAPIKey,
"linear": egress.CredentialOAuth,
},
services: []string{"github", "linear"},
},
{
name: "another org's binding never resolves",
org: "other",
want: map[string]egress.CredentialKind{"github": egress.CredentialAPIKey},
services: []string{
"github",
},
},
{
name: "unknown org binds nothing",
org: "nobody",
want: map[string]egress.CredentialKind{},
services: []string{},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, services := orgCredentialKinds(bindings, tc.org, creds)
if len(got) != len(tc.want) {
t.Fatalf("kinds = %v, want %v", got, tc.want)
}
for service, kind := range tc.want {
if got[service] != kind {
t.Errorf("kind[%s] = %q, want %q", service, got[service], kind)
}
}
if len(services) != len(tc.services) {
t.Fatalf("services = %v, want %v", services, tc.services)
}
})
}
}
+9 -1
View File
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"sort"
@@ -197,10 +198,17 @@ func (s *Server) handleHarnessTerminal(w http.ResponseWriter, r *http.Request) {
// bridgeTerminal pumps bytes between the dashboard WebSocket and the setup
// session until either side closes, then closes both so the surviving copy
// unblocks. It is a free function so a test can drive it without a listener.
// A session failure is written to the socket first: the terminal would
// otherwise only close, leaving the operator no reason.
func bridgeTerminal(ws *websocket.Conn, session io.ReadWriteCloser) {
done := make(chan struct{}, 2)
go func() { _, _ = io.Copy(session, ws); done <- struct{}{} }()
go func() { _, _ = io.Copy(ws, session); done <- struct{}{} }()
go func() {
if _, err := io.Copy(ws, session); err != nil {
_, _ = fmt.Fprintf(ws, "\r\n\x1b[31m%s\x1b[0m\r\n", err)
}
done <- struct{}{}
}()
<-done
_ = ws.Close()
_ = session.Close()
+62
View File
@@ -0,0 +1,62 @@
syntax = "proto3";
package harness.v1;
// HarnessService opens one duplex PTY session into a Kit container the daemon
// owns. The dashboard's setup terminal consumes the setup target; the
// run-scoped live view consumes the run target (archie-core-qi91), which this
// version does not serve. See docs/prds/external-agent-harness.md,
// "Session contract".
service HarnessService {
// Open carries a session: the first request frame names the target, later
// frames carry terminal input and sizing, and the response stream carries
// terminal output until the command exits or the session ends.
rpc Open(stream OpenRequest) returns (stream OpenResponse);
}
// OpenRequest is one client frame. The first frame must be target; a frame
// with no field set is a keepalive the server ignores.
message OpenRequest {
oneof frame {
// target names the session and must be the first frame.
OpenTarget target = 1;
// stdin is terminal input bytes.
bytes stdin = 2;
}
}
// OpenResponse is one server frame.
message OpenResponse {
oneof frame {
// stdout is terminal output bytes.
bytes stdout = 1;
// exit_code ends the session: the command's exit status.
int32 exit_code = 2;
// error ends the session when it could not be opened or was aborted.
string error = 3;
}
}
// OpenTarget addresses one session. A setup session is addressed by Kit
// profile and is ephemeral; a run session is addressed by task and attempt and
// is bounded by the task.
message OpenTarget {
string org = 1;
oneof target {
SetupTarget setup = 2;
RunTarget run = 3;
}
}
// SetupTarget runs the CLI's own login in an ephemeral container built from
// the named agent profile's Kit.
message SetupTarget {
string profile = 1;
}
// RunTarget is a running task's container. It is reserved for the run-scoped
// live view; this version answers Unimplemented for it.
message RunTarget {
int64 task_id = 1;
int64 attempt = 2;
}