Replaces stdin/stdout subprocess transport with NATS JetStream request-reply pattern. The Runner interface and workflow orchestration are unchanged. New NATSRunner (implements Runner): - Publishes AgentRequestMessage to JetStream with X-Archie-Reply header - Blocks on core NATS inbox subscription for response - Same Run(ctx, workspace, req) (Result, error) contract Rewritten archie-agent (cmd/archie-agent): - Long-running NATS-connected process (was 28-line stdin/stdout) - Pull-consumes from ARCHIE_TASKS with archie.agent.> filter - Builds InProcessRunner, executes agent loop, replies to inbox - Graceful shutdown via signal handling Extended internal/nats: - SubjectAgentWildcard, SubjectForAgentRequest, ReplyHeader - Conn() accessor, NewReplyInbox(), PublishMsg() for JetStream - Stream subjects extended to ["archie.task.>", "archie.agent.>"] Config: "nats" added as valid agent.mode alongside inprocess/subprocess Tests: NATSRunner round-trip with embedded NATS server + handler
16 KiB
agentexec
Now I have all the information needed. Here is the comprehensive report.
1. All Fields of Invocation, Request, Result, Response
All types are defined in /work/apps/archie-core/internal/agentexec/protocol.go.
Invocation (lines 139-144):
Version int `json:"version"`
Workspace string `json:"workspace"`
Request Request `json:"request"`
Providers map[string]Provider `json:"providers"`
Request (lines 59-74):
Version int `json:"version"`
TaskID int64 `json:"task_id"`
Attempt int `json:"attempt"`
Stage string `json:"stage"`
Model string `json:"model"`
Mission string `json:"mission"`
ExtraRules string `json:"extra_rules,omitempty"`
ReadOnly bool `json:"read_only,omitempty"`
Budget Budget `json:"budget"`
Gate Gate `json:"gate"`
Preflight []Command `json:"preflight,omitempty"`
Protection Protection `json:"protection"`
Notes string `json:"notes,omitempty"`
CaptureTools []CaptureTool `json:"capture_tools,omitempty"`
Result (lines 97-111):
Version int `json:"version"`
TaskID int64 `json:"task_id"`
Attempt int `json:"attempt"`
Stage string `json:"stage"`
Status string `json:"status"`
StopReason string `json:"stop_reason,omitempty"`
Changes []string `json:"changes,omitempty"`
Iterations int `json:"iterations,omitempty"`
TokensUsed int `json:"tokens_used,omitempty"`
Summary string `json:"summary,omitempty"`
Detail string `json:"detail,omitempty"`
AppendedNotes []string `json:"appended_notes,omitempty"`
Captures map[string][]json.RawMessage `json:"captures,omitempty"`
Response (lines 168-172):
Version int `json:"version"`
Result Result `json:"result"`
Error string `json:"error,omitempty"`
Supporting types referenced above:
- Budget (lines 31-35):
MaxSteps int,MaxTokens int,WallClock time.Duration - Gate (lines 25-28):
Commands []Command,MaxConsecutiveFailures int - Command (lines 18-22):
Name string,Argv []string,ExpectFailure bool - Protection (lines 38-41):
Suffixes []string,Globs []string - CaptureTool (lines 45-53):
Name string,Description string,Parameters json.RawMessage,RequiredFields []string,NonEmptyStrings []string,BooleanFields []string,MaxCalls int - Provider (lines 115-119):
Class string,APIKeyEnv string,BaseURL string
2. ServeOne Step by Step
Defined in /work/apps/archie-core/internal/agentexec/worker.go.
ServeOne (lines 16-20) is the public entry point. It delegates to serveOne with a factory closure that, given an Invocation, returns a runner:
func ServeOne(ctx context.Context, in io.Reader, out io.Writer, log *slog.Logger) error {
return serveOne(ctx, in, out, func(invocation Invocation) Runner {
return NewInProcessRunner(NewRuntime(invocation.Providers), log)
})
}
serveOne (lines 22-43) does:
-
Decode invocation (line 24): Calls
decodeOne(in, &invocation)which reads up tomaxProtocolBytes+1bytes (8 MB + 1) fromin. Rejects if the value exceeds 8 MB. JSON-decodes into the value (here,Invocation). Also checks there is exactly one JSON value (rejects trailing JSON values, line 58-61). -
Validate invocation (line 27): Calls
invocation.Validate()which checks:Version == ProtocolVersion(must be 1)Workspaceis non-emptyProvidersmap is non-empty, and every entry'sClassis non-empty andBaseURLis parseable without userinfo/query/fragment- The inner
Request.Validate():Versionmust be 1,TaskID> 0,Attempt> 0,Stagenon-empty,Modelnon-empty
-
Create runner (line 30): Calls
newRunner(invocation)-- in ServeOne's case,NewInProcessRunner(NewRuntime(invocation.Providers), log). -
Run agent (line 34):
runner.Run(ctx, invocation.Workspace, invocation.Request)returnsresult, runErr. -
Build response (lines 35-38): Constructs
Response{Version: ProtocolVersion, Result: result}. IfrunErr != nil, setsresponse.Error = runErr.Error(). -
Encode response to stdout (lines 39-41):
json.NewEncoder(out).Encode(response). Returns error if encoding fails. -
Return nil on success.
Key design detail: The serveOne generic function separates the wire protocol (stdin/stdout JSON framing) from the runner creation strategy. The factory func(Invocation) Runner allows different strategies (in-process vs. subprocess vs. future NATS). Currently ServeOne hardcodes the in-process factory, but a future NATS-based archie-agent could call serveOne with a different factory or skip it entirely.
3. SubprocessRunner.Run Step by Step
Defined in /work/apps/archie-core/internal/agentexec/subprocess.go.
The SubprocessRunner struct (lines 17-27):
type SubprocessRunner struct {
Command string
Args []string
Environ []string // daemon's os.Environ()
AdditionalEnv []string // operator-approved env var names
Diagnostics io.Writer // typically os.Stderr
Providers map[string]Provider
}
Run (lines 29-94):
-
Extract provider name from model (lines 30-32): Splits
req.Modelat the first "/". E.g."anthropic/claude-sonnet-4"yields providerName="anthropic". If no "/" or empty, returns error. -
Look up provider (lines 34-37): Indexes
r.Providers[providerName]. Missing returns error. -
Build and validate invocation (lines 38-42): Creates
Invocation{Version, Workspace, Request, Providers}with just that one provider entry. Validates. -
Check command (lines 43-45):
strings.TrimSpace(r.Command)must be non-empty. -
Marshal invocation (lines 46-50):
json.Marshal(invocation)+ appends newline. -
Create OS command (line 52):
exec.CommandContext(ctx, r.Command, r.Args...). -
Configure process cancellation (line 53):
configureProcessCancellation(cmd)-- on Unix this setsSetpgid: trueand aCancelfunction that doessyscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)(see/work/apps/archie-core/internal/agentexec/subprocess_process_unix.golines 15-27). This ensures the worker process AND all its grandchildren die on context cancellation. -
Set stdin (line 54):
cmd.Stdin = bytes.NewReader(payload). -
Set environment (line 55):
WorkerEnvironment(r.Environ, r.AdditionalEnv, providers)which (lines 155-162):- Starts with
DefaultEnvironmentNames():PATH, HOME, TMPDIR, TMP, TEMP, LANG, LC_ALL, SSL_CERT_FILE, SSL_CERT_DIR - Appends
r.AdditionalEnv - Appends each provider's
APIKeyEnv - Calls
SelectEnvironmentwhich copies only those named variables fromr.Environ, preserving order.
- Starts with
-
Capture stdout and stderr (lines 56-62):
- Stdout:
cappedBuffer{limit: maxProtocolBytes}(8 MB). Ifr.Diagnostics != nil, stderr goes toio.MultiWriter(r.Diagnostics, stderr)(tee to both the diagnostics writer and the capped buffer). Otherwise just the capped buffer (64 KB limit viamaxDiagnosticBytes).
- Stdout:
-
Run the process (lines 63-71):
cmd.Run(). On error:- If
context.Cause(ctx)is set (cancellation), returns the cause wrapped with any stderr diagnostic. - Otherwise returns the exec error with stderr output.
- If
-
Check stdout truncation (lines 72-74): If
stdout.truncatedis true, return error. -
Decode response (lines 75-78):
decodeOne(bytes.NewReader(stdout.Bytes()), &response)-- same framing as the reader side. -
Validate response version (lines 79-81): Must match
ProtocolVersion(1). -
Validate result against request (lines 82-91):
- If
response.Result.Version != 0, callsresponse.Result.ValidateFor(req)-- checks Version==1, TaskID/Attempt/Stage identity match, Status non-empty. - If
response.Error != "", returns result witherrors.New(response.Error). - Validates one more time at line 90 for the non-error case.
- If
-
Return (line 93):
response.Result, nil.
4. NewRuntime and InProcessRunner
NewRuntime in /work/apps/archie-core/internal/agentexec/runtime.go (lines 7-23):
- Takes
providers map[string]Provider(agentexec.Provider type). - Returns
*runtime.Runtime(fromgithub.com/samcharles93/ai-sdk/runtime). - If providers is empty, returns
nil. - Calls
runtime.RegisterBuiltinClasses()(global, side-effecting). - For each entry: creates
runtime.ProviderConfig{ID: name, Class: provider.Class, BaseURL: provider.BaseURL}. - If
provider.APIKeyEnv == "", setsAuth = AuthConfig{Type: AuthTypeNone}, otherwiseAuth = AuthConfig{APIKeyEnv: provider.APIKeyEnv}. - Calls
runtime.NewRuntime(runtime.Config{Providers: catalog})and returns it.
InProcessRunner in /work/apps/archie-core/internal/agentexec/inprocess.go (lines 29-33):
type InProcessRunner struct {
runtime *runtime.Runtime
log *slog.Logger
run loopFunc // func(context.Context, agentloop.Config) (agentloop.Result, error)
}
NewInProcessRunner (lines 35-37): Stores runtime, log, and sets run = agentloop.Run (the exported function from the agentloop package).
InProcessRunner.Run (lines 39-86):
- Validates
reqviareq.Validate(). - Checks
r.runtime != nilandr.run != nil. - Creates
memoryNotesstruct (lines 219-228 of inprocess.go) -- implements a simpleLoad/Appendinterface. Initialized withreq.Notes, collectsappendedentries. - Creates
capturesmap:map[string][]json.RawMessage. - Calls
r.run(ctx, agentloop.Config{...})with all fields mapped fromreq:Runtime: r.runtimeModelRef: req.ModelWorkDir: workspaceMission: req.Mission,ExtraRules: req.ExtraRulesNotes: notes(the memoryNotes)Gate: toAgentGate(req.Gate)(converts agentexec.Gate -> agentloop.GateConfig)Preflight: toAgentCommands(req.Preflight)Budget: agentloop.Budget(req.Budget)(direct cast)ReadOnly: req.ReadOnlyProtectPaths: protectionMatcher(...)(closure that checks suffixes/globs)Extra: mergeToolSets(captureToolSet(req.CaptureTools, captures), scriptToolSet(workspace))Logger: r.logger(req)(structured with task/attempt/stage/model)
- Builds
Resultfrom the returnedagentloop.Resultfields. - Checks
context.Cause(ctx)for cancellation -- if cancelled, returns result with cancellation cause as error. - Returns
result, err.
5. Provider Types
There are two separate Provider structs with identical fields but different packages and tags:
agentexec.Provider (/work/apps/archie-core/internal/agentexec/protocol.go, lines 115-119):
type Provider struct {
Class string `json:"class"`
APIKeyEnv string `json:"api_key_env,omitempty"`
BaseURL string `json:"base_url,omitempty"`
}
Used for: JSON serialization to the subprocess worker, and as the type consumed by NewRuntime and SubprocessRunner.Providers.
config.Provider (/work/apps/archie-core/internal/config/config.go, lines 125-129):
type Provider struct {
Class string `toml:"class"`
APIKeyEnv string `toml:"api_key_env"`
BaseURL string `toml:"base_url"`
}
Used for: TOML deserialization from the config file.
Bridge (/work/apps/archie-core/cmd/archied/main.go, lines 182-188):
func executionProviders(cfg config.Config) map[string]agentexec.Provider {
providers := make(map[string]agentexec.Provider, len(cfg.Providers))
for name, p := range cfg.Providers {
providers[name] = agentexec.Provider{Class: p.Class, APIKeyEnv: p.APIKeyEnv, BaseURL: p.BaseURL}
}
return providers
}
The daemon composition root converts config.Provider to agentexec.Provider by field-wise copy and then passes them to both NewRuntime and SubprocessRunner.
6. How drainNATS Interacts with SubprocessRunner
Current state: drainNATS only affects task distribution, not agent execution.
The flow:
-
Daemon.Cycle (daemon.go lines 94-103): If
d.Nats != nil, callsd.drainNATS(ctx), elsed.drainSQLite(ctx). -
drainNATS (lines 126-151): Calls
d.Nats.Fetch(ctx)to get a JetStream message. If found, callsprocessNATSTask. Falls back todrainSQLitelogic (SQLiteClaimNextthend.process) if no NATS message. -
processNATSTask (lines 219-253): Decodes
TaskMessage, enqueues to SQLite, claims the task, then callsd.process(ctx, task)-- the sameprocessfunction used by the SQLite path. -
d.process (lines 472-493): Routes the task to the correct workflow, then calls:
workflow.Run(ctx, wf, &workflow.TaskContext{ ... Agent: d.Agent, // line 488 -- the agentexec.Runner interface ... }) -
workflow.Run (workflow.go line 145): Iterates stages. For LLM-driven stages, calls
AgentStage.Stage()(agent.go line 110):res, err := tc.Agent.Run(ctx, tc.Dir, req)This
Agentfield is theagentexec.Runnerinterface. It was wired incmd/archied/main.go(lines 124-138):- If
cfg.Agent.Mode == "subprocess": creates&agentexec.SubprocessRunner{...}withCommand,Environ,AdditionalEnv,Diagnostics,Providers. - If
cfg.Agent.Mode == "inprocess": createsagentexec.NewInProcessRunner(llm, log).
- If
-
SubprocessRunner.Run spawns a fresh
archie-agentbinary per stage, sends JSON on stdin, reads JSON from stdout. -
The daemon's NATS
processNATSTaskdoes NOT interact with SubprocessRunner directly -- it only fetches the task message from NATS, enqueues it to SQLite, and claims it. The actual agent execution happens throughd.Agent.Runwhich remains either in-process or subprocess regardless of the NATS path.
Design Implications for NATSRunner and NATS-based archie-agent
Runner interface (/work/apps/archie-core/internal/agentexec/inprocess.go, line 20-22):
type Runner interface {
Run(ctx context.Context, workspace string, req Request) (Result, error)
}
A new NATSRunner would need to implement this same interface. It would:
- Accept a NATS connection, a subject prefix, and the provider map (or the whole Invocation fields).
- In
Run(ctx, workspace, req):- Build an
Invocation(same asSubprocessRunner.Rundoes on lines 38-42). - Publish it to a NATS subject (e.g.
archie.stage.request.<taskID>). - Include a reply subject in the message headers or as part of the payload.
- Wait for the response on the reply subject (with the context's deadline).
- Decode the
Response, validate, returnResult.
- Build an
- No need for stdin/stdout marshalling or process management.
New archie-agent (cmd/archie-agent/main.go): Instead of ServeOne(ctx, os.Stdin, os.Stdout, log), it would:
- Connect to NATS.
- Subscribe to a subject like
archie.stage.request.>. - For each message: decode
Invocationfrom the message body, callserveOne-style logic (validate, build runner, run, build response), publishResponseback to the reply subject, ack. - Could be a long-running daemon with a pool of subscribers.
Key integration point: The daemon composition root (cmd/archied/main.go lines 124-138) would gain a third branch:
case "nats":
agentRunner = &agentexec.NATSRunner{
Conn: natsClient.Conn(), // or a new NATS connection
Subject: "archie.stage.request",
Providers: providers,
}
And in config.go the Agent.Mode validation (lines 298-305) would accept "nats" in addition to "inprocess" and "subprocess".
The serveOne generic function in worker.go (lines 22-43) is already well-factored: it separates wire protocol from runner creation. A NATS-based archie-agent could reuse serveOne by providing a factory that creates an in-process runner, or it could have its own message handler that calls the same Validate -> Runner.Run -> Response logic.