diff --git a/.agents/skills/archie-architecture-contract/SKILL.md b/.agents/skills/archie-architecture-contract/SKILL.md index 2130edc3b..0c894fddc 100644 --- a/.agents/skills/archie-architecture-contract/SKILL.md +++ b/.agents/skills/archie-architecture-contract/SKILL.md @@ -53,7 +53,7 @@ Do not simplify these facts into the target model: | Durable work | `store.Task` combines forge/chat source data, workflow position, output, retry, identity, and PR state. | `internal/store/store.go` | | Workflow | `internal/workflow` routes a `store.Task` into a `map[string]Workflow`, mutates a broad `TaskContext` through sequential stages. | `internal/workflow/workflow.go` | | Agent-stage boundary | Versioned `agentexec.Request`/`Result` carries one autonomous stage; validation correlates task, attempt, and stage. | `internal/agentexec/protocol.go` | -| Container handoff | `internal/taskrun` carries entire task plus `config.Repo` and `config.TaskConfig`; `archie-agent` runs routing and sequencing. | `internal/taskrun/taskrun.go`; `cmd/archie-agent/main.go` | +| Container handoff | `internal/taskrun` carries entire task plus `config.Repo` and `config.TaskConfig`; `internal/app/agentworker` owns routing and sequencing while its NATS adapter owns wire mechanics. | `internal/taskrun/taskrun.go`; `internal/app/agentworker/`; `internal/infrastructure/agenttransport/nats/` | | Git ownership | Daemon-owned worktree operations clone, commit, push, diff, and clean up. | `internal/worktree/`; `internal/workflow/steps.go`; `ARCHITECTURE.md` | | Enforcement | Gates, read-only/protected paths, TDD inverted test gates, and diff caps are represented outside prompt prose. | `internal/agentexec/inprocess.go`; `internal/workflow/{agent,tdd,steps}.go` | | Configuration | `internal/config.Config`, `IdentityConfig`, `Repo`, and task snapshots mix input decoding with runtime concerns. | `internal/config/config.go` | diff --git a/.agents/skills/archie-codebase-discovery/SKILL.md b/.agents/skills/archie-codebase-discovery/SKILL.md index c6428f638..674929a42 100644 --- a/.agents/skills/archie-codebase-discovery/SKILL.md +++ b/.agents/skills/archie-codebase-discovery/SKILL.md @@ -8,7 +8,8 @@ description: "Trace Archie behavior and blast radius through Go source, types, s Use evidence from the current checkout to explain a behavior before changing it. Volatile tool facts below were verified on **2026-07-28**. -**Composition root**: `cmd/archied/main.go` and `cmd/archie-agent/`. +**Composition roots**: `cmd/archied/main.go` (legacy wiring) and +`internal/app/agentworker` (invoked by `cmd/archie-agent/main.go`). **Production wiring**: a path selected by a shipped entrypoint — not a constructor, decoder, or passing unit test. **State owner**: component with authority to validate and persist a state transition. @@ -63,7 +64,7 @@ Start at these verified anchors: | Concern | Current anchor | |---|---| | Resident daemon | `cmd/archied/main.go` → `run()` → `daemon.Daemon` | -| Sandboxed worker | `cmd/archie-agent/main.go`, `cmd/archie-agent/taskrun.go` | +| Sandboxed worker | `cmd/archie-agent/main.go` → `internal/app/agentworker/`; NATS boundary in `internal/infrastructure/agenttransport/nats/` | | Task orchestration | `internal/daemon/daemon.go`, `internal/workflow/` | | Task contracts and SQLite | `internal/store/interface.go`, `internal/store/` | | Production task persistence | `internal/store/`; `store.Open` in `cmd/archied/main.go` | @@ -176,7 +177,8 @@ test-only. 1. Start at `cmd/archied/main.go:run`. 2. Record config condition per branch. 3. Follow concrete construction into `daemon.Daemon` and gateway start. -4. For container/NATS behavior, also trace `cmd/archie-agent`. +4. For container/NATS behavior, trace `cmd/archie-agent/main.go` into + `internal/app/agentworker` and `internal/infrastructure/agenttransport/nats`. 5. Trace both sides of every RPC/transport and error/timeout paths. 6. Read contract/architecture tests. 7. Confirm deploy inputs in Compose and CI. diff --git a/.agents/skills/archie-debugging-playbook/SKILL.md b/.agents/skills/archie-debugging-playbook/SKILL.md index 554793fd8..59a025295 100644 --- a/.agents/skills/archie-debugging-playbook/SKILL.md +++ b/.agents/skills/archie-debugging-playbook/SKILL.md @@ -91,7 +91,7 @@ commit `308c199`, not proof that Yaegi cannot run external commands. | Task discovery | `archie.task.>` in `ARCHIE_TASKS` | Work-queue retention; daemon durable `archie-daemon`; max deliver 3 | | Per-stage agent request | `archie.agent..request` | Reply inbox in `X-Archie-Reply`; wall-clock budget or 30m | | Full task handoff | `archie.taskrun.` | Core request/reply; no-responder retry 20s every 250ms | -| Store RPC | `archie.store.update`, `.transition` | Client timeout 60s by `cmd/archie-agent/taskrun.go` | +| Store RPC | `archie.store.update`, `.transition` | Client timeout 60s composed by `internal/app/agentworker/worker.go` through the infrastructure transport | | Forge RPC | `archie.forge.*` | Error travels in a JSON envelope | | Worktree RPC | `archie.worktree.prepare`, `.push` | Server default handler bound 15m | | Discovery dedup | `Nats-Msg-Id: archie://` | JetStream duplicate window 2m | diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index a8115fb55..acf88d035 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -18,7 +18,11 @@ See `docs/architecture/organisation.md` for the target structure and | Package | Role | | ---------------------------- | ------------------------------------------------------------------------- | | `cmd/archied/` | Binary entry: config→forge→store→runtime→workflows→daemon | -| `cmd/archie-agent/` | One-invocation autonomous stage worker (JSON stdin/stdout) | +| `cmd/archie-agent/` | Binary entry: flags, environment, signals, and agentworker invocation | +| `internal/app/agentworker/` | Agent worker composition, lifecycle, routing, and workflow execution | +| `internal/infrastructure/agentboot/` | Worker boot-brief filesystem decoding | +| `internal/infrastructure/agentgit/` | Worker git safe-directory setup | +| `internal/infrastructure/agenttransport/nats/` | Worker NATS transport, stage/task wire boundaries, and RPC client construction | | `internal/config/` | Config types only — loading lives in `infrastructure/configuration` | | `internal/daemon/` | Resident loop: poll, enqueue, claim, route, process, reconcile | | `internal/domain/workflow/` | Engine: stage lists, routing, shared steps, workflow definitions | diff --git a/CLAUDE.md b/CLAUDE.md index d56ec54bc..a7c78c4f7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -343,7 +343,7 @@ boundaries. Summary of what's evolved since that doc was last fully accurate: conflict and the later one silently receives nothing. `nats.Client.Subscribe` creates a NEW consumer, so calling it on subjects the client's own durable consumer already filters will never deliver — drain the existing consumer with - `Fetch` instead, as `cmd/archie-agent` and the agentexec round-trip fake do. + `Fetch` instead, as `internal/app/agentworker` and the agentexec round-trip fake do. (2) *Reply inboxes are not stream subjects.* Answering a request must use core NATS publish, not the JetStream path: a `Message.ReplyAddress` is an ephemeral `_INBOX.*` belonging to one waiting caller and part of no stream, so a durable @@ -375,8 +375,8 @@ boundaries. Summary of what's evolved since that doc was last fully accurate: `.gitignore` and `.git/info/exclude`, so anything in the worktree root is pushed onto the task branch. Nothing under `.git` can be tracked; `worktree.go:41-46` documents this reasoning for the prepared sentinel. If you - change the path, update `cmd/archie-agent/main.go` `bootTaskID` (the reader) - in lockstep. + change the path, update `internal/infrastructure/agentboot/task.go` `TaskID` + (the reader) in lockstep. - **MCP providers are registered by the daemon as _optional_** (`cmd/archied/ main.go`, `providerRegistry.RegisterOptional`). One that cannot start is logged at error level, excluded from the running set, and reported diff --git a/cmd/archie-agent/architecture_test.go b/cmd/archie-agent/architecture_test.go new file mode 100644 index 000000000..c5a337a23 --- /dev/null +++ b/cmd/archie-agent/architecture_test.go @@ -0,0 +1,108 @@ +package main + +import ( + "go/ast" + "go/parser" + "go/token" + "path/filepath" + "strconv" + "strings" + "testing" +) + +func TestCommandBoundary(t *testing.T) { + paths, err := filepath.Glob("*.go") + if err != nil { + t.Fatal(err) + } + for _, path := range paths { + if !strings.HasSuffix(path, "_test.go") && path != "main.go" { + t.Errorf("cmd/archie-agent contains substantive production file %q; runtime ownership belongs in internal/app/agentworker", path) + } + } + + file, err := parser.ParseFile(token.NewFileSet(), "main.go", nil, 0) + if err != nil { + t.Fatal(err) + } + + const ( + projectInternal = "github.com/samcharles93/archie-core/internal/" + appImport = projectInternal + "app/agentworker" + ) + appIdent := "agentworker" + for _, imported := range file.Imports { + path, err := strconv.Unquote(imported.Path.Value) + if err != nil { + t.Fatal(err) + } + if strings.HasPrefix(path, projectInternal) && path != appImport { + t.Errorf("cmd/archie-agent imports substantive internal package %q; only %q is allowed", path, appImport) + } + if path == appImport && imported.Name != nil { + if imported.Name.Name == "." || imported.Name.Name == "_" { + t.Errorf("cmd/archie-agent imports %q with unsupported alias %q", appImport, imported.Name.Name) + } else { + appIdent = imported.Name.Name + } + } + } + + allowedFunctions := map[string]bool{ + "main": true, + "run": true, + "runCommand": true, + "natsConnectionSettings": true, + } + for _, declaration := range file.Decls { + function, ok := declaration.(*ast.FuncDecl) + if !ok { + continue + } + if function.Recv != nil || !allowedFunctions[function.Name.Name] { + t.Errorf("cmd/archie-agent main.go declares unexpected function %q; orchestration belongs in internal/app/agentworker", function.Name.Name) + } + delete(allowedFunctions, function.Name.Name) + } + for name := range allowedFunctions { + t.Errorf("cmd/archie-agent main.go is missing intended process function %q", name) + } + + var runFunction *ast.FuncDecl + for _, declaration := range file.Decls { + function, ok := declaration.(*ast.FuncDecl) + if ok && function.Name.Name == "run" { + runFunction = function + break + } + } + if runFunction == nil { + t.Fatal("cmd/archie-agent has no run function") + } + + workerCalls := 0 + ast.Inspect(runFunction.Body, func(node ast.Node) bool { + call, ok := node.(*ast.CallExpr) + if !ok { + return true + } + callee, ok := call.Fun.(*ast.Ident) + if !ok || callee.Name != "runCommand" { + return true + } + for _, argument := range call.Args { + selector, ok := argument.(*ast.SelectorExpr) + if !ok || selector.Sel.Name != "Run" { + continue + } + pkg, ok := selector.X.(*ast.Ident) + if ok && pkg.Name == appIdent { + workerCalls++ + } + } + return true + }) + if workerCalls != 1 { + t.Fatalf("cmd/archie-agent run passes agentworker.Run to runCommand %d times, want exactly 1", workerCalls) + } +} diff --git a/cmd/archie-agent/main.go b/cmd/archie-agent/main.go index 6fe96d681..d3b3564b7 100644 --- a/cmd/archie-agent/main.go +++ b/cmd/archie-agent/main.go @@ -1,230 +1,63 @@ -// Command archie-agent is a long-running NATS-connected worker. It answers -// two independent request shapes that serve different deployment modes: -// -// 1. Single-stage requests on archie.agent.> over the ARCHIE_TASKS -// JetStream stream -- used when agent.mode is "nats" without -// container sandboxing. One stage is dispatched per message, and the -// agent replies on the caller's inbox. -// -// 2. Full-task handoffs on archie.taskrun.> (core NATS, queue-grouped) -// -- used with Docker container sandboxing. The agent runs -// workflow.Route/workflow.Run itself, proxying Store/Forge/worktree -// operations back to archied over NATS instead of holding those -// credentials directly. +// Command archie-agent runs Archie's NATS-connected agent worker. package main import ( "context" - "encoding/json" "errors" "flag" "fmt" + "io" "log/slog" "os" "os/signal" - "path/filepath" - "strings" "syscall" - "time" - "github.com/nats-io/nats.go" - - "github.com/samcharles93/archie-core/internal/agentexec" - "github.com/samcharles93/archie-core/internal/config" - "github.com/samcharles93/archie-core/internal/container" - "github.com/samcharles93/archie-core/internal/domain/workintake" - "github.com/samcharles93/archie-core/internal/eventbus" - arnats "github.com/samcharles93/archie-core/internal/infrastructure/eventbus/nats" - "github.com/samcharles93/archie-core/internal/storage" - "github.com/samcharles93/archie-core/internal/taskrun" - "github.com/samcharles93/archie-core/internal/tools" - "github.com/samcharles93/archie-core/internal/tools/mcp" - toolprovider "github.com/samcharles93/archie-core/internal/tools/provider" - mcptoolprovider "github.com/samcharles93/archie-core/internal/tools/provider/mcp" + "github.com/samcharles93/archie-core/internal/app/agentworker" ) -const ( - consumerName = "archie-agent" - pollTimeout = 5 * time.Second +const consumerName = "archie-agent" - // ackWait exceeds the longest stage runtime: a stage still working must - // not have its message redelivered to another worker. - ackWait = 30 * time.Minute - - // taskRunSubjectWildcard matches full-task handoffs (archie.taskrun.). - // A queue group means only one archie-agent instance answers each - // message, whether deployed as a shared worker pool (today) or spawned - // one-per-task (the sandboxed container target). - taskRunSubjectWildcard = "archie.taskrun.>" - taskRunQueueGroup = "archie-taskrun-workers" -) +type workerRunner func(context.Context, agentworker.Settings, *slog.Logger) error func main() { os.Exit(run()) } func run() int { - natsURLFlag := flag.String("nats-url", "", "NATS server URL (defaults to NATS_URL)") - consumer := flag.String("consumer", consumerName, "JetStream consumer name") - flag.Parse() + return runCommand(os.Args[1:], os.Getenv, os.Stderr, agentworker.Run) +} - natsURL, natsToken := natsConnectionSettings(*natsURLFlag, os.Getenv) +func runCommand(args []string, getenv func(string) string, stderr io.Writer, runWorker workerRunner) int { + flags := flag.NewFlagSet("archie-agent", flag.ContinueOnError) + flags.SetOutput(stderr) + natsURLFlag := flags.String("nats-url", "", "NATS server URL (defaults to NATS_URL)") + consumer := flags.String("consumer", consumerName, "JetStream consumer name") + if err := flags.Parse(args); err != nil { + if errors.Is(err, flag.ErrHelp) { + return 0 + } + return 2 + } + + natsURL, natsToken := natsConnectionSettings(*natsURLFlag, getenv) if natsURL == "" { - fmt.Fprintln(os.Stderr, "error: -nats-url or NATS_URL is required") - flag.Usage() + fmt.Fprintln(stderr, "error: -nats-url or NATS_URL is required") + flags.Usage() return 1 } - log := slog.New(slog.NewJSONHandler(os.Stderr, nil)) - + log := slog.New(slog.NewJSONHandler(stderr, nil)) ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() - // Before anything can run a gate: the worktree is a host bind mount owned - // by another UID, and git refuses to work in it until it is marked safe. - markWorktreeSafe(ctx, storage.WorktreeMountDir, runGit, log) - - // One bus client owns the stream definition. archied declares the same - // stream with the same subjects; this process previously declared its - // own copy, free to disagree about retention, dedup window and TTLs. - bus, err := arnats.Connect(ctx, arnats.Config{ - URL: natsURL, - Token: natsToken, - Subjects: []string{workintake.SubjectTaskWildcard, agentexec.SubjectAgentWildcard}, - ConsumerName: *consumer, - FilterSubject: agentexec.SubjectAgentWildcard, - PollTimeout: pollTimeout, - AckWait: ackWait, - }, log) - if err != nil { - log.Error("nats connect failed", "err", err) + if err := runWorker(ctx, agentworker.Settings{ + NATSURL: natsURL, + NATSToken: natsToken, + Consumer: *consumer, + }, log); err != nil { return 1 } - defer bus.Close() - - // Full-task handoffs use core NATS request/reply with a queue group, - // which the JetStream contract does not express. - nc, err := bus.CoreConn() - if err != nil { - log.Error("nats connection unavailable", "err", err) - return 1 - } - - // A dedicated per-task subscription (below) means this container was - // spawned for one task, which is also when its stderr is otherwise - // unrecoverable: container.Pool creates it with AutoRemove: true. Once - // nc exists, ship this process's own logs to the daemon over - // SubjectForSystem(taskID) so they survive the container's exit. Shared - // queue-group mode serves many unrelated tasks per process, so there is - // no single task subject to ship to -- it keeps the stderr-only logger. - // - // This only covers logging done through the `log` variable from here on. - // bus (arnats.Client) was constructed above with the pre-wrap logger and - // keeps it for its own lifetime, so its own wire-activity Debug lines - // (e.g. "published" on every bus.Respond) stay stderr-only regardless. - // That's a real, known gap, not an oversight: it's the bus's internal - // chatter about NATS itself, not the agent's own reasoning or tool - // output, and reordering construction to close it would mean connecting - // to NATS before knowing whether NATS is even needed for this call. - taskID, hasTaskID := bootTaskID(storage.WorktreeMountDir, log) - if hasTaskID { - log = slog.New(agentexec.NewSystemLogHandler(log.Handler(), nc, taskID)) - log.Info("system log publisher attached", "task", taskID) - } - - // Subscribe to task run messages. - taskRunSub, err := subscribeTaskRuns(ctx, nc, log, taskID, hasTaskID) - if err != nil { - log.Error("taskrun subscribe failed", "err", err) - return 1 - } - defer func() { - if err := taskRunSub.Unsubscribe(); err != nil { - log.Warn("task run unsubscribe failed", "err", err) - } - }() - - log.Info("archie-agent ready", "nats", natsURL, "consumer", *consumer) - return runMainLoop(ctx, bus, log) -} - -// subscribeTaskRuns prefers a dedicated per-task subscription over the -// shared queue group whenever this container was spawned for a specific -// task. taskID/hasTaskID come from bootTaskID (.git/task.json present at the -// worktree mount) -- the caller resolves it once, since it also decides -// whether to attach a per-task log publisher to log before this is called. -func subscribeTaskRuns(ctx context.Context, nc *nats.Conn, log *slog.Logger, taskID int64, hasTaskID bool) (*nats.Subscription, error) { - if hasTaskID { - log.Info("taskrun: dedicated per-task subscription", "task", taskID) - return nc.Subscribe(taskrun.SubjectForTask(taskID), func(msg *nats.Msg) { - handleTaskRun(ctx, msg, nc, log) - }) - } - log.Info("taskrun: shared queue-group subscription (no .git/task.json found)") - return nc.QueueSubscribe(taskRunSubjectWildcard, taskRunQueueGroup, func(msg *nats.Msg) { - handleTaskRun(ctx, msg, nc, log) - }) -} - -// bootTaskID reads the boot-time task.json brief the daemon writes into -// the worktree before container acquire (container.WriteTaskJSON) and -// returns its ID. It lives under .git so the agent's own commit can't sweep -// it onto the task branch. Returns (0, false) when the file is absent, -// unparseable, or has no ID -- any of which mean "shared pool mode, no -// dedicated task." -func bootTaskID(mountDir string, log *slog.Logger) (int64, bool) { - data, err := os.ReadFile(filepath.Join(mountDir, ".git", "task.json")) - if err != nil { - return 0, false - } - var payload container.TaskPayload - if err := json.Unmarshal(data, &payload); err != nil { - log.Warn("task.json present but unparseable -- falling back to shared queue subscription", "err", err) - return 0, false - } - if payload.ID <= 0 { - return 0, false - } - return payload.ID, true -} - -// stageBus is the messaging this loop needs: pull the next stage request and -// answer it on the requester's inbox. -type stageBus interface { - Fetch(ctx context.Context) (eventbus.Message, error) - Respond(ctx context.Context, replyAddress string, payload []byte) error -} - -// runMainLoop runs the fetch-handle-ack loop until the context is cancelled. -func runMainLoop(ctx context.Context, bus stageBus, log *slog.Logger) int { - for { - if ctx.Err() != nil { - log.Info("archie-agent shutting down") - return 0 - } - msg, err := bus.Fetch(ctx) - switch { - case errors.Is(err, eventbus.ErrNoMessage): - continue - case err != nil: - if ctx.Err() != nil { - return 0 - } - log.Error("fetch failed", "err", err) - time.Sleep(1 * time.Second) - continue - } - if err := handle(ctx, msg, bus, log); err != nil { - log.Error("handle failed", "err", err) - if err := msg.Nak(); err != nil { - log.Warn("nak failed", "err", err) - } - continue - } - if err := msg.Ack(); err != nil { - log.Warn("ack failed", "err", err) - } - } + return 0 } func natsConnectionSettings(flagURL string, getenv func(string) string) (url, token string) { @@ -234,182 +67,3 @@ func natsConnectionSettings(flagURL string, getenv func(string) string) (url, to } return url, getenv("NATS_TOKEN") } - -// mcpProviderSet holds the resources needed to clean up MCP providers. -type mcpProviderSet struct { - providers []toolprovider.Engine - registry *tools.Registry -} - -// startMCPProviders creates MCP transports from config, starts them, -// discovers tools, and registers them into a local registry. Returns -// nil when there are no servers configured. -func startMCPProviders(ctx context.Context, servers []config.MCPServer, log *slog.Logger) (*mcpProviderSet, error) { - if len(servers) == 0 { - return nil, nil - } - reg := tools.NewRegistry() - var providers []toolprovider.Engine - var firstErr error - - for _, srv := range servers { - name := strings.TrimSpace(srv.Name) - if name == "" { - log.Warn("mcp server skipped: empty name") - continue - } - transportType := strings.ToLower(strings.TrimSpace(srv.Transport)) - if transportType == "" { - transportType = "stdio" - } - - var transport mcptoolprovider.LifecycleTransport - switch transportType { - case "stdio": - command := strings.TrimSpace(srv.Command) - if command == "" { - err := fmt.Errorf("MCP stdio server %q requires a command", name) - log.Warn("mcp server skipped", "name", name, "err", err) - if firstErr == nil { - firstErr = err - } - continue - } - transport = mcp.NewStdioTransport(mcp.StdioTransportConfig{ - Command: command, - Args: append([]string(nil), srv.Args...), - }) - case "http", "streamablehttp": - url := strings.TrimSpace(srv.URL) - if url == "" { - err := fmt.Errorf("MCP http server %q requires a url", name) - log.Warn("mcp server skipped", "name", name, "err", err) - if firstErr == nil { - firstErr = err - } - continue - } - transport = mcp.NewHTTPTransport(mcp.HTTPTransportConfig{ - Endpoint: url, - Headers: srv.Headers, - }) - case "sse": - sseEndpoint := strings.TrimSpace(srv.SSEEndpoint) - if sseEndpoint == "" { - err := fmt.Errorf("MCP sse server %q requires an sse_endpoint", name) - log.Warn("mcp server skipped", "name", name, "err", err) - if firstErr == nil { - firstErr = err - } - continue - } - transport = mcp.NewSSETransport(mcp.SSETransportConfig{ - SSEEndpoint: sseEndpoint, - MessageEndpoint: strings.TrimSpace(srv.MessageEndpoint), - Headers: srv.Headers, - }) - default: - log.Warn("mcp server skipped: unknown transport", "name", name, "transport", transportType) - continue - } - - provider := mcptoolprovider.New(name, transport) - if err := provider.Start(ctx); err != nil { - log.Warn("mcp provider start failed", "name", name, "err", err) - if firstErr == nil { - firstErr = err - } - continue - } - - entries, err := provider.Discover(ctx) - if err != nil { - log.Warn("mcp tool discovery failed", "name", name, "err", err) - // Stop the provider since we can't use it. - _ = provider.Stop(ctx) - if firstErr == nil { - firstErr = err - } - continue - } - - for _, entry := range entries { - if err := reg.Register(entry); err != nil { - log.Warn("mcp tool registration failed", "tool", entry.Name, "err", err) - } - } - providers = append(providers, provider) - log.Info("mcp provider started", "name", name, "transport", transportType, "tools", len(entries)) - } - - if len(providers) == 0 { - return nil, firstErr - } - return &mcpProviderSet{providers: providers, registry: reg}, nil -} - -func (s *mcpProviderSet) cleanup(ctx context.Context, log *slog.Logger) { - for _, p := range s.providers { - if err := p.Stop(ctx); err != nil { - log.Warn("mcp provider stop failed", "err", err) - } - } -} - -func handle(ctx context.Context, msg eventbus.Message, bus stageBus, log *slog.Logger) error { - var req agentexec.AgentRequestMessage - if err := json.Unmarshal(msg.Data(), &req); err != nil { - return fmt.Errorf("decode request: %w", err) - } - - replyTo, err := msg.ReplyAddress() - if err != nil { - return fmt.Errorf("stage request on %s: %w", msg.Subject(), err) - } - - log.Info("processing stage", - "task", req.TaskID, - "attempt", req.Attempt, - "stage", req.Stage, - "workflow", req.Workflow, - "reply_to", replyTo, - ) - - // Start MCP providers and build a local tool registry. - mcpSet, mcpErr := startMCPProviders(ctx, req.MCPServers, log) - if mcpSet != nil { - defer mcpSet.cleanup(ctx, log) - } - if mcpErr != nil { - log.Warn("mcp providers had errors, continuing with available tools", "err", mcpErr) - } - - // Build a runner factory that includes MCP-discovered tools. - factory := func(providers map[string]agentexec.Provider, log *slog.Logger) agentexec.Runner { - var registries []*tools.Registry - if mcpSet != nil { - registries = append(registries, mcpSet.registry) - } - return agentexec.NewInProcessRunner(agentexec.NewRuntime(providers), log, registries...) - } - - resp, err := agentexec.HandleMessage(ctx, req, log, factory) - if err != nil { - return fmt.Errorf("handle: %w", err) - } - - data, err := json.Marshal(resp) - if err != nil { - return fmt.Errorf("encode response: %w", err) - } - if err := bus.Respond(ctx, replyTo, data); err != nil { - return fmt.Errorf("publish response: %w", err) - } - - log.Info("stage complete", - "task", req.TaskID, - "stage", req.Stage, - "status", resp.Result.Status, - ) - return nil -} diff --git a/cmd/archie-agent/main_test.go b/cmd/archie-agent/main_test.go index 398824d50..ed60d8fbc 100644 --- a/cmd/archie-agent/main_test.go +++ b/cmd/archie-agent/main_test.go @@ -1,73 +1,107 @@ package main import ( - "encoding/json" + "bytes" + "context" + "errors" "log/slog" - "os" - "path/filepath" + "strings" "testing" - "github.com/samcharles93/archie-core/internal/container" + "github.com/samcharles93/archie-core/internal/app/agentworker" ) -// gitDir returns a temp mount dir with the .git directory the daemon writes -// the boot brief into. -func gitDir(t *testing.T) string { - t.Helper() - dir := t.TempDir() - if err := os.MkdirAll(filepath.Join(dir, ".git"), 0o700); err != nil { - t.Fatal(err) +func TestRunCommandPreservesCLIExitSemantics(t *testing.T) { + tests := []struct { + name string + args []string + getenv func(string) string + worker workerRunner + wantCode int + }{ + {name: "help", args: []string{"-h"}}, + {name: "flag parse error", args: []string{"-unknown"}, wantCode: 2}, + {name: "missing NATS URL", wantCode: 1}, + { + name: "worker failure", + args: []string{"-nats-url", "nats://test"}, + worker: func(context.Context, agentworker.Settings, *slog.Logger) error { + return errors.New("worker failed") + }, + wantCode: 1, + }, + { + name: "success", + args: []string{"-nats-url", "nats://test"}, + worker: func(context.Context, agentworker.Settings, *slog.Logger) error { + return nil + }, + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + getenv := test.getenv + if getenv == nil { + getenv = func(string) string { return "" } + } + worker := test.worker + if worker == nil { + worker = func(context.Context, agentworker.Settings, *slog.Logger) error { + t.Fatal("worker called") + return nil + } + } + if got := runCommand(test.args, getenv, &bytes.Buffer{}, worker); got != test.wantCode { + t.Fatalf("runCommand() = %d, want %d", got, test.wantCode) + } + }) } - return dir } -func TestBootTaskID(t *testing.T) { - log := slog.New(slog.DiscardHandler) - - t.Run("no task.json -- shared pool mode", func(t *testing.T) { - id, ok := bootTaskID(t.TempDir(), log) - if ok || id != 0 { - t.Fatalf("bootTaskID() = (%d, %v), want (0, false)", id, ok) - } +func TestRunCommandRequiresNATSURL(t *testing.T) { + var stderr bytes.Buffer + called := false + code := runCommand(nil, func(string) string { return "" }, &stderr, func(context.Context, agentworker.Settings, *slog.Logger) error { + called = true + return nil }) + if code != 1 || called { + t.Fatalf("(code, worker called) = (%d, %v), want (1, false)", code, called) + } + for _, want := range []string{"error: -nats-url or NATS_URL is required", "Usage of archie-agent:"} { + if !strings.Contains(stderr.String(), want) { + t.Fatalf("stderr %q does not contain %q", stderr.String(), want) + } + } +} - t.Run("valid task.json -- dedicated per-task mode", func(t *testing.T) { - dir := gitDir(t) - data, err := json.Marshal(container.TaskPayload{ID: 42, Owner: "acme", Repo: "widget"}) - if err != nil { - t.Fatal(err) - } - if err := os.WriteFile(filepath.Join(dir, ".git", "task.json"), data, 0o644); err != nil { - t.Fatal(err) - } - id, ok := bootTaskID(dir, log) - if !ok || id != 42 { - t.Fatalf("bootTaskID() = (%d, %v), want (42, true)", id, ok) - } - }) - - t.Run("malformed task.json falls back to shared pool mode", func(t *testing.T) { - dir := gitDir(t) - if err := os.WriteFile(filepath.Join(dir, ".git", "task.json"), []byte("not json"), 0o644); err != nil { - t.Fatal(err) - } - id, ok := bootTaskID(dir, log) - if ok || id != 0 { - t.Fatalf("bootTaskID() = (%d, %v), want (0, false)", id, ok) - } - }) - - t.Run("zero ID falls back to shared pool mode", func(t *testing.T) { - dir := gitDir(t) - data, _ := json.Marshal(container.TaskPayload{Owner: "acme", Repo: "widget"}) - if err := os.WriteFile(filepath.Join(dir, ".git", "task.json"), data, 0o644); err != nil { - t.Fatal(err) - } - id, ok := bootTaskID(dir, log) - if ok || id != 0 { - t.Fatalf("bootTaskID() = (%d, %v), want (0, false)", id, ok) - } - }) +func TestRunCommandMapsWorkerOutcomeToExitStatus(t *testing.T) { + workerErr := errors.New("startup failed") + tests := []struct { + name string + workerErr error + wantCode int + }{ + {name: "success"}, + {name: "worker failure", workerErr: workerErr, wantCode: 1}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + var stderr bytes.Buffer + code := runCommand([]string{"-nats-url", "nats://flag", "-consumer", "custom"}, func(string) string { return "token" }, &stderr, func(_ context.Context, settings agentworker.Settings, _ *slog.Logger) error { + if settings.NATSURL != "nats://flag" || settings.NATSToken != "token" || settings.Consumer != "custom" { + t.Fatalf("settings = %#v", settings) + } + return test.workerErr + }) + if code != test.wantCode { + t.Fatalf("code = %d, want %d", code, test.wantCode) + } + if test.workerErr != nil && stderr.Len() != 0 { + t.Fatalf("stderr = %q, want command not to duplicate application logging", stderr.String()) + } + }) + } } func TestNATSConnectionSettings(t *testing.T) { diff --git a/docs/architecture/agent-system.md b/docs/architecture/agent-system.md index 5b8b1f472..6c0890f21 100644 --- a/docs/architecture/agent-system.md +++ b/docs/architecture/agent-system.md @@ -218,11 +218,11 @@ each meaning deliberately; a global text replacement is prohibited. | Task status and transition history | `store.Status*`, `transitions` | WorkflowExecution lifecycle and domain events | Enforce legal transitions atomically. Domain events and state changes cannot diverge. Remove duplicated status strings from consumers. | | Claimed or queued task | `ClaimNext`, `ClaimByIssue`, `RecoverStale` | Scheduled WorkflowExecution | Claim by WorkflowExecution ID and expected version/state. Crash recovery and leases must not silently create a new execution. | | `workflow.TaskContext` | `internal/workflow` | WorkflowExecution context | Replace the mutable service bag with execution state plus narrow step-required contracts. Complete application configuration and unrelated services are forbidden. | -| `taskrun.Request` and `taskrun.Response` | `internal/taskrun`, `cmd/archie-agent` | WorkflowExecution handoff | This is the versioned worker contract for executing a WorkflowExecution. It carries domain-owned execution data, not store rows or global configuration DTOs. | +| `taskrun.Request` and `taskrun.Response` | `internal/taskrun`; consumed by `internal/app/agentworker`, decoded by `internal/infrastructure/agenttransport/nats` | WorkflowExecution handoff | This is the versioned worker contract for executing a WorkflowExecution. It carries domain-owned execution data, not store rows or global configuration DTOs. | | Per-stage `agentexec.Request.TaskID` | `internal/agentexec` | WorkflowExecutionID correlation | An agent invocation is a bounded WorkflowStep execution. Correlate request/result with WorkflowExecutionID, StepExecutionID, attempt, and protocol version. | | NATS `TaskMessage` and `archie.task.*` | `internal/nats` | Channel request or WorkflowExecution transport | Discovery messages and execution dispatch are different contracts and subjects. Domain meaning stays outside NATS infrastructure. Identity and source-message idempotency must be explicit. | | `taskDispatcher` work item | `internal/daemon` | WorkflowExecution scheduling | Scheduling operates on execution references and declared concurrency policy. Repository serialization is policy, not identity derived from an incomplete task struct. | -| `container.TaskPayload` and `task.json` | `internal/container`, `cmd/archie-agent` | WorkflowExecution brief | Rename the boot artifact and schema. It identifies the mounted execution and required workspace only; it is not an alternative execution definition. Malformed input must fail safely rather than silently joining an unrelated shared queue. | +| `container.TaskPayload` and `task.json` | `internal/container`; read by `internal/infrastructure/agentboot` | WorkflowExecution brief | Rename the boot artifact and schema. It identifies the mounted execution and required workspace only; it is not an alternative execution definition. Malformed input must fail safely rather than silently joining an unrelated shared queue. | | `storage.TaskRef` | `internal/storage` | WorkflowExecution workspace lease | Storage receives the minimal execution/workspace reference and policy required to provision and release resources. It does not receive the WorkflowExecution aggregate. | | Gateway `TaskCreator` | `internal/gateway` | Accepted work request handoff | A channel may carry a request for durable workflow-backed work, but it does not request or create a WorkflowExecution directly. Work intake admits and routes the request; the Agent System consumes the accepted work request and is the sole owner of selecting a Workflow version and creating the WorkflowExecution. Ordinary agent interactions do not require this handoff. | | Gateway `TaskController` and `ChatTaskStatus` | `internal/gateway/tasks.go` | WorkflowExecution commands and query view | Approve, cancel, and status use Agent System contracts. Channel code does not duplicate lifecycle constants or decide transitions. | diff --git a/docs/architecture/observability.md b/docs/architecture/observability.md index 1131fec94..171975e59 100644 --- a/docs/architecture/observability.md +++ b/docs/architecture/observability.md @@ -3,21 +3,23 @@ **Status:** Implemented **Date:** 2026-08-09 -## Problem +## Historical problem -archied's own daemon logging (`internal/logging/`) is solid: a rotating file -sink, a live `Feed` the dashboard subscribes to, and a reader the `/api/logs` -endpoint serves from. None of that reaches inside a task. +archied's own daemon logging (`internal/logging/`) was already solid: a +rotating file sink, a live `Feed` the dashboard subscribes to, and a reader +the `/api/logs` endpoint serves from. Before this implementation, none of +that reached inside a task. A task's actual work happens in an `archie-agent` container, dispatched over NATS (`[agent] mode = "nats"` is the only supported mode; the in-process runner exists for tests and is being migrated away — see [Decision: no new functionality on non-NATS transports](#decision-no-new-functionality-on-non-nats-transports)). -Four things currently destroy the evidence of what that container did: +Before this design was implemented, five things destroyed the evidence of what +that container did: -1. **`cmd/archie-agent/main.go`** logs to `os.Stderr` only - (`log := slog.New(slog.NewJSONHandler(os.Stderr, nil))`). Nothing captures - it outside the container. +1. **Worker startup** constructed an stderr-only logger and never replaced its + handler after discovering a dedicated boot task, so runtime records could + not leave the container. 2. **`internal/container/pool.go`** creates every task container with `AutoRemove: true`. The container — and its stderr — is gone at exit, success or failure alike. @@ -42,9 +44,9 @@ auto-fix (status: parked)` and there is no way — not from the dashboard, not from Telegram chat, not by SSH'ing into the deployment host — to see *why* `go build` failed. The container that knew is already gone. -This matters beyond debugging convenience: the deployment model -(`CLAUDE.md` "Deployment model") is a systemd daemon plus ephemeral Docker -containers that may run unattended for weeks between operator logins, and is +This matters beyond debugging convenience: supported deployments can combine +a resident daemon with ephemeral Docker containers that run unattended for +weeks between operator logins, and the system is intended to be usable by people other than the original operator. A system that is only debuggable by an operator who happens to be watching `docker ps` in real time does not meet that bar. @@ -62,7 +64,7 @@ failure (connectivity/config) from the one this document addresses (a connected container's own output being unrecoverable), and is out of scope here. -## Design +## Implemented design The key property: **logs leave the container while it is alive**, so its death at `AutoRemove` no longer matters. Persistence happens in the daemon @@ -79,25 +81,26 @@ task-scoped writer producing the same `logging.Entry` JSONL shape at ### Agent side: make the designed subject real -`cmd/archie-agent/main.go` replaces the stderr-only `slog.Handler` with one +`internal/app/agentworker` replaces the stderr-only `slog.Handler` with one that tees to stderr (kept for container-level operator debugging when a shell is attached) and publishes each record to -`agentexec.SubjectForSystem(taskID)`, which already exists for this purpose -and already has zero consumers to conflict with. The agent already holds its -NATS client and task ID (`bootTaskID`, read from -`/.git/task.json` per the container/pool.go trap documented in -`CLAUDE.md`). Publishing is fire-and-forget with a bounded local buffer: log -shipping must never block or fail the run it is reporting on. +`agentexec.SubjectForSystem(taskID)`, which already exists for this purpose. +The application layer obtains only a narrow publisher from +`internal/infrastructure/agenttransport/nats`; raw core-NATS types remain in +infrastructure. `internal/infrastructure/agentboot.TaskID` reads the task ID +from `/.git/task.json` per the container/pool.go trap documented in +`CLAUDE.md`. Publishing is fire-and-forget: log shipping must never block or +fail the run it is reporting on. ### Daemon side -Subscribe `agentexec.SubjectAgentWildcard` ("archie.agent.>") for the -`.system` suffix once at startup (app-layer wiring — this composes the -domain's log sink with the infrastructure NATS subscription, so it lives in -`internal/app/`, not in `internal/domain/workflow` or `internal/nats` -directly). Demux by task ID, append to that task's sink, and also publish -into the existing `logging.Feed` so live task output appears on the -dashboard exactly like daemon output does today. +`cmd/archied/main.go` currently registers `subscribeSystemLogs` once at +startup for the `.system` suffix of `agentexec.SubjectAgentWildcard` +(`"archie.agent.>"`). It demultiplexes by task ID, appends to that task's +sink, and publishes into the existing `logging.Feed` so live task output +appears on the dashboard exactly like daemon output. The approved target is +for this composition to move to `internal/app/archied`; that package does not +yet exist, so this document does not claim the migration is complete. ### Gate output: stop discarding it @@ -135,16 +138,14 @@ lifecycle rather than accumulating independently. `internal/logging` remains cross-cutting per `organisation.md` — it imports no domain, infrastructure, or app package, so extending it to be task-scoped -does not change its classification. The workflow domain writes through an -interface it declares (a `TaskLogSink` or equivalent, not a direct -`internal/logging` import into `internal/domain/workflow`, unless -`internal/logging` is judged to already be an acceptable cross-cutting -dependency for a domain package — confirm against `organisation.md`'s -concrete rule before writing the interface). `internal/app` wires the NATS -subscription to the sink; `internal/agentexec` and `internal/nats` supply -the transport underneath. +does not change its classification. The workflow domain writes through the +cross-cutting logging contract. On the agent side, +`internal/app/agentworker` composes the narrow publisher supplied by +`internal/infrastructure/agenttransport/nats`. On the daemon side, +`cmd/archied/main.go` currently composes the NATS subscription with the task +sink until the approved `internal/app/archied` migration is implemented. -## Sequencing +## Implementation sequence (completed) 1. Gate-output capture (~30 lines, no NATS changes) — fixes the immediate symptom that prompted this document. diff --git a/internal/agentexec/stage_message.go b/internal/agentexec/stage_message.go new file mode 100644 index 000000000..b80552de8 --- /dev/null +++ b/internal/agentexec/stage_message.go @@ -0,0 +1,19 @@ +package agentexec + +import "context" + +// StageMessage is one decoded single-stage request together with the transport +// operations that complete its delivery. Infrastructure owns the wire payload, +// reply inbox, and acknowledgement mechanism; the worker only sees typed data. +type StageMessage interface { + Request() (AgentRequestMessage, error) + Respond(context.Context, *AgentResponseEnvelope) error + Ack() error + Nak() error + LogAttributes() []any +} + +// StageConsumer receives decoded single-stage requests. +type StageConsumer interface { + FetchStage(context.Context) (StageMessage, error) +} diff --git a/internal/agentexec/systemlog_test.go b/internal/agentexec/systemlog_test.go index 2622b48ee..dd6ced53a 100644 --- a/internal/agentexec/systemlog_test.go +++ b/internal/agentexec/systemlog_test.go @@ -37,7 +37,7 @@ func (f *fakePublisher) calls() int { } // capturingHandler is a minimal slog.Handler that records every record it -// receives, standing in for cmd/archie-agent's real stderr handler. +// receives, standing in for the agentworker's wrapped process stderr handler. type capturingHandler struct { mu sync.Mutex records []slog.Record diff --git a/internal/app/agentworker/architecture_test.go b/internal/app/agentworker/architecture_test.go new file mode 100644 index 000000000..fb6a091d7 --- /dev/null +++ b/internal/app/agentworker/architecture_test.go @@ -0,0 +1,51 @@ +package agentworker + +import ( + "go/ast" + "go/parser" + "go/token" + "path/filepath" + "strconv" + "strings" + "testing" +) + +func TestApplicationOwnsNoTransportWireMechanics(t *testing.T) { + files, err := filepath.Glob("*.go") + if err != nil { + t.Fatal(err) + } + for _, path := range files { + if strings.HasSuffix(path, "_test.go") { + continue + } + file, err := parser.ParseFile(token.NewFileSet(), path, nil, 0) + if err != nil { + t.Fatal(err) + } + for _, imported := range file.Imports { + name, err := strconv.Unquote(imported.Path.Value) + if err != nil { + t.Fatal(err) + } + switch { + case name == "encoding/json": + t.Errorf("%s imports JSON wire encoding", path) + case name == "github.com/samcharles93/archie-core/internal/domain/workintake": + t.Errorf("%s imports NATS topology subjects", path) + case name == "github.com/nats-io/nats.go" || strings.HasPrefix(name, "github.com/nats-io/nats.go/"): + t.Errorf("%s imports raw NATS SDK package %q", path, name) + } + } + ast.Inspect(file, func(node ast.Node) bool { + selector, ok := node.(*ast.SelectorExpr) + if !ok { + return true + } + if selector.Sel.Name == "ReplyAddress" || selector.Sel.Name == "Subject" || selector.Sel.Name == "Data" { + t.Errorf("%s inspects transport wire selector %s", path, selector.Sel.Name) + } + return true + }) + } +} diff --git a/internal/app/agentworker/mcp_providers.go b/internal/app/agentworker/mcp_providers.go new file mode 100644 index 000000000..9c8b8670e --- /dev/null +++ b/internal/app/agentworker/mcp_providers.go @@ -0,0 +1,173 @@ +package agentworker + +import ( + "context" + "errors" + "fmt" + "log/slog" + "strings" + + "github.com/samcharles93/archie-core/internal/config" + "github.com/samcharles93/archie-core/internal/tools" + "github.com/samcharles93/archie-core/internal/tools/mcp" + toolprovider "github.com/samcharles93/archie-core/internal/tools/provider" + mcptoolprovider "github.com/samcharles93/archie-core/internal/tools/provider/mcp" +) + +var errUnknownMCPTransport = errors.New("unknown MCP transport") + +// mcpProviderSet holds the resources needed to clean up MCP providers. +type mcpProviderSet struct { + providers []toolprovider.Engine + registry *tools.Registry +} + +type mcpProviderBuilder func(config.MCPServer) (toolprovider.Engine, string, error) + +// startMCPProviders creates MCP transports from config, starts them, +// discovers tools, and registers them into a local registry. Returns +// nil when there are no servers configured. +func startMCPProviders(ctx context.Context, servers []config.MCPServer, log *slog.Logger) (*mcpProviderSet, error) { + return startMCPProvidersWith(ctx, servers, log, buildMCPProvider) +} + +func startMCPProvidersWith( + ctx context.Context, + servers []config.MCPServer, + log *slog.Logger, + build mcpProviderBuilder, +) (*mcpProviderSet, error) { + if len(servers) == 0 { + return nil, nil + } + + set := &mcpProviderSet{registry: tools.NewRegistry()} + var firstErr error + for _, server := range servers { + name := strings.TrimSpace(server.Name) + if name == "" { + log.Warn("mcp server skipped: empty name") + continue + } + + provider, transportType, err := build(server) + if errors.Is(err, errUnknownMCPTransport) { + log.Warn("mcp server skipped: unknown transport", "name", name, "transport", transportType) + continue + } + if err != nil { + log.Warn("mcp server skipped", "name", name, "err", err) + firstErr = retainFirstError(firstErr, err) + continue + } + + toolCount, err := startMCPProvider(ctx, name, provider, set.registry, log) + if err != nil { + firstErr = retainFirstError(firstErr, err) + continue + } + + set.providers = append(set.providers, provider) + log.Info("mcp provider started", "name", name, "transport", transportType, "tools", toolCount) + } + + if len(set.providers) == 0 { + return nil, firstErr + } + return set, nil +} + +func buildMCPProvider(server config.MCPServer) (toolprovider.Engine, string, error) { + name := strings.TrimSpace(server.Name) + transportType := strings.ToLower(strings.TrimSpace(server.Transport)) + if transportType == "" { + transportType = "stdio" + } + + transport, err := mcpTransportForServer(name, transportType, server) + if err != nil { + return nil, transportType, err + } + return mcptoolprovider.New(name, transport), transportType, nil +} + +func mcpTransportForServer( + name string, + transportType string, + server config.MCPServer, +) (mcptoolprovider.LifecycleTransport, error) { + switch transportType { + case "stdio": + command := strings.TrimSpace(server.Command) + if command == "" { + return nil, fmt.Errorf("MCP stdio server %q requires a command", name) + } + return mcp.NewStdioTransport(mcp.StdioTransportConfig{ + Command: command, + Args: append([]string(nil), server.Args...), + }), nil + case "http", "streamablehttp": + url := strings.TrimSpace(server.URL) + if url == "" { + return nil, fmt.Errorf("MCP http server %q requires a url", name) + } + return mcp.NewHTTPTransport(mcp.HTTPTransportConfig{ + Endpoint: url, + Headers: server.Headers, + }), nil + case "sse": + sseEndpoint := strings.TrimSpace(server.SSEEndpoint) + if sseEndpoint == "" { + return nil, fmt.Errorf("MCP sse server %q requires an sse_endpoint", name) + } + return mcp.NewSSETransport(mcp.SSETransportConfig{ + SSEEndpoint: sseEndpoint, + MessageEndpoint: strings.TrimSpace(server.MessageEndpoint), + Headers: server.Headers, + }), nil + default: + return nil, fmt.Errorf("%w: %s", errUnknownMCPTransport, transportType) + } +} + +func startMCPProvider( + ctx context.Context, + name string, + provider toolprovider.Engine, + registry *tools.Registry, + log *slog.Logger, +) (int, error) { + if err := provider.Start(ctx); err != nil { + log.Warn("mcp provider start failed", "name", name, "err", err) + return 0, err + } + + entries, err := provider.Discover(ctx) + if err != nil { + log.Warn("mcp tool discovery failed", "name", name, "err", err) + _ = provider.Stop(ctx) + return 0, err + } + + for _, entry := range entries { + if err := registry.Register(entry); err != nil { + log.Warn("mcp tool registration failed", "tool", entry.Name, "err", err) + } + } + return len(entries), nil +} + +func retainFirstError(firstErr, err error) error { + if firstErr != nil { + return firstErr + } + return err +} + +func (s *mcpProviderSet) cleanup(ctx context.Context, log *slog.Logger) { + for _, provider := range s.providers { + if err := provider.Stop(ctx); err != nil { + log.Warn("mcp provider stop failed", "err", err) + } + } +} diff --git a/internal/app/agentworker/mcp_providers_test.go b/internal/app/agentworker/mcp_providers_test.go new file mode 100644 index 000000000..e76c7aaf4 --- /dev/null +++ b/internal/app/agentworker/mcp_providers_test.go @@ -0,0 +1,414 @@ +package agentworker + +import ( + "context" + "errors" + "fmt" + "log/slog" + "reflect" + "testing" + + "github.com/samcharles93/archie-core/internal/config" + "github.com/samcharles93/archie-core/internal/plugin" + "github.com/samcharles93/archie-core/internal/tools" + protocolmcp "github.com/samcharles93/archie-core/internal/tools/mcp" + toolprovider "github.com/samcharles93/archie-core/internal/tools/provider" +) + +type fakeMCPEngine struct { + id string + entries []tools.ToolEntry + startErr error + discoverErr error + stopErr error + starts int + discovers int + stops int +} + +func (f *fakeMCPEngine) Manifest() plugin.Manifest { + return plugin.Manifest{ + ID: "mcp." + f.id, + Name: f.id, + Version: "1.0.0", + APIVersion: plugin.HostAPIVersion, + Capabilities: []plugin.CapabilityKind{"tools"}, + } +} + +func (f *fakeMCPEngine) Start(context.Context) error { + f.starts++ + return f.startErr +} + +func (f *fakeMCPEngine) Discover(context.Context) ([]tools.ToolEntry, error) { + f.discovers++ + return f.entries, f.discoverErr +} + +func (f *fakeMCPEngine) Health(context.Context) plugin.Health { + return plugin.Health{Status: plugin.HealthHealthy} +} + +func (f *fakeMCPEngine) Stop(context.Context) error { + f.stops++ + return f.stopErr +} + +func testTool(name string) tools.ToolEntry { + return tools.ToolEntry{ + Name: name, + Handler: func(context.Context, map[string]any) (any, error) { + return nil, nil + }, + } +} + +func TestBuildMCPProviderSelectsTransport(t *testing.T) { + tests := []struct { + name string + server config.MCPServer + wantTransport string + wantType any + wantCommand string + wantArgs []string + wantEndpoint string + wantMessageEndpoint string + wantHeaders map[string]string + wantErr string + wantUnknown bool + }{ + { + name: "default stdio trims command", + server: config.MCPServer{Name: " local ", Command: " server ", Args: []string{"--flag"}}, + wantTransport: "stdio", + wantType: (*protocolmcp.StdioTransport)(nil), + wantCommand: "server", + wantArgs: []string{"--flag"}, + }, + { + name: "explicit stdio", + server: config.MCPServer{Name: "local", Transport: " STDIO ", Command: "server"}, + wantTransport: "stdio", + wantType: (*protocolmcp.StdioTransport)(nil), + wantCommand: "server", + }, + { + name: "http", + server: config.MCPServer{Name: "remote", Transport: "http", URL: " https://example.test/mcp ", Headers: map[string]string{"Authorization": "Bearer test"}}, + wantTransport: "http", + wantType: (*protocolmcp.HTTPTransport)(nil), + wantEndpoint: "https://example.test/mcp", + wantHeaders: map[string]string{"Authorization": "Bearer test"}, + }, + { + name: "streamable http alias", + server: config.MCPServer{Name: "remote", Transport: "StreamableHTTP", URL: "https://example.test/mcp"}, + wantTransport: "streamablehttp", + wantType: (*protocolmcp.HTTPTransport)(nil), + wantEndpoint: "https://example.test/mcp", + }, + { + name: "sse", + server: config.MCPServer{Name: "events", Transport: "sse", SSEEndpoint: " https://example.test/events ", MessageEndpoint: " https://example.test/messages ", Headers: map[string]string{"X-Test": "value"}}, + wantTransport: "sse", + wantType: (*protocolmcp.SSETransport)(nil), + wantEndpoint: "https://example.test/events", + wantMessageEndpoint: "https://example.test/messages", + wantHeaders: map[string]string{"X-Test": "value"}, + }, + { + name: "stdio requires command", + server: config.MCPServer{Name: "local", Transport: "stdio"}, + wantTransport: "stdio", + wantErr: `MCP stdio server "local" requires a command`, + }, + { + name: "http requires url", + server: config.MCPServer{Name: "remote", Transport: "http"}, + wantTransport: "http", + wantErr: `MCP http server "remote" requires a url`, + }, + { + name: "sse requires endpoint", + server: config.MCPServer{Name: "events", Transport: "sse"}, + wantTransport: "sse", + wantErr: `MCP sse server "events" requires an sse_endpoint`, + }, + { + name: "unknown transport is a nonfatal skip", + server: config.MCPServer{Name: "other", Transport: "websocket"}, + wantTransport: "websocket", + wantErr: "unknown MCP transport: websocket", + wantUnknown: true, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + provider, transportType, err := buildMCPProvider(tc.server) + if transportType != tc.wantTransport { + t.Fatalf("transport = %q, want %q", transportType, tc.wantTransport) + } + if tc.wantErr != "" { + if err == nil || err.Error() != tc.wantErr { + t.Fatalf("error = %v, want %q", err, tc.wantErr) + } + if errors.Is(err, errUnknownMCPTransport) != tc.wantUnknown { + t.Fatalf("errors.Is(err, errUnknownMCPTransport) = %v, want %v", errors.Is(err, errUnknownMCPTransport), tc.wantUnknown) + } + if provider != nil { + t.Fatalf("provider = %T, want nil on invalid configuration", provider) + } + return + } + if err != nil { + t.Fatalf("buildMCPProvider() error = %v", err) + } + if provider == nil { + t.Fatal("buildMCPProvider() returned nil provider") + } + transport := reflect.ValueOf(provider).Elem().FieldByName("transport") + if transport.IsNil() { + t.Fatal("provider transport is nil") + } + if got, want := transport.Elem().Type(), reflect.TypeOf(tc.wantType); got != want { + t.Fatalf("transport type = %v, want %v", got, want) + } + assertMCPTransportConfig(t, transport.Elem(), tc.wantCommand, tc.wantArgs, tc.wantEndpoint, tc.wantMessageEndpoint, tc.wantHeaders) + }) + } +} + +func assertMCPTransportConfig( + t *testing.T, + transport reflect.Value, + wantCommand string, + wantArgs []string, + wantEndpoint string, + wantMessageEndpoint string, + wantHeaders map[string]string, +) { + t.Helper() + config := transport.Elem().FieldByName("config") + assertStringField := func(name, want string) { + t.Helper() + if want == "" { + return + } + if got := config.FieldByName(name).String(); got != want { + t.Errorf("transport config %s = %q, want %q", name, got, want) + } + } + assertStringField("Command", wantCommand) + if wantEndpoint != "" { + field := "Endpoint" + if config.FieldByName(field).Kind() == reflect.Invalid { + field = "SSEEndpoint" + } + assertStringField(field, wantEndpoint) + } + assertStringField("MessageEndpoint", wantMessageEndpoint) + + if wantArgs != nil { + args := config.FieldByName("Args") + got := make([]string, args.Len()) + for i := range args.Len() { + got[i] = args.Index(i).String() + } + if !reflect.DeepEqual(got, wantArgs) { + t.Errorf("transport config Args = %v, want %v", got, wantArgs) + } + } + if wantHeaders != nil { + headers := config.FieldByName("Headers") + for key, want := range wantHeaders { + if got := headers.MapIndex(reflect.ValueOf(key)).String(); got != want { + t.Errorf("transport config Headers[%q] = %q, want %q", key, got, want) + } + } + } +} + +func TestStartMCPProviderLifecycle(t *testing.T) { + startErr := errors.New("start failed") + discoverErr := errors.New("discover failed") + log := slog.New(slog.DiscardHandler) + + tests := []struct { + name string + provider *fakeMCPEngine + wantErr error + wantTools int + wantStarts int + wantDiscovers int + wantStops int + wantRegistered []string + }{ + { + name: "start failure does not attempt discovery or stop", + provider: &fakeMCPEngine{id: "start", startErr: startErr}, + wantErr: startErr, + wantStarts: 1, + }, + { + name: "discovery failure stops started provider", + provider: &fakeMCPEngine{id: "discover", discoverErr: discoverErr, stopErr: errors.New("ignored cleanup failure")}, + wantErr: discoverErr, + wantStarts: 1, + wantDiscovers: 1, + wantStops: 1, + }, + { + name: "registration failure is nonfatal", + provider: &fakeMCPEngine{id: "registration", entries: []tools.ToolEntry{ + testTool("available"), + {Name: "invalid"}, + testTool("available"), + }}, + wantTools: 3, + wantStarts: 1, + wantDiscovers: 1, + wantRegistered: []string{"available"}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + registry := tools.NewRegistry() + gotTools, err := startMCPProvider(t.Context(), tc.name, tc.provider, registry, log) + if !errors.Is(err, tc.wantErr) { + t.Fatalf("error = %v, want %v", err, tc.wantErr) + } + if gotTools != tc.wantTools { + t.Fatalf("tool count = %d, want %d", gotTools, tc.wantTools) + } + if tc.provider.starts != tc.wantStarts || tc.provider.discovers != tc.wantDiscovers || tc.provider.stops != tc.wantStops { + t.Fatalf("lifecycle counts = start:%d discover:%d stop:%d, want start:%d discover:%d stop:%d", tc.provider.starts, tc.provider.discovers, tc.provider.stops, tc.wantStarts, tc.wantDiscovers, tc.wantStops) + } + for _, name := range tc.wantRegistered { + if _, ok := registry.Get(name); !ok { + t.Errorf("tool %q was not registered", name) + } + } + }) + } +} + +func TestStartMCPProvidersPreservesOptionalProviderBehavior(t *testing.T) { + configErr := errors.New("invalid config") + startErr := errors.New("start failed") + discoverErr := errors.New("discover failed") + stopErr := errors.New("stop failed") + providers := map[string]*fakeMCPEngine{ + "start-failure": {id: "start-failure", startErr: startErr}, + "discover-failure": {id: "discover-failure", discoverErr: discoverErr}, + "valid-one": {id: "valid-one", entries: []tools.ToolEntry{testTool("shared")}, stopErr: stopErr}, + "valid-two": {id: "valid-two", entries: []tools.ToolEntry{testTool("shared")}}, + } + buildCalls := make(map[string]int) + build := func(server config.MCPServer) (toolprovider.Engine, string, error) { + buildCalls[server.Name]++ + switch server.Name { + case "unknown": + return nil, "websocket", fmt.Errorf("%w: websocket", errUnknownMCPTransport) + case "invalid": + return nil, "stdio", configErr + default: + return providers[server.Name], "stdio", nil + } + } + + servers := []config.MCPServer{ + {Name: " "}, + {Name: "unknown"}, + {Name: "invalid"}, + {Name: "start-failure"}, + {Name: "discover-failure"}, + {Name: "valid-one"}, + {Name: "valid-two"}, + } + set, err := startMCPProvidersWith(t.Context(), servers, slog.New(slog.DiscardHandler), build) + if err != nil { + t.Fatalf("error = %v, want nil when valid providers remain available", err) + } + if set == nil { + t.Fatal("provider set = nil, want valid providers to remain available") + } + if got := len(set.providers); got != 2 { + t.Fatalf("available providers = %d, want 2", got) + } + if _, ok := set.registry.Get("shared"); !ok { + t.Fatal("tool from valid provider was not available") + } + if buildCalls[" "] != 0 { + t.Fatal("empty-name config reached provider builder") + } + if providers["start-failure"].stops != 0 { + t.Fatal("start failure unexpectedly stopped provider") + } + if providers["discover-failure"].stops != 1 { + t.Fatalf("discovery failure stop count = %d, want 1", providers["discover-failure"].stops) + } + + set.cleanup(t.Context(), slog.New(slog.DiscardHandler)) + if providers["valid-one"].stops != 1 || providers["valid-two"].stops != 1 { + t.Fatalf("cleanup stop counts = (%d, %d), want (1, 1)", providers["valid-one"].stops, providers["valid-two"].stops) + } +} + +func TestStartMCPProvidersReturnsNilWithoutAvailableProviders(t *testing.T) { + firstErr := errors.New("first provider failed") + secondErr := errors.New("second provider failed") + tests := []struct { + name string + servers []config.MCPServer + build mcpProviderBuilder + wantErr error + }{ + { + name: "no configured servers", + }, + { + name: "unknown transport is skipped without error", + servers: []config.MCPServer{{Name: "unknown"}}, + build: func(config.MCPServer) (toolprovider.Engine, string, error) { + return nil, "unknown", errUnknownMCPTransport + }, + }, + { + name: "first provider error is returned", + servers: []config.MCPServer{{Name: "first"}, {Name: "second"}}, + build: func(server config.MCPServer) (toolprovider.Engine, string, error) { + if server.Name == "first" { + return nil, "stdio", firstErr + } + return nil, "stdio", secondErr + }, + wantErr: firstErr, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + build := tc.build + if build == nil { + build = func(config.MCPServer) (toolprovider.Engine, string, error) { + t.Fatal("builder called without configured servers") + return nil, "", nil + } + } + set, err := startMCPProvidersWith(t.Context(), tc.servers, slog.New(slog.DiscardHandler), build) + if set != nil { + t.Fatalf("provider set = %#v, want nil", set) + } + if tc.wantErr == nil { + if err != nil { + t.Fatalf("error = %v, want nil", err) + } + } else if reflect.ValueOf(err).Pointer() != reflect.ValueOf(tc.wantErr).Pointer() { + t.Fatalf("error = %v, want exact first error %v", err, tc.wantErr) + } + }) + } +} diff --git a/internal/app/agentworker/single_stage_request.go b/internal/app/agentworker/single_stage_request.go new file mode 100644 index 000000000..010374e82 --- /dev/null +++ b/internal/app/agentworker/single_stage_request.go @@ -0,0 +1,94 @@ +package agentworker + +import ( + "context" + "fmt" + "log/slog" + + "github.com/samcharles93/archie-core/internal/agentexec" + "github.com/samcharles93/archie-core/internal/config" + "github.com/samcharles93/archie-core/internal/tools" +) + +type singleStageRequestDependencies struct { + handleMessage func(context.Context, agentexec.AgentRequestMessage, *slog.Logger, agentexec.RunnerFactory) (*agentexec.AgentResponseEnvelope, error) + startProviders func(context.Context, []config.MCPServer, *slog.Logger) (*mcpProviderSet, error) + newRunner func(map[string]agentexec.Provider, *slog.Logger, ...*tools.Registry) agentexec.Runner +} + +func productionSingleStageDependencies() singleStageRequestDependencies { + return singleStageRequestDependencies{ + handleMessage: agentexec.HandleMessage, + startProviders: startMCPProviders, + newRunner: newSingleStageRunner, + } +} + +func newSingleStageRunner( + providers map[string]agentexec.Provider, + log *slog.Logger, + registries ...*tools.Registry, +) agentexec.Runner { + return agentexec.NewInProcessRunner(agentexec.NewRuntime(providers), log, registries...) +} + +func handleSingleStageRequest( + ctx context.Context, + message agentexec.StageMessage, + log *slog.Logger, + dependencies singleStageRequestDependencies, +) error { + request, err := message.Request() + if err != nil { + return err + } + + attributes := []any{ + "task", request.TaskID, + "attempt", request.Attempt, + "stage", request.Stage, + "workflow", request.Workflow, + } + attributes = append(attributes, message.LogAttributes()...) + log.Info("processing stage", attributes...) + + startProviders := dependencies.startProviders + if startProviders == nil { + startProviders = startMCPProviders + } + newRunner := dependencies.newRunner + if newRunner == nil { + newRunner = newSingleStageRunner + } + + mcpSet, mcpErr := startProviders(ctx, request.MCPServers, log) + if mcpSet != nil { + defer mcpSet.cleanup(ctx, log) + } + if mcpErr != nil { + log.Warn("mcp providers had errors, continuing with available tools", "err", mcpErr) + } + + factory := func(providers map[string]agentexec.Provider, log *slog.Logger) agentexec.Runner { + var registries []*tools.Registry + if mcpSet != nil { + registries = append(registries, mcpSet.registry) + } + return newRunner(providers, log, registries...) + } + + response, err := dependencies.handleMessage(ctx, request, log, factory) + if err != nil { + return fmt.Errorf("handle: %w", err) + } + if err := message.Respond(ctx, response); err != nil { + return err + } + + log.Info("stage complete", + "task", request.TaskID, + "stage", request.Stage, + "status", response.Result.Status, + ) + return nil +} diff --git a/internal/app/agentworker/single_stage_request_test.go b/internal/app/agentworker/single_stage_request_test.go new file mode 100644 index 000000000..49e668f6f --- /dev/null +++ b/internal/app/agentworker/single_stage_request_test.go @@ -0,0 +1,123 @@ +package agentworker + +import ( + "bytes" + "context" + "errors" + "log/slog" + "strings" + "testing" + + "github.com/samcharles93/archie-core/internal/agentexec" + "github.com/samcharles93/archie-core/internal/config" + "github.com/samcharles93/archie-core/internal/tools" + toolprovider "github.com/samcharles93/archie-core/internal/tools/provider" +) + +func singleStageLogger() (*slog.Logger, *bytes.Buffer) { + var output bytes.Buffer + return slog.New(slog.NewTextHandler(&output, nil)), &output +} + +func TestHandleSingleStageRequestReturnsTypedBoundaryAndExecutionFailures(t *testing.T) { + requestErr := errors.New("decode request: invalid") + handleErr := errors.New("agent execution failed") + respondErr := errors.New("publish response: unavailable") + tests := []struct { + name string + message *stageMessageStub + handleErr error + wantErr error + wantHandle int + wantReply int + }{ + {name: "typed boundary", message: &stageMessageStub{requestErr: requestErr}, wantErr: requestErr}, + {name: "execution", message: &stageMessageStub{request: agentexec.AgentRequestMessage{TaskID: 17}}, handleErr: handleErr, wantErr: handleErr, wantHandle: 1}, + {name: "response", message: &stageMessageStub{request: agentexec.AgentRequestMessage{TaskID: 23}, respondErr: respondErr}, wantErr: respondErr, wantHandle: 1, wantReply: 1}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + handleCalls := 0 + dependencies := singleStageRequestDependencies{ + handleMessage: func(context.Context, agentexec.AgentRequestMessage, *slog.Logger, agentexec.RunnerFactory) (*agentexec.AgentResponseEnvelope, error) { + handleCalls++ + return &agentexec.AgentResponseEnvelope{Result: agentexec.Result{Status: agentexec.StatusPassed}}, test.handleErr + }, + } + err := handleSingleStageRequest(t.Context(), test.message, slog.New(slog.DiscardHandler), dependencies) + if !errors.Is(err, test.wantErr) { + t.Fatalf("error = %v, want %v", err, test.wantErr) + } + if handleCalls != test.wantHandle || test.message.respondCalls != test.wantReply { + t.Fatalf("handle/respond calls = (%d,%d), want (%d,%d)", handleCalls, test.message.respondCalls, test.wantHandle, test.wantReply) + } + }) + } +} + +func TestHandleSingleStageRequestAttachesAndCleansUpMCPProviders(t *testing.T) { + message := &stageMessageStub{request: agentexec.AgentRequestMessage{ + TaskID: 29, + Stage: "implement", + MCPServers: []config.MCPServer{{Name: "available"}}, + }} + registry := tools.NewRegistry() + if err := registry.Register(testTool("available")); err != nil { + t.Fatal(err) + } + provider := &fakeMCPEngine{id: "available"} + set := &mcpProviderSet{providers: []toolprovider.Engine{provider}, registry: registry} + startCalls := 0 + factoryCalls := 0 + dependencies := singleStageRequestDependencies{ + startProviders: func(_ context.Context, servers []config.MCPServer, _ *slog.Logger) (*mcpProviderSet, error) { + startCalls++ + if len(servers) != 1 || servers[0].Name != "available" { + t.Fatalf("MCP servers = %#v", servers) + } + return set, nil + }, + newRunner: func(_ map[string]agentexec.Provider, _ *slog.Logger, registries ...*tools.Registry) agentexec.Runner { + factoryCalls++ + if len(registries) != 1 || registries[0] != registry { + t.Fatalf("runner registries = %#v", registries) + } + return panicRunner{t: t} + }, + handleMessage: func(_ context.Context, request agentexec.AgentRequestMessage, _ *slog.Logger, factory agentexec.RunnerFactory) (*agentexec.AgentResponseEnvelope, error) { + if runner := factory(request.Providers, slog.New(slog.DiscardHandler)); runner == nil { + t.Fatal("runner factory returned nil") + } + return &agentexec.AgentResponseEnvelope{Result: agentexec.Result{Status: agentexec.StatusPassed}}, nil + }, + } + if err := handleSingleStageRequest(t.Context(), message, slog.New(slog.DiscardHandler), dependencies); err != nil { + t.Fatal(err) + } + if startCalls != 1 || factoryCalls != 1 || provider.stops != 1 || message.respondCalls != 1 { + t.Fatalf("start/factory/stops/respond = (%d,%d,%d,%d), want all 1", startCalls, factoryCalls, provider.stops, message.respondCalls) + } +} + +func TestHandleSingleStageRequestPreservesLifecycleLogs(t *testing.T) { + message := &stageMessageStub{request: agentexec.AgentRequestMessage{ + TaskID: 31, Attempt: 4, Stage: "test", Workflow: "tdd", + MCPServers: []config.MCPServer{{Name: "broken", Transport: "stdio"}}, + }} + dependencies := productionSingleStageDependencies() + dependencies.handleMessage = func(context.Context, agentexec.AgentRequestMessage, *slog.Logger, agentexec.RunnerFactory) (*agentexec.AgentResponseEnvelope, error) { + return &agentexec.AgentResponseEnvelope{Result: agentexec.Result{Status: agentexec.StatusPassed}}, nil + } + log, output := singleStageLogger() + if err := handleSingleStageRequest(t.Context(), message, log, dependencies); err != nil { + t.Fatal(err) + } + for _, expected := range []string{ + "processing stage", "task=31", "attempt=4", "stage=test", "workflow=tdd", "reply_to=_INBOX.test", + "mcp providers had errors, continuing with available tools", "stage complete", "status=passed", + } { + if !strings.Contains(output.String(), expected) { + t.Fatalf("log output %q does not contain %q", output.String(), expected) + } + } +} diff --git a/internal/app/agentworker/stage_loop.go b/internal/app/agentworker/stage_loop.go new file mode 100644 index 000000000..d52d17d9d --- /dev/null +++ b/internal/app/agentworker/stage_loop.go @@ -0,0 +1,52 @@ +package agentworker + +import ( + "context" + "errors" + "log/slog" + "time" + + "github.com/samcharles93/archie-core/internal/agentexec" + "github.com/samcharles93/archie-core/internal/eventbus" +) + +// stageBus is the typed single-stage delivery capability required by the loop. +type stageBus interface { + agentexec.StageConsumer +} + +// runMainLoop runs the fetch-handle-ack loop until the context is cancelled. +func runMainLoop(ctx context.Context, bus stageBus, log *slog.Logger) int { + return runMainLoopWithDelay(ctx, bus, log, time.Sleep) +} + +func runMainLoopWithDelay(ctx context.Context, bus stageBus, log *slog.Logger, delay func(time.Duration)) int { + for { + if ctx.Err() != nil { + log.Info("archie-agent shutting down") + return 0 + } + message, err := bus.FetchStage(ctx) + switch { + case errors.Is(err, eventbus.ErrNoMessage): + continue + case err != nil: + if ctx.Err() != nil { + return 0 + } + log.Error("fetch failed", "err", err) + delay(time.Second) + continue + } + if err := handleSingleStageRequest(ctx, message, log, productionSingleStageDependencies()); err != nil { + log.Error("handle failed", "err", err) + if err := message.Nak(); err != nil { + log.Warn("nak failed", "err", err) + } + continue + } + if err := message.Ack(); err != nil { + log.Warn("ack failed", "err", err) + } + } +} diff --git a/internal/app/agentworker/stage_loop_test.go b/internal/app/agentworker/stage_loop_test.go new file mode 100644 index 000000000..fcfc37a10 --- /dev/null +++ b/internal/app/agentworker/stage_loop_test.go @@ -0,0 +1,191 @@ +package agentworker + +import ( + "bytes" + "context" + "errors" + "log/slog" + "strings" + "testing" + "time" + + "github.com/samcharles93/archie-core/internal/agentexec" + "github.com/samcharles93/archie-core/internal/eventbus" +) + +type stageFetch struct { + message agentexec.StageMessage + err error + before func() +} + +type stageBusStub struct { + fetches []stageFetch + fetchCalls int +} + +func (b *stageBusStub) FetchStage(context.Context) (agentexec.StageMessage, error) { + if b.fetchCalls >= len(b.fetches) { + panic("unexpected FetchStage call") + } + fetch := b.fetches[b.fetchCalls] + b.fetchCalls++ + if fetch.before != nil { + fetch.before() + } + return fetch.message, fetch.err +} + +type stageMessageStub struct { + request agentexec.AgentRequestMessage + requestErr error + respondErr error + ackErr error + nakErr error + respondCancel context.CancelFunc + respondCalls int + ackCalls int + nakCalls int +} + +func (m *stageMessageStub) Request() (agentexec.AgentRequestMessage, error) { + return m.request, m.requestErr +} + +func (m *stageMessageStub) Respond(context.Context, *agentexec.AgentResponseEnvelope) error { + m.respondCalls++ + if m.respondCancel != nil { + m.respondCancel() + } + return m.respondErr +} + +func (m *stageMessageStub) Ack() error { m.ackCalls++; return m.ackErr } +func (m *stageMessageStub) Nak() error { m.nakCalls++; return m.nakErr } +func (*stageMessageStub) LogAttributes() []any { return []any{"reply_to", "_INBOX.test"} } + +func stageLoopLogger() (*slog.Logger, *bytes.Buffer) { + var output bytes.Buffer + return slog.New(slog.NewTextHandler(&output, nil)), &output +} + +func noDelay(t *testing.T) (func(time.Duration), *[]time.Duration) { + t.Helper() + var calls []time.Duration + return func(duration time.Duration) { calls = append(calls, duration) }, &calls +} + +func TestRunMainLoopExitsImmediatelyWhenAlreadyCancelled(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + cancel() + bus := &stageBusStub{} + log, output := stageLoopLogger() + if code := runMainLoop(ctx, bus, log); code != 0 { + t.Fatalf("runMainLoop() = %d, want 0", code) + } + if bus.fetchCalls != 0 || !strings.Contains(output.String(), "archie-agent shutting down") { + t.Fatalf("fetches/log = (%d, %q), want no fetch and shutdown log", bus.fetchCalls, output.String()) + } +} + +func TestRunMainLoopFetchOutcomes(t *testing.T) { + fetchErr := errors.New("temporary fetch failure") + tests := []struct { + name string + firstErr error + wantDelay bool + wantLog bool + }{ + {name: "idle", firstErr: eventbus.ErrNoMessage}, + {name: "failure", firstErr: fetchErr, wantDelay: true, wantLog: true}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + bus := &stageBusStub{fetches: []stageFetch{ + {err: test.firstErr}, + {err: errors.New("fetch after cancellation"), before: cancel}, + }} + log, output := stageLoopLogger() + delay, delays := noDelay(t) + if code := runMainLoopWithDelay(ctx, bus, log, delay); code != 0 { + t.Fatalf("runMainLoopWithDelay() = %d, want 0", code) + } + if got := len(*delays) == 1 && (*delays)[0] == time.Second; got != test.wantDelay { + t.Fatalf("delayed = %v (%v), want %v", got, *delays, test.wantDelay) + } + if got := strings.Contains(output.String(), "fetch failed"); got != test.wantLog { + t.Fatalf("fetch failure logged = %v, want %v; %q", got, test.wantLog, output.String()) + } + }) + } +} + +func TestRunMainLoopStopsWhenFetchCancelsContext(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + bus := &stageBusStub{fetches: []stageFetch{{err: errors.New("interrupted"), before: cancel}}} + log, output := stageLoopLogger() + delay, delays := noDelay(t) + if code := runMainLoopWithDelay(ctx, bus, log, delay); code != 0 { + t.Fatalf("runMainLoopWithDelay() = %d, want 0", code) + } + if len(*delays) != 0 || output.Len() != 0 { + t.Fatalf("delay/log = (%v, %q), want none", *delays, output.String()) + } +} + +func TestRunMainLoopNaksHandleErrorsAndContinues(t *testing.T) { + for _, test := range []struct { + name string + nakErr error + wantNakLog bool + }{ + {name: "nak succeeds"}, + {name: "nak fails", nakErr: errors.New("nak unavailable"), wantNakLog: true}, + } { + t.Run(test.name, func(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + message := &stageMessageStub{requestErr: errors.New("decode request: invalid"), nakErr: test.nakErr} + bus := &stageBusStub{fetches: []stageFetch{{message: message}, {err: errors.New("cancelled"), before: cancel}}} + log, output := stageLoopLogger() + delay, _ := noDelay(t) + runMainLoopWithDelay(ctx, bus, log, delay) + if message.nakCalls != 1 || message.ackCalls != 0 { + t.Fatalf("nak/ack = (%d,%d), want (1,0)", message.nakCalls, message.ackCalls) + } + if got := strings.Contains(output.String(), "nak failed"); got != test.wantNakLog { + t.Fatalf("nak failure logged = %v, want %v", got, test.wantNakLog) + } + }) + } +} + +func TestRunMainLoopAcksSuccessfulHandleAndContinues(t *testing.T) { + for _, test := range []struct { + name string + ackErr error + wantAckLog bool + }{ + {name: "ack succeeds"}, + {name: "ack fails", ackErr: errors.New("ack unavailable"), wantAckLog: true}, + } { + t.Run(test.name, func(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + message := &stageMessageStub{ + request: agentexec.AgentRequestMessage{Providers: map[string]agentexec.Provider{"test": {Class: "openai"}}}, + ackErr: test.ackErr, + respondCancel: cancel, + } + bus := &stageBusStub{fetches: []stageFetch{{message: message}}} + log, output := stageLoopLogger() + delay, _ := noDelay(t) + runMainLoopWithDelay(ctx, bus, log, delay) + if message.respondCalls != 1 || message.ackCalls != 1 || message.nakCalls != 0 { + t.Fatalf("respond/ack/nak = (%d,%d,%d), want (1,1,0)", message.respondCalls, message.ackCalls, message.nakCalls) + } + if got := strings.Contains(output.String(), "ack failed"); got != test.wantAckLog { + t.Fatalf("ack failure logged = %v, want %v", got, test.wantAckLog) + } + }) + } +} diff --git a/cmd/archie-agent/taskrun.go b/internal/app/agentworker/task_execution.go similarity index 56% rename from cmd/archie-agent/taskrun.go rename to internal/app/agentworker/task_execution.go index bb1abe0e8..0a1495e8d 100644 --- a/cmd/archie-agent/taskrun.go +++ b/internal/app/agentworker/task_execution.go @@ -1,28 +1,19 @@ -package main +package agentworker import ( "context" - "encoding/json" "fmt" "log/slog" - "time" - - natsio "github.com/nats-io/nats.go" "github.com/samcharles93/archie-core/internal/agentexec" "github.com/samcharles93/archie-core/internal/domain/workflow" "github.com/samcharles93/archie-core/internal/domain/workflow/skillbuild" "github.com/samcharles93/archie-core/internal/domain/workflow/wfeval" - "github.com/samcharles93/archie-core/internal/forgerpc" - "github.com/samcharles93/archie-core/internal/storage" - "github.com/samcharles93/archie-core/internal/storerpc" + "github.com/samcharles93/archie-core/internal/store" "github.com/samcharles93/archie-core/internal/taskrun" "github.com/samcharles93/archie-core/internal/worktree" - "github.com/samcharles93/archie-core/internal/worktreerpc" ) -const rpcTimeout = 60 * time.Second - // hybridTrees implements workflow.Trees by splitting operations: Prepare // and Push (network ops needing the daemon's forge credential) proxy over // worktreerpc; CommitAll/Diff/ChangedFiles/ChangedLines are local git @@ -31,8 +22,13 @@ const rpcTimeout = 60 * time.Second // Prepare deliberately discards the RPC response's directory (the // daemon's host-side path) and returns localDir instead -- archied and // archie-agent see the same files at different paths. +type remoteTrees interface { + Prepare(ctx context.Context, owner, repo, base string, issue int, title, body, labels string) (dir, branch string, err error) + Push(ctx context.Context, owner, repo string, issue int, branch string) error +} + type hybridTrees struct { - push *worktreerpc.Client + push remoteTrees local *worktree.Manager owner, repo string issue int @@ -69,12 +65,16 @@ func (h *hybridTrees) ChangedLines(ctx context.Context, dir, base string) (int, var _ workflow.Trees = (*hybridTrees)(nil) -// runTask builds a workflow.Registry from the container's own mounted -// worktree, routes and runs req.Task's entire workflow, and reports the -// terminal outcome. Store/Forge/worktree-push calls proxy back to archied -// over nc; Store is archied's sole authority for task state -- the -// returned Response.Task is a best-effort snapshot for logging only. -func runTask(ctx context.Context, req taskrun.Request, nc *natsio.Conn, newRunner agentexec.RunnerFactory, workDir string, log *slog.Logger) (*taskrun.Response, error) { +type taskDependencies struct { + forge workflow.Forger + store store.WorkflowStore + trees remoteTrees +} + +// runTask builds a workflow.Registry from the container's mounted worktree, +// routes and runs the entire workflow, and reports its terminal outcome. +// Store remains archied's authority; Response.Task is a logging snapshot. +func runTask(ctx context.Context, req taskrun.Request, dependencies taskDependencies, newRunner agentexec.RunnerFactory, workDir string, log *slog.Logger) (*taskrun.Response, error) { registry, err := skillbuild.BuildRegistry(workDir) if err != nil { return nil, fmt.Errorf("build registry: %w", err) @@ -83,7 +83,7 @@ func runTask(ctx context.Context, req taskrun.Request, nc *natsio.Conn, newRunne wf := workflow.Route(req.Task, registry) trees := &hybridTrees{ - push: &worktreerpc.Client{Conn: nc, Timeout: rpcTimeout, Identity: req.Task.Identity}, + push: dependencies.trees, local: &worktree.Manager{WorkDir: workDir}, owner: req.Task.Owner, repo: req.Task.Repo, @@ -115,8 +115,8 @@ func runTask(ctx context.Context, req taskrun.Request, nc *natsio.Conn, newRunne Task: req.Task, Repo: req.Repo, Cfg: req.Cfg.ToConfig(), - Forge: &forgerpc.Client{Conn: nc, Timeout: rpcTimeout, Identity: req.Task.Identity}, - Store: &storerpc.Client{Conn: nc, Timeout: rpcTimeout}, + Forge: dependencies.forge, + Store: dependencies.store, Trees: trees, Agent: agent, Log: log, @@ -130,45 +130,3 @@ func runTask(ctx context.Context, req taskrun.Request, nc *natsio.Conn, newRunne Status: tc.Outcome.Status, }, nil } - -// handleTaskRun decodes a taskrun.Request from msg, runs it via runTask -// against the container's fixed worktree mount, and replies with the -// outcome. Errors are reported in the reply, not returned -- there's no -// caller to return them to; this is a NATS subscription callback. -func handleTaskRun(ctx context.Context, msg *natsio.Msg, nc *natsio.Conn, log *slog.Logger) { - var req taskrun.Request - if err := json.Unmarshal(msg.Data, &req); err != nil { - log.Error("taskrun decode failed", "err", err) - respondTaskRun(msg, nil, fmt.Errorf("decode taskrun request: %w", err), log) - return - } - - log.Info("running task", "task", req.Task.ID, "repo", req.Repo.FullName(), "issue", req.Task.IssueNumber) - - resp, err := runTask(ctx, req, nc, agentexec.DefaultRunnerFactory, storage.WorktreeMountDir, log) - if err != nil { - log.Error("task run failed", "task", req.Task.ID, "err", err) - respondTaskRun(msg, nil, err, log) - return - } - - log.Info("task run complete", "task", req.Task.ID, "status", resp.Status) - respondTaskRun(msg, resp, nil, log) -} - -func respondTaskRun(msg *natsio.Msg, resp *taskrun.Response, runErr error, log *slog.Logger) { - if resp == nil { - resp = &taskrun.Response{} - } - if runErr != nil { - resp.Error = runErr.Error() - } - data, err := json.Marshal(resp) - if err != nil { - log.Error("taskrun encode response failed", "err", err) - return - } - if err := msg.Respond(data); err != nil { - log.Error("taskrun respond failed", "err", err) - } -} diff --git a/cmd/archie-agent/taskrun_test.go b/internal/app/agentworker/task_execution_test.go similarity index 54% rename from cmd/archie-agent/taskrun_test.go rename to internal/app/agentworker/task_execution_test.go index 030279a31..3f9640504 100644 --- a/cmd/archie-agent/taskrun_test.go +++ b/internal/app/agentworker/task_execution_test.go @@ -1,4 +1,4 @@ -package main +package agentworker import ( "context" @@ -16,6 +16,7 @@ import ( "github.com/samcharles93/archie-core/internal/config" "github.com/samcharles93/archie-core/internal/forge" "github.com/samcharles93/archie-core/internal/forgerpc" + agentnats "github.com/samcharles93/archie-core/internal/infrastructure/agenttransport/nats" "github.com/samcharles93/archie-core/internal/store" "github.com/samcharles93/archie-core/internal/storerpc" "github.com/samcharles93/archie-core/internal/taskrun" @@ -23,25 +24,6 @@ import ( "github.com/samcharles93/archie-core/internal/worktreerpc" ) -func startEmbeddedForTaskRun(t *testing.T) *server.Server { - t.Helper() - srv := natssrv.RunRandClientPortServer() - t.Cleanup(srv.Shutdown) - return srv -} - -func connectForTaskRun(t *testing.T, url string) *natsio.Conn { - t.Helper() - nc, err := natsio.Connect(url) - if err != nil { - t.Fatalf("nats connect: %v", err) - } - t.Cleanup(nc.Close) - return nc -} - -// panicRunner fails the test if the bootstrap workflow (deterministic, -// no LLM stages) ever invokes it. type panicRunner struct{ t *testing.T } func (r panicRunner) Run(context.Context, string, agentexec.Request) (agentexec.Result, error) { @@ -49,8 +31,6 @@ func (r panicRunner) Run(context.Context, string, agentexec.Request) (agentexec. return agentexec.Result{}, nil } -// prCapturingForge records CreatePR calls and answers the four Forger -// methods the bootstrap workflow needs; everything else panics. type prCapturingForge struct { prs []struct{ owner, repo, title, head, base, body string } } @@ -92,29 +72,39 @@ func (f *prCapturingForge) React(context.Context, string, string, int, string) e func (f *prCapturingForge) VerifyPush(context.Context, string, string) error { panic("unexpected call") } +func (f *prCapturingForge) LinkBranch(context.Context, string, string, int, string) error { return nil } -func (f *prCapturingForge) LinkBranch(context.Context, string, string, int, string) error { - return nil +type remoteManager struct { + manager *worktree.Manager + dir string +} + +func (r *remoteManager) Prepare(ctx context.Context, owner, repo, base string, issue int, title, body, labels string) (string, string, error) { + dir, branch, err := r.manager.Prepare(ctx, owner, repo, base, issue, title, body, labels) + if err == nil { + r.dir = dir + } + return dir, branch, err +} + +func (r *remoteManager) Push(ctx context.Context, _, _ string, _ int, branch string) error { + return r.manager.Push(ctx, r.dir, branch) } -// newLocalRemote mirrors worktree_test.go's helper: a bare repo with one -// commit on main, reachable via file:////.git. func newLocalRemote(t *testing.T, owner, repo string) string { t.Helper() host := t.TempDir() bare := filepath.Join(host, owner, repo+".git") seed := filepath.Join(t.TempDir(), "seed") - run := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := exec.CommandContext(t.Context(), "git", args...) cmd.Dir = dir cmd.Env = append(os.Environ(), "GIT_TERMINAL_PROMPT=0") if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v: %v\n%s", args, err, out) } } - if err := os.MkdirAll(bare, 0o755); err != nil { t.Fatal(err) } @@ -134,20 +124,32 @@ func newLocalRemote(t *testing.T, owner, repo string) string { return host } -// TestRunTaskExecutesBootstrapWorkflowEndToEnd drives runTask through the -// deterministic "bootstrap" workflow (no LLM calls) with real storerpc/ -// forgerpc/worktreerpc servers backing an RPC-constructed TaskContext, -// proving the full archie-agent-side wiring: registry build, Route, Run, -// and the hybrid Trees split between proxied Prepare/Push and local git. +func startEmbeddedTaskRPCServer(t *testing.T) *server.Server { + t.Helper() + srv := natssrv.RunRandClientPortServer() + t.Cleanup(srv.Shutdown) + if err := srv.EnableJetStream(&server.JetStreamConfig{StoreDir: t.TempDir()}); err != nil { + t.Fatal(err) + } + return srv +} + +func connectTaskRPC(t *testing.T, url string) *natsio.Conn { + t.Helper() + connection, err := natsio.Connect(url) + if err != nil { + t.Fatal(err) + } + t.Cleanup(connection.Close) + return connection +} + func TestRunTaskExecutesBootstrapWorkflowEndToEnd(t *testing.T) { if _, err := exec.LookPath("git"); err != nil { t.Skip("git not installed") } - ctx := context.Background() + ctx := t.Context() host := newLocalRemote(t, "acme", "widget") - - // archied side: real Store, a capturing Forge, and the Manager that - // actually holds the (unused-for-file-remotes) push token. st := store.OpenTest(t) if _, err := st.EnqueueIssue(ctx, "acme", "widget", 1, "feat: bootstrap test", "", "bootstrap", ""); err != nil { t.Fatalf("enqueue: %v", err) @@ -156,8 +158,50 @@ func TestRunTaskExecutesBootstrapWorkflowEndToEnd(t *testing.T) { if err != nil || task == nil { t.Fatalf("claim: (%v, %v)", task, err) } - fg := &prCapturingForge{} + manager := &worktree.Manager{WorkDir: t.TempDir(), Token: "unused", BotUser: "archie-bot", BotEmail: "archie-bot@example.com", BaseURL: "file://" + host} + remote := &remoteManager{manager: manager} + hostDir, _, err := remote.Prepare(ctx, "acme", "widget", "main", 1, task.Title, "", "bootstrap") + if err != nil { + t.Fatalf("prepare: %v", err) + } + req := taskrun.Request{Task: task, Repo: config.Repo{Owner: "acme", Name: "widget", Base: "main"}, Cfg: config.Config{}.ForTask()} + fakeRunner := agentexec.RunnerFactory(func(map[string]agentexec.Provider, *slog.Logger) agentexec.Runner { return panicRunner{t} }) + response, err := runTask(ctx, req, taskDependencies{forge: fg, store: st, trees: remote}, fakeRunner, hostDir, slog.New(slog.DiscardHandler)) + if err != nil { + t.Fatalf("runTask: %v", err) + } + if response.Status != store.StatusPROpen || len(fg.prs) != 1 { + t.Fatalf("response status/PRs = (%q, %d), want (%q, 1)", response.Status, len(fg.prs), store.StatusPROpen) + } + stored, err := st.TaskByIssue(ctx, "acme", "widget", 1) + if err != nil || stored == nil || stored.Status != store.StatusPROpen || stored.PRNumber != 5 { + t.Fatalf("stored task = (%+v, %v), want PR-open task #5", stored, err) + } + if _, err := os.Stat(filepath.Join(hostDir, ".archie", "bootstrap.md")); err != nil { + t.Fatalf("expected bootstrap marker file: %v", err) + } +} + +// TestExecuteTaskRequestUsesInfrastructureRPCDependencies preserves the +// application-to-infrastructure proof: the application asks Transport for its +// forge/store/tree contracts, and those concrete clients cross embedded NATS +// to daemon-side RPC servers before the bootstrap workflow can complete. +func TestExecuteTaskRequestUsesInfrastructureRPCDependencies(t *testing.T) { + if _, err := exec.LookPath("git"); err != nil { + t.Skip("git not installed") + } + ctx := t.Context() + host := newLocalRemote(t, "acme", "rpc-widget") + st := store.OpenTest(t) + if _, err := st.EnqueueIssue(ctx, "acme", "rpc-widget", 2, "feat: RPC bootstrap test", "", "bootstrap", ""); err != nil { + t.Fatal(err) + } + task, err := st.ClaimNext(ctx) + if err != nil || task == nil { + t.Fatalf("claim = (%+v, %v)", task, err) + } + forge := &prCapturingForge{} daemonTrees := &worktree.Manager{ WorkDir: t.TempDir(), Token: "unused-for-file-remotes", @@ -165,72 +209,54 @@ func TestRunTaskExecutesBootstrapWorkflowEndToEnd(t *testing.T) { BotEmail: "archie-bot@example.com", BaseURL: "file://" + host, } - - // The daemon always prepares the worktree before container acquire -- - // reproduce that here so the container's bind mount has real content. - hostDir, _, err := daemonTrees.Prepare(ctx, "acme", "widget", "main", 1, task.Title, "", "bootstrap") + hostDir, _, err := daemonTrees.Prepare(ctx, "acme", "rpc-widget", "main", 2, task.Title, "", "bootstrap") if err != nil { - t.Fatalf("daemon-side prepare: %v", err) + t.Fatal(err) } - srv := startEmbeddedForTaskRun(t) - url := srv.ClientURL() - serverConn := connectForTaskRun(t, url) - + srv := startEmbeddedTaskRPCServer(t) + serverConn := connectTaskRPC(t, srv.ClientURL()) unsubStore, err := (&storerpc.Server{Store: st, Log: slog.New(slog.DiscardHandler)}).Register(serverConn) if err != nil { - t.Fatalf("register storerpc: %v", err) + t.Fatal(err) } t.Cleanup(unsubStore) - unsubForge, err := (&forgerpc.Server{Forge: fg, Log: slog.New(slog.DiscardHandler)}).Register(serverConn) + unsubForge, err := (&forgerpc.Server{Forge: forge, Log: slog.New(slog.DiscardHandler)}).Register(serverConn) if err != nil { - t.Fatalf("register forgerpc: %v", err) + t.Fatal(err) } t.Cleanup(unsubForge) unsubTrees, err := (&worktreerpc.Server{Trees: daemonTrees, Log: slog.New(slog.DiscardHandler)}).Register(serverConn) if err != nil { - t.Fatalf("register worktreerpc: %v", err) + t.Fatal(err) } t.Cleanup(unsubTrees) - // archie-agent side: its own NATS connection, "bind mount" is just - // hostDir since this test runs in a single process/filesystem. - agentConn := connectForTaskRun(t, url) + transport, err := agentnats.Connect(ctx, agentnats.Config{ + URL: srv.ClientURL(), + ConsumerName: "rpc-bootstrap-test", + }, slog.New(slog.DiscardHandler)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(transport.Close) - req := taskrun.Request{ + request := taskrun.Request{ Task: task, - Repo: config.Repo{Owner: "acme", Name: "widget", Base: "main"}, + Repo: config.Repo{Owner: "acme", Name: "rpc-widget", Base: "main"}, Cfg: config.Config{}.ForTask(), } - - fakeRunner := agentexec.RunnerFactory(func(map[string]agentexec.Provider, *slog.Logger) agentexec.Runner { - return panicRunner{t} - }) - - resp, err := runTask(ctx, req, agentConn, fakeRunner, hostDir, slog.New(slog.DiscardHandler)) + response, err := executeTaskRequest(ctx, request, transport, hostDir, slog.New(slog.DiscardHandler)) if err != nil { - t.Fatalf("runTask: %v", err) + t.Fatal(err) } - if resp.Status != store.StatusPROpen { - t.Fatalf("resp.Status = %q, want %q (task: %+v)", resp.Status, store.StatusPROpen, resp.Task) + if response.Status != store.StatusPROpen || len(forge.prs) != 1 { + t.Fatalf("response status/PRs = (%q, %d), want (%q, 1)", response.Status, len(forge.prs), store.StatusPROpen) } - if len(fg.prs) != 1 { - t.Fatalf("expected 1 CreatePR call, got %d: %+v", len(fg.prs), fg.prs) + stored, err := st.TaskByIssue(ctx, "acme", "rpc-widget", 2) + if err != nil || stored == nil || stored.Status != store.StatusPROpen || stored.PRNumber != 5 { + t.Fatalf("stored task = (%+v, %v), want RPC-persisted PR-open task #5", stored, err) } - - // The authoritative state landed in archied's store via storerpc, not - // just in the in-memory tc.Task. - stored, err := st.TaskByIssue(ctx, "acme", "widget", 1) - if err != nil || stored == nil { - t.Fatalf("TaskByIssue: (%+v, %v)", stored, err) - } - if stored.Status != store.StatusPROpen { - t.Fatalf("stored task status = %q, want %q", stored.Status, store.StatusPROpen) - } - if stored.PRNumber != 5 { - t.Fatalf("stored task PR number = %d, want 5", stored.PRNumber) - } - if _, err := os.Stat(filepath.Join(hostDir, ".archie", "bootstrap.md")); err != nil { t.Fatalf("expected bootstrap marker file: %v", err) } diff --git a/internal/app/agentworker/worker.go b/internal/app/agentworker/worker.go new file mode 100644 index 000000000..9b365ddff --- /dev/null +++ b/internal/app/agentworker/worker.go @@ -0,0 +1,148 @@ +// Package agentworker composes and runs Archie's NATS-connected agent worker. +package agentworker + +import ( + "context" + "errors" + "log/slog" + "time" + + "github.com/samcharles93/archie-core/internal/agentexec" + "github.com/samcharles93/archie-core/internal/domain/workflow" + "github.com/samcharles93/archie-core/internal/infrastructure/agentboot" + "github.com/samcharles93/archie-core/internal/infrastructure/agentgit" + agentnats "github.com/samcharles93/archie-core/internal/infrastructure/agenttransport/nats" + "github.com/samcharles93/archie-core/internal/storage" + "github.com/samcharles93/archie-core/internal/store" + "github.com/samcharles93/archie-core/internal/taskrun" +) + +const rpcTimeout = 60 * time.Second + +// StartupError reports which worker startup operation failed. The application +// logs the operation-specific failure once; the command only maps it to exit 1. +type StartupError struct { + Operation string + Err error +} + +func (e *StartupError) Error() string { return e.Operation + ": " + e.Err.Error() } +func (e *StartupError) Unwrap() error { return e.Err } + +// Settings are process-derived inputs for one worker runtime. +type Settings struct { + NATSURL string + NATSToken string + Consumer string + WorkDir string +} + +type workerTransport interface { + agentexec.StageConsumer + Close() + LogPublisher() agentexec.LogPublisher + SubscribeTasks(context.Context, int64, bool, agentnats.TaskHandler, *slog.Logger) (agentnats.Subscription, error) + Forger(string, time.Duration) workflow.Forger + Store(time.Duration) store.WorkflowStore + Trees(string, time.Duration) agentnats.RemoteTrees +} + +type workerDependencies struct { + connect func(context.Context, agentnats.Config, *slog.Logger) (workerTransport, error) + markSafe func(context.Context, string, *slog.Logger) bool + bootID func(string) (int64, bool, error) + runLoop func(context.Context, stageBus, *slog.Logger) int +} + +func productionWorkerDependencies() workerDependencies { + return workerDependencies{ + connect: func(ctx context.Context, config agentnats.Config, log *slog.Logger) (workerTransport, error) { + return agentnats.Connect(ctx, config, log) + }, + markSafe: agentgit.MarkSafe, + bootID: agentboot.TaskID, + runLoop: runMainLoop, + } +} + +// Run starts the worker, serves full-task and single-stage requests, and +// returns after ctx cancellation or a startup failure. +func Run(ctx context.Context, settings Settings, log *slog.Logger) error { + return run(ctx, settings, log, productionWorkerDependencies()) +} + +func run(ctx context.Context, settings Settings, log *slog.Logger, dependencies workerDependencies) error { + if log == nil { + log = slog.New(slog.DiscardHandler) + } + workDir := settings.WorkDir + if workDir == "" { + workDir = storage.WorktreeMountDir + } + + dependencies.markSafe(ctx, workDir, log) + + transport, err := dependencies.connect(ctx, agentnats.Config{ + URL: settings.NATSURL, + Token: settings.NATSToken, + ConsumerName: settings.Consumer, + }, log) + if err != nil { + operation := "nats connect failed" + if errors.Is(err, agentnats.ErrCoreConnectionUnavailable) { + operation = "nats connection unavailable" + } + log.Error(operation, "err", err) + return &StartupError{Operation: operation, Err: err} + } + defer transport.Close() + + taskID, dedicated, err := dependencies.bootID(workDir) + if err != nil { + log.Error("task boot failed", "err", err) + return &StartupError{Operation: "task boot failed", Err: err} + } + if dedicated { + log = slog.New(agentexec.NewSystemLogHandler(log.Handler(), transport.LogPublisher(), taskID)) + log.Info("system log publisher attached", "task", taskID) + } + + subscription, err := transport.SubscribeTasks(ctx, taskID, dedicated, func(ctx context.Context, request taskrun.Request) (*taskrun.Response, error) { + return executeTaskRequest(ctx, request, transport, workDir, log) + }, log) + if err != nil { + log.Error("taskrun subscribe failed", "err", err) + return &StartupError{Operation: "taskrun subscribe failed", Err: err} + } + defer func() { + if err := subscription.Close(); err != nil { + log.Warn("task run unsubscribe failed", "err", err) + } + }() + + log.Info("archie-agent ready", "nats", settings.NATSURL, "consumer", settings.Consumer) + dependencies.runLoop(ctx, transport, log) + return nil +} + +type taskServiceTransport interface { + Forger(string, time.Duration) workflow.Forger + Store(time.Duration) store.WorkflowStore + Trees(string, time.Duration) agentnats.RemoteTrees +} + +func executeTaskRequest(ctx context.Context, request taskrun.Request, transport taskServiceTransport, workDir string, log *slog.Logger) (*taskrun.Response, error) { + log.Info("running task", "task", request.Task.ID, "repo", request.Repo.FullName(), "issue", request.Task.IssueNumber) + dependencies := taskDependencies{ + forge: transport.Forger(request.Task.Identity, rpcTimeout), + store: transport.Store(rpcTimeout), + trees: transport.Trees(request.Task.Identity, rpcTimeout), + } + response, err := runTask(ctx, request, dependencies, agentexec.DefaultRunnerFactory, workDir, log) + if err != nil { + log.Error("task run failed", "task", request.Task.ID, "err", err) + return nil, err + } + log.Info("task run complete", "task", request.Task.ID, "status", response.Status) + return response, nil +} diff --git a/internal/app/agentworker/worker_test.go b/internal/app/agentworker/worker_test.go new file mode 100644 index 000000000..a52f7e4cf --- /dev/null +++ b/internal/app/agentworker/worker_test.go @@ -0,0 +1,229 @@ +package agentworker + +import ( + "bytes" + "context" + "errors" + "fmt" + "log/slog" + "strings" + "testing" + "time" + + "github.com/samcharles93/archie-core/internal/agentexec" + "github.com/samcharles93/archie-core/internal/domain/workflow" + agentnats "github.com/samcharles93/archie-core/internal/infrastructure/agenttransport/nats" + "github.com/samcharles93/archie-core/internal/store" +) + +type workerPublisher struct{ messages *[][]byte } + +func (p workerPublisher) Publish(_ string, data []byte) error { + if p.messages != nil { + *p.messages = append(*p.messages, append([]byte(nil), data...)) + } + return nil +} + +type workerSubscription struct { + close func() + closeErr error +} + +func (s workerSubscription) Close() error { + s.close() + return s.closeErr +} + +type workerTransportStub struct { + events *[]string + subscribeErr error + subCloseErr error + publisher agentexec.LogPublisher + dedicated bool +} + +func (*workerTransportStub) FetchStage(context.Context) (agentexec.StageMessage, error) { + return nil, nil +} +func (t *workerTransportStub) Close() { *t.events = append(*t.events, "transport-close") } +func (t *workerTransportStub) LogPublisher() agentexec.LogPublisher { + *t.events = append(*t.events, "log-publisher") + if t.publisher != nil { + return t.publisher + } + return workerPublisher{} +} + +func (t *workerTransportStub) SubscribeTasks(_ context.Context, _ int64, dedicated bool, _ agentnats.TaskHandler, _ *slog.Logger) (agentnats.Subscription, error) { + *t.events = append(*t.events, "subscribe") + t.dedicated = dedicated + if t.subscribeErr != nil { + return nil, t.subscribeErr + } + return workerSubscription{ + close: func() { *t.events = append(*t.events, "subscription-close") }, + closeErr: t.subCloseErr, + }, nil +} +func (*workerTransportStub) Forger(string, time.Duration) workflow.Forger { return nil } +func (*workerTransportStub) Store(time.Duration) store.WorkflowStore { return nil } +func (*workerTransportStub) Trees(string, time.Duration) agentnats.RemoteTrees { return nil } + +func TestRunPassesCompleteTransportConfiguration(t *testing.T) { + var got agentnats.Config + var events []string + transport := &workerTransportStub{events: &events} + ctx, cancel := context.WithCancel(t.Context()) + dependencies := workerDependencies{ + markSafe: func(context.Context, string, *slog.Logger) bool { return true }, + connect: func(_ context.Context, config agentnats.Config, _ *slog.Logger) (workerTransport, error) { + got = config + return transport, nil + }, + bootID: func(string) (int64, bool, error) { return 0, false, nil }, + runLoop: func(context.Context, stageBus, *slog.Logger) int { cancel(); return 0 }, + } + settings := Settings{NATSURL: "nats://test", NATSToken: "secret", Consumer: "worker", WorkDir: "/worktree"} + if err := run(ctx, settings, slog.New(slog.DiscardHandler), dependencies); err != nil { + t.Fatal(err) + } + if got.URL != settings.NATSURL || got.Token != settings.NATSToken || got.ConsumerName != settings.Consumer { + t.Fatalf("connection config = %#v, want URL/token/consumer from settings", got) + } +} + +func TestRunPreservesStartupAndReverseCleanupOrder(t *testing.T) { + var events []string + transport := &workerTransportStub{events: &events} + dependencies := workerDependencies{ + markSafe: func(context.Context, string, *slog.Logger) bool { + events = append(events, "git-safety") + return true + }, + connect: func(context.Context, agentnats.Config, *slog.Logger) (workerTransport, error) { + events = append(events, "connect") + return transport, nil + }, + bootID: func(string) (int64, bool, error) { + events = append(events, "boot-id") + return 42, true, nil + }, + runLoop: func(context.Context, stageBus, *slog.Logger) int { + events = append(events, "stage-loop") + return 0 + }, + } + if err := run(t.Context(), Settings{NATSURL: "nats://test", Consumer: "worker", WorkDir: "/worktree"}, slog.New(slog.DiscardHandler), dependencies); err != nil { + t.Fatal(err) + } + want := []string{"git-safety", "connect", "boot-id", "log-publisher", "subscribe", "stage-loop", "subscription-close", "transport-close"} + if len(events) != len(want) { + t.Fatalf("events = %v, want %v", events, want) + } + for i := range want { + if events[i] != want[i] { + t.Fatalf("events = %v, want %v", events, want) + } + } + if !transport.dedicated { + t.Fatal("dedicated boot ID did not select dedicated subscription") + } +} + +func TestRunWarnsWhenNormalSubscriptionCleanupFails(t *testing.T) { + closeErr := errors.New("unsubscribe failed") + var events []string + transport := &workerTransportStub{events: &events, subCloseErr: closeErr} + dependencies := workerDependencies{ + markSafe: func(context.Context, string, *slog.Logger) bool { return true }, + connect: func(context.Context, agentnats.Config, *slog.Logger) (workerTransport, error) { + return transport, nil + }, + bootID: func(string) (int64, bool, error) { return 0, false, nil }, + runLoop: func(context.Context, stageBus, *slog.Logger) int { return 0 }, + } + var output bytes.Buffer + if err := run(t.Context(), Settings{NATSURL: "nats://test", Consumer: "worker"}, slog.New(slog.NewTextHandler(&output, nil)), dependencies); err != nil { + t.Fatal(err) + } + if !strings.Contains(output.String(), "task run unsubscribe failed") || !strings.Contains(output.String(), closeErr.Error()) { + t.Fatalf("cleanup log = %q, want warning containing %q", output.String(), closeErr) + } +} + +func TestRunStartupFailuresCleanUpOwnedResources(t *testing.T) { + connectErr := errors.New("connect failed") + bootErr := errors.New("invalid boot metadata") + subscribeErr := errors.New("subscribe failed") + tests := []struct { + name string + connectErr error + bootErr error + subscribeErr error + dedicated bool + wantErr error + wantOperation string + wantClosed bool + }{ + {name: "connect", connectErr: connectErr, wantErr: connectErr, wantOperation: "nats connect failed"}, + {name: "core connection", connectErr: fmt.Errorf("%w: unavailable", agentnats.ErrCoreConnectionUnavailable), wantErr: agentnats.ErrCoreConnectionUnavailable, wantOperation: "nats connection unavailable"}, + {name: "boot metadata", bootErr: bootErr, wantErr: bootErr, wantOperation: "task boot failed", wantClosed: true}, + {name: "subscribe", subscribeErr: subscribeErr, dedicated: true, wantErr: subscribeErr, wantOperation: "taskrun subscribe failed", wantClosed: true}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + var events []string + var published [][]byte + transport := &workerTransportStub{events: &events, subscribeErr: test.subscribeErr, publisher: workerPublisher{messages: &published}} + dependencies := workerDependencies{ + markSafe: func(context.Context, string, *slog.Logger) bool { return true }, + connect: func(context.Context, agentnats.Config, *slog.Logger) (workerTransport, error) { + return transport, test.connectErr + }, + bootID: func(string) (int64, bool, error) { + if test.bootErr != nil { + return 0, false, test.bootErr + } + if test.dedicated { + return 42, true, nil + } + return 0, false, nil + }, + runLoop: func(context.Context, stageBus, *slog.Logger) int { t.Fatal("stage loop called"); return 0 }, + } + var output bytes.Buffer + err := run(t.Context(), Settings{NATSURL: "nats://test", Consumer: "worker"}, slog.New(slog.NewTextHandler(&output, nil)), dependencies) + if !errors.Is(err, test.wantErr) { + t.Fatalf("error = %v, want %v", err, test.wantErr) + } + var startupErr *StartupError + if !errors.As(err, &startupErr) || startupErr.Operation != test.wantOperation { + t.Fatalf("startup error = %#v, want operation %q", startupErr, test.wantOperation) + } + closed := len(events) > 0 && events[len(events)-1] == "transport-close" + if closed != test.wantClosed { + t.Fatalf("transport closed = %v, want %v; events %v", closed, test.wantClosed, events) + } + if test.bootErr != nil { + for _, event := range events { + if event == "subscribe" { + t.Fatalf("invalid boot metadata reached task subscription: events %v", events) + } + } + } + if got := strings.Count(output.String(), test.wantOperation); got != 1 { + t.Fatalf("operation log count = %d, want 1; output %q", got, output.String()) + } + if test.subscribeErr != nil { + var joined strings.Builder + for _, data := range published { + joined.WriteString(string(data)) + } + if !strings.Contains(joined.String(), test.wantOperation) { + t.Fatalf("task-scoped logs %q do not contain %q", joined.String(), test.wantOperation) + } + } + }) + } +} diff --git a/internal/daemon/daemon.go b/internal/daemon/daemon.go index e4001e3a1..adaf95ef9 100644 --- a/internal/daemon/daemon.go +++ b/internal/daemon/daemon.go @@ -24,6 +24,7 @@ import ( "github.com/samcharles93/archie-core/internal/eventbus" "github.com/samcharles93/archie-core/internal/events" "github.com/samcharles93/archie-core/internal/forge" + agentnats "github.com/samcharles93/archie-core/internal/infrastructure/agenttransport/nats" "github.com/samcharles93/archie-core/internal/logging" "github.com/samcharles93/archie-core/internal/memory" "github.com/samcharles93/archie-core/internal/plugin" @@ -1004,8 +1005,8 @@ func (d *Daemon) acquireTaskContainer( } // containerEnv returns the environment variables passed to agent containers. -// runViaAgent publishes a full-task handoff to archie-agent over -// archie.taskrun. and waits for its completion report. Every durable +// runViaAgent publishes a full-task handoff to archie-agent over the +// infrastructure-owned task subject and waits for its completion report. Every durable // side effect (Store transitions, Forge calls, git push) happens inside // archie-agent's workflow.Run, proxied back to this daemon over // storerpc/forgerpc/worktreerpc -- those RPC servers are the sole place a @@ -1084,7 +1085,7 @@ func (d *Daemon) taskRunRetryBackoff() time.Duration { // that's already done, etc.) is returned immediately without retrying, // since those don't mean "not ready yet". func (d *Daemon) requestTaskRun(ctx context.Context, taskID int64, data []byte) ([]byte, error) { - subject := taskrun.SubjectForTask(taskID) + subject := agentnats.SubjectForTask(taskID) deadline := time.Now().Add(d.taskRunReadyTimeout()) backoff := d.taskRunRetryBackoff() for { diff --git a/internal/daemon/daemon_test.go b/internal/daemon/daemon_test.go index 9cf7f5dfb..2219d5ec7 100644 --- a/internal/daemon/daemon_test.go +++ b/internal/daemon/daemon_test.go @@ -18,6 +18,7 @@ import ( "github.com/samcharles93/archie-core/internal/config" "github.com/samcharles93/archie-core/internal/domain/workintake" "github.com/samcharles93/archie-core/internal/forge" + agentnats "github.com/samcharles93/archie-core/internal/infrastructure/agenttransport/nats" arnats "github.com/samcharles93/archie-core/internal/infrastructure/eventbus/nats" "github.com/samcharles93/archie-core/internal/logging" "github.com/samcharles93/archie-core/internal/secret" @@ -455,7 +456,7 @@ func TestRunViaAgentRetriesUntilResponderAppears(t *testing.T) { t.Errorf("CoreConn: %v", connErr) return } - sub, err := coreConn.Subscribe(taskrun.SubjectForTask(task.ID), func(msg *natsio.Msg) { + sub, err := coreConn.Subscribe(agentnats.SubjectForTask(task.ID), func(msg *natsio.Msg) { data, _ := json.Marshal(taskrun.Response{Status: store.StatusPROpen}) _ = msg.Respond(data) }) @@ -516,7 +517,7 @@ func TestRunViaAgentParksOnRunError(t *testing.T) { t.Fatalf("claim: (%v, %v)", task, err) } - sub, err := mustCoreConn(t, busClient).Subscribe(taskrun.SubjectForTask(task.ID), func(msg *natsio.Msg) { + sub, err := mustCoreConn(t, busClient).Subscribe(agentnats.SubjectForTask(task.ID), func(msg *natsio.Msg) { data, _ := json.Marshal(taskrun.Response{Error: "registry build failed"}) _ = msg.Respond(data) }) @@ -552,7 +553,7 @@ func TestRunViaAgentSendsExpectedRequest(t *testing.T) { } received := make(chan taskrun.Request, 1) - sub, err := mustCoreConn(t, busClient).Subscribe(taskrun.SubjectForTask(task.ID), func(msg *natsio.Msg) { + sub, err := mustCoreConn(t, busClient).Subscribe(agentnats.SubjectForTask(task.ID), func(msg *natsio.Msg) { var req taskrun.Request _ = json.Unmarshal(msg.Data, &req) received <- req diff --git a/internal/domain/workflow/workflow.go b/internal/domain/workflow/workflow.go index e1354bc92..57404501e 100644 --- a/internal/domain/workflow/workflow.go +++ b/internal/domain/workflow/workflow.go @@ -58,11 +58,11 @@ type TaskContext struct { Log *slog.Logger // CustomStages discovers a repo's per-repo custom stages (Yaegi- // interpreted from .archie/stages/*.go in the given worktree - // directory), returning them in the order they should run. Wired up - // by the composition root (cmd/archied) -- nil disables discovery, so - // StageRepoStages is a no-op. Kept as an injected function rather - // than a direct import to avoid workflow depending on its own - // generated Yaegi symbol table (an import cycle). + // directory), returning them in the order they should run. Application + // composition injects it; nil disables discovery, so StageRepoStages is a + // no-op. The composition layer provides the implementation rather + // than importing the generated Yaegi symbol table here, which would create + // a workflow package import cycle. CustomStages func(dir string) ([]Stage, error) // SkillBody is the Markdown body of the loaded SKILL.md for this diff --git a/internal/infrastructure/agentboot/task.go b/internal/infrastructure/agentboot/task.go new file mode 100644 index 000000000..8ac59ca17 --- /dev/null +++ b/internal/infrastructure/agentboot/task.go @@ -0,0 +1,36 @@ +package agentboot + +import ( + "encoding/json" + "errors" + "fmt" + "io/fs" + "os" + "path/filepath" + + "github.com/samcharles93/archie-core/internal/container" +) + +// TaskID reads the daemon-owned boot brief at /.git/task.json. +// An absent brief selects shared-worker mode. Any other read error, malformed +// JSON, or non-positive task ID is a startup error: a container with invalid +// dedicated-task metadata must never join the shared queue. +func TaskID(mountDir string) (taskID int64, dedicated bool, err error) { + path := filepath.Join(mountDir, ".git", "task.json") + data, err := os.ReadFile(path) + if errors.Is(err, fs.ErrNotExist) { + return 0, false, nil + } + if err != nil { + return 0, false, fmt.Errorf("read %s: %w", path, err) + } + + var payload container.TaskPayload + if err := json.Unmarshal(data, &payload); err != nil { + return 0, false, fmt.Errorf("decode %s: %w", path, err) + } + if payload.ID <= 0 { + return 0, false, fmt.Errorf("decode %s: task ID must be positive, got %d", path, payload.ID) + } + return payload.ID, true, nil +} diff --git a/internal/infrastructure/agentboot/task_test.go b/internal/infrastructure/agentboot/task_test.go new file mode 100644 index 000000000..987e91ac4 --- /dev/null +++ b/internal/infrastructure/agentboot/task_test.go @@ -0,0 +1,78 @@ +package agentboot + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/samcharles93/archie-core/internal/container" +) + +func bootGitDir(t *testing.T) string { + t.Helper() + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, ".git"), 0o700); err != nil { + t.Fatal(err) + } + return dir +} + +func TestTaskIDDistinguishesSharedDedicatedAndInvalidBoots(t *testing.T) { + tests := []struct { + name string + payload []byte + write bool + wantID int64 + wantOK bool + wantErr bool + }{ + {name: "absent uses shared mode"}, + {name: "valid uses dedicated mode", payload: mustTaskPayload(t, container.TaskPayload{ID: 42}), write: true, wantID: 42, wantOK: true}, + {name: "malformed fails closed", payload: []byte("not json"), write: true, wantErr: true}, + {name: "zero ID fails closed", payload: mustTaskPayload(t, container.TaskPayload{}), write: true, wantErr: true}, + {name: "negative ID fails closed", payload: mustTaskPayload(t, container.TaskPayload{ID: -1}), write: true, wantErr: true}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + dir := bootGitDir(t) + if test.write { + if err := os.WriteFile(filepath.Join(dir, ".git", "task.json"), test.payload, 0o600); err != nil { + t.Fatal(err) + } + } + id, ok, err := TaskID(dir) + if (err != nil) != test.wantErr { + t.Fatalf("TaskID() error = %v, wantErr %v", err, test.wantErr) + } + if id != test.wantID || ok != test.wantOK { + t.Fatalf("TaskID() = (%d, %v), want (%d, %v)", id, ok, test.wantID, test.wantOK) + } + }) + } +} + +func TestTaskIDFailsClosedOnNonMissingReadError(t *testing.T) { + dir := bootGitDir(t) + path := filepath.Join(dir, ".git", "task.json") + if err := os.Mkdir(path, 0o700); err != nil { + t.Fatal(err) + } + + id, dedicated, err := TaskID(dir) + if err == nil { + t.Fatal("TaskID() error = nil for unreadable task.json path") + } + if id != 0 || dedicated { + t.Fatalf("TaskID() = (%d, %v), want (0, false)", id, dedicated) + } +} + +func mustTaskPayload(t *testing.T, payload container.TaskPayload) []byte { + t.Helper() + data, err := json.Marshal(payload) + if err != nil { + t.Fatal(err) + } + return data +} diff --git a/cmd/archie-agent/gitsafe.go b/internal/infrastructure/agentgit/safety.go similarity index 82% rename from cmd/archie-agent/gitsafe.go rename to internal/infrastructure/agentgit/safety.go index b14b2da30..d2fb1c661 100644 --- a/cmd/archie-agent/gitsafe.go +++ b/internal/infrastructure/agentgit/safety.go @@ -1,4 +1,4 @@ -package main +package agentgit import ( "context" @@ -17,7 +17,12 @@ func runGit(ctx context.Context, args ...string) ([]byte, error) { return exec.CommandContext(ctx, "git", args...).CombinedOutput() } -// markWorktreeSafe adds mountDir to git's global safe.directory list. +// MarkSafe adds mountDir to git's global safe.directory list on a best-effort basis. +func MarkSafe(ctx context.Context, mountDir string, log *slog.Logger) bool { + return markSafe(ctx, mountDir, runGit, log) +} + +// markSafe adds mountDir to git's global safe.directory list. // // The daemon bind-mounts the task worktree into the container from the host, // so the directory is owned by a UID the container user is not. Git refuses to @@ -32,7 +37,7 @@ func runGit(ctx context.Context, args ...string) ([]byte, error) { // The mount is checked first so that running archie-agent directly on a host // (shared queue-group mode, no container) does not write to the operator's own // ~/.gitconfig for a path that was never mounted. -func markWorktreeSafe(ctx context.Context, mountDir string, run gitRunner, log *slog.Logger) bool { +func markSafe(ctx context.Context, mountDir string, run gitRunner, log *slog.Logger) bool { info, err := os.Stat(mountDir) if err != nil || !info.IsDir() { log.Debug("git safe.directory skipped: no worktree mounted", "dir", mountDir) diff --git a/cmd/archie-agent/gitsafe_test.go b/internal/infrastructure/agentgit/safety_test.go similarity index 64% rename from cmd/archie-agent/gitsafe_test.go rename to internal/infrastructure/agentgit/safety_test.go index 8451f45e9..10aa77b84 100644 --- a/cmd/archie-agent/gitsafe_test.go +++ b/internal/infrastructure/agentgit/safety_test.go @@ -1,11 +1,8 @@ -package main +package agentgit import ( "context" "errors" - "go/ast" - "go/parser" - "go/token" "log/slog" "os" "path/filepath" @@ -13,38 +10,6 @@ import ( "testing" ) -func TestMainConfiguresWorktreeSafetyOnce(t *testing.T) { - file, err := parser.ParseFile(token.NewFileSet(), "main.go", nil, 0) - if err != nil { - t.Fatal(err) - } - - var markCalls, inlineCalls int - ast.Inspect(file, func(node ast.Node) bool { - call, ok := node.(*ast.CallExpr) - if !ok { - return true - } - if ident, ok := call.Fun.(*ast.Ident); ok && ident.Name == "markWorktreeSafe" { - markCalls++ - } - for _, arg := range call.Args { - literal, ok := arg.(*ast.BasicLit) - if ok && literal.Kind == token.STRING && literal.Value == `"safe.directory"` { - inlineCalls++ - } - } - return true - }) - - if markCalls != 1 { - t.Fatalf("main.go calls markWorktreeSafe %d times, want 1", markCalls) - } - if got := markCalls + inlineCalls; got != 1 { - t.Fatalf("main.go configures safe.directory %d times, want one authoritative setup path", got) - } -} - func TestMarkWorktreeSafe(t *testing.T) { log := slog.New(slog.DiscardHandler) @@ -95,10 +60,10 @@ func TestMarkWorktreeSafe(t *testing.T) { return []byte("output"), tc.runErr } - got := markWorktreeSafe(context.Background(), dir, run, log) + got := markSafe(context.Background(), dir, run, log) if got != tc.want { - t.Fatalf("markWorktreeSafe() = %v, want %v", got, tc.want) + t.Fatalf("markSafe() = %v, want %v", got, tc.want) } if len(runs) != tc.wantRuns { t.Fatalf("git invoked %d times, want %d (%v)", len(runs), tc.wantRuns, runs) @@ -129,8 +94,8 @@ func TestMarkWorktreeSafeIgnoresNonDirectory(t *testing.T) { return nil, nil } - if got := markWorktreeSafe(context.Background(), file, run, slog.New(slog.DiscardHandler)); got { - t.Fatal("markWorktreeSafe() = true for a non-directory mount path, want false") + if got := markSafe(context.Background(), file, run, slog.New(slog.DiscardHandler)); got { + t.Fatal("markSafe() = true for a non-directory mount path, want false") } if invoked { t.Fatal("git was invoked for a non-directory mount path") diff --git a/internal/infrastructure/agenttransport/nats/architecture_test.go b/internal/infrastructure/agenttransport/nats/architecture_test.go new file mode 100644 index 000000000..2ce11f76b --- /dev/null +++ b/internal/infrastructure/agenttransport/nats/architecture_test.go @@ -0,0 +1,35 @@ +package nats + +import ( + "go/parser" + "go/token" + "path/filepath" + "strconv" + "strings" + "testing" +) + +func TestInfrastructureDoesNotImportApplication(t *testing.T) { + files, err := filepath.Glob("*.go") + if err != nil { + t.Fatal(err) + } + for _, path := range files { + if strings.HasSuffix(path, "_test.go") { + continue + } + file, err := parser.ParseFile(token.NewFileSet(), path, nil, parser.ImportsOnly) + if err != nil { + t.Fatal(err) + } + for _, imported := range file.Imports { + name, err := strconv.Unquote(imported.Path.Value) + if err != nil { + t.Fatal(err) + } + if strings.Contains(name, "/internal/app/") { + t.Errorf("%s imports application package %q", path, name) + } + } + } +} diff --git a/internal/infrastructure/agenttransport/nats/transport.go b/internal/infrastructure/agenttransport/nats/transport.go new file mode 100644 index 000000000..d98b0060c --- /dev/null +++ b/internal/infrastructure/agenttransport/nats/transport.go @@ -0,0 +1,328 @@ +// Package nats implements the agent worker's broker and RPC boundaries over NATS. +// Raw NATS SDK types and wire encoding stay in this infrastructure package. +package nats + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "log/slog" + "strconv" + "strings" + "time" + + natsio "github.com/nats-io/nats.go" + + "github.com/samcharles93/archie-core/internal/agentexec" + "github.com/samcharles93/archie-core/internal/domain/workflow" + "github.com/samcharles93/archie-core/internal/domain/workintake" + "github.com/samcharles93/archie-core/internal/eventbus" + "github.com/samcharles93/archie-core/internal/forgerpc" + busnats "github.com/samcharles93/archie-core/internal/infrastructure/eventbus/nats" + "github.com/samcharles93/archie-core/internal/store" + "github.com/samcharles93/archie-core/internal/storerpc" + "github.com/samcharles93/archie-core/internal/taskrun" + "github.com/samcharles93/archie-core/internal/worktreerpc" +) + +// Config contains the process and deployment inputs for the worker transport. +// Stream subjects, filtering, and delivery timing are transport invariants. +type Config struct { + URL string + Token string + ConsumerName string +} + +const ( + stagePollTimeout = 5 * time.Second + stageAckWait = 30 * time.Minute + taskSubscriptionFlushTimeout = 5 * time.Second +) + +// ErrCoreConnectionUnavailable identifies failure to obtain the core-NATS +// connection after the durable bus connected successfully. +var ErrCoreConnectionUnavailable = errors.New("core NATS connection unavailable") + +type sdkSubscription interface { + Unsubscribe() error +} + +// Transport owns the worker's durable event bus and its shared core-NATS connection. +type Transport struct { + bus *busnats.Client + conn *natsio.Conn + + subscribe func(string, natsio.MsgHandler) (sdkSubscription, error) + queueSubscribe func(string, string, natsio.MsgHandler) (sdkSubscription, error) + flush func(time.Duration) error +} + +type busConnector func(context.Context, busnats.Config, *slog.Logger) (*busnats.Client, error) + +// Connect establishes both worker transports over one NATS connection. +func Connect(ctx context.Context, config Config, log *slog.Logger) (*Transport, error) { + return connect(ctx, config, log, busnats.Connect) +} + +func connect(ctx context.Context, config Config, log *slog.Logger, connectBus busConnector) (*Transport, error) { + bus, err := connectBus(ctx, busnats.Config{ + URL: config.URL, + Token: config.Token, + Subjects: []string{workintake.SubjectTaskWildcard, agentexec.SubjectAgentWildcard}, + ConsumerName: config.ConsumerName, + FilterSubject: agentexec.SubjectAgentWildcard, + PollTimeout: stagePollTimeout, + AckWait: stageAckWait, + }, log) + if err != nil { + return nil, err + } + return transportFromBus(bus) +} + +func transportFromBus(bus *busnats.Client) (*Transport, error) { + conn, err := bus.CoreConn() + if err != nil { + bus.Close() + return nil, fmt.Errorf("%w: %w", ErrCoreConnectionUnavailable, err) + } + return &Transport{ + bus: bus, + conn: conn, + subscribe: func(subject string, handler natsio.MsgHandler) (sdkSubscription, error) { + return conn.Subscribe(subject, handler) + }, + queueSubscribe: func(subject, queue string, handler natsio.MsgHandler) (sdkSubscription, error) { + return conn.QueueSubscribe(subject, queue, handler) + }, + flush: conn.FlushTimeout, + }, nil +} + +// Close releases all broker resources. +func (t *Transport) Close() { t.bus.Close() } + +type stageMessage struct { + wire eventbus.Message + request agentexec.AgentRequestMessage + requestErr error + replyAddress string + encode func(any) ([]byte, error) + respond func(context.Context, string, []byte) error +} + +var _ agentexec.StageMessage = (*stageMessage)(nil) + +// FetchStage receives and decodes the next durable single-stage request while +// retaining its reply inbox and acknowledgement controls inside infrastructure. +func (t *Transport) FetchStage(ctx context.Context) (agentexec.StageMessage, error) { + wire, err := t.bus.Fetch(ctx) + if err != nil { + return nil, err + } + return newStageMessage(wire, t.bus.Respond), nil +} + +func newStageMessage(wire eventbus.Message, respond func(context.Context, string, []byte) error) agentexec.StageMessage { + message := &stageMessage{wire: wire, encode: json.Marshal, respond: respond} + if err := json.Unmarshal(wire.Data(), &message.request); err != nil { + message.requestErr = fmt.Errorf("decode request: %w", err) + return message + } + var err error + message.replyAddress, err = wire.ReplyAddress() + if err != nil { + message.requestErr = fmt.Errorf("stage request on %s: %w", wire.Subject(), err) + } + return message +} + +func (m *stageMessage) Request() (agentexec.AgentRequestMessage, error) { + return m.request, m.requestErr +} + +func (m *stageMessage) Respond(ctx context.Context, response *agentexec.AgentResponseEnvelope) error { + data, err := m.encode(response) + if err != nil { + return fmt.Errorf("encode response: %w", err) + } + if err := m.respond(ctx, m.replyAddress, data); err != nil { + return fmt.Errorf("publish response: %w", err) + } + return nil +} + +func (m *stageMessage) Ack() error { return m.wire.Ack() } +func (m *stageMessage) Nak() error { return m.wire.Nak() } + +func (m *stageMessage) LogAttributes() []any { + if m.replyAddress == "" { + return nil + } + return []any{"reply_to", m.replyAddress} +} + +// LogPublisher returns the narrow fire-and-forget capability used by system logs. +func (t *Transport) LogPublisher() agentexec.LogPublisher { return t.conn } + +// RemoteTrees is the proxied half of workflow.Trees. +type RemoteTrees interface { + Prepare(ctx context.Context, owner, repo, base string, issue int, title, body, labels string) (dir, branch string, err error) + Push(ctx context.Context, owner, repo string, issue int, branch string) error +} + +// The top-level forgerpc, storerpc, and worktreerpc packages remain legacy +// infrastructure with an open final destination. This adapter is their single +// worker composition point until that broader migration is approved. + +// Forger constructs the identity-scoped forge RPC client. +func (t *Transport) Forger(identity string, timeout time.Duration) workflow.Forger { + return &forgerpc.Client{Conn: t.conn, Timeout: timeout, Identity: identity} +} + +// Store constructs the workflow store RPC client. +func (t *Transport) Store(timeout time.Duration) store.WorkflowStore { + return &storerpc.Client{Conn: t.conn, Timeout: timeout} +} + +// Trees constructs the identity-scoped worktree RPC client. +func (t *Transport) Trees(identity string, timeout time.Duration) RemoteTrees { + return &worktreerpc.Client{Conn: t.conn, Timeout: timeout, Identity: identity} +} + +// TaskHandler executes one decoded full-task request. +type TaskHandler func(context.Context, taskrun.Request) (*taskrun.Response, error) + +// Subscription is a taskrun subscription owned by the worker lifecycle. +type Subscription interface { + Close() error +} + +type subscription struct{ sub sdkSubscription } + +func (s subscription) Close() error { return s.sub.Unsubscribe() } + +const ( + taskRunSubjectPrefix = "archie.taskrun." + taskRunSubjectWildcard = taskRunSubjectPrefix + ">" + taskRunQueueGroup = "archie-taskrun-workers" +) + +// SubjectForTask returns the canonical core-NATS subject for one full-task handoff. +func SubjectForTask(taskID int64) string { + return fmt.Sprintf("%s%d", taskRunSubjectPrefix, taskID) +} + +func taskIDFromSubject(subject string) (int64, error) { + value, ok := strings.CutPrefix(subject, taskRunSubjectPrefix) + if !ok || value == "" { + return 0, fmt.Errorf("invalid taskrun subject %q", subject) + } + taskID, err := strconv.ParseInt(value, 10, 64) + if err != nil || taskID <= 0 || SubjectForTask(taskID) != subject { + return 0, fmt.Errorf("invalid taskrun subject %q", subject) + } + return taskID, nil +} + +// SubscribeTasks selects a dedicated task subject when a boot ID is present, +// otherwise the shared worker queue. Wire decoding and inbox replies remain here. +func (t *Transport) SubscribeTasks(ctx context.Context, taskID int64, dedicated bool, handler TaskHandler, log *slog.Logger) (Subscription, error) { + callback := func(msg *natsio.Msg) { t.handleTask(ctx, msg, taskID, dedicated, handler, log) } + subscribe := t.subscribe + if subscribe == nil { + subscribe = func(subject string, handler natsio.MsgHandler) (sdkSubscription, error) { + return t.conn.Subscribe(subject, handler) + } + } + queueSubscribe := t.queueSubscribe + if queueSubscribe == nil { + queueSubscribe = func(subject, queue string, handler natsio.MsgHandler) (sdkSubscription, error) { + return t.conn.QueueSubscribe(subject, queue, handler) + } + } + flush := t.flush + if flush == nil { + flush = t.conn.FlushTimeout + } + + var ( + sub sdkSubscription + err error + ) + if dedicated { + log.Info("taskrun: dedicated per-task subscription", "task", taskID) + sub, err = subscribe(SubjectForTask(taskID), callback) + } else { + log.Info("taskrun: shared queue-group subscription (no .git/task.json found)") + sub, err = queueSubscribe(taskRunSubjectWildcard, taskRunQueueGroup, callback) + } + if err != nil { + return nil, err + } + if err := flush(taskSubscriptionFlushTimeout); err != nil { + if cleanupErr := sub.Unsubscribe(); cleanupErr != nil { + log.Warn("task subscription cleanup failed after flush failure", "err", cleanupErr) + } + return nil, fmt.Errorf("flush task subscription: %w", err) + } + return subscription{sub: sub}, nil +} + +func (t *Transport) handleTask( + ctx context.Context, + msg *natsio.Msg, + bootTaskID int64, + dedicated bool, + handler TaskHandler, + log *slog.Logger, +) { + var request taskrun.Request + if err := json.Unmarshal(msg.Data, &request); err != nil { + log.Error("taskrun decode failed", "err", err) + t.respondTask(msg, nil, fmt.Errorf("decode taskrun request: %w", err), log) + return + } + if err := validateTaskRequest(msg.Subject, request, bootTaskID, dedicated); err != nil { + log.Error("taskrun validation failed", "subject", msg.Subject, "err", err) + t.respondTask(msg, nil, fmt.Errorf("validate taskrun request: %w", err), log) + return + } + response, err := handler(ctx, request) + t.respondTask(msg, response, err, log) +} + +func validateTaskRequest(subject string, request taskrun.Request, bootTaskID int64, dedicated bool) error { + subjectTaskID, err := taskIDFromSubject(subject) + if err != nil { + return err + } + if err := request.Validate(); err != nil { + return err + } + if request.Task.ID != subjectTaskID { + return fmt.Errorf("task ID %d does not match subject task ID %d", request.Task.ID, subjectTaskID) + } + if dedicated && subjectTaskID != bootTaskID { + return fmt.Errorf("task ID %d does not match dedicated boot task ID %d", subjectTaskID, bootTaskID) + } + return nil +} + +func (*Transport) respondTask(msg *natsio.Msg, response *taskrun.Response, runErr error, log *slog.Logger) { + if response == nil { + response = &taskrun.Response{} + } + if runErr != nil { + response.Error = runErr.Error() + } + data, err := json.Marshal(response) + if err != nil { + log.Error("taskrun encode response failed", "err", err) + return + } + if err := msg.Respond(data); err != nil { + log.Error("taskrun respond failed", "err", err) + } +} diff --git a/internal/infrastructure/agenttransport/nats/transport_test.go b/internal/infrastructure/agenttransport/nats/transport_test.go new file mode 100644 index 000000000..880d9bbd7 --- /dev/null +++ b/internal/infrastructure/agenttransport/nats/transport_test.go @@ -0,0 +1,460 @@ +package nats + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "log/slog" + "reflect" + "strings" + "testing" + "time" + + "github.com/nats-io/nats-server/v2/server" + natstest "github.com/nats-io/nats-server/v2/test" + natsio "github.com/nats-io/nats.go" + + "github.com/samcharles93/archie-core/internal/agentexec" + "github.com/samcharles93/archie-core/internal/domain/workintake" + "github.com/samcharles93/archie-core/internal/forgerpc" + busnats "github.com/samcharles93/archie-core/internal/infrastructure/eventbus/nats" + "github.com/samcharles93/archie-core/internal/store" + "github.com/samcharles93/archie-core/internal/storerpc" + "github.com/samcharles93/archie-core/internal/taskrun" + "github.com/samcharles93/archie-core/internal/worktreerpc" +) + +func testTransport(t *testing.T) (*Transport, *natsio.Conn) { + t.Helper() + srv := natstest.RunRandClientPortServer() + t.Cleanup(srv.Shutdown) + conn, err := natsio.Connect(srv.ClientURL()) + if err != nil { + t.Fatal(err) + } + t.Cleanup(conn.Close) + return &Transport{conn: conn}, conn +} + +func TestTaskRunSubjectOwnsCanonicalNATSTopology(t *testing.T) { + if got := SubjectForTask(42); got != "archie.taskrun.42" { + t.Fatalf("SubjectForTask(42) = %q", got) + } + + for _, test := range []struct { + name string + subject string + wantID int64 + wantErr bool + }{ + {name: "canonical", subject: "archie.taskrun.42", wantID: 42}, + {name: "missing prefix", subject: "taskrun.42", wantErr: true}, + {name: "missing ID", subject: "archie.taskrun.", wantErr: true}, + {name: "nondecimal", subject: "archie.taskrun.any", wantErr: true}, + {name: "zero", subject: "archie.taskrun.0", wantErr: true}, + {name: "negative", subject: "archie.taskrun.-1", wantErr: true}, + {name: "noncanonical leading zero", subject: "archie.taskrun.042", wantErr: true}, + {name: "extra token", subject: "archie.taskrun.42.extra", wantErr: true}, + } { + t.Run(test.name, func(t *testing.T) { + got, err := taskIDFromSubject(test.subject) + if test.wantErr { + if err == nil { + t.Fatalf("taskIDFromSubject(%q) = %d, want error", test.subject, got) + } + return + } + if err != nil || got != test.wantID { + t.Fatalf("taskIDFromSubject(%q) = (%d, %v), want (%d, nil)", test.subject, got, err, test.wantID) + } + }) + } +} + +func TestSubscribeTasksPreservesDedicatedAndSharedSubjects(t *testing.T) { + tests := []struct { + name string + taskID int64 + dedicated bool + subject string + requestID int64 + }{ + {name: "dedicated", taskID: 42, dedicated: true, subject: SubjectForTask(42), requestID: 42}, + {name: "shared queue", subject: SubjectForTask(7), requestID: 7}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + transport, conn := testTransport(t) + calls := 0 + subscription, err := transport.SubscribeTasks(t.Context(), test.taskID, test.dedicated, func(_ context.Context, request taskrun.Request) (*taskrun.Response, error) { + calls++ + if request.Task.ID != test.requestID { + t.Fatalf("task ID = %d, want %d", request.Task.ID, test.requestID) + } + return &taskrun.Response{Status: "done"}, nil + }, slog.New(slog.DiscardHandler)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = subscription.Close() }) + + payload, err := json.Marshal(taskrun.Request{Task: &store.Task{ID: test.requestID}}) + if err != nil { + t.Fatal(err) + } + message, err := conn.Request(test.subject, payload, time.Second) + if err != nil { + t.Fatal(err) + } + var response taskrun.Response + if err := json.Unmarshal(message.Data, &response); err != nil { + t.Fatal(err) + } + if calls != 1 || response.Status != "done" { + t.Fatalf("handler/response = (%d, %q), want (1, done)", calls, response.Status) + } + }) + } +} + +func TestSubscribeTasksOwnsWireErrors(t *testing.T) { + transport, conn := testTransport(t) + handlerCalls := 0 + subscription, err := transport.SubscribeTasks(t.Context(), 9, true, func(context.Context, taskrun.Request) (*taskrun.Response, error) { + handlerCalls++ + return nil, errors.New("run failed") + }, slog.New(slog.DiscardHandler)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = subscription.Close() }) + + for _, test := range []struct { + name string + payload []byte + want string + wantHandlerCalls int + }{ + {name: "decode", payload: []byte("not json"), want: "decode taskrun request:"}, + {name: "missing task", payload: []byte(`{}`), want: "validate taskrun request: task is required"}, + {name: "null task", payload: []byte(`{"task":null}`), want: "validate taskrun request: task is required"}, + {name: "nonpositive task ID", payload: []byte(`{"task":{"id":0}}`), want: "validate taskrun request: task ID must be positive, got 0"}, + {name: "payload subject mismatch", payload: []byte(`{"task":{"id":8}}`), want: "validate taskrun request: task ID 8 does not match subject task ID 9"}, + {name: "handler", payload: []byte(`{"task":{"id":9}}`), want: "run failed", wantHandlerCalls: 1}, + } { + t.Run(test.name, func(t *testing.T) { + before := handlerCalls + message, err := conn.Request(SubjectForTask(9), test.payload, time.Second) + if err != nil { + t.Fatal(err) + } + var response taskrun.Response + if err := json.Unmarshal(message.Data, &response); err != nil { + t.Fatal(err) + } + if response.Error == "" || !strings.Contains(response.Error, test.want) { + t.Fatalf("response error = %q, want containing %q", response.Error, test.want) + } + if got := handlerCalls - before; got != test.wantHandlerCalls { + t.Fatalf("handler calls = %d, want %d", got, test.wantHandlerCalls) + } + }) + } +} + +func TestHandleTaskRejectsSubjectCorrelationErrors(t *testing.T) { + for _, test := range []struct { + name string + subject string + bootTaskID int64 + dedicated bool + requestID int64 + want string + }{ + { + name: "malformed subject", + subject: "archie.taskrun.invalid", + requestID: 7, + want: `validate taskrun request: invalid taskrun subject "archie.taskrun.invalid"`, + }, + { + name: "dedicated boot mismatch", + subject: SubjectForTask(9), + bootTaskID: 42, + dedicated: true, + requestID: 9, + want: "validate taskrun request: task ID 9 does not match dedicated boot task ID 42", + }, + } { + t.Run(test.name, func(t *testing.T) { + transport, conn := testTransport(t) + handlerCalls := 0 + subscription, err := conn.Subscribe(test.subject, func(message *natsio.Msg) { + transport.handleTask(t.Context(), message, test.bootTaskID, test.dedicated, func(context.Context, taskrun.Request) (*taskrun.Response, error) { + handlerCalls++ + return &taskrun.Response{Status: "unexpected"}, nil + }, slog.New(slog.DiscardHandler)) + }) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = subscription.Unsubscribe() }) + if err := conn.Flush(); err != nil { + t.Fatal(err) + } + + payload, err := json.Marshal(taskrun.Request{Task: &store.Task{ID: test.requestID}}) + if err != nil { + t.Fatal(err) + } + message, err := conn.Request(test.subject, payload, time.Second) + if err != nil { + t.Fatal(err) + } + var response taskrun.Response + if err := json.Unmarshal(message.Data, &response); err != nil { + t.Fatal(err) + } + if response.Error != test.want { + t.Fatalf("response error = %q, want %q", response.Error, test.want) + } + if handlerCalls != 0 { + t.Fatalf("handler calls = %d, want 0", handlerCalls) + } + }) + } +} + +type recordingSDKSubscription struct { + unsubscribeCalls int + unsubscribeErr error +} + +func (s *recordingSDKSubscription) Unsubscribe() error { + s.unsubscribeCalls++ + return s.unsubscribeErr +} + +func TestSubscribeTasksReportsSetupFailuresAndCleansUpFlushFailure(t *testing.T) { + subscribeErr := errors.New("subscribe failed") + queueErr := errors.New("queue subscribe failed") + flushErr := errors.New("flush failed") + cleanupErr := errors.New("unsubscribe failed") + tests := []struct { + name string + dedicated bool + subscribeErr error + queueErr error + flushErr error + cleanupErr error + wantErr error + wantUnsubscribes int + wantCleanupLog bool + }{ + {name: "direct subscribe", dedicated: true, subscribeErr: subscribeErr, wantErr: subscribeErr}, + {name: "queue subscribe", queueErr: queueErr, wantErr: queueErr}, + {name: "flush cleanup succeeds", dedicated: true, flushErr: flushErr, wantErr: flushErr, wantUnsubscribes: 1}, + {name: "flush cleanup fails", dedicated: true, flushErr: flushErr, cleanupErr: cleanupErr, wantErr: flushErr, wantUnsubscribes: 1, wantCleanupLog: true}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + sub := &recordingSDKSubscription{unsubscribeErr: test.cleanupErr} + transport := &Transport{ + subscribe: func(string, natsio.MsgHandler) (sdkSubscription, error) { + return sub, test.subscribeErr + }, + queueSubscribe: func(string, string, natsio.MsgHandler) (sdkSubscription, error) { + return sub, test.queueErr + }, + flush: func(timeout time.Duration) error { + if timeout != taskSubscriptionFlushTimeout { + t.Fatalf("flush timeout = %s, want %s", timeout, taskSubscriptionFlushTimeout) + } + return test.flushErr + }, + } + var output bytes.Buffer + _, err := transport.SubscribeTasks(t.Context(), 42, test.dedicated, func(context.Context, taskrun.Request) (*taskrun.Response, error) { + t.Fatal("handler called") + return nil, nil + }, slog.New(slog.NewTextHandler(&output, nil))) + if !errors.Is(err, test.wantErr) { + t.Fatalf("error = %v, want %v", err, test.wantErr) + } + if sub.unsubscribeCalls != test.wantUnsubscribes { + t.Fatalf("unsubscribe calls = %d, want %d", sub.unsubscribeCalls, test.wantUnsubscribes) + } + if got := strings.Contains(output.String(), "task subscription cleanup failed after flush failure"); got != test.wantCleanupLog { + t.Fatalf("cleanup warning logged = %v, want %v; output %q", got, test.wantCleanupLog, output.String()) + } + }) + } +} + +func TestConnectOwnsWorkerTopology(t *testing.T) { + connectErr := errors.New("stop after config capture") + var got busnats.Config + _, err := connect(t.Context(), Config{ + URL: "nats://worker.test:4222", + Token: "secret", + ConsumerName: "agent-worker", + }, slog.New(slog.DiscardHandler), func(_ context.Context, config busnats.Config, _ *slog.Logger) (*busnats.Client, error) { + got = config + return nil, connectErr + }) + if !errors.Is(err, connectErr) { + t.Fatalf("connect error = %v, want %v", err, connectErr) + } + if got.URL != "nats://worker.test:4222" || got.Token != "secret" || got.ConsumerName != "agent-worker" { + t.Fatalf("deployment config = %#v", got) + } + wantSubjects := []string{workintake.SubjectTaskWildcard, agentexec.SubjectAgentWildcard} + if !reflect.DeepEqual(got.Subjects, wantSubjects) { + t.Fatalf("subjects = %v, want %v", got.Subjects, wantSubjects) + } + if got.FilterSubject != agentexec.SubjectAgentWildcard || got.PollTimeout != stagePollTimeout || got.AckWait != stageAckWait { + t.Fatalf("topology config = %#v, want filter=%q poll=%s ack=%s", got, agentexec.SubjectAgentWildcard, stagePollTimeout, stageAckWait) + } +} + +type rawStageMessage struct { + data []byte + subject string + reply string + replyErr error + ackErr error + nakErr error + ackCalls int + nakCalls int +} + +func (m *rawStageMessage) Data() []byte { return m.data } +func (m *rawStageMessage) Subject() string { return m.subject } +func (m *rawStageMessage) ReplyAddress() (string, error) { return m.reply, m.replyErr } +func (m *rawStageMessage) Ack() error { m.ackCalls++; return m.ackErr } +func (m *rawStageMessage) Nak() error { m.nakCalls++; return m.nakErr } + +func TestStageMessageOwnsDecodeReplyAndAcknowledgement(t *testing.T) { + replyErr := errors.New("no reply") + for _, test := range []struct { + name string + wire *rawStageMessage + wantErr string + }{ + {name: "decode", wire: &rawStageMessage{data: []byte("not json"), subject: "archie.agent.bad"}, wantErr: "decode request:"}, + {name: "reply", wire: &rawStageMessage{data: []byte(`{"task_id":7}`), subject: "archie.agent.no-reply", replyErr: replyErr}, wantErr: "stage request on archie.agent.no-reply:"}, + {name: "typed request", wire: &rawStageMessage{data: []byte(`{"task_id":7,"stage":"test"}`), reply: "_INBOX.7"}}, + } { + t.Run(test.name, func(t *testing.T) { + message := newStageMessage(test.wire, func(context.Context, string, []byte) error { return nil }) + request, err := message.Request() + if test.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), test.wantErr) { + t.Fatalf("Request error = %v, want containing %q", err, test.wantErr) + } + } else if err != nil || request.TaskID != 7 || request.Stage != "test" { + t.Fatalf("Request = (%#v, %v), want typed task 7/test", request, err) + } + if err := message.Ack(); err != nil { + t.Fatal(err) + } + if err := message.Nak(); err != nil { + t.Fatal(err) + } + if test.wire.ackCalls != 1 || test.wire.nakCalls != 1 { + t.Fatalf("ack/nak calls = (%d,%d), want (1,1)", test.wire.ackCalls, test.wire.nakCalls) + } + }) + } +} + +func TestStageMessageDelegatesAcknowledgementErrors(t *testing.T) { + ackErr := errors.New("ack failed") + nakErr := errors.New("nak failed") + wire := &rawStageMessage{data: []byte(`{"task_id":7}`), reply: "_INBOX.7", ackErr: ackErr, nakErr: nakErr} + message := newStageMessage(wire, func(context.Context, string, []byte) error { return nil }) + if err := message.Ack(); !errors.Is(err, ackErr) { + t.Fatalf("Ack error = %v, want %v", err, ackErr) + } + if err := message.Nak(); !errors.Is(err, nakErr) { + t.Fatalf("Nak error = %v, want %v", err, nakErr) + } +} + +func TestStageMessageOwnsResponseEncodingAndInboxDelivery(t *testing.T) { + encodeErr := errors.New("encode failed") + respondErr := errors.New("respond failed") + response := &agentexec.AgentResponseEnvelope{Result: agentexec.Result{Status: agentexec.StatusPassed}} + + wire := &rawStageMessage{data: []byte(`{"task_id":7}`), reply: "_INBOX.7"} + message := newStageMessage(wire, func(_ context.Context, reply string, data []byte) error { + if reply != "_INBOX.7" || !strings.Contains(string(data), `"status":"passed"`) { + t.Fatalf("reply/data = (%q,%s)", reply, data) + } + return nil + }) + if err := message.Respond(t.Context(), response); err != nil { + t.Fatal(err) + } + + concrete, ok := message.(*stageMessage) + if !ok { + t.Fatalf("message type = %T, want *stageMessage", message) + } + concrete.encode = func(any) ([]byte, error) { return nil, encodeErr } + if err := concrete.Respond(t.Context(), response); !errors.Is(err, encodeErr) || !strings.Contains(err.Error(), "encode response") { + t.Fatalf("encode error = %v", err) + } + concrete.encode = json.Marshal + concrete.respond = func(context.Context, string, []byte) error { return respondErr } + if err := concrete.Respond(t.Context(), response); !errors.Is(err, respondErr) || !strings.Contains(err.Error(), "publish response") { + t.Fatalf("respond error = %v", err) + } +} + +func TestRPCFactoriesPreserveIdentityAndTimeout(t *testing.T) { + transport, _ := testTransport(t) + timeout := 60 * time.Second + forgeClient, ok := transport.Forger("identity-a", timeout).(*forgerpc.Client) + if !ok || forgeClient.Identity != "identity-a" || forgeClient.Timeout != timeout { + t.Fatalf("forge client = %#v", forgeClient) + } + storeClient, ok := transport.Store(timeout).(*storerpc.Client) + if !ok || storeClient.Timeout != timeout { + t.Fatalf("store client = %#v", storeClient) + } + treeClient, ok := transport.Trees("identity-a", timeout).(*worktreerpc.Client) + if !ok || treeClient.Identity != "identity-a" || treeClient.Timeout != timeout { + t.Fatalf("tree client = %#v", treeClient) + } +} + +func TestTransportFromBusClassifiesUnavailableCoreConnection(t *testing.T) { + transport, err := transportFromBus(&busnats.Client{}) + if transport != nil { + t.Fatalf("transport = %#v, want nil", transport) + } + if !errors.Is(err, ErrCoreConnectionUnavailable) { + t.Fatalf("error = %v, want ErrCoreConnectionUnavailable", err) + } +} + +func TestConnectOwnsJetStreamAndCoreConnection(t *testing.T) { + srv := natstest.RunRandClientPortServer() + t.Cleanup(srv.Shutdown) + if err := srv.EnableJetStream(&server.JetStreamConfig{StoreDir: t.TempDir()}); err != nil { + t.Fatal(err) + } + transport, err := Connect(t.Context(), Config{ + URL: srv.ClientURL(), + ConsumerName: "worker-test", + }, slog.New(slog.DiscardHandler)) + if err != nil { + t.Fatal(err) + } + transport.Close() + if _, err := transport.FetchStage(t.Context()); err == nil { + t.Fatal("FetchStage after Close returned nil error") + } +} diff --git a/internal/taskrun/taskrun.go b/internal/taskrun/taskrun.go index c5bf8e7cd..bfed0a721 100644 --- a/internal/taskrun/taskrun.go +++ b/internal/taskrun/taskrun.go @@ -7,6 +7,7 @@ package taskrun import ( + "errors" "fmt" "github.com/samcharles93/archie-core/internal/agentexec" @@ -14,11 +15,6 @@ import ( "github.com/samcharles93/archie-core/internal/store" ) -// SubjectForTask returns the per-task NATS subject for a full-task handoff. -func SubjectForTask(taskID int64) string { - return fmt.Sprintf("archie.taskrun.%d", taskID) -} - // Request carries everything archie-agent needs to run a task's entire // workflow itself: the task row (already claimed by archied), the repo's // gate/ecosystem config, a non-secret config snapshot, and LLM provider @@ -35,6 +31,18 @@ type Request struct { MCPServers []config.MCPServer `json:"mcp_servers,omitempty"` } +// Validate rejects a full-task request that cannot be correlated to a real +// task. Transport-specific subject correlation is enforced by the adapter. +func (r Request) Validate() error { + if r.Task == nil { + return errors.New("task is required") + } + if r.Task.ID <= 0 { + return fmt.Errorf("task ID must be positive, got %d", r.Task.ID) + } + return nil +} + // Response reports the task's final state after archie-agent runs its // workflow. Task carries the last-known field values for logging; the // authoritative state already landed in archied's store via storerpc diff --git a/internal/taskrun/taskrun_test.go b/internal/taskrun/taskrun_test.go index 57d8fedaa..698b19e31 100644 --- a/internal/taskrun/taskrun_test.go +++ b/internal/taskrun/taskrun_test.go @@ -50,6 +50,26 @@ func TestRequestJSONRoundTrip(t *testing.T) { } } +func TestRequestValidateRequiresPositiveTaskID(t *testing.T) { + for _, test := range []struct { + name string + request Request + wantErr bool + }{ + {name: "missing task", wantErr: true}, + {name: "zero ID", request: Request{Task: &store.Task{}}, wantErr: true}, + {name: "negative ID", request: Request{Task: &store.Task{ID: -1}}, wantErr: true}, + {name: "positive ID", request: Request{Task: &store.Task{ID: 1}}}, + } { + t.Run(test.name, func(t *testing.T) { + err := test.request.Validate() + if (err != nil) != test.wantErr { + t.Fatalf("Validate() error = %v, wantErr %v", err, test.wantErr) + } + }) + } +} + func TestResponseJSONRoundTrip(t *testing.T) { resp := Response{ Task: &store.Task{ID: 1, Status: store.StatusPROpen}, @@ -67,9 +87,3 @@ func TestResponseJSONRoundTrip(t *testing.T) { t.Fatalf("Response did not round-trip: %+v", got) } } - -func TestSubjectForTask(t *testing.T) { - if got := SubjectForTask(42); got != "archie.taskrun.42" { - t.Fatalf("SubjectForTask(42) = %q", got) - } -}