Commit Graph
20 Commits
Author SHA1 Message Date
sam 6ebbd3b2cf fix: add missing LinkBranch stubs to test doubles
Fix go vet/test failures by adding LinkBranch methods to stubForge, prCapturingForge, testForge, and fakeForge. Update test data from 'sam/archie' to 'acme/widget' and state label defaults from 'archie:*' to 'agent:*'. Add Taskfile.yml for project build automation. Implement container self-network detection so spawned agents join the daemon's Docker network.
2026-07-24 13:32:50 +10:00
sam d80401d595 feat: add config overlay and shared RPC/Yaegi packages
- Introduce `-config-overlay` flag and `config.LoadOverlay()` to merge a base config with an overlay file, only overriding fields the overlay sets. This enables docker-compose to mount the base config from the host and apply only deployment-specific overrides.
- Create `internal/natsrpc` package with reusable NATS request/reply primitives (Envelope, Client, Call, RegisterAll, Respond). Migrate storerpc, worktreerpc, and forgerpc to use it, eliminating duplicated marshal/unmarshal and error handling.
- Create `internal/yaegiutil` package with shared interpreter construction (stdlib + extra symbols), generic export resolution (Resolve), and panic recovery (Safe). Migrate all Yaegi-based surfaces (gate scripts, daemon plugins, skill plugins, workflow stages, skill scripts) to use it, reducing boilerplate and centralizing panic safety.
- Simplify `config.docker.toml` to contain only overlay fields (paths, container config) and remove the duplicated repos list – the base config is now the single source of truth.
- Update docker-compose.yml to pass both -config and -config-overlay flags and mount the base config from the host.
- Add tests for LoadOverlay, DockerConfigOverlaysExampleConfig, Envelope, and yaegiutil utilities.
2026-07-24 12:57:10 +10:00
sam e4368f70af feat: persistent volumes with TTL for container/worktree storage (archie-core-j7v)
Add [containers].volume_ttl (default 72h). Persistent-storage repos get a
per-repo bare Git cache (--reference-if-able --dissociate) so task
worktrees clone fast without repeated full network clones, while staying
fully independent of the cache's lifecycle — cache expiry can never
corrupt an active or parked task's worktree. Docker repo volumes are
labeled as Archie-owned and swept non-forced (a volume still attached to a
running container survives and retries next cycle); the repo cache
directory is swept by mtime. Both cleanups run at daemon startup and each
poll cycle.

Filed archie-core-756 (repo-wide golangci-lint baseline) and archie-core-340
(intermittent JetStream TempDir teardown flake) as separate follow-ups
rather than scope-creeping them into this change.
2026-07-24 10:22:11 +10:00
sam 75f237d1ad feat: daemon hands off tasks to archie-agent in sandboxed mode (archie-core-5zh.8)
daemon.process() branches on ContainerPool != nil: sandboxed tasks publish
a taskrun.Request to archie.taskrun.<id> and await the response via the
new runViaAgent method, instead of running workflow.Route/workflow.Run
locally. Non-container mode is unchanged. Every durable side effect
(Store, Forge, git push) already landed via the storerpc/forgerpc/
worktreerpc proxies during archie-agent's run by the time runViaAgent
returns — it only parks the task itself on transport/setup failure, never
duplicating a terminal state archie-agent's own workflow.Run already
recorded.

Also fixes a real correctness gap: with concurrent multi-task dispatch
(archie-core-u64/g1l) plus containerized archie-agent, a shared queue-group
subscription could let one container answer a sibling's taskrun message
while having a *different* task's worktree mounted. archie-agent now reads
its boot-time task.json and subscribes to its own exact per-task subject
when present, falling back to the shared queue group only in dev/pool mode
(no task.json).

Adds agentexec.ProvidersFromConfig, deduped from cmd/archied's former
executionProviders.

Closes the archie-core-5zh epic: archie-agent runs an entire task's
workflow per invocation instead of one stage per round trip, with all
credentials staying daemon-side throughout.
2026-07-24 09:14:56 +10:00
sam 81ce920434 feat: register storerpc/forgerpc/worktreerpc servers in archied (archie-core-5zh.9)
Add registerTaskRPCServers() in cmd/archied/main.go: subscribes all three
RPC servers on the daemon's NATS connection whenever [nats] is configured,
using the same Store/Forge/worktree.Manager instances the daemon already
owns. archie-agent containers now have a live responder for the Store,
Forge, and worktree Prepare/Push calls proxied in archie-core-5zh.1-3.

Test spins an embedded NATS server and round-trips real storerpc/forgerpc/
worktreerpc clients against the registered handlers.
2026-07-24 08:59:04 +10:00
sam 4be28e0e48 feat: optional NATS token authentication (archie-core-l1q)
Add token_env to [nats] config, resolved and validated at daemon startup
(fails early if configured but the named env var is unset). Container
workers get NATS_URL/NATS_TOKEN; archie-agent honors both while CLI-supplied
URL still takes precedence. Anonymous auth remains the default for
single-node deployments.

Unblocks archie-core-5zh (promote archie-agent to per-task process).
2026-07-24 07:51:06 +10:00
sam f8df051b19 feat: bounded concurrent task dispatch in daemon (archie-core-u64)
Daemon previously processed one task at a time. Add bounded concurrent
dispatch for SQLite and NATS agent modes, reusing [containers].max_concurrency
(0 = unlimited). Same-repository tasks stay FIFO-serialized to avoid worktree
conflicts; different repositories now run concurrently.

Unblocks archie-core-l1q (NATS transport) and archie-core-g1l
(per-repo allow_concurrent override).
2026-07-24 07:42:13 +10:00
sam edd05aea6b feat: add Gitea forge support alongside GitHub
Implements the Forge interface for Gitea instances (tested against
git.catlow.cloud, Gitea v1.26.4). The forge.New factory now dispatches
on forge type, and config validation accepts 'gitea'.

Changes:
- internal/forge/gitea.go — GiteaClient implementing all 11 Forge
  methods: issue polling (assigned + label), comments, PR creation,
  PR state reconciliation, issue closure, reactions, state labels,
  push verification. Handles edge cases: not-found PRs return
  'closed', GetUserByName errors from the Gitea SDK are mapped to
  user-friendly messages.
- internal/forge/forge.go — New() factory dispatching on forge type
  (github/gitea). GitHub constructor renamed to NewGitHub.
- internal/forge/github_test.go — updated to use New() signature
- internal/config/config.go — forge.type accepts 'gitea', BotEmail
  defaults change per forge type (gitea uses @gitea.local)
- cmd/archied/main.go — passes cfg.Forge.Type to forge.New()

Dependencies: added code.gitea.io/sdk/gitea v0.25.1

End-to-end verified against live Gitea instance: daemon starts clean,
registry built (5 workflows), web UI listening, no errors or warnings.
2026-07-22 05:15:09 +10:00
sam fd360fcc0b fix: resolve 8 remaining design-decision audit items
Fixes (4 items):
  S4 — Add TypeScript, Deno, Bun cache ecosystems to
       cacheVolumesByEcosystem. TypeScript shares node+pnpm volumes;
       Deno and Bun get dedicated cache volumes.
  R4 — Share single Docker client between container.Pool and
       storage.DockerBackend. NewPool now accepts an optional
       DockerClient in Config; if nil, creates its own. Pool
       tracks ownership (ownCli) and only closes its own client.
  S2 — Wire Storage.Teardown() call in daemon.process() after
       workflow.Run(). Docker backend is a no-op; this closes the
       interface contract for future backends.
  R5 — Emit observability event on AugmentRegistry failure.
       Previously only logged a warning; now emits a
       'registry_augment_failed' event with error detail.
       Promoted log level to Error.

Documentation (4 items):
  D1 — Document PluginRegistry as reserved for future daemon
       extension points (forge resolvers, storage backends, etc.)
  S1 — Fix /data/ layout comment to reflect actual task.json path
       (/data/worktree/task.json, not /data/task.json)
  S3 — Add TODO(PRD §3) for session.jsonl, memory.jsonl, plugins/
       volume — these need agent-side implementation first
  N6 — Document that VolumeInspectOptions{} requires moby v25+
       (go.mod pins compatible version)
  N7 — Document that _Plugin.IValue is required by Yaegi interface
       bridging; warn that go generate strips nil-guards

Test: TestCacheMountsForTypeScript verifies typescript ecosystem
returns node+pnpm volumes sorted by destination.

Verification: single client.New(client.FromEnv) in main.go;
all 21 storage tests pass; full suite green.
2026-07-22 05:00:37 +10:00
sam 04175a1a6a fix: address 9 'Now' adversarial audit findings
P1  (panic):      nil Storage guard in daemon.process()
C1  (correctness): check WriteTaskJSON error — don't start blind
C2  (correctness): move 'working' label after container setup
C3  (correctness): warn on duplicate workflow names in registry
C4  (correctness): test that generated _Plugin wrapper has nil-guards
C5  (correctness): .agents/skills as file (ENOTDIR) not fatal
C6  (correctness): ecosystem case-insensitive in cacheMounts
N1  (correctness): TOCTOU race — treat 'already exists' as success
N2  (correctness): recover from panicking plugin.Name() at startup

--- Details ---

P1: d.Storage.Setup() is now guarded by a nil check. In normal
operation Storage is always set when ContainerPool is set, but
the invariant is now explicit and enforced.

C1: container.WriteTaskJSON() error was discarded with _. Now it
returns early with a park transition — the container never starts
without a boot-time brief.

C2: SetStateLabel('working') was called before container setup.
If Acquire failed, the task was parked but the label stayed
'working' permanently. Now the label is set only after successful
container setup (or immediately if no containers are configured).

C3: BuildRegistry and AugmentRegistry now log a warning when two
skills declare the same workflow name and one overwrites the other.

C4: TestGeneratedWrapperHasNilGuards verifies that nil WName/
WVersion don't panic. If go generate strips the nil-guards, this
test catches it before the daemon crashes.

C5: skill.Catalog() now returns nil (no skills) instead of
propagating ENOTDIR errors when .agents/skills is a file.

C6: cacheMounts() normalizes ecosystem to lowercase. 'Python',
'NODE', 'Go' all resolve correctly now.

N1: ensureVolume() treats Docker's 'already exists' error as
success, closing the TOCTOU window between VolumeInspect and
VolumeCreate.

N2: safePluginInfo() wraps plugin.Name()/Version() calls in a
recover. A plugin with a panicking Name() no longer crashes the
daemon at startup.
2026-07-22 04:50:11 +10:00
sam 7dbb17a4cb feat: pluggable storage backends with Docker volumes and /data/ layout
Introduces the storage.Backend interface for pluggable container mount
management. The Docker implementation (DockerBackend) is the MVP —
no NFS/SMB/S3, just Docker volumes + bind mounts.

/data/ volume layout:
  /data/worktree/  — bind mount of host worktree (always)
  /data/repo/      — per-repo Docker volume (when persistent_storage = true)
  /data/cache/go/  — shared GOMODCACHE volume
  /data/cache/node/— shared npm cache volume
  /data/cache/pnpm/— shared pnpm store volume
  /data/cache/pip/ — shared pip cache volume
  /data/cache/cargo/— shared Rust cargo cache volume
  /data/task.json  — boot-time brief (existing WriteTaskJSON)

Changes:
- internal/storage/: Backend interface, DockerBackend, Mount types,
  ecosystem-to-cache-volume mapping, ConvertMounts helper
- container.Pool.Acquire: now accepts []storage.Mount instead of
  hardcoding a single bind mount — mount list comes from the storage
  backend
- daemon.process(): calls d.Storage.Setup() to build the mount list,
  passes Ecosystem + PersistentStorage from repo config
- config.Repo.PersistentStorage: when true, a named Docker volume
  (archie-repo-<owner>-<repo>) is mounted at /data/repo and survives
  task completion
- main.go: creates DockerBackend when containers are enabled

21 tests: interface contract, ecosystem cache mounts (go/node/
python/rust/custom), worktree bind mount ordering, persistent
storage on/off/empty-owner guard, mount list sort+dedup, adversarial
(nonexistent worktree, empty ecosystem, unknown ecosystem).

Why: daemon-managed cache volumes mean warm GOMODCACHE/npm cache
across tasks. Per-repo persistent volumes preserve expensive build
artifacts. The interface leaves room for future backends without
rewiring container orchestration.
2026-07-22 04:28:30 +10:00
sam f38ba5d515 feat: wire Layer 2 daemon plugin loader into archied startup
Daemon plugins (Yaegi-interpreted .go files) are now loaded from the
configured plugin_dir at startup. Each file must export a 'Plugin'
variable using the _Plugin wrapper struct, which bridges Yaegi types
to the Go plugin.Plugin interface.

Changes:
- internal/plugin/pluginextract/ — generated Yaegi symbol table for
  the plugin package (mirrors wfextract pattern), including the
  _Plugin interface wrapper that lets interpreted code implement
  Go interfaces
- LoadDir now accepts optional symbol tables via variadic parameter,
  avoiding an import cycle between plugin and pluginextract
- Plugin tests rewritten as behavioral (not structural), verifying
  actual Yaegi interpretation of valid, invalid, and missing plugins
- config.PluginDir field for specifying the plugin directory
- daemon.PluginRegistry field for holding loaded plugins
- main.go wires plugin loading: LoadDir at startup, register into
  daemon, log each loaded plugin

Why: the Plugin interface and LoadDir existed but weren't connected
to the daemon. Skills define workflows; daemon plugins are the
extension point for forge resolvers, storage backends, and other
core-daemon concerns.
2026-07-22 04:11:02 +10:00
sam 8958a82233 feat: replace hardcoded workflow registry with catalog-driven skillbuild.BuildRegistry()
The daemon no longer hardcodes workflow.Registry{...} in main.go.
Instead, skillbuild.BuildRegistry() scans .agents/skills/ for catalog
entries declaring metadata.archie.workflow and builds each workflow
from its Yaegi-interpreted stage plugins.

Built-in workflows (bootstrap, implement, tdd, feasibility, default)
serve as fallbacks when no skill declares a given workflow name. A
skill that declares metadata.archie.workflow: implement overrides the
built-in — skills define workflows, plugins define stages.

Why: skills must be extension primitives. Adding a workflow should
mean creating .agents/skills/archie-wf-<name>/plugins/*.go, not
modifying Go code in main.go. The catalog is the single source of
truth for what workflows exist.
2026-07-22 04:02:58 +10:00
sam de57f72cce feat: add Docker container sandbox for archie-agent
Dockerfile: multi-stage fat image (golang:1.26 builder → ubuntu:24.04)
with Go 1.26, Python 3.12/3.13 via deadsnakes, UV, Node 22, pnpm,
TypeScript, esbuild, git, gh, golangci-lint, gofumpt.

Container pool (internal/container): manages Docker container lifecycle
via moby/moby client. Per-task containers with bind-mounted workspaces.

- Acquire: create, start, mount workspace, inject NATS + API key env
- Release: stop, remove (AutoRemove as safety net)
- Orphan recovery on daemon startup (label: archie-daemon=true)
- MaxConcurrency enforcement, MaxUptime config (unused in MVP)
- Image pull with "missing"/"always" policy

Config: [containers] block (enabled, image, max_concurrency, max_uptime,
pull_policy). Validation: requires agent.mode = "nats" and [nats] url.

Daemon: acquires container before workflow.Run, releases after.
Container environment auto-populated from provider API keys.
2026-07-22 02:39:17 +10:00
sam 8c4bb1ef1d feat: promote archie-agent to NATS-connected per-task worker
Replaces stdin/stdout subprocess transport with NATS JetStream
request-reply pattern. The Runner interface and workflow orchestration
are unchanged.

New NATSRunner (implements Runner):
- Publishes AgentRequestMessage to JetStream with X-Archie-Reply header
- Blocks on core NATS inbox subscription for response
- Same Run(ctx, workspace, req) (Result, error) contract

Rewritten archie-agent (cmd/archie-agent):
- Long-running NATS-connected process (was 28-line stdin/stdout)
- Pull-consumes from ARCHIE_TASKS with archie.agent.> filter
- Builds InProcessRunner, executes agent loop, replies to inbox
- Graceful shutdown via signal handling

Extended internal/nats:
- SubjectAgentWildcard, SubjectForAgentRequest, ReplyHeader
- Conn() accessor, NewReplyInbox(), PublishMsg() for JetStream
- Stream subjects extended to ["archie.task.>", "archie.agent.>"]

Config: "nats" added as valid agent.mode alongside inprocess/subprocess
Tests: NATSRunner round-trip with embedded NATS server + handler
2026-07-22 02:13:01 +10:00
sam bb2193e5e4 feat: add optional NATS JetStream task distribution
Adds optional NATS-based task distribution alongside the existing
SQLite flow. When [nats] is absent from config, zero behaviour change.

New internal/nats package:
- Subject constants matching PRD §2 (archie.task.{bug,feature,default,bootstrap})
- Client: Connect (stream + durable pull consumer), PublishTask
  (Nats-Msg-Id dedup), Fetch, DecodeTask
- Tests with embedded NATS server (connect/publish, dedup, subject routing)

Config: NATSConfig{URL, TokenEnv} on Config, commented-out [nats] block
in config.example.toml

Store: ClaimByIssue atomic claim by owner/repo/number for NATS consumer

Daemon: Nats client field, poll() branches to pollNATS/pollSQLite,
Cycle() branches to drainNATS/drainSQLite (NATS fetch then SQLite
ClaimNext fallback for requeued tasks), processNATSTask
(decode → EnqueueIssue → ClaimByIssue → process → Ack)

Scratch: explorer and plan agent outputs from parallel-explore phase
2026-07-22 02:02:09 +10:00
sam c0d108a5d0 feat: add Yaegi-based plugin system (gates, custom stages, skill scripts)
Implements the full PRD (docs/prds/yaegi-plugin-system.md): repos can
now define .archie/gate.go for diff-inspecting quality rules shell
commands can't express, .archie/stages/*.go for custom pipeline steps,
and skills can ship Go scripts run via a new run_go_script agent tool —
all interpreted in-process with Yaegi, with panics recovered so a bad
script can't take down the daemon. Hot reload (phase 4) needed no code:
every loader reads from disk per call and every task gets a fresh
clone, so nothing is ever cached to go stale.
2026-07-22 00:54:46 +10:00
sam 86339b4d0d feat: add subprocess agent runner 2026-07-21 23:21:24 +10:00
sam a8d005a6f6 feat: extract agent execution boundary 2026-07-21 22:16:50 +10:00
sam 251cc7022c feat: add configurable forge and workflow behavior 2026-07-21 15:03:41 +10:00