chore: cleanup old docs

This commit is contained in:
sam committed 2026-09-28 01:48:20 +10:00
1 parent 6d9d0e44d9
commit 657c8feafd
9 files changed
-4364

No files matched your search

-45
View File
@@ -1,45 +0,0 @@
# archied -- design notes
Archie is a resident daemon that keeps personal projects moving: it polls GitHub for issues labelled
`archie`, works each one in an isolated fresh-clone worktree through a routed **workflow**, and opens a
pull request for human review. Sam reviews and merges; the machine grinds in between.
Guiding principle: **environmental constraints over prompt instructions.** Agents don't get asked nicely
to write good code -- the quality gate (vet/build/test/lint, configurable per repo and per workflow stage)
refuses to let a run succeed until it passes. The gate mechanics live in `ai-sdk/agentloop`; this repo
owns orchestration only.
## Workflows, not a pipeline
`internal/workflow` is the engine: a Workflow is an ordered list of stages over a shared TaskContext;
stages are deterministic steps (worktree, gate, diff cap, PR, comments -- the shared step library in
`steps.go`) or agent stages (`agentloop.Run` with a stage-specific mission/toolset/gate). Routing is
label-driven (`bug` → tdd, `feature` → feasibility, default → implement), refinable by an LLM triage
stage. Adding a workflow must never require touching the engine or reimplementing steps.
Planned workflows:
- **bootstrap** (registered): deterministic marker-file PR proving invites/clone/push/PR plumbing.
- **implement**: planner (read-only agentloop, plan posted to the issue) → builder (full toolset,
repo gate) → diff cap → PR.
- **tdd** (bugfixes): analyse → problem surface → write failing repro tests (gate stage with
`ExpectFailure`) → fix (gate: suite passes) → PR.
- **feasibility** (features): read-only analysis → roadmap fit → won't-do close with reasons, or
plan + PRD → email Sam → `waiting_human` → LLM-judged reply → hand off to implement.
## Task lifecycle
`queued → running(workflow:stage) → waiting_human | pr_open → merged | parked | rejected | closed_wont_do`
SQLite (`internal/store`), WAL, one row per issue with a transitions audit table. Crash recovery:
`running` tasks re-queue on startup with attempt++. Parks are never silent -- every park comments on the
issue with the reason and gate output.
## Security posture
- Fine-grained PAT on the bot account only (Contents RW, PRs RW, Issues RW, Metadata R), via
`ARCHIE_GITHUB_TOKEN`; git auth through an askpass helper so the token stays out of argv and
`.git/config`.
- The model never runs git: `internal/worktree` owns all repo operations deterministically.
- Agent tools are jailed to the task worktree (toolkit path jail); intended deployment is a dedicated
Incus container so the blast radius is environmental, not prompt-negotiated.
-519
View File
@@ -1,519 +0,0 @@
# archie-core v2 Architecture -- Design Document
**Author:** Archie
**Date:** 2026-07-21 (implementation status updated 2026-07-22)
**Status:** Final (substantially implemented -- remaining aspirational items are pluggable storage backends and Layer 2 daemon plugin loader)
---
## 0. Implementation status (as of 2026-07-22, updated 2026-07-22)
This document describes a target end state. The table below is the ground truth; read it before treating any section as already built.
| Area | Status | What actually exists today (post-e4d11b1) |
| --------------------------------------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Daemon/agent binary split | **Implemented** | `cmd/archied` and `cmd/archie-agent` both exist. `archie-agent` is a 189-line NATS-connected per-task worker that subscribes to task subjects, runs the full multi-stage workflow, and reports results. Workflow routing and sequencing (`workflow.Route`, `workflow.Run`) now run inside the agent, not the daemon. Three agent modes: `inprocess`, `subprocess`, `nats`. |
| Sandbox / "daemon never runs untrusted code" | **Partial** | Default mode is still `inprocess` for development. Docker sandbox mode (`agent.mode = "nats"` + `containers.enabled = true`) runs each task in an ephemeral Docker container with workspace bind-mounted at `/workspace`. `internal/container/pool.go` manages full lifecycle: pull, create, start, release, orphan recovery. Daemon dispatch and container lifecycle both enforce `max_concurrency`; same-repo tasks remain serialised. |
| NATS / JetStream messaging | **Implemented** | `internal/nats/client.go`: JetStream connect, stream creation (WorkQueue retention, file storage), pull consumer (durable, ack-wait 5min, max-deliver 3, dedup by Msg-Id), `PublishTask`, `Fetch`, `NewReplyInbox`, and optional token authentication. Subject hierarchy lives in `internal/nats/subjects.go`; `internal/agentexec/nats.go` implements request/reply agent execution. |
| Docker container sandbox | **Implemented** | `internal/container/pool.go` (210 lines): Docker API client, image pull (always/missing/never), container create with workspace bind mount, start, release (stop+remove), orphan recovery on daemon restart. `Dockerfile` at repo root builds archie-agent fat image. `[containers]` config block: enabled, image, max_concurrency, max_uptime, pull_policy. `agent.mode = "nats"` required when containers enabled. |
| `/data/` volume, `task.json` | **Partial** | `container.WriteTaskJSON()` writes task payload to the worktree before container acquire. The container pool bind-mounts the isolated task worktree at `/data/worktree`; configured repos also receive a persistent `/data/repo` Docker volume. Task payload travels over NATS as the primary channel; `/data/task.json` is a boot-time brief for the container. |
| `[containers]` config block | **Implemented** | `ContainerConfig` struct in config.go: enabled, image, max_concurrency, max_uptime, pull_policy. Validated on load. |
| Yaegi extensibility (gates, custom stages, skill scripts) | **Implemented** | Three surfaces: `.archie/gate.go`, `.archie/stages/*.go`, skill scripts via `run_go_script` tool. All in-process via Yaegi with generated symbol tables and panic recovery. Full detail in `docs/prds/yaegi-plugin-system.md`. |
| Layer 1: skill-bundled plugins | **Implemented** | `skill.DiscoverPlugins()` scans `.agents/skills/<name>/plugins/*.go`. `skill.Plugin.Run()` executes them via Yaegi. `skillbuild.BuildWorkflow()` constructs `workflow.Workflow` from plugin-defined stages. `TaskContext.SkillPlugins` makes them available during stage execution. `skillbuild.BuildRegistry()` replaces the hardcoded `workflow.Registry` with catalogue-driven construction -- skills define workflows, plugins define stages. |
| Layer 2: core daemon plugins | **Partial** | `internal/plugin/plugin.go`: `Plugin` interface (`Name()`, `Version()`), `Registry` type, `LoadDir()` using Yaegi to interpret `.go` files. No `~/.config/archie/plugins/` loader wired into `cmd/archied/main.go` yet -- the interface and registry exist but are not connected to the daemon's startup. |
| Skills (agentskills.io spec) | **Implemented** | `internal/skill/skill.go`: parses SKILL.md YAML frontmatter, discovers skills from `.agents/skills/*/SKILL.md`, extracts body. Skills follow `archie-wf-*` naming convention. Progressive disclosure: `Catalog()` (Tier 1 -- frontmatter-only scan, ~100 tokens per entry), `LoadBody()` (Tier 2 -- full body on activation), `Discover()` (Tier 3 -- full parse with plugins). `metadata.archie.workflow` declares skill→workflow affinity. `skill.SkillForWorkflow()` is catalogue-driven, no hardcoded maps. `skillbuild.BuildRegistry()` constructs the daemon's workflow registry from the catalogue. |
| Worktree/repo management | **Implemented** | Each issue keeps an isolated worktree. Repos with `persistent_storage = true` use a per-repo bare Git object cache to avoid repeated full network clones; caches and Archie-owned repo volumes expire via `containers.volume_ttl`. |
| Config schema | **Closer to sec.4** | `[agent]` block: mode (inprocess/subprocess/nats) + command + env. `[nats]` block: url + token_env. `[containers]` block: enabled, image, max_concurrency, max_uptime, volume_ttl, pull_policy. Per-repo `persistent_storage` is supported as a boolean Docker-volume/Git-cache opt-in. |
**Net read post-1dca419:** Phases 1, 2, and 3 are substantially implemented. NATS, Docker sandbox, skill parsing, skill-bundled plugins (Layer 1), skill progressive disclosure, and catalogue-driven workflow construction (`skillbuild.BuildRegistry()`) all landed. The remaining aspirational work is Layer 2 daemon plugin loader (interface exists, not wired), pluggable storage backends, and per-worktree dynamic registry augmentation.
---
## 1. Binary split
### archied (daemon)
The resident orchestrator. Owns:
- **Task discovery** -- polls configured forges for issues labelled `archie` (or configured label)
- **NATS publishing** -- publishes discovered tasks to JetStream subjects
- **Container lifecycle** -- spawns Docker containers, kills them after grace period, cleans up volumes
- **Message gatekeeping** -- reads agent responses from NATS, reviews them, forwards to human channels
- **Core plugin loading** -- loads Yaegi `.go` plugins from `~/.config/archie/plugins/` at startup
- **Web dashboard** -- SSE observability endpoint
- **Human communication** -- issue comments, labels, PR management, notify webhook
The daemon never runs untrusted code. Agent containers are the sandbox boundary.
> **Target state, not current state.** Today, default `[agent] mode = "inprocess"` runs the agent loop (and therefore
> all LLM-directed tool calls) directly inside `archied`. `subprocess` mode exists (`internal/agentexec/subprocess.go`)
> but its own code comment says: _"This is not an OS security boundary: a same-UID subprocess may still access
> daemon-readable host resources."_ This claim becomes true only once the Docker sandbox (sec.4, Phase 2) lands.
### archie-agent (ephemeral container)
Spawned per task. Lives in a Docker container. Receives its brief via `/data/task.json` (volume-mounted). Owns:
- **Workflow execution** -- runs the routed workflow stages (implement, tdd, feasibility, bootstrap)
- **LLM orchestration** -- via `ai-sdk/agentloop`, the agent drives LLM calls
- **Bundled plugin loading** -- loads Yaegi `.go` plugins from the skill's `plugins/` directory and the `/data/plugins/`
volume
- **Skill loading** -- reads `.agents/skills/archie-wf-*` from the worktree, follows the agentskills.io progressive
disclosure model
- **NATS reporting** -- publishes results to `archie.agent.<task_id>.response`, system messages to
`archie.agent.<task_id>.system`
Lifecycle: spawn -> load task -> run workflow -> report -> wait for follow-ups (grace period) -> killed by daemon.
> **Target state, not current state.** `cmd/archie-agent/main.go` today is 28 lines: read one JSON `Invocation` from
> stdin, run one stage via `agentexec.ServeOne`, write one `Response` to stdout, exit. It is a single-stage subprocess
> worker invoked by `archied`'s `SubprocessRunner`, not a persistent per-task container process. Workflow routing and
> multi-stage sequencing (`workflow.Route`, `workflow.Run`) run entirely inside `archied`
> (`internal/daemon/daemon.go:process()`), not inside archie-agent. There is no `/data/task.json`, no bundled-plugin
> loading, no `.agents/skills/archie-wf-*` discovery, and no NATS reporting -- see sec.0, sec.2, sec.3, sec.5. The
> existing `agentexec.Runner` interface (`InProcessRunner`/`SubprocessRunner`) is, however, deliberately shaped so a
> future container-backed runner can be a drop-in swap without changing workflow orchestration.
---
## 2. NATS subject hierarchy
> **Status: aspirational, zero code.** No NATS/JetStream dependency exists anywhere in `go.mod` or the codebase. Today,
> task discovery is GitHub-poll-driven (`daemon.Run()` ticks on `PollInterval`, calls `pollIssues()`), distribution is a
> SQLite `tasks` table used as a FIFO queue (`store.ClaimNext` atomically claims the oldest `queued` row), and
> results/status flow through an in-process `internal/events.Bus` (Go channels, multi-subscriber fan-out,
> drop-on-overflow) that already feeds both a SQLite `events` table and the SSE dashboard. `events.Bus` has roughly the
> right shape to become the NATS publish path -- same `Event` schema, swap the transport -- which meaningfully de-risks
> this phase, but it is currently single-process only with no network transport.
### Task distribution
```events
archie.task.bug -- TDD workflow tasks
archie.task.feature -- Feasibility workflow tasks
archie.task.default -- Implement workflow tasks
archie.task.bootstrap -- Bootstrap diagnostic tasks
```
Daemon publishes to the appropriate subject based on label routing. Agents subscribe to subjects they're configured to
handle.
### Agent communication (per-task)
```events
archie.agent.<task_id>.response -- Agent output: summaries, PR bodies, results. Daemon reviews before human delivery.
archie.agent.<task_id>.system -- Internal comms: log dumps, data requests, health, PII warnings. Never forwarded unprompted.
archie.agent.<task_id>.request -- Optional: agent requests additional data from the daemon.
```
The daemon gatekeeps the response channel -- no agent output reaches a human without daemon review.
### JetStream durability
Tasks are published with JetStream persistence. On agent container failure, NATS redelivers. The daemon doesn't need
retry logic -- NATS handles it.
---
## 3. Persistent volume
> **Status: partially implemented.** `task.json`, `/data/worktree`, optional per-repo `/data/repo` Docker volumes,
> ecosystem cache volumes, and TTL cleanup exist. Task worktrees remain isolated by issue, while configured repos use a
> per-repo bare Git object cache to seed clones without repeating full network transfers. `session.jsonl`,
> `memory.jsonl`, and staged plugin volumes remain unimplemented.
### /data/ layout
```tree
/data/
|---- task.json # Full task payload (owner, repo, issue, title, body, labels, workflow)
|---- session.jsonl # Agent session transcript (appended per stage)
|---- memory.jsonl # Optional: cross-session memory (persisted per project)
|---- plugins/ # Bundled plugins from the skill (daemon-staged, overrides worktree)
|---- worktree/ # Read-only mount of the cloned repo (managed by daemon)
```
### task.json
```json
{
"id": 42,
"owner": "sam",
"repo": "example-service",
"issue_number": 170,
"title": "Add /copy command for messages and session transcripts",
"body": "...",
"labels": ["enhancement", "archie"],
"workflow": "implement",
"branch": "archie/issue-170",
"plan": "",
"config": {
"base_branch": "main",
"gate": [
["gofumpt", "-w", "."],
["go", "vet", "./..."],
["go", "test", "-race", "./..."]
],
"protect": ["_templ.go"],
"ecosystem": "go",
"test_glob": "*_test.go"
},
"nats": {
"url": "nats://archie-core:4222",
"response_subject": "archie.agent.42.response",
"system_subject": "archie.agent.42.system",
"request_subject": "archie.agent.42.request"
}
}
```
The daemon writes `task.json` to the volume before the container starts. The agent reads it on boot.
### Volume lifecycle
- Created on task start (persistent per-repo if configured)
- Kept until terminal outcome (`pr_open`, `merged`, `rejected`, `closed_wont_do`) OR TTL expiry
- Parked tasks keep their volume for inspection and requeue reuse
- Cleanup is a daemon responsibility
---
## 4. Container management
> **Status: implemented with a flatter schema than the sketch below.** `[containers]` configures the image,
> concurrency, uptime, pull policy, and persistent-storage TTL. `Repo.PersistentStorage` opts a repository into the
> per-repo Docker volume and Git object cache. The backend abstraction exists in `internal/storage`; additional storage
> types and structured per-backend configuration remain future work.
### Daemon config
```toml
# Container lifecycle
[containers]
max_concurrency = 4 # max simultaneous agent containers
max_uptime = "30m" # grace period after task completion before kill
volume_ttl = "72h" # max volume age before cleanup (overrides terminal wait)
# Image strategy
[containers.image]
registry = "ghcr.io/sam/archie-agent:latest" # primary: pull from registry
registry_pull = true # allow pulls
dockerfile = "./Dockerfile.agent" # fallback: build locally
dockerfile_build = true # allow builds
# Persistent storage
[containers.persistent_storage]
type = "docker" # docker (volume), bind, s3, sftp, nfs
path = "archie-data" # volume name or mount path
```
### Per-repo persistent storage override
```toml
[[repos]]
owner = "sam"
name = "example-service"
persistent_storage = { type = "bind", path = "./archie-core-data/projects/example-service" }
```
When configured, each task for this repo mounts the same bind path. Memory persists across sessions. The agent writes to
`MEMORY.md` in this directory.
### Agent image
A fat Docker image containing:
- archie-agent binary
- NATS client library
- LLM provider SDKs (ai-sdk runtime)
- Base tools: git, gh, go, golangci-lint, gofumpt, node, python, ruff, pytest, etc.
- esbuild (for TypeScript plugin transpilation)
- Yaegi interpreter (bundled in the binary)
Pre-built and published to a registry. The daemon pulls on startup. Users can provide their own image via config.
---
## 5. Extensibility layers
> **Reconciliation note.** This section describes a two-layer plugin architecture (skill-bundled plugins loaded
> per-task, core daemon plugins loaded at startup) that does not exist in code. What _does_ exist -- and is implemented,
> tested, and already in production -- is described in the sibling PRD `docs/prds/yaegi-plugin-system.md`, whose three
> phases (custom gate functions, custom workflow stages, skill scripts) are marked "implemented" and match disk state
> exactly as of commit `d45f21d`. That implementation is narrower and shaped differently than what follows:
>
> - It has **no skill-bundling concept** -- `.archie/gate.go` and `.archie/stages/*.go` are repo-level, not shipped
> inside a skill's `plugins/` directory; skill scripts (`.archie/skills/<skill>/scripts/*.go`) are the closest analog
> but none of the five current skills ship a `scripts/` directory yet.
> - It has **no daemon-side core plugin registry** -- no `Plugin` interface, no `Register(daemon *Daemon)`, no
> `~/.config/archie/plugins/` loader. Yaegi today only interprets small, narrowly-scoped `.go` files (gates, stages,
> scripts) in-process inside `archied`, using generated symbol tables (`internal/gate/gateextract`,
> `internal/workflow/wfextract`) rather than a general extension-registration interface.
> - Custom stages are **auto-discovered and run in filename order**, not routed by name from `config.toml` -- the
> workflow engine has no data-driven stage graph today, so the `[[workflows.stages]]` config sketch below does not
> correspond to real config.
>
> Before implementation begins, decide: does Layer 1/Layer 2 as described below _replace_ the existing three Yaegi
> surfaces, or does it _extend_ them (e.g., teach the existing loaders to also read from a skill's `plugins/` directory,
> and add a genuinely new daemon-side `Plugin` interface as a separate, later addition)? The existing surfaces are
> working code with real symbol-table generation and panic recovery -- treat them as the implementation substrate for
> Layer 1, not as something to discard.
### Layer 1: Skills (`archie-wf-*`)
> **Status: cosmetic only.** Five `SKILL.md` files exist today (`ecosystem-node`, `ecosystem-python`, `go-quality-gate`,
> `security-audit`, `tdd-bugfix`) at `.archie/skills/<name>/SKILL.md` -- a single flat repo-relative location, no
> worktree-vs-user-level split, no `archie-wf-*` naming. Each has a `metadata.archie` block with only `tools` and
> `engine` keys (no `plugins`, no `budget_usd`). **No Go code parses SKILL.md frontmatter, discovers skills, or does
> progressive disclosure** -- grepping for a skill loader, catalog, or frontmatter parser across the repo returns
> nothing. The workflow `Mission` strings that drive agent behavior today are hand-written Go string literals in
> `internal/workflow/{implement,tdd,feasibility}.go`, not content injected from SKILL.md. The stage names described in
> the PRD's SKILL.md prose closely mirror the hardcoded Go stage names (one drift found: `tdd-bugfix/SKILL.md` says the
> last stage is "deliver"; the code calls it `open-pr`) -- these read as retrofitted documentation of existing v1
> behavior wearing v2-flavored frontmatter, not a functioning specification the daemon consumes.
Skills are Markdown documents following the [agentskills.io](https://agentskills.io/specification) specification with
archie-specific extensions.
#### Discovery
```tree
.agents/skills/archie-wf-tdd/SKILL.md (worktree -- highest precedence)
.agents/skills/archie-wf-implement/SKILL.md
.agents/skills/archie-wf-feasibility/SKILL.md
~/.archie/skills/archie-wf-default/SKILL.md (user-level -- fallback)
```
The agent discovers skills at workflow start. The skill matching the task's workflow type is loaded first. If absent,
falls back to `archie-wf-default`. If that's absent too, the agent works from AGENTS.md conventions and best practices.
#### Extended agentskills.io format
Skills follow the standard spec (`name`, `description`, `version`, `SKILL.md` body) with archie-specific metadata:
```yaml
---
name: archie-wf-tdd
description: >
Test-driven bugfix workflow for archie-core. Reproduce the bug with failing tests, prove the repro, fix the code
without touching tests.
version: 1.0.0
metadata:
archie:
plugins: # bundled Yaegi plugins shipped with this skill
- plugins/custom-gate.go # repo-specific gate logic
- plugins/gosec-check.go # security scanner integration
tools: [go, golangci-lint, gofumpt, task]
engine: any
budget_usd: 2.00
---
```
#### Progressive disclosure
Per the agentskills.io spec:
1. **Catalogue** (~100 tokens): `name` + `description` loaded at workflow start
2. **Instructions** (<5000 tokens): Full `SKILL.md` body when the skill is activated
3. **Resources** (as needed): References, scripts, and bundled plugins loaded on demand
#### Bundled plugins
A skill's `plugins/` directory contains Yaegi `.go` files. These are loaded by the agent at workflow start alongside the
skill instructions. The daemon copies them to `/data/plugins/` before the container starts.
If `/data/plugins/` already has a plugin with the same name (from persistent project storage), the volume version wins.
Conflicts are logged by the daemon.
### Layer 2: Core plugins
> **Status: aspirational, zero code.** No `Plugin` interface, no plugin registry, no `~/.config/archie/plugins/` loader
> exists anywhere in the repo (confirmed by grep for `Plugin`, plugin registry, and `Register(daemon` -- zero matches).
> Yaegi is a real dependency (`traefik/yaegi v0.16.1`) and is genuinely wired up, but only for the three narrow surfaces
> in sec.5's reconciliation note above, all interpreted in-process inside `archied` -- not for extending the daemon with
> new forge/ticketing/storage/secrets/notify implementations as described below.
Core plugins live at `~/.config/archie/plugins/` and extend the daemon itself:
```tree
~/.config/archie/plugins/
|---- forge-gitea.go # Gitea forge implementation
|---- forge-gitlab.go # GitLab forge implementation
|---- ticketing-jira.go # Jira ticketing integration
|---- ticketing-linear.go # Linear ticketing integration
|---- storage-s3.go # S3 persistent storage backend
|---- vault-secrets.go # HashiCorp Vault integration
|---- notify-servicenow.go # ServiceNow integration
```
#### Loading
Loaded by the daemon at startup via Yaegi. Each plugin registers itself with the daemon's extension registry:
```go
// Plugin interface (Go, interpreted via Yaegi)
type Plugin interface {
Name() string
Version() string
Register(daemon *Daemon) error
}
```
The daemon discovers `.go` files, evaluates them, extracts the plugin symbol, and calls `Register()`. Failed plugins are
logged and skipped -- the daemon starts with remaining plugins.
#### What core plugins can do
- **Forges** -- implement the `forge.Forge` interface. Register themselves as available forge types.
- **Ticketing** -- implement a ticketing interface. Poll external systems, format issues into the standard `forge.Issue`
struct.
- **Storage backends** -- implement a storage interface. Provide volume population and cleanup.
- **Secrets** -- implement a vault interface. Inject credentials into `/data/secrets/` before agent start.
- **Notifications** -- implement a notify interface. Deliver PRDs, status updates beyond the built-in webhook.
---
## 6. Task lifecycle (v2)
> **Status: implemented.** NATS JetStream is now the task distribution path (`internal/nats/client.go`, `internal/daemon/daemon.go:drainNATS()`). Docker containers spawn per task (`internal/container/pool.go`). The flow below matches the running code post-e4d11b1.
```
Discover -> Publish to JetStream -> Agent consumes -> running(workflow:stage)
v
pr_open -> merged | rejected
v
waiting_human -> (approved) -> requeue
v
parked
```
### Detailed flow
1. **Discover** -- daemon polls forges for labelled/assigned issues
2. **Publish** -- daemon publishes task to `archie.task.<workflow_type>` JetStream subject
3. **Spawn** -- daemon creates task volume, writes `task.json`, stages bundled plugins
4. **Start container** -- `docker run` with volume mounts, network (NATS), resource limits
5. **Agent boots** -- reads `task.json` from `/data/`, connects to NATS, discovers skills, loads bundled plugins
6. **Run workflow** -- agent executes routed workflow stages. Reports progress to system channel. Result to response
channel.
7. **Daemon reviews** -- reads agent response. Forwards to human channels (issue comment, label update, PR). System
messages stay internal.
8. **Grace period** -- agent stays alive for `max_uptime`. Handles follow-ups (gate re-runs, human replies forwarded by
daemon).
9. **Kill** -- daemon kills container. Volume cleaned up on terminal outcome or TTL expiry.
---
## 7. Agent -> daemon -> human flow
```flow
,---------, response ,---------, reviewed ,----------,
| Agent |------------ -> | Daemon |------------ -> | Human |
| | system | | queried | channels |
| |------------ -> | |<-------------| |
|------------ |------------ |-------------
|
|---- Issue comments (parked, PR, waiting)
|---- Label updates (archie:queued -> archie:working -> archie:pr)
|---- Notify webhook (feasibility PRD delivered)
|---- SSE dashboard (real-time observability)
```
The agent writes whatever it wants to `response`. The daemon reviews it. No agent output reaches a human unprompted. The
`system` channel is purely internal -- the daemon reads it for observability and debugging, never forwards it.
---
## 8. Implementation phases
> Updated 2026-07-24 against actual repo state (see sec.0). Items marked **done** exist in the current tree; nothing else in
> this document is implemented beyond them. Phase ordering below is unchanged from the original sketch except Phase 3,
> which now builds on top of existing Yaegi work instead of assuming a blank slate.
### Phase 1: Binary split + NATS
- ~~Split `archied` into daemon binary~~ -- **done**, `cmd/archied` exists
- ~~Create `archie-agent` binary~~ -- **done, but partial**: `cmd/archie-agent` exists and executes one workflow _stage_
per invocation over a versioned stdin/stdout JSON protocol (`internal/agentexec`, protocol v1). Still needed: promote
it from a single-stage subprocess worker to the PRD's per-task process, and move workflow routing/sequencing
(`workflow.Route`/`workflow.Run`) out of `archied` and into it.
- ~~Add NATS to both (JetStream for task queue, request-reply for comms)~~ -- **done**. `[nats]` is optional, supports
anonymous local mode and token authentication via `token_env`, and leaves `internal/events.Bus` as the local
observability fan-out.
- ~~Daemon publishes, agent consumes, results return via NATS~~ -- **done**. Task discovery uses JetStream work-queue
subjects; autonomous stages use JetStream requests plus core NATS reply inboxes.
- Current single-process mode preserved for development (`--local` flag) -- effectively already true: `inprocess`
`[agent] mode` is the _default_, not an opt-in fallback, so this constraint is satisfied but should be revisited once
subprocess/container modes are the default.
### Phase 2: Docker sandbox
- Docker agent image (fat, with tools) -- not started
- Daemon container lifecycle: spawn, volume create, env inject, kill -- not started
- ~~Persistent volumes with TTL~~ -- **done**: configured repos receive a persistent Docker volume plus a per-repo Git
object cache; both are automatically reaped using `containers.volume_ttl`.
- ~~`max_concurrency` enforcement~~ -- **done**: SQLite and NATS drains dispatch different repositories concurrently
up to `[containers].max_concurrency`; same-repo tasks remain serialised by default.
- Grace period `max_uptime` -- not started
- **Groundwork already in place:** `agentexec.Runner` (`InProcessRunner`/`SubprocessRunner`) is an interface designed,
per its own doc comments, to be swapped for a container-backed runner without touching workflow orchestration --
implement `ContainerRunner` against this interface rather than redesigning the execution boundary.
### Phase 3: Skills + bundled plugins
- agentskills.io spec implementation (progressive disclosure) -- not started; current SKILL.md files are unparsed
documentation (see sec.5)
- `archie-wf-*` skill discovery from worktree and user paths -- not started; current skills live at a single flat
`.archie/skills/` path with different naming
- Bundled plugin loading via Yaegi from skill's `plugins/` directory -- not started, but **has a working substrate to
extend**: `internal/gate/gateeval`, `internal/workflow/wfeval`, and `internal/skillscript` already interpret `.go`
files via Yaegi with generated symbol tables and panic recovery (see `docs/prds/yaegi-plugin-system.md`, Phases 1--3,
done). This phase should teach those loaders to also discover files from a skill's `plugins/` directory, not build a
second Yaegi integration from scratch.
- Volume-staged plugins with conflict resolution -- not started, and depends on Phase 2's volume work existing first
### Phase 4: Core plugins
- Yaegi plugin loader for `~/.config/archie/plugins/` -- not started (no `Plugin` interface, no registry, no loader
exist today)
- Plugin registry in daemon -- not started
- Gitea forge as first core plugin (proof of concept) -- not started (current `internal/forge` only implements GitHub
REST polling)
- Linear ticketing as second core plugin -- not started
### Phase 5: Pluggable storage
- Docker volume backend (built-in) -- not started
- Bind mount backend (built-in) -- not started
- S3, SFTP, NFS backends as core plugins -- not started
---
## 9. Open questions
1. **Agent image versioning:** Use tagged versions (`:v1.2.3`) with a "never repoint tags" policy. `:latest` is for dev
only, not the config default. No digest complexity needed -- archie-core is self-hosted, not a public registry.
2. **NATS auth:** Anonymous for single-node deployments (daemon + NATS + agents on same host). Credentials required for
multi-node/cluster. Add an optional `[nats]` config block (default: anonymous, validated only when present).
3. **Worktree ownership:** ~~Answered by current implementation.~~ Fresh `git clone` per task attempt, no shared cache.
This is fine for v2. Revisit if concurrent agents on the same repo become a bottleneck.
4. **Grace period interaction:** Reset `max_uptime` on human reply -- the agent just did useful work. But cap total
lifetime with `max_total_uptime` (e.g. 4 hours) to prevent infinite hanging. `max_uptime` = grace after inactivity,
`max_total_uptime` = absolute kill switch.
5. **Concurrent agents on the same repo:** ~~Resolved.~~ Serialised by default (dispatcher chains tasks per
`owner/repo`). `[[repos]]` now supports `allow_concurrent = true` (`config.Repo.AllowConcurrent`) to opt a repo out
of that serialisation; the global `[containers].max_concurrency` slot limit still applies. Worktrees are already
keyed per issue number (`worktree.Manager.Dir`), so opted-in repos don't collide on disk.
6. **Plugin architecture reconciliation:** Extend the existing Yaegi surfaces (`internal/gate/gateeval`,
`internal/workflow/wfeval`, `internal/skillscript`) rather than building a parallel `Plugin.Register(daemon)`
interface. Core plugins follow the same pattern: `Eval()` a `.go` file, extract known symbols. Settled before Phase
3/4.
7. **Skill loading scope:** "Parse frontmatter, inject body into system prompt" is sufficient for v2. The gap is that
nothing reads SKILL.md today -- closing that gap is the value. Full progressive disclosure (catalogue -> activate ->
resources) is deferred to a later phase.
-389
View File
@@ -1,389 +0,0 @@
# archie-core Skills System -- PRD
**Author:** Archie
**Date:** 2026-07-18
**Status:** Draft
---
## What archie-core is
arcie-core is an LLM coding agent that responds to issues, queries, and commands on connected interfaces. The current MVP runs on GitHub: it receives issue assignments, analyses scope and requirements, implements changes in isolated worktrees, runs quality gates, and opens pull requests.
The skills system extends this with reusable playbooks that tell archie-core **how** to work on any given project.
---
## 1. The workflow file (`archie.yaml`)
Every project can ship an `archie.yaml` that defines how archie-core should behave when working on that codebase. Without one, archie-core uses sensible defaults (read files, run bash, open PRs). With one, it knows the exact gates to pass, tools available, commit format, and which skills are relevant.
### 1.1 Discovery
```
.archie.yaml (root of repo -- highest precedence)
archie.yaml (repo root)
.archie/archie.yaml (subdirectory, one level deep)
```
### 1.2 Schema
```yaml
# archie.yaml
version: "1"
# Project identity
project:
name: my-go-service
language: go
repo: github.com/sam/my-go-service
# How archie-core connects to this project
connection:
interfaces:
- github # respond to assigned issues
- linear # respond to labelled tickets
- webhook # respond to HTTP POSTs
github:
repo: sam/my-go-service
issue_labels:
ready: "archie:ready" # issues with this label get claimed
claimed: "archie:claimed" # archie sets this when working
done: "archie:done" # archie sets this after PR
pr:
base: main
draft: false
# Skills -- playbooks archie-core loads to know how to work
skills:
paths:
- .archie/skills/ # project-local skills
- ~/.archie/skills/ # per-user skills
remotes:
- github.com/sam/archie-skills@v1 # shared skill library
# Sandbox -- what's available in the container
sandbox:
image: ghcr.io/sam/archie-sandbox:latest
tools:
- go
- golangci-lint
- gofumpt
- task
- gh
- docker
env:
GOFLAGS: "-mod=readonly"
# Gates -- archie-core must pass these before committing
gates:
test:
run: go test -race -count=1 ./...
timeout: 300
lint:
run: golangci-lint run ./...
timeout: 120
format:
run: gofumpt -w . && git diff --exit-code
deadcode:
run: task deadcode
optional: true
# Quality pipeline -- which gates run when
quality:
pre_commit: [test, lint, format]
pre_push: [deadcode]
# Commit conventions
commit:
format: conventional
signoff: false
# Branch conventions
branch:
format: "{type}/cat-{number}-{kebab-description}"
types:
feat: "Feature work"
fix: "Bug fixes"
test: "Test additions"
chore: "Maintenance"
refactor: "Code restructuring"
docs: "Documentation"
```
---
## 2. Skills (playbooks)
Skills are Markdown documents that tell archie-core **how** to perform specific tasks. They follow the [agentskills.io](https://agentskills.io/specification) format, making them portable across compatible LLM coding agents.
### 2.1 Format
```
go-testing/
├── SKILL.md # Required: frontmatter + instructions
├── references/ # Optional: supplementary docs
│ └── coverage.md
├── scripts/ # Optional: executable helpers
│ └── bench.sh
└── assets/ # Optional: templates
└── test.vscode.code-snippets
```
### 2.2 SKILL.md
```markdown
---
name: go-quality-gate
description: >
Run the full Go quality pipeline:
gofumpt, go fix, golangci-lint, go test -race, and task deadcode.
Use before any commit in a Go project.
version: 1.0.0
metadata:
archie:
# Which gates this skill covers
gates: [test, lint, format, deadcode]
# Required sandbox tools
tools: [go, golangci-lint, gofumpt, task]
# Preferred engine (if multiple available)
engine: any
# Max spend for this skill
budget_usd: 2.00
---
# Go Quality Gate
## When to use
Before committing any Go change. After implementing a feature or fix.
## Procedure
Run in this exact order. If any step fails, fix the issue before continuing.
1. `gofumpt -w .`
2. `go fix ./...`
3. `golangci-lint run ./...`
4. `go test -race -count=1 ./...`
5. `task deadcode`
## Verification
All five steps must exit 0. If `task deadcode` fails on unrelated packages,
scope it to only changed packages: `golangci-lint run --tests=false --enable-only=unused,staticcheck ./changed/pkg/...`
## Common failures
- `golangci-lint`: unused function -- remove it or add `//nolint:unused` comment
- `go test -race`: data race -- protect with mutex or channel
- `task deadcode`: pre-existing dead code in unrelated package -- scope to changed packages
```
### 2.3 Skill metadata (arcie-specific)
Skills carry archie-specific metadata under `metadata.archie`:
| Field | Purpose |
| ------------ | ------------------------------------------------ |
| `gates` | Which gates from `archie.yaml` this skill covers |
| `tools` | Sandbox tools this skill needs |
| `engine` | Preferred execution engine |
| `model` | Preferred model for this skill |
| `budget_usd` | Max spend ceiling |
| `timeout` | Max seconds |
| `worktree` | Whether to create isolated worktree |
### 2.4 Progressive disclosure
Following the skills.sh model to keep context lean:
1. **Advertise** (~100 tokens per skill): `name` + `description` loaded at startup
2. **Activate** (<5000 tokens): Full SKILL.md body loaded when skill is relevant
3. **Resources** (as needed): References, scripts, assets loaded on demand
---
## 3. How archie-core processes an issue
This is the core loop. When an issue lands with the configured label:
### 3.1 Analyse
1. Read issue title, body, labels, comments
2. Load relevant skills from the project's `archie.yaml`
3. Determine scope: which packages, files, and gates are in play
4. Determine branch type (`feat/`, `fix/`, `test/`, etc.) from issue labels or conventional commit prefix
### 3.2 Claim
1. Set issue label to `archie:claimed`
2. Comment: "Archie claimed this. Branch: `<branch>`"
3. Create isolated worktree from `main`
### 3.3 Implement
1. Load full skill bodies for each relevant skill
2. Read project source and tests in the affected area
3. Implement changes, writing tests first where applicable
4. Run targeted tests iteratively
### 3.4 Gate
1. Run all gates from the quality pipeline in order
2. Fix any failures, re-running from the failing gate
3. Audit diff for secrets, unrelated changes, attribution
### 3.5 Deliver
1. Commit with conventional format: `feat(scope): description`
2. Push branch
3. Open PR to `main` with:
- Summary of changes
- Acceptance criteria addressed
- Gate results
- Link back to source issue/Linear ticket
4. Set issue label to `archie:done`
5. Comment with PR URL
### 3.6 On failure
If any gate fails irrecoverably:
1. Leave issue as `archie:claimed`
2. Comment with exact blocker, failed command, and worktree path
3. Preserve worktree for inspection
4. Do not open a broken PR
---
## 4. Engine abstraction
arcie-core can delegate implementation to different LLM engines. The workflow file declares preference order; archie-core resolves the first available engine at runtime.
```yaml
engines:
- type: claude # Claude Code CLI -- full agent loop
model: sonnet
budget_usd: 5.00
- type: codex # OpenAI Codex CLI
model: gpt-5.1-codex-max
- type: raw # Direct LLM -- no agent loop, single prompt
model: gpt-4o-mini
```
Each engine provides the same interface:
```
Engine:
- Run(prompt, tools, budget, timeout) -> Result
- Skills() -> []SkillName
- Capabilities() -> {streaming, reasoning, vision, agentic}
```
For skills that need a specific engine, the skill metadata overrides:
```yaml
metadata:
archie:
engine: claude # this skill only works with Claude
```
---
## 5. Skills in practice
### 5.1 Shared skill library
```yaml
# archie.yaml
skills:
remotes:
- github.com/sam/archie-skills@v1
```
Skills from the shared library are available alongside project-local skills. The agent loads them on demand when an issue references relevant concepts.
### 5.3 Skill that wraps an external tool
```yaml
---
name: trivy-scan
description: >
Scan container images and filesystems for vulnerabilities using Trivy.
Use when asked about security scanning, CVEs, or container hardening.
metadata:
archie:
tools: [docker, trivy]
engine: any
---
## Procedure
1. `trivy image --severity HIGH,CRITICAL <image>`
2. If findings exist, report each CVE with package, version, and fixed version
3. If no findings, report clean
```
---
## 6. Comparison: archie-core vs existing systems
| | archie-core | Claude Code | Codex CLI | GitHub Actions | External Agent |
| ------------------------ | ---------------------------------------------- | ------------------------------- | ------------------------------- | ---------------------------------- | ------------------------------- |
| **How it receives work** | GitHub issue assignment, Linear label, webhook | Chat, slash commands | Chat, slash commands | YAML triggers (push, PR, schedule) | Chat, cron, webhooks |
| **What it does** | Analyse → implement → gate → PR | Analyse → implement → gate → PR | Analyse → implement → gate → PR | Run arbitrary shell commands | Analyse → implement → gate → PR |
| **Skills** | SKILL.md (agentskills.io) | SKILL.md (native) | SKILL.md (agentskills.io) | Composite actions | SKILL.md (native) |
| **Gates** | Named checkpoints in archie.yaml | Ad-hoc (CLAUDE.md + lint) | Ad-hoc | Steps in jobs | Skills with gates |
| **Engine pluggable** | Yes | No (fixed to Claude) | No (fixed to Codex) | No (fixed to GitHub) | Yes (provider config) |
| **Worktree isolation** | Yes | Yes (`--worktree`) | No | No (ephemeral runner) | No |
| **Blueprint/cron** | Export externally | No | No | Schedule trigger | Native |
| **Where it runs** | Docker sandbox | Host | Host or sandbox | GitHub runner | Host, Docker, Modal, SSH |
---
## 7. Implementation phases
### Phase 1: Workflow file and gates (current MVP)
- `archie.yaml` parser
- Gate runner executing shell commands and checking exit codes
- Skill discovery from project and user paths
- GitHub interface: claim issues, create worktrees, open PRs
### Phase 2: Skills
- SKILL.md progressive disclosure (advertise → activate → resources)
- archie-specific metadata parsing
- Skill chaining: skills that depend on other skills
- `archie skills list/install` CLI
### Phase 3: Engine abstraction
- Engine interface definition
- Claude Code adapter
- Raw LLM adapter
- Engine fallback chain
- Per-skill engine preferences
### Phase 4: Advanced features
- Gate overrides per skill
- Skill templates for project scaffolding (`archie init --skill go-service`)
- Remote skill libraries (git repos as skill sources)
- Blueprint export to an external scheduler
### Phase 5: Additional interfaces
- Linear webhook integration
- Generic webhook interface for arbitrary sources
- CLI command interface (`archie run <command>`)
---
## 8. Open questions
1. **Claim protocol:** Single-claim (arcie takes one issue at a time) or queue-based (claims N issues, works through them sequentially)?
2. **Concurrent worktrees:** Allow multiple archie-core instances working on different issues simultaneously?
3. **PR merging:** Should archie-core ever auto-merge, or always require human review?
4. **Skill registry:** Self-host on git.catlow.cloud, use agentskills.io, or both?
5. **Engine budget enforcement:** Per-issue, per-skill, or monthly cap?
6. **Sandbox building:** Pre-built Docker images vs. project-defined Dockerfiles vs. Nix shells?
-400
View File
@@ -1,400 +0,0 @@
# archie-core Workflows & Skills -- Product Requirements Document
**Author:** Archie
**Date:** 2026-07-18
**Status:** Draft
---
## Overview
This PRD defines a workflow and skills system for archie-core. Workflow files declare gates, tool availability, execution engines, and skill references. Skills are reusable playbook documents the agent loads on demand. Together they enable any project to define how agents work within their sandbox and select an available execution engine.
---
## 1. Workflow Files (`archie.yaml`)
### 1.1 Discovery
A project declares its workflow in `archie.yaml` at the project root. archie-core discovers it the same way Task (Taskfile) and devcontainers do:
```
.archie.yaml (root of repo -- highest precedence)
archie.yaml (repo root)
.archie/archie.yaml (subdirectory, one level deep)
```
If no file is found, archie-core uses built-in defaults (read, write, bash).
### 1.2 Schema
```yaml
# archie.yaml -- complete schema
version: "1"
# Project identity
project:
name: my-go-service
language: go
# Available skills (directories or individual files)
skills:
paths:
- .archie/skills/ # project-local skills
- ~/.archie/skills/ # per-user skills
- github.com/org/shared-skills@v1 # remote skill repo
skills:
- go-testing # named skill from any path
- security-audit
# Execution engines in preference order
engines:
- type: claude # Claude Code CLI
model: sonnet
- type: codex # OpenAI Codex
- type: pi # Pi agent harness
- type: raw # Direct LLM without agent wrapper
# Sandbox definition (what the Docker container provides)
sandbox:
image: ghcr.io/samcharles93/archie-sandbox:latest
tools:
- go
- golangci-lint
- gofumpt
- task
- gh
- docker
mounts:
- source: /var/run/docker.sock
target: /var/run/docker.sock
env:
GOFLAGS: "-mod=readonly"
CGO_ENABLED: "0"
# Gates -- ordered checkpoints that must pass
gates:
test:
run: go test -race -count=1 ./...
timeout: 300
description: "All tests must pass"
lint:
run: golangci-lint run ./...
timeout: 120
description: "Zero lint issues"
format:
run: gofumpt -w . && git diff --exit-code
description: "No formatting drift"
vet:
run: go vet ./...
description: "No vet warnings"
deadcode:
run: task deadcode
optional: true # non-blocking gate
description: "Dead code check (advisory)"
# Quality pipeline -- ordered sequence of gate groups
quality:
pre_commit:
- test
- lint
- format
- vet
pre_push:
- deadcode
# Commit conventions
commit:
format: conventional # conventional commits
signoff: false
template: "feat({scope}): {description}"
# PR conventions
pr:
template: .github/PULL_REQUEST_TEMPLATE.md
checks:
- test
- lint
# Project-specific commands (what `archie run <name>` executes)
commands:
build:
run: go build -o bin/app ./cmd/server
description: "Build the binary"
serve:
run: go run ./cmd/server
description: "Start dev server"
test:
run: go test ./...
description: "Run all tests"
lint:
run: golangci-lint run ./...
description: "Run linter"
all:
run:
- gofumpt -w .
- go fix ./...
- golangci-lint run ./...
- go test -race ./...
- task deadcode
description: "Full quality pipeline"
```
### 1.3 Design decisions (with rationale)
| Decision | Rationale |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| YAML over TOML/JSON | Already dominant in CI (GitHub Actions, GitLab CI, Taskfile). Human-readable with comments. |
| Single file over multi-file | Task discovery (like `Taskfile.yml`), devcontainer conventions. Avoids directory sprawl. |
| Gates as named checkpoints | GitHub Actions job model. Gates can be composed into quality pipelines, selected individually, or skipped with `--skip`. |
| Engines as ordered preference list | Supports deterministic fallback between available engines. Each engine declares its own tool surface. |
| Skills as paths + references | Matches devcontainer features (paths resolve to directories) and skills.sh (namespaced references). |
| Commands section for direct invocation | Arnie can `archie run test` without an agent turn. Thin wrapper over the sandbox. |
| Quality pipeline groups gates | Pre-commit vs pre-push vs pre-merge distinction. Some gates are required, some advisory. |
---
## 2. Skills
### 2.1 Format
Skills follow the [agentskills.io](https://agentskills.io/specification) specification:
```
skill-name/
├── SKILL.md # Required: YAML frontmatter + Markdown instructions
├── scripts/ # Optional: executable helpers
├── references/ # Optional: longer docs, checklists
└── assets/ # Optional: templates, configs
```
**SKILL.md frontmatter:**
```yaml
---
name: go-testing
description: >
Run Go tests with race detection, coverage, and benchmark analysis.
Use when asked to test Go code, run the test suite, or check coverage.
version: 1.0.0
metadata:
archie:
gates: [test] # gates this skill requires
tools: [go, golangci-lint] # sandbox tools it needs
engine: any # engine preference
compatibility: Requires Go 1.22+
---
```
### 2.2 Progressive disclosure
Following the skills.sh and Claude Code patterns:
1. **Advertise** (~100 tokens): `name` + `description` loaded at session start
2. **Load** (<5000 tokens recommended): Full SKILL.md body when skill activates
3. **Resources** (as needed): Scripts, references, assets loaded on demand
### 2.3 archie-specific metadata
Skills can carry archie-specific frontmatter under `metadata.archie`:
```yaml
metadata:
archie:
gates: [test, lint] # must-pass gates before skill runs
tools: [go, docker] # sandbox tools required
engine: any # preferred engine
model: gpt-4o # preferred model
budget_usd: 2.00 # max spend ceiling
timeout: 600 # max seconds
worktree: true # create isolated worktree
```
### 2.4 Skill sources (inspired by devcontainer features + Dagger modules)
```
skills:
paths:
- .archie/skills/ # project-local (committed)
- ~/.archie/skills/ # per-user (personal)
skills:
- go-testing # by name from any path
remotes:
- github.com/samcharles93/archie-skills@v1 # tagged git repo
- https://agentskills.io/registry # community registry
```
---
## 3. Execution Engine Model
### 3.1 Engine abstraction
Each engine provides the same interface. The workflow file declares preference order; archie-core resolves the first available engine at runtime.
```
Engine interface:
- Run(prompt, tools[], budget, timeout) -> Result
- Tools() -> []ToolSchema
- Capabilities() -> {streaming, reasoning, vision, agentic}
```
### 3.2 Supported engines
| Engine | Type | What it provides |
| -------- | ---------------- | ------------------------------------------------------ |
| `claude` | Claude Code CLI | Full agent loop, tool use, worktrees, subagents |
| `codex` | OpenAI Codex CLI | Agentic coding with sandbox |
| `pi` | Pi agent harness | Minimalist coding agent, stdio protocol |
| `raw` | Direct LLM call | Single-turn prompt, no agent loop. For simple scripts. |
### 3.3 Engine selection logic
```yaml
engines:
- type: claude
model: sonnet
budget_usd: 5.00
- type: raw
model: gpt-4o-mini
```
arcie-core tries engines in order and selects the first available,
authenticated engine. Each engine declares its budget ceiling.
---
## 4. Additional Features
### 4.1 Worktree isolation (`worktree: true`)
Inspired by Claude Code's `--worktree` flag. When a skill or command sets `worktree: true`, archie-core:
1. Creates an isolated git worktree at `.archie/worktrees/<skill-name>-<timestamp>/`
2. Runs all gates and implementation in that worktree
3. Cleans up on success, preserves on failure for inspection
This prevents concurrent agent sessions from conflicting and enables parallel agent execution.
### 4.2 Skill chaining and composition
Skills can declare dependencies, forming a DAG:
```yaml
---
name: full-ci
description: Complete CI pipeline including security audit
metadata:
archie:
requires: [go-testing, security-audit]
run_sequential: true
---
```
arcie-core resolves the dependency graph and runs skills in order (parallel where possible, respecting `run_sequential`). This is conceptually similar to GitHub Actions `needs` and Taskfile `deps`.
### 4.3 Gate overrides per skill
A skill can override project-level gates:
```yaml
metadata:
archie:
gates:
test:
run: go test -short ./... # override: faster test for this skill
lint: skip # skip lint for this skill
```
Three override modes: `override` (replace command), `skip` (omit gate), `add` (add new gate requirement).
### 4.4 Skill templates (scaffolding)
Skills can carry an `assets/` directory with templates. When archie-core initialises a new project, it can apply skill templates:
```
skill-name/
├── SKILL.md
└── assets/
├── .archie.yaml.tmpl # Template workflow file
├── Taskfile.yml.tmpl # Template task runner
└── .golangci.yml.tmpl # Template linter config
```
`archie init --skill go-service` scaffolds a new project with the go-service skill's templates applied.
### 4.5 Blueprint export (external cron integration)
Any archie-core workflow can be exported as an automation blueprint:
```yaml
metadata:
archie:
blueprint:
schedule: "every 6h"
deliver: telegram
name: "CI monitor"
```
The export produces a `SKILL.md` with scheduler metadata. The blueprint's prompt is the archie workflow's quality pipeline.
---
## 5. Comparison to Existing Systems
| Feature | archie-core | GitHub Actions | Dagger | Taskfile | devcontainer | External Skills |
| ---------------------- | ------------------- | ----------------- | -------------- | ---------- | ----------------- | --------------- |
| Language | YAML | YAML | Go/Python/TS | YAML | JSON | Markdown |
| Gates | Named + composable | Steps in jobs | Functions | Tasks | Lifecycle scripts | N/A |
| Engine abstraction | First-class | GitHub only | Dagger Engine | Shell only | None | Provider |
| Skills | agentskills.io spec | Composite actions | Dagger Modules | Includes | Features | SKILL.md |
| Worktree isolation | Built-in | No | Containers | No | Container | No |
| Blueprint export | To external cron | No | No | No | No | Native |
| Multi-engine fallback | Yes (ordered list) | No | No | No | No | Provider config |
| Progressive disclosure | Yes (skill spec) | No | No | No | Feature metadata | Yes |
---
## 6. Implementation Phases
### Phase 1: Core workflow file
- `archie.yaml` parser and schema validation
- Gate runner (test, lint, format, vet)
- `archie run <command>` CLI
- Sandbox Docker integration
### Phase 2: Skills
- SKILL.md discovery and progressive disclosure
- Skill metadata with gate/tool/engine requirements
- `archie skills list/install/search`
### Phase 3: Engine abstraction
- Engine interface definition
- Claude Code adapter
- Raw LLM adapter
- Engine fallback chain
### Phase 4: Advanced features
- Worktree isolation
- Skill chaining (dependency DAG)
- Gate overrides
- Blueprint export
### Phase 5: Ecosystem
- Public skill registry
- `archie init --skill <name>` scaffolding
- Community skill contributions
---
## 7. Open Questions
1. **Skill registry hosting:** Self-hosted on git.catlow.cloud, or use agentskills.io?
2. **External integration:** Should archie-core be exposed as a tool, a skill, or standalone?
3. **Budget enforcement:** Per-skill budget vs. per-workflow budget vs. monthly cap?
4. **Sandbox image building:** Pre-built images vs. Dockerfile in each skill?
File diff suppressed because it is too large. Load diff
-220
View File
@@ -1,220 +0,0 @@
# Archie Design Decisions
Record of architectural choices, reversals, and lessons learned during
development. Each entry has a date, a decision, the reasoning, and the
context that led to it. This document is the first thing to read before
changing anything that contradicts a prior decision.
---
## 2026-08-24 -- Turn continuation is never output-volume or token gated
**Decision**: A turn is never stopped because an aggregate token count or
aggregate tool-result character count was reached. `max_tokens` and
`turn_budget_chars` are not runtime continuation controls. Tool output is
bounded only per invocation with `max_result_chars`; oversized individual
results are spilled when a workspace-readable spill directory is configured,
or explicitly truncated otherwise.
**Reasoning**: Aggregate caps discard useful work and caused the model to emit
repeated `turn budget exceeded (200000 chars)` tool failures instead of
finishing. Model context/output accounting and tool-result volume are distinct:
provider output limits may describe one model response, but they must never be
used to terminate the multi-step turn.
**Context**: GitHub #604 showed full file and module listings followed by four
identical read failures and a shell failure at the old 200,000-character cap.
The renderer now summarises those legacy events for replay, while current
dispatch never creates them. Find and grep are confined to the active
workspace/worktree; the Go module cache remains readable only by explicit
path, not by discovery.
---
## 2026-07-23 -- Forge-agnostic issue creation via CreateIssue
**Decision**: Add `CreateIssue(ctx, owner, repo, title, body, labels)
(int, error)` to the `forge.Forge` interface. Every forge backend
(Gitea, GitHub, future beads) implements it.
**Reasoning**: Agents discover out-of-scope findings during planning
and building. These should be filed as tickets immediately rather than
lost. Making `CreateIssue` part of the Forge interface means the agent
calls one method regardless of the underlying ticketing system.
**Future**: A `beads` forge backend will implement `CreateIssue` by
calling `bd create`. This makes beads the internal/default issue tracker
while Gitea/GitHub are external forges. The agent doesn't know which
it's talking to -- the daemon resolves the forge from repo config.
---
## 2026-07-23 -- Dolt and Beads for agent-native issue tracking
**Decision**: Install dolt (version-controlled SQL) and beads (`bd`,
a graph-based issue tracker for AI agents) as the foundation for
multi-repo scaling and multi-agent coordination.
**Reasoning**: Beads provides persistent memory across sessions, first-
class dependency tracking, JSON output for programmatic use, and hash-
based IDs that prevent merge conflicts in multi-agent workflows. It's
built on Dolt which gives cell-level merge, native branching, and
remote sync -- the same primitives git gives code but applied to issues.
**Context**: Gas Town (`~/gt/`) is cloned for reference -- it implements
a multi-agent workspace on top of beads. The archie-core scaling
strategy will follow a similar model: a coordinator daemon with per-repo
agents, all sharing a beads-backed task graph.
**Current state**: dolt 2.2.2, beads 1.1.0 installed. Beads initialised
in archie-core (`.beads/`). Six issues seeded for the scaling roadmap.
---
## 2026-07-23 -- Token budgets must be generous, not predictive
> **Superseded by the 2026-08-24 decision above.** There is no configured
> aggregate token cap; model token usage remains accounting only.
**Decision**: Set `max_tokens` to 100,000,000 (100M). Do not attempt to
predict what a task needs without data. Lower limits later based on
actual per-workflow/per-repo metrics.
**Reasoning**: An agent killed mid-task at 1M tokens wastes every token
it already consumed. There is no compaction or checkpointing yet, so a
killed agent is a total loss. Long-running agents regularly use 160+ steps and
hundreds of thousands of tokens per request. Large repositories need
proportionally more.
**Context**: The planner burned 357K tokens just to discover a fix
already existed. The builder hit 1M at step 25 of 90. Neither task was
unreasonable -- they were just exploring a large codebase.
**Future**: Track actual token consumption per workflow/stage/repo. Use
that data to set informed limits. Build compaction or checkpointing so
a killed agent can resume rather than restart.
---
## 2026-07-23 -- baseline-fix: auto-repair pre-existing gate failures
**Decision**: When `StageBaselineGate` detects a red gate (gofumpt, go
vet, etc. failing on clean main), launch the builder to fix the
failures via TDD rather than parking. Only park if the builder cannot
fix them.
**Reasoning**: A red baseline means the repo was broken before archie
touched it. Parking creates an infinite loop -- the next retry hits the
same failure. The agent should fix pre-existing issues, not fail on them.
**Implementation**: `StageBaselineGate` now calls `tc.Agent.Run()` with
a TDD mission when the gate fails. Uses the same builder agent. Note:
this required wiring `tc.Agent` (the runner) through the stage context.
---
## 2026-07-23 -- No hardcoded MaxSteps in workflow stages
**Decision**: Remove all hardcoded `MaxSteps` values from workflow
stages. Every stage inherits from `[budgets].max_steps` in the config.
**Reasoning**: Hardcoded values (15 for plan, 15 for TDD analyse, 15
for feasibility assess, 20 for PRD) were arbitrary and broke on large
repos. The config should be the single source of truth.
**Files affected**: `implement.go`, `tdd.go`, `feasibility.go`.
---
## 2026-07-23 -- No-op detection: close issues when build has 0 changes
**Decision**: When the builder completes with `StatusPassed` and 0 file
changes, close the issue with a comment rather than parking on "worktree
has no changes to commit".
**Reasoning**: A builder that passes with 0 changes means the fix
already exists or the issue is a no-op. The old behaviour parked in an
infinite loop (commit-push fails because nothing changed, task re-queues,
builder runs again, same result).
**Implementation**: `TaskContext.BuildNoChanges` is set in the build
stage's `OnResult` when `Status == Passed && len(Changes) == 0`.
`StageCommitPush` checks this flag and calls `Forge.CloseIssue()` +
sets `Outcome = Merged` to stop the workflow.
**Test**: `TestStageCommitPushClosesIssueWhenBuildNoChanges` in
`implement_test.go`.
---
## 2026-07-23 -- assignee trigger over label trigger
**Decision**: Default to `trigger = "assignee"` instead of `trigger =
"label"`. The daemon only claims issues explicitly assigned to the bot
user.
**Reasoning**: Codex created 80+ issues with the `archie` label, and
the daemon auto-claimed every one. Label-based triggering is too
promiscuous for repos shared between humans and agents.
---
## 2026-07-23 -- DeepSeek v4 Pro as primary model
**Decision**: Use `deepseek/deepseek-v4-pro` as the planner and builder
model. Qwythos-9B (local llama.cpp) is the fallback for offline/cheap
tasks.
**Reasoning**: Qwythos couldn't complete a representative large-repository
plan stage before timing out. DeepSeek completed it in 21 iterations. Large
repositories need a larger model.
**Provider setup**: `class = "deepseek"`, API key via `DEEPSEEK_API_KEY`
env var from Bitwarden Secrets Manager (bws).
---
## 2026-07-22 -- v1.0.0 tagged
**Decision**: Tagged `v1.0.0` at commit incorporating the daemon/agent
split, NATS JetStream, Docker sandbox, Yaegi plugins, agentskills.io
skills, and Gitea forge support.
---
## 2026-07-22 -- Docker Compose stack with NATS + containers
**Decision**: The production deployment uses `docker-compose.yml` with
NATS, archied, and archie-agent containers. Config at `config.docker.toml`.
**Key fixes**:
- NATS healthcheck uses `wget localhost:8222/healthz` (not `nats-server check`)
- `host.docker.internal:host-gateway` extra_hosts for Linux
- Config mount path: `config.docker.toml` → `/etc/archie/config.toml`
- No stale `image:` tag on archied service (forces local build)
---
## 2026-07-22 -- Worktree Prepare before Container Acquire
**Decision**: Clone the worktree (`Trees.Prepare()`) before acquiring
a Docker container. Made `Prepare()` idempotent -- skips if `.git`
already exists.
**Reasoning**: Docker bind-mounts fail with "bind source path does not
exist" if the worktree hasn't been cloned yet. The old code acquired
the container first, then the prepare stage cloned into a path that
didn't exist on the host.
---
## 2026-07-22 -- Gofumpt and Go tools must be on PATH
**Decision**: The daemon's `exec.Command` for gate checks inherits the
process PATH. `~/go/bin` (where `gofumpt`, `golangci-lint`, etc. live)
must be in the PATH before starting the daemon.
**Symptom**: `baseline red -- gofumpt -w . fails` but manual run passes.
Root cause: `which gofumpt` returns nothing because `~/go/bin` isn't on
PATH in external agent sessions.
@@ -1,235 +0,0 @@
# Multi-Agent Collaboration -- PRD
**Status:** Draft (discovery interview complete, 2026-07-25)
**Author:** Discovery interview (archie-core-abg.1), synthesised from Sam's answers
**Parent epic:** archie-core-abg
---
## 0. Framing
Today, "running multiple agents" means running multiple independent `archied`
processes on the same host (`carina`): `archie` (bot_user=archie, multiple
repositories, deepseek models) and `winter` (bot_user=winter, a restricted
repository set, local llama.cpp models). Confirmed by diffing
`~/.config/archie/config.toml` and `~/.config/archie/winter.toml` on carina:
they differ in `bot_user`, `[[repos]]` list, and `[models]`/`[providers]` --
not just identity. `Config.BotUser` is a single global field, so one process
= one identity today.
The goal stated for archie-core overall is to become a fully extensible
agentic workhorse. This PRD
scopes the "multi-agent" slice of that: what it takes to go from N processes
to one daemon that can do what N processes do, plus the surrounding
capabilities (chat channels, plugins, memory) needed to reach that bar.
## 1. Scope (confirmed in interview)
All of the following are **in scope** for the epic (not necessarily the first
implementation slice -- see section 6):
1. **Daemon-side concurrency** -- one daemon process running multiple bot
identities concurrently, replacing N separate `archied` processes.
2. **Cross-repo / cross-config flexibility** -- identities are not statically
bound to one repo list at process startup; repo/model/provider
configuration varies per identity within a single daemon.
3. **Agent-to-agent collaboration on a single task** -- one agent's output
feeding another (e.g. plan -> implement -> review handoff), not just N
identities running in parallel unaware of each other.
4. **Conversational access** -- chat channels (Telegram, web UI, more via
plugins) where Sam talks to archie directly, outside the Gitea
issue-driven flow. Confirmed requirement: **full parity** with the
issue-driven flow -- chat is an alternate entry point into the same
triage/planner/builder dispatch pipeline, not a separate toy Q&A surface.
Chat must be able to: answer questions read-only, spawn/control real work
(start an agent, check status, approve/reject a PR, cancel a run), and
drive the same pipeline Gitea assignment drives today.
5. **General extensibility** -- plugins/channels/integrations should be
addable, not hardcoded:
- Additional forges beyond GitHub/Gitea (e.g. GitLab) via the existing
`forge.Forge` interface, as Layer 2 core plugins (see
`docs/prds/archie-core-v2-architecture.md` sec.5).
- MCP server connections (archie as an MCP client, to consume external
tools/data sources).
- SSH / remote execution as an agent capability.
- Arbitrary API integrations, callable by agents as tools.
- Additional chat channels beyond Telegram/web UI, as plugins.
6. **Memory systems for multi-session chat** -- conversations with archie
need to persist across sessions (distinct from the existing per-task
`/data/memory.jsonl` sketch in the v2 PRD, which is task/project scoped,
not conversation scoped).
7. **Slash-command capabilities** in chat channels (e.g. `/status`, `/spawn`,
`/approve`) for direct channel interaction.
8. **Multi-user support**, as a capability the architecture should not
foreclose -- "if I want to extend my agent to multiple people, I should be
able to." Not required to ship in the first slice (see section 5), but
the identity/session model must not assume single-operator forever.
## 2. Explicitly out of scope (for now)
- **Voice/audio interfaces.** Text-based chat only.
- **Building GitHub support** -- it already exists (`internal/forge`
implements GitHub REST polling per `ARCHITECTURE.md`).
- Anything not listed in section 1 that isn't a natural extension of the
plugin/forge/channel interfaces below.
Note: multi-user auth and GitLab support are **in scope as capabilities the
architecture must support**, but are not committed to the first
implementation slice -- see section 5.
## 3. Constraints from existing architecture
From `ARCHITECTURE.md` and `docs/prds/archie-core-v2-architecture.md`
(read in full before implementation; summarised here):
- **`Config.BotUser` is a single global field** (`internal/config`) -- the
root blocker for daemon-side multi-identity. Becomes a per-identity field
in a new config shape (see section 4).
- **NATS/JetStream task distribution is implemented**: `internal/nats`
publishes tasks to `archie.task.<workflow_type>` subjects; agents
(in-process, subprocess, or Docker containers per `internal/container`)
consume and report back over `archie.agent.<task_id>.{response,system,request}`.
This subject hierarchy is a natural fit for routing by identity too
(e.g. `archie.<identity>.task.<type>`) but that's a design decision, not
yet made.
- **Dispatch is `daemon.Run()` polling per configured forge/repo list**,
serialised per `owner/repo` by default (`AllowConcurrent` opts out).
Multi-identity daemon means multiple poll loops (or one loop iterating
multiple identity configs) feeding the same NATS/container dispatch
machinery.
- **Layer 2 core plugins (`~/.config/archie/plugins/`, Yaegi-loaded,
`Plugin.Register(daemon)`) are designed but not wired** -- interface
exists (`internal/plugin/plugin.go`), no loader in `cmd/archied/main.go`.
This is the natural extension point for new forges, MCP client
connections, and chat channel plugins. Building the multi-agent epic on
top of an unwired plugin layer means **wiring Layer 2 is a prerequisite**,
not parallel work.
- **No chat/messaging code exists today** -- no Telegram, no web UI beyond
the read-only SSE dashboard (`internal/webui`). This is greenfield.
- **The daemon gatekeeps all agent output** -- "no agent output reaches a
human without daemon review" (`archie-core-v2-architecture.md` sec.7).
Chat channels must respect this: a chat message that triggers a spawn goes
through the same review/labelling machinery PRs and comments do today, not
a side channel that bypasses it.
- **Environmental constraints over prompt rules** (`ARCHITECTURE.md` design
decision #1) -- chat-triggered actions (spawn, approve, cancel) should be
enforced by code (permission checks, rate limits), not by asking the LLM
nicely.
## 4. Identity model (design direction, not yet finalized)
Confirmed requirement: within one daemon, identities differ by bot identity,
repo list, _and_ model/provider config (per the archie/winter diff). Proposed
shape -- **not yet approved, needs a follow-up design pass before
implementation**:
```toml
# ~/.config/archie/config.toml (sketch)
[[identities]]
name = "archie"
bot_user = "archie"
forge = { type = "gitea", host = "...", token_env = "..." }
models = { triage = "...", planner = "...", builder = "..." }
repos = [ ... ]
[[identities]]
name = "winter"
bot_user = "winter"
forge = { type = "gitea", host = "...", token_env = "..." }
models = { triage = "...", planner = "...", builder = "..." }
repos = [ ... ]
```
```
**Design answer (2026-07-25): per-identity goroutine, shared infrastructure.**
Each identity gets its own goroutine running an independent poll loop with
its own forge client, worktree manager, and NATS subject namespace
(`archie.<identity>.task.<type>`). They share only the infrastructure
pieces where isolation adds cost without benefit:
| Shared (one instance for all identities) | Per-identity (one per goroutine) |
|---|---|
| SQLite store (`internal/store`) | Forge client (different token per identity) |
| NATS connection (`internal/nats`) | Worktree manager (different `BotUser` per identity) |
| Container pool (`internal/container`) | Config subset: forge, repos, models, providers, dispatch, poll_interval, bot_user, bot_email |
| Event bus (`internal/events`) | NATS subject namespace: `archie.<identity>.task.<type>` |
| Web UI (`internal/webui`) | Poll loop goroutine |
| Plugin registry (`internal/plugin`) | |
| Gateway router (`internal/gateway`) | |
Decision rationale: per-identity goroutines guarantee failure isolation (one
identity's forge outage cannot block another's poll tick), each identity
gets its own forge rate-limit domain, and the existing `Daemon` struct maps
cleanly -- each goroutine holds a thin `Identity` value (forge client + config
subset + worktree manager + NATS subject prefix) and dispatches tasks with
an `identity_name` column on the task row. The shared NATS connection
multiplexes subjects by identity prefix; the shared store already handles
concurrent ClaimNext via SQLite's serialized writes. NATS subjects carry
the identity name so agent containers (which consume from NATS) route
responses back to the correct daemon identity.
This design resolves the open question in section 4 of the original PRD
draft (per-identity goroutine vs single loop iterating identities -- the
former, for failure isolation).
## 5. Priority / phasing
Sam deferred ordering to this PRD (interview answer: "let the interview
settle scope, I'll decide after seeing it written up"). Proposed
dependency-ordered phases, for Sam to confirm or reorder:
**Phase A -- Foundation (blocks everything else)**
- Wire Layer 2 core plugin loading (`~/.config/archie/plugins/`,
`Plugin.Register(daemon)`) into `cmd/archied/main.go`. Currently designed,
unwired -- both chat channels and new forges depend on this existing.
**Phase B -- Multi-identity daemon core**
- Replace single `Config.BotUser` with the `[[identities]]` shape (section
4, after design confirmation).
- Multi-identity poll/dispatch, still issue-driven (Gitea only, no chat yet).
- Migrate the two carina processes (archie, winter) onto one daemon as a
real-world validation.
**Phase C -- Chat channel foundation**
- Chat channel plugin interface (parallel to `forge.Forge`).
- Telegram channel plugin: read-only Q&A against task/daemon state first,
then spawn/control actions gated through the same review pipeline as
issue-driven dispatch.
- Web UI: extend `internal/webui` from read-only SSE dashboard to
bidirectional chat.
- Multi-session conversation memory (distinct persistence from per-task
`/data/memory.jsonl`).
- Slash commands (`/status`, `/spawn`, `/approve`, `/cancel`).
**Phase D -- Agent-to-agent collaboration**
- Plan -> implement -> review handoff within a single task's workflow
(builds on existing `workflow.Stage` sequencing -- this may already be
close given the `implement` workflow's plan/build stage split; needs
investigation into how much is genuinely new vs. reframing).
**Phase E -- Extended integrations (as needed, each is an independent
plugin, no fixed order)**
- GitLab forge plugin.
- MCP client capability (archie consuming external MCP servers as tools).
- SSH / remote execution tool.
- Additional chat channels.
- Multi-user support (identity/session model extended beyond
single-operator -- needs its own design pass; not blocking Phase A-D).
Sam's explicit instruction was to identify next requirements **and begin
work**, not just deliver a document. Phase A (wiring the Layer 2 plugin
loader) is the correct starting point: it's small, well-specified already in
`docs/prds/archie-core-v2-architecture.md` sec.5, and unblocks both the chat
channel work and new forge plugins that everything else in this PRD depends
on.
## 6. Acceptance criteria for this PRD
- [x] Reflects Sam's actual interview answers, not assumptions.
- [x] Confirms the archie/winter config diff (repo lists differ, not just
identity).
- [ ] Broken into scoped, dependency-ordered beads issues under
`archie-core-abg`, the way the 5zh epic was. (Next step, this session.)
```
@@ -1,461 +0,0 @@
# archie-core TypeScript Plugin Engine -- PRD
**Author:** Archie
**Date:** 2026-07-21
**Status:** Draft
---
## Summary
A TypeScript-based plugin engine that lets repositories ship interpreted, dynamically-loaded workflow extensions. Plugins are `.ts` files placed in `.archie/workflows/`. archie-core discovers them on its next poll cycle, transpiles them to JavaScript via esbuild, and executes them in an embedded [goja](https://github.com/dop251/goja) runtime -- a pure-Go ECMAScript interpreter with zero CGO dependencies.
---
## Why TypeScript + goja
### TypeScript as the plugin language
TypeScript is the most widely understood language among developers writing automation, CI, and workflow scripts. It already ships in most projects. Gate authors already know it. The ecosystem for linting, formatting, and type-checking is mature and free.
Having a TypeScript-native plugin engine means:
- Plugin authors get familiar syntax, type safety, and IDE support
- No separate toolchain -- just `.ts` files in a folder
- Skills, gates, and workflow stages share the same language
### Why goja
| Competitor | Problem |
| ------------------ | -------------------------------------------------------------------- |
| V8 CGO bindings | Requires CGO, ~50MB static libraries, platform-specific builds |
| QuickJS CGO | Requires CGO, smaller than V8 but still a C dependency |
| QJS (Wazero) | Pure Go WASM, but sandboxed -- can't expose Go functions efficiently |
| Ramune | Heavy dependency, bundles an entire TypeScript 7 compiler |
| Node.js subprocess | Process overhead per execution, IPC serialisation cost |
goja is a pure-Go ECMAScript 5.1+ interpreter at ~20k lines. It:
- Has **zero CGO** -- cross-compiles everywhere Go does
- Passes nearly all TC39 Test262 tests for ES5.1 + significant ES6 coverage
- Can run Babel and the TypeScript compiler itself (proven capability)
- Provides direct Go↔JS interop via `Set()`/`Get()`/`Runtime.RunString()`
- Has a companion `goja_nodejs` package for `require()`, `setTimeout`, and `console`
- Is used in production by k6 (Grafana's load testing tool, 30k+ stars)
### How TypeScript becomes JavaScript
arcie-core bundles esbuild via `go:embed`. On plugin load, TypeScript source is transpiled to ES2017 JavaScript in-process:
```
.archie/workflows/security-gate.ts → esbuild.Transform() → goja.Runtime.RunString()
```
esbuild is pure Go, takes ~3ms for typical gate scripts, and strips type annotations without type-checking (the plugin author does that in their editor). If the transpilation fails, the plugin is skipped with a diagnostic comment on the issue.
---
## Design
### Plugin discovery
During the poll cycle, archie-core scans each repo's worktree for `.archie/workflows/*.ts`:
```
.archie/workflows/
├── custom-gate.ts # gate functions: exported `check()`
├── db-migration.ts # custom stage: exported `stage()`
└── notify-slack.ts # skill script: exported `run()`
```
Each file is a self-contained module. No `package.json`, no `node_modules`, no external imports. If you need external deps, they need to be exposed via archie-core's Go→JS bridge.
### Plugin types
A plugin declares its role via an exported symbol:
| Export | Role | When called |
| ---------------------------------------------------- | -------------- | --------------------------------------------- |
| `export function check(ctx: GateContext): Finding[]` | Gate function | After shell-based gate, before commit |
| `export function stage(): Stage` | Workflow stage | When a workflow references the stage name |
| `export function run(ctx: ScriptContext): void` | Skill script | When a skill's instructions invoke the script |
arcie-core discovers the role by checking which exports exist. A file can export multiple roles.
### Gate functions
```typescript
// .archie/workflows/custom-gate.ts
interface GateContext {
diff: string; // unified diff of changes
changedFiles: string[]; // list of changed file paths
dir: string; // absolute worktree path
repo: string; // "owner/name"
baseRef: string; // "origin/main"
}
interface Finding {
level: "error" | "warn";
file?: string;
line?: number;
message: string;
}
interface Block {
name: string;
message: string;
}
export function check(ctx: GateContext): Finding[] {
const findings: Finding[] = [];
// Rule: no new panic() calls
for (const line of ctx.diff.split("\n")) {
if (line.startsWith("+") && !line.startsWith("+++")) {
if (line.includes("panic(")) {
findings.push({
level: "error",
message: "new panic() call -- use error returns instead",
});
}
}
}
// Rule: no time.Sleep in non-test files
for (const file of ctx.changedFiles) {
if (file.endsWith("_test.go")) continue;
const content = archie.readFile(`${ctx.dir}/${file}`);
if (content.includes("time.Sleep")) {
findings.push({
level: "warn",
file: file,
message: "time.Sleep in non-test code -- use time.Ticker instead",
});
}
}
return findings;
}
```
After transpilation and execution, archie-core calls `check(ctx)` via goja. Error-level findings park the task. Warn-level findings are logged.
### Custom workflow stages
```typescript
// .archie/workflows/run-migrations.ts
export function stage(): Stage {
return {
name: "db-migration",
run: (ctx: TaskContext): string | null => {
// Check if migrations directory exists
const hasMigrations = archie.fileExists(`${ctx.dir}/migrations`);
if (!hasMigrations) return null; // nothing to do
// Run migration tests
const result = archie.exec("go", ["test", "./migrations/..."]);
if (result.exitCode !== 0) {
return `migration tests failed: ${result.stderr}`;
}
return null; // success
},
};
}
```
### Skill scripts
```typescript
// .archie/workflows/security-audit.ts
export function run(ctx: ScriptContext): void {
const result = archie.exec("gitleaks", ["detect", "--no-git"]);
if (result.exitCode !== 0) {
archie.log("warn", `gitleaks found secrets:\n${result.stdout}`);
}
archie.log("info", "security audit complete");
}
```
### archie Go→JS bridge
arcie-core exposes a `archie` global in the goja runtime:
```typescript
interface Archie {
// File operations (scoped to worktree)
readFile(path: string): string;
fileExists(path: string): boolean;
listDir(path: string): string[];
walkDir(path: string): string[];
// Shell execution
exec(command: string, args?: string[]): ExecResult;
// Logging
log(level: string, message: string): void;
// Path utilities
resolve(...parts: string[]): string;
ext(path: string): string;
// Environment
env(key: string): string;
// HTTP (optional, gated)
fetch(url: string, options?: FetchOptions): FetchResult;
}
interface ExecResult {
stdout: string;
stderr: string;
exitCode: number;
}
```
All file operations are **jailed to the worktree**. `archie.readFile("/etc/passwd")` fails. The bridge enforces this in Go before passing the path to the OS.
### Reload cycle
Plugins are loaded fresh on every poll cycle -- archie-core already fresh-clones per task. No hot-reload complexity, no cache invalidation. The worktree is the source of truth.
For the rare case where a plugin errors at parse/transpile time, archie-core comments on the issue with the diagnostic and parks the task. The plugin author fixes the `.ts` file and requeues.
---
## Implementation
### Architecture
```
internal/plugins/
├── loader.go # scan .archie/workflows/, transpile, evaluate
├── runtime.go # goja runtime pool management
├── bridge.go # archie global (file I/O, exec, log, fetch)
├── types.go # GateContext, Finding, Stage, TaskContext → JS types
├── loader_test.go
└── testdata/
└── workflows/
├── gate.ts # test gate plugin
└── stage.ts # test stage plugin
```
### goja runtime pool
Workflow stages and gates need a clean goja runtime per execution. Creating a new runtime per call is slow (~50ms for stdlib setup). A pool of pre-warmed runtimes solves this:
```go
type Pool struct {
warm chan *goja.Runtime
}
func NewPool(size int) *Pool {
p := &Pool{warm: make(chan *goja.Runtime, size)}
for i := 0; i < size; i++ {
rt := goja.New()
installBridge(rt) // register the `archie` global
installStdlib(rt) // console, setTimeout, etc.
p.warm <- rt
}
return p
}
func (p *Pool) Run(ctx context.Context, src string, globals map[string]any) (goja.Value, error) {
rt := <-p.warm
defer func() { p.warm <- rt }()
// Set per-execution globals
for k, v := range globals {
rt.Set(k, v)
}
return rt.RunString(src)
}
```
### Transpile pipeline
```go
func (l *Loader) transpile(filename string, ts []byte) (string, error) {
result := l.esbuild.Transform(string(ts), api.TransformOptions{
Loader: api.LoaderTS,
Target: api.ES2017,
Sourcefile: filename,
})
if len(result.Errors) > 0 {
return "", fmt.Errorf("transpile %s: %v", filename, result.Errors)
}
return string(result.Code), nil
}
```
### load pipeline
```go
func (l *Loader) LoadPlugins(worktreeDir string) ([]Plugin, error) {
dir := filepath.Join(worktreeDir, ".archie", "workflows")
entries, err := os.ReadDir(dir)
if os.IsNotExist(err) {
return nil, nil // no plugins -- not an error
}
if err != nil {
return nil, err
}
var plugins []Plugin
for _, e := range entries {
if !strings.HasSuffix(e.Name(), ".ts") {
continue
}
path := filepath.Join(dir, e.Name())
src, err := os.ReadFile(path)
if err != nil {
continue
}
js, err := l.transpile(e.Name(), src)
if err != nil {
l.log.Warn("plugin transpile failed", "file", e.Name(), "err", err)
continue
}
rt := l.pool.Acquire()
_, err = rt.RunString(js)
l.pool.Release(rt)
if err != nil {
l.log.Warn("plugin eval failed", "file", e.Name(), "err", err)
continue
}
plugins = append(plugins, discover(rt, e.Name())...)
}
return plugins, nil
}
```
### Export discovery
```go
func discover(rt *goja.Runtime, filename string) []Plugin {
var out []Plugin
if check := rt.Get("check"); check != nil {
out = append(out, Plugin{Kind: "gate", Name: filename, Func: check})
}
if stage := rt.Get("stage"); stage != nil {
out = append(out, Plugin{Kind: "stage", Name: filename, Func: stage})
}
if run := rt.Get("run"); run != nil {
out = append(out, Plugin{Kind: "script", Name: filename, Func: run})
}
return out
}
```
### Wiring into the daemon
In `daemon.process()`:
```go
// After worktree prepare, before gate stage:
plugins, err := d.Plugins.LoadPlugins(tc.Dir)
if err != nil {
d.Log.Warn("plugin load failed", "err", err)
}
tc.Plugins = plugins // available to all stages via TaskContext
// In the gate stage:
if gatePlugin := tc.Plugins.Gate(); gatePlugin != nil {
findings := callGatePlugin(gatePlugin, gateContext(tc))
for _, f := range findings {
if f.Level == "error" {
// park the task
}
}
}
```
---
## Limitations
### goja's ECMAScript coverage
goja natively supports ES5.1 with significant ES6 features. Modern syntax like `async/await`, `??`, `?.`, or `class` fields may not parse. The transpile step (esbuild targeting ES2017) handles most of this, but there will be edge cases:
| Feature | Status |
| ----------------- | -------------------------------------- |
| Arrow functions | ✅ via transpile |
| Template literals | ✅ via transpile |
| Destructuring | ✅ via transpile |
| `const`/`let` | ✅ via transpile |
| `async/await` | ❌ not supported (goja is synchronous) |
| `Promise` | ❌ requires `goja_nodejs` event loop |
| Generators | ❌ not supported |
| `Proxy` | ❌ not supported |
For gate scripts, this is fine -- gates are synchronous by nature. If async is genuinely needed later, the Bun backend (via `tsgo`) is the escape hatch.
### Performance
Gate scripts run once per task. Even a 100ms transpile+eval overhead is negligible compared to the LLM calls in the workflow. If a plugin becomes hot-path, it should be promoted to a compiled Go stage.
### No external imports
Plugins can't `import` from npm or other files. They're self-contained modules. Complexity that needs external deps should be a compiled Go stage, not a plugin. The `archie` global provides filesystem and shell access, which covers the common cases.
---
## Comparison: Yaegi vs goja
| | Yaegi (Go) | goja + esbuild (TypeScript) |
| -------------------- | ----------------------------------------------------------- | --------------------------------------------------------- |
| **Language** | Go | TypeScript |
| **Pure Go** | ✅ | ✅ (goja + esbuild both pure Go) |
| **CGO** | None | None |
| **Transpile step** | None (interpreted directly) | esbuild TS→JS (~3ms) |
| **Author audience** | Go developers | Anyone who knows TypeScript |
| **Type annotations** | N/A (Go is typed) | Stripped by esbuild (no type-check at runtime) |
| **Interop** | Direct Go↔Go via `Use()` | Bridge via `archie` global |
| **Performance** | Interpreted Go (~10x slower than compiled) | Interpreted JS (~50x slower than V8, fine for gates) |
| **Maturity** | 8k stars, used by Traefik for plugins | 7k stars, used by k6, goja is battle-tested |
| **File extension** | `.go` | `.ts` |
| **Plugin folder** | `.archie/workflows/` | `.archie/workflows/` |
| **Best for** | Custom stages needing Go's stdlib, tight daemon integration | Custom gates and scripts where author familiarity matters |
**Recommendation: support both.** They solve different problems. Yaegi for repo owners who want full Go power in their custom stages. goja for repo owners who want to write gates and scripts in TypeScript. Both load from `.archie/workflows/` -- archie-core discovers which engine to use by file extension (`.go` → Yaegi, `.ts` → goja+esbuild).
---
## Implementation phases
### Phase 1: goja runtime + esbuild
- Add `github.com/dop251/goja` and `github.com/evanw/esbuild` as dependencies
- Build `internal/plugins/` with loader, runtime pool, bridge
- Wire gate plugin discovery into the implement and TDD workflows
- Gate plugins run after shell-based gate, before commit
### Phase 2: Custom stages
- Stage plugins discovered and registered alongside built-in stages
- `config.toml` can reference plugin stages by name
- TaskContext exposed to JS plugins
### Phase 3: Skill scripts
- Skill SKILL.md can reference `scripts/security-audit.ts`
- archie-core loads, transpiles, and evaluates when the skill activates
- Script stdout captured and returned to the agent as tool output
### Phase 4: Yaegi side-by-side
- Go plugin files (`.go`) loaded via Yaegi from the same `.archie/workflows/` directory
- Same discovery pattern, different interpreter
- archie-core chooses engine by file extension
---
## Open questions
1. **Jailing:** How strict should the `archie` bridge be? File I/O jailed to worktree, no network access by default? Or trust the repo author?
2. **Error UX:** When a plugin fails to transpile, should archie-core comment on the issue with the specific error, or just park silently?
3. **Runtime pool sizing:** 4 warmed runtimes? Dynamic?
4. **Plugin versioning:** Do plugins get a `version` export? Or is the git commit the version?
5. **TypeScript type definitions:** Ship a `.d.ts` file for the `archie` global so plugin authors get IntelliSense?
-337
View File
@@ -1,337 +0,0 @@
# archie-core Yaegi Plugin System -- PRD
**Author:** Archie
**Date:** 2026-07-21
**Status:** Draft
---
## Summary
Embed the [Yaegi](https://github.com/traefik/yaegi) Go interpreter to let
archie-core load and execute `.go` files at runtime. This enables three things
that shell commands can't do: **custom gates with deep worktree inspection**,
**custom workflow stages defined per-repo**, and **skill scripts written in Go
instead of shell**.
---
## Why Yaegi
### The problem with shell-only gates
`[[repos.gate]]` is command lists. That's fine for standard tooling (`go vet`,
`golangci-lint`), but it can't express project-specific rules:
- "No new `panic()` calls in the diff"
- "No new dependencies in `go.mod` without justification"
- "All public functions in `pkg/api/` must be documented"
- "No `time.Sleep` in production code paths"
These are rules that need to read the diff, parse Go ASTs, or walk the worktree
-- things shell scripts can do but shouldn't have to.
### Why not shell scripts?
Shell scripts for gates are fine until they're not. They can't import the
archie-core types, can't use the Go parser, and are a second language to
maintain alongside the daemon. Yaegi runs Go -- the same language as the daemon,
the same language as the repos archie-core works on.
### Why not Go's native `plugin` package?
Go's `plugin` package (`.so` files) requires:
- Exact same Go version and deps as the host binary
- Linux/macOS only
- No unload
- Painful distribution
Yaegi interprets `.go` source files directly -- no compilation step, portable
everywhere, loads from filesystem paths.
### Production precedent
- **Traefik:** 200+ community plugins via Yaegi, loaded from plugins.traefik.io
- **Vikunja:** Adopted Yaegi for plugin system (see PR #2178), loads plugins
from `plugins/` directory
---
## Design
### Three extension surfaces
| Surface | Location | When loaded | What it does |
| -------------------------- | ------------------------------------- | -------------------------------------------- | --------------------------------------------------------------------- |
| **Custom gate functions** | `.archie/gate.go` | During the gate stage (after shell commands) | Returns `[]Finding` -- each finding has severity, message, file, line |
| **Custom workflow stages** | `.archie/stages/<name>.go` | When the workflow references the stage name | Returns a `workflow.Stage` implementation |
| **Skill scripts** | `.archie/skills/<skill>/scripts/*.go` | When the parent skill is activated | Called by the skill's instructions |
### Gate functions (`.archie/gate.go`)
The most important extension surface. After the shell-based gate passes,
archie-core loads and evaluates `.archie/gate.go` in the worktree directory. The
script exposes a `Check` function:
```go
// .archie/gate.go -- per-repo gate rules interpreted at runtime
package gate
import (
"fmt"
"os"
"path/filepath"
"strings"
)
// GateContext carries everything the gate checker can inspect.
type GateContext struct {
Diff string // unified diff of all changes
ChangedFiles []string // list of changed file paths (repo-relative)
Dir string // absolute worktree path
BaseRef string // base branch (e.g. "origin/main")
Repo string // "owner/name"
}
// Finding is one gate violation.
type Finding struct {
Level string // "error" (blocks commit) or "warn" (advisory)
File string // optional file path
Line int // optional line number
Message string
}
// Block defines a named rule that can prevent passing.
type Block struct {
Name string
Message string
}
// Check returns findings for the gate. Return nil for a clean gate.
func Check(ctx GateContext) []Finding {
var findings []Finding
// Rule: no new panic() calls in committed code
for _, line := range strings.Split(ctx.Diff, "\n") {
if strings.HasPrefix(line, "+") && !strings.HasPrefix(line, "+++") {
if strings.Contains(line, "panic(") {
findings = append(findings, Finding{
Level: "error",
Message: "new panic() call -- use error returns instead",
})
}
}
}
// Rule: no time.Sleep in non-test files
for _, f := range ctx.ChangedFiles {
if strings.HasSuffix(f, "_test.go") {
continue
}
path := filepath.Join(ctx.Dir, f)
b, err := os.ReadFile(path)
if err != nil {
continue
}
if strings.Contains(string(b), "time.Sleep") {
findings = append(findings, Finding{
Level: "warn",
File: f,
Message: "time.Sleep in non-test code -- use time.Ticker or context.WithTimeout instead",
})
}
}
return findings
}
```
The daemon:
1. Reads `.archie/gate.go` from the worktree
2. Creates a Yaegi interpreter with `GateContext` and `Finding` types pre-loaded
via `Use()`
3. Evaluates the file
4. Calls `Check(ctx)` via `Eval("gate.Check")` + `Interface()`
5. Logs findings and blocks the gate if any have `Level: "error"`
### Custom workflow stages (`.archie/stages/`)
A repo can define custom pipeline stages that archie-core discovers and makes
routable:
```go
// .archie/stages/run-migrations.go
package stages
import "github.com/samcharles93/archie-core/internal/workflow"
func Stage() workflow.Stage {
return workflow.Stage{
Name: "db-migration",
Run: func(ctx context.Context, tc *workflow.TaskContext) error {
// Check if go.mod contains migration tools
// Run migration tests
// Verify schema compatibility
return nil
},
}
}
```
Registered stages appear alongside built-in stages. A workflow in `config.toml`
can reference them by name:
```toml
[[workflows.stages]]
name = "db-migration" # matches the .go file
source = "repo" # loaded from .archie/stages/
```
### Skill scripts (`.archie/skills/<skill>/scripts/`)
Skills can ship Go scripts that archie-core interprets:
```go
// .archie/skills/security-audit/scripts/gitleaks.go
package main
import (
"fmt"
"os/exec"
"context"
)
func main() {
ctx := context.Background()
cmd := exec.CommandContext(ctx, "gitleaks", "detect", "--no-git", "--verbose")
out, _ := cmd.CombinedOutput()
fmt.Println(string(out))
}
```
Skills reference these scripts naturally in their instructions: "Run
`scripts/gitleaks.go` to scan for secrets."
---
## API surface archie-core exposes to Yaegi
Via `i.Use()`, the daemon pre-loads symbols that interpreted code can import:
| Package | What's exposed |
| ------------------------------- | ----------------------------------------- |
| `archie-core/internal/workflow` | `Stage`, `TaskContext`, `Outcome` |
| `archie-core/internal/gate` | `GateContext`, `Finding`, `Block` |
| `stdlib` | Full Go standard library (stdlib.Symbols) |
Generated with `yaegi extract` -- same approach Traefik and Vikunja use. A
`//go:generate yaegi extract ...` line in the source triggers symbol table
generation during `go generate`.
---
## Limitations
From Yaegi's own docs:
- **No assembly files** (`.s` not supported)
- **No CGo** (can't call C code -- the daemon is pure Go, so this doesn't
matter)
- **No compiler/linker directives** (no `//go:embed`, etc. -- fine for gate
scripts)
- **Interfaces from pre-compiled code can't be added dynamically** -- the daemon
must pre-compile wrapper types for any interface interpreted code needs to
implement
- **Slower than compiled code** -- gate scripts run once per commit, not in hot
paths; negligible impact
The Vikunja PR documents a specific Yaegi quirk: **typed factory functions
required.** You can't do `v.Interface().(MyInterface)` on a value returned from
interpreted code. Each interface needs its own factory:
```go
func NewGate() Gate { return &MyGate{} }
func NewReportable() Reportable { return &MyGate{} }
```
---
## Implementation phases
### Phase 1: Gate functions -- implemented
- `yaegi` added as a dependency
- Symbol table for `GateContext`, `Finding` generated via `yaegi extract` into
`internal/gate/gateextract` (a separate package from `internal/gate` itself,
avoiding the self-import cycle extracting a package's own symbols creates)
- `internal/gate/gateeval.Evaluate`: loads `.archie/gate.go` from the worktree,
interprets it, calls `Check()`; panics are recovered
- Wired in as `workflow.StageYaegiGate`, run after the final commit in both the
`implement` and `tdd` workflows
- Error findings park the task; warn findings are logged only
### Phase 2: Custom stages -- implemented
- Symbol table for `workflow.Stage`/`TaskContext`/`Outcome` generated into
`internal/workflow/wfextract`
- `internal/workflow/wfeval.Discover` scans `.archie/stages/*.go` in filename
order and calls each file's exported `Stage()` function
- `TaskContext.CustomStages` is an injected `func(dir string) ([]Stage, error)`
-- the composition root (`cmd/archied`) wires it to `wfeval.Discover`, keeping
`internal/workflow` decoupled from its own generated symbol table
- `workflow.StageRepoStages` runs discovered stages in order right after the
worktree is prepared, in both `implement` and `tdd`
- Simplification vs. the original sketch: stages are auto-discovered and always
run (like `.archie/gate.go`) rather than referenced by name from `config.toml`
-- the workflow engine has no data-driven stage graph today, so wiring
arbitrary named stages into arbitrary positions would need a larger
config-driven rework. Auto-run-in-order covers the PRD's actual use case (a
repo-defined pre-flight/setup hook) without it.
### Phase 3: Skill scripts -- implemented
- `internal/skillscript.Run` interprets a `.go` file via `i.EvalPath`, which
(like `go run`) evaluates and calls its `main()`; stdout/stderr are captured
via `interp.Options`
- Uses `stdlib.Symbols` + `stdlib/unrestricted.Symbols` (not just `stdlib`) --
`os/exec` lives in the unrestricted set, and the PRD's own example script
shells out via `os/exec`
- No archie-core-specific symbol table needed here -- skill scripts only touch
the Go standard library
- Exposed to the agent loop as a new `run_go_script` tool
(`internal/agentexec/inprocess.go`), available in every agent stage; path is
validated to stay inside the workspace
- This is how "the skill's instructions reference a script" cashes out
concretely: the agent calls `run_go_script` with the path instead of needing a
Go toolchain in the sandbox
### Phase 4: Hot reload -- not needed, by construction
- All three loaders (`gate.Evaluate`, `wfeval.Discover`, `skillscript.Run`) read
their `.go` source from disk on every call -- no in-memory cache exists
anywhere in the implementation
- Every task additionally gets a fresh `git clone` (`worktree.Manager.Prepare`
removes and re-clones the worktree directory per task)
- Consequently every stage already re-reads `.archie/gate.go`,
`.archie/stages/*.go`, and skill scripts fresh, every run -- there is no stale
state a file watcher would ever need to invalidate
- A file-watching daemon-runtime reload mechanism would add machinery with
nothing to do given this architecture; revisit only if the daemon starts
reusing long-lived worktrees across tasks
---
## Open questions
1. **Symbol table generation:** Run `yaegi extract` as a `go generate` step, or
generate on daemon startup?
2. **Error handling:** If the interpreted gate panics or times out, park the
task? Or fall back to shell-only gate?
3. **Budget:** Interpreted gate scripts run in the gate stage -- should they
have their own timeout/step budget?
4. **Template system:** Should archie-core ship with common gate templates (e.g.
`go: no panic, no Sleep, no hardcoded paths`)?
5. **Remote gates:** Could `.archie/gate.go` be pulled from a shared remote
registry, like skills?