Commit Graph
7 Commits
Author SHA1 Message Date
sam d80401d595 feat: add config overlay and shared RPC/Yaegi packages
- Introduce `-config-overlay` flag and `config.LoadOverlay()` to merge a base config with an overlay file, only overriding fields the overlay sets. This enables docker-compose to mount the base config from the host and apply only deployment-specific overrides.
- Create `internal/natsrpc` package with reusable NATS request/reply primitives (Envelope, Client, Call, RegisterAll, Respond). Migrate storerpc, worktreerpc, and forgerpc to use it, eliminating duplicated marshal/unmarshal and error handling.
- Create `internal/yaegiutil` package with shared interpreter construction (stdlib + extra symbols), generic export resolution (Resolve), and panic recovery (Safe). Migrate all Yaegi-based surfaces (gate scripts, daemon plugins, skill plugins, workflow stages, skill scripts) to use it, reducing boilerplate and centralizing panic safety.
- Simplify `config.docker.toml` to contain only overlay fields (paths, container config) and remove the duplicated repos list – the base config is now the single source of truth.
- Update docker-compose.yml to pass both -config and -config-overlay flags and mount the base config from the host.
- Add tests for LoadOverlay, DockerConfigOverlaysExampleConfig, Envelope, and yaegiutil utilities.
2026-07-24 12:57:10 +10:00
sam f8df051b19 feat: bounded concurrent task dispatch in daemon (archie-core-u64)
Daemon previously processed one task at a time. Add bounded concurrent
dispatch for SQLite and NATS agent modes, reusing [containers].max_concurrency
(0 = unlimited). Same-repository tasks stay FIFO-serialized to avoid worktree
conflicts; different repositories now run concurrently.

Unblocks archie-core-l1q (NATS transport) and archie-core-g1l
(per-repo allow_concurrent override).
2026-07-24 07:42:13 +10:00
sam 8a8e676bd6 fix: address 6 'Soon' resilience audit findings
R2  (resilience): skip broken stage plugins, don't abort daemon startup.
    BuildWorkflow now skips individual broken plugins with a warning
    instead of hard-failing. BuildRegistry and AugmentRegistry skip
    broken skills with a warning instead of returning an error. One
    bad plugin in one skill no longer prevents the daemon from starting
    or blocks other skills from registering their workflows.

R1  (resilience): log skipped daemon plugins in LoadDir. Every failure
    path now emits a WARN log via slog.Default() — unreadable files,
    compile errors, missing Plugin exports, wrong types. Operators can
    now see WHY a plugin was skipped. PRD says 'logged and skipped';
    code now matches spec.

N3  (resilience): log symbol-table load errors separately from plugin
    source errors. A broken pluginextract.Symbols no longer causes
    silent skip of ALL plugins — each failure is logged with the
    symbol table index that failed.

N4  (resilience): centralize Yaegi interpreter setup in
    newStageInterpreter(). Each plugin file still needs its own
    interpreter (package main with duplicate Stage() symbols cannot
    co-evaluate), but the setup is now explicit and documented.

N5  (resilience): document tuple fallback positional-field fragility.
    The func() (string, func(...)) fallback in loadStage now warns
    that it constructs Stage with positional fields and will silently
    zero-value any future Stage fields.

R3  (resilience): document nil-client edge case in ensureVolume.

Tests: TestBuildWorkflowSkipsBrokenPlugin (broken plugin skipped,
surviving stages execute), TestBuildRegistrySkipsBrokenSkill (bad
skill doesn't block others, built-in fallbacks preserved).
2026-07-22 04:55:26 +10:00
sam 04175a1a6a fix: address 9 'Now' adversarial audit findings
P1  (panic):      nil Storage guard in daemon.process()
C1  (correctness): check WriteTaskJSON error — don't start blind
C2  (correctness): move 'working' label after container setup
C3  (correctness): warn on duplicate workflow names in registry
C4  (correctness): test that generated _Plugin wrapper has nil-guards
C5  (correctness): .agents/skills as file (ENOTDIR) not fatal
C6  (correctness): ecosystem case-insensitive in cacheMounts
N1  (correctness): TOCTOU race — treat 'already exists' as success
N2  (correctness): recover from panicking plugin.Name() at startup

--- Details ---

P1: d.Storage.Setup() is now guarded by a nil check. In normal
operation Storage is always set when ContainerPool is set, but
the invariant is now explicit and enforced.

C1: container.WriteTaskJSON() error was discarded with _. Now it
returns early with a park transition — the container never starts
without a boot-time brief.

C2: SetStateLabel('working') was called before container setup.
If Acquire failed, the task was parked but the label stayed
'working' permanently. Now the label is set only after successful
container setup (or immediately if no containers are configured).

C3: BuildRegistry and AugmentRegistry now log a warning when two
skills declare the same workflow name and one overwrites the other.

C4: TestGeneratedWrapperHasNilGuards verifies that nil WName/
WVersion don't panic. If go generate strips the nil-guards, this
test catches it before the daemon crashes.

C5: skill.Catalog() now returns nil (no skills) instead of
propagating ENOTDIR errors when .agents/skills is a file.

C6: cacheMounts() normalizes ecosystem to lowercase. 'Python',
'NODE', 'Go' all resolve correctly now.

N1: ensureVolume() treats Docker's 'already exists' error as
success, closing the TOCTOU window between VolumeInspect and
VolumeCreate.

N2: safePluginInfo() wraps plugin.Name()/Version() calls in a
recover. A plugin with a panicking Name() no longer crashes the
daemon at startup.
2026-07-22 04:50:11 +10:00
sam a9a3f7b3f9 feat: per-worktree dynamic registry augmentation
When a worktree already exists on disk (retry, waiting_human ->
implement handoff), the daemon now scans it for .agents/skills/ that
declare metadata.archie.workflow and augments the startup registry
with any newly discovered workflows. Worktree skills take precedence
over the startup registry for that task only.

Changes:
- skillbuild.AugmentRegistry(worktree, base) — scans a worktree
  directory for skills, builds workflows from plugins, returns a
  new registry with worktree entries merged over base. Base is never
  mutated. Mirrors BuildRegistry but starts from caller's registry
  instead of builtins.
- daemon.process() — computes worktree path, calls AugmentRegistry
  if the directory exists, uses augmented registry for routing.
  Degrades gracefully to startup registry on error.

Tests (10 new):
- Adds worktree workflows to base registry
- Worktree skills override same-named base entries
- Base registry is not mutated (copy semantics)
- Nonexistent worktree returns base unchanged
- Worktree with no .agents/skills/ returns base unchanged
- Worktree skill with no plugins does not override base
- Adversarial: concurrent augmentation is safe
- Adversarial: broken SKILL.md is skipped
- Adversarial: bare worktree (no .agents/) does not error
- Adversarial: nil base is treated as empty

Why: skills are per-project (.agents/skills/ in each repo). The
startup registry only knows about a shared skills directory. Per-
worktree augmentation means a repo can define its own workflows
without restarting the daemon.
2026-07-22 04:18:12 +10:00
sam 8958a82233 feat: replace hardcoded workflow registry with catalog-driven skillbuild.BuildRegistry()
The daemon no longer hardcodes workflow.Registry{...} in main.go.
Instead, skillbuild.BuildRegistry() scans .agents/skills/ for catalog
entries declaring metadata.archie.workflow and builds each workflow
from its Yaegi-interpreted stage plugins.

Built-in workflows (bootstrap, implement, tdd, feasibility, default)
serve as fallbacks when no skill declares a given workflow name. A
skill that declares metadata.archie.workflow: implement overrides the
built-in — skills define workflows, plugins define stages.

Why: skills must be extension primitives. Adding a workflow should
mean creating .agents/skills/archie-wf-<name>/plugins/*.go, not
modifying Go code in main.go. The catalog is the single source of
truth for what workflows exist.
2026-07-22 04:02:58 +10:00
sam bba6827ec2 feat: build workflows from skill stage plugins
New skillbuild package: BuildWorkflow() loads Yaegi-interpreted stage
plugins from a skill's plugins/ directory and constructs a runnable
workflow.Workflow. Uses wfextract symbol tables for type access.

A skill directory with metadata.archie.workflow: greet and stage plugins
in plugins/*.go IS the workflow definition — no Go code changes needed.

Stage plugins export:
  func Stage() workflow.Stage
or the fallback signature:
  func Stage() (string, func(context.Context, *workflow.TaskContext) error)

Stages are loaded in filename order. Missing plugins dir is not an error
(matches wfeval.Discover pattern).

Test: skill with two plugins (hello, world) produces a 2-stage workflow
that modifies BuildSummary sequentially.
2026-07-22 03:50:00 +10:00