feat: wire Layer 2 daemon plugin loader into archied startup

Daemon plugins (Yaegi-interpreted .go files) are now loaded from the
configured plugin_dir at startup. Each file must export a 'Plugin'
variable using the _Plugin wrapper struct, which bridges Yaegi types
to the Go plugin.Plugin interface.

Changes:
- internal/plugin/pluginextract/ — generated Yaegi symbol table for
  the plugin package (mirrors wfextract pattern), including the
  _Plugin interface wrapper that lets interpreted code implement
  Go interfaces
- LoadDir now accepts optional symbol tables via variadic parameter,
  avoiding an import cycle between plugin and pluginextract
- Plugin tests rewritten as behavioral (not structural), verifying
  actual Yaegi interpretation of valid, invalid, and missing plugins
- config.PluginDir field for specifying the plugin directory
- daemon.PluginRegistry field for holding loaded plugins
- main.go wires plugin loading: LoadDir at startup, register into
  daemon, log each loaded plugin

Why: the Plugin interface and LoadDir existed but weren't connected
to the daemon. Skills define workflows; daemon plugins are the
extension point for forge resolvers, storage backends, and other
core-daemon concerns.
This commit is contained in:
sam committed 2026-07-22 04:11:02 +10:00
1 parent 8958a82233
commit f38ba5d515
8 files changed
+256 -35

No files matched your search

+21 -2
View File
@@ -21,6 +21,8 @@ import (
"github.com/samcharles93/archie-core/internal/events"
"github.com/samcharles93/archie-core/internal/forge"
"github.com/samcharles93/archie-core/internal/nats"
"github.com/samcharles93/archie-core/internal/plugin"
"github.com/samcharles93/archie-core/internal/plugin/pluginextract"
"github.com/samcharles93/archie-core/internal/store"
"github.com/samcharles93/archie-core/internal/webui"
"github.com/samcharles93/archie-core/internal/workflow/skillbuild"
@@ -176,6 +178,22 @@ func run() int {
}
log.Info("workflow registry built", "workflows", len(registry))
// Load daemon plugins from the configured plugin directory (Layer 2).
// Failed plugins are skipped — the daemon starts with the remaining set.
var pluginReg *plugin.Registry
if cfg.PluginDir != "" {
plugins, err := plugin.LoadDir(cfg.PluginDir, pluginextract.Symbols)
if err != nil {
log.Error("plugin load failed", "dir", cfg.PluginDir, "err", err)
return 1
}
pluginReg = &plugin.Registry{}
for _, p := range plugins {
pluginReg.Register(p)
log.Info("daemon plugin loaded", "name", p.Name(), "version", p.Version())
}
}
d := &daemon.Daemon{
Cfg: cfg,
Store: st,
@@ -190,8 +208,9 @@ func run() int {
},
Runtime: llm,
Agent: agentRunner,
Workflows: registry,
Log: log,
Workflows: registry,
PluginRegistry: pluginReg,
Log: log,
CustomStages: wfeval.Discover,
Nats: natsClient,
ContainerPool: containerPool,
+6 -6
View File
@@ -2,7 +2,7 @@
**Author:** Archie (Hermes agent)
**Date:** 2026-07-21 (implementation status updated 2026-07-22)
**Status:** Final
**Status:** Final (substantially implemented — remaining aspirational items are pluggable storage backends and Layer 2 daemon plugin loader)
---
@@ -16,16 +16,16 @@ This document describes a target end state. The table below is the ground truth;
| Sandbox / "daemon never runs untrusted code" | **Partial** | Default mode is still `inprocess` for development. Docker sandbox mode (`agent.mode = "nats"` + `containers.enabled = true`) runs each task in an ephemeral Docker container with workspace bind-mounted at `/workspace`. `internal/container/pool.go` manages full lifecycle: pull, create, start, release, orphan recovery. `max_concurrency` enforced. |
| NATS / JetStream messaging | **Implemented** | `internal/nats/client.go` (176 lines): JetStream connect, stream creation (WorkQueue retention, file storage), pull consumer (durable, ack-wait 5min, max-deliver 3, dedup by Msg-Id), `PublishTask`, `Fetch`, `NewReplyInbox`. Subject hierarchy in `internal/nats/subjects.go`. `[nats]` config block in config.go. `internal/agentexec/nats.go` implements NATS-based agent runner. |
| Docker container sandbox | **Implemented** | `internal/container/pool.go` (210 lines): Docker API client, image pull (always/missing/never), container create with workspace bind mount, start, release (stop+remove), orphan recovery on daemon restart. `Dockerfile` at repo root builds archie-agent fat image. `[containers]` config block: enabled, image, max_concurrency, max_uptime, pull_policy. `agent.mode = "nats"` required when containers enabled. |
| `/data/` volume, `task.json` | **Partial** | No `/data/task.json` file convention exists. Containers mount a workspace directory at `/workspace` (bind mount, host path). Task payload travels over NATS, not a volume file. Worktrees are still ephemeral fresh clones per task. |
| `/data/` volume, `task.json` | **Partial** | `container.WriteTaskJSON()` writes task payload to the worktree before container acquire. The container pool bind-mounts the worktree at `/data/worktree`. Task payload travels over NATS as the primary channel; `/data/task.json` is a boot-time brief for the container. Worktrees are still ephemeral fresh clones per task. |
| `[containers]` config block | **Implemented** | `ContainerConfig` struct in config.go: enabled, image, max_concurrency, max_uptime, pull_policy. Validated on load. |
| Yaegi extensibility (gates, custom stages, skill scripts) | **Implemented** | Three surfaces: `.archie/gate.go`, `.archie/stages/*.go`, skill scripts via `run_go_script` tool. All in-process via Yaegi with generated symbol tables and panic recovery. Full detail in `docs/prds/yaegi-plugin-system.md`. |
| Layer 1: skill-bundled plugins | **Aspirational** | No skill ships a `plugins/` directory. No code reads `metadata.archie.plugins`. No volume-staged plugin loading. |
| Layer 2: core daemon plugins | **Aspirational** | No `Plugin` interface, no registry, no `~/.config/archie/plugins/` loader. |
| Skills (agentskills.io spec) | **Partial** | `internal/skill/skill.go` (116 lines): parses SKILL.md YAML frontmatter, discovers skills from `.agents/skills/*/SKILL.md`, extracts body. Skills moved from `.archie/skills/` to `.agents/skills/`. No `archie-wf-*` naming convention yet. No progressive disclosure -- body injected directly. |
| Layer 1: skill-bundled plugins | **Implemented** | `skill.DiscoverPlugins()` scans `.agents/skills/<name>/plugins/*.go`. `skill.Plugin.Run()` executes them via Yaegi. `skillbuild.BuildWorkflow()` constructs `workflow.Workflow` from plugin-defined stages. `TaskContext.SkillPlugins` makes them available during stage execution. `skillbuild.BuildRegistry()` replaces the hardcoded `workflow.Registry` with catalog-driven construction — skills define workflows, plugins define stages. |
| Layer 2: core daemon plugins | **Partial** | `internal/plugin/plugin.go`: `Plugin` interface (`Name()`, `Version()`), `Registry` type, `LoadDir()` using Yaegi to interpret `.go` files. No `~/.config/archie/plugins/` loader wired into `cmd/archied/main.go` yet — the interface and registry exist but are not connected to the daemon's startup. |
| Skills (agentskills.io spec) | **Implemented** | `internal/skill/skill.go`: parses SKILL.md YAML frontmatter, discovers skills from `.agents/skills/*/SKILL.md`, extracts body. Skills follow `archie-wf-*` naming convention. Progressive disclosure: `Catalog()` (Tier 1 — frontmatter-only scan, ~100 tokens per entry), `LoadBody()` (Tier 2 — full body on activation), `Discover()` (Tier 3 — full parse with plugins). `metadata.archie.workflow` declares skill→workflow affinity. `skill.SkillForWorkflow()` is catalog-driven, no hardcoded maps. `skillbuild.BuildRegistry()` constructs the daemon's workflow registry from the catalog. |
| Worktree/repo management | **Implemented, ephemeral** | Fresh `git clone` per task attempt, no shared cache, no persistence, no TTL. |
| Config schema | **Closer to sec.4** | `[agent]` block: mode (inprocess/subprocess/nats) + command + env. `[nats]` block: url + token_env. `[containers]` block: enabled, image, max_concurrency, max_uptime, pull_policy. No per-repo persistent_storage override yet. |
**Net read post-e4d11b1:** Phases 1 and 2 are substantially implemented -- NATS, Docker sandbox, skill parsing all landed in three commits. The remaining aspirational work is core plugin registry (Layer 2), skill-bundled plugins (Layer 1), skill progressive disclosure, and pluggable storage backends. The PRD is now more "current state with aspirational sections" than "entirely aspirational."
**Net read post-1dca419:** Phases 1, 2, and 3 are substantially implemented. NATS, Docker sandbox, skill parsing, skill-bundled plugins (Layer 1), skill progressive disclosure, and catalog-driven workflow construction (`skillbuild.BuildRegistry()`) all landed. The remaining aspirational work is Layer 2 daemon plugin loader (interface exists, not wired), pluggable storage backends, and per-worktree dynamic registry augmentation.
---
+6
View File
@@ -210,6 +210,12 @@ type Config struct {
// (plugin-defined workflows override built-ins). When empty,
// only built-in workflows are available.
SkillsDir string `toml:"skills_dir"`
// PluginDir is an optional path to a directory of Yaegi-interpreted
// daemon plugins (*.go files). Each file must export a "Plugin"
// variable satisfying the plugin.Plugin interface. Failed plugins
// are skipped — the daemon starts with the remaining set. When
// empty, no daemon plugins are loaded.
PluginDir string `toml:"plugin_dir"`
DBPath string `toml:"db_path"`
PollInterval Duration `toml:"poll_interval"`
// Label marks issues archie should pick up.
+4
View File
@@ -24,6 +24,7 @@ import (
"github.com/samcharles93/archie-core/internal/events"
"github.com/samcharles93/archie-core/internal/forge"
arnats "github.com/samcharles93/archie-core/internal/nats"
"github.com/samcharles93/archie-core/internal/plugin"
"github.com/samcharles93/archie-core/internal/store"
"github.com/samcharles93/archie-core/internal/workflow"
"github.com/samcharles93/archie-core/internal/worktree"
@@ -45,6 +46,9 @@ type Daemon struct {
// ContainerPool manages Docker container lifecycle. Nil when [containers]
// is not configured. When non-nil, every task gets a fresh container.
ContainerPool *container.Pool
// PluginRegistry holds core daemon plugins loaded from the configured
// plugin_dir at startup (Layer 2). Nil when no plugin_dir is configured.
PluginRegistry *plugin.Registry
// CustomStages discovers a repo's per-repo Yaegi custom stages
// (.archie/stages/*.go) from its prepared worktree. Set by the
// composition root (cmd/archied) to wfeval.Discover; nil disables
+12 -1
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"os"
"path/filepath"
"reflect"
"sort"
"strings"
@@ -40,7 +41,12 @@ func (r *Registry) Plugins() []Plugin { return r.plugins }
// Each file must export a variable named "Plugin" that implements the
// Plugin interface. Failed plugins are logged and skipped — the daemon
// starts with the remaining plugins (PRD section 5).
func LoadDir(dir string) ([]Plugin, error) {
//
// extraSymbols are additional Yaegi symbol tables made available to
// interpreted code. Callers should pass pluginextract.Symbols so that
// interpreted types can satisfy the plugin.Plugin interface across
// the Yaegi/Go boundary.
func LoadDir(dir string, extraSymbols ...map[string]map[string]reflect.Value) ([]Plugin, error) {
entries, err := os.ReadDir(dir)
if os.IsNotExist(err) {
return nil, nil
@@ -68,6 +74,11 @@ func LoadDir(dir string) ([]Plugin, error) {
if err := i.Use(stdlib.Symbols); err != nil {
continue
}
for _, s := range extraSymbols {
if err := i.Use(s); err != nil {
continue
}
}
if _, err := i.Eval(string(src)); err != nil {
continue // skip files that don't compile
}
+161 -26
View File
@@ -1,45 +1,180 @@
package plugin
package plugin_test
import "testing"
import (
"os"
"path/filepath"
"testing"
// ── regression: Core plugin registry nonexistent ─────────────────────
"github.com/samcharles93/archie-core/internal/plugin"
"github.com/samcharles93/archie-core/internal/plugin/pluginextract"
)
// ── Plugin interface and registry ────────────────────────────────────
func TestPluginInterfaceExists(t *testing.T) {
// PRD section 5 Layer 2: plugins implement Name(), Version().
// The Plugin interface must be usable for type assertions from
// Yaegi-interpreted code.
var p Plugin
_ = p // ensures the type exists at compile time
// Verify the methods exist on the interface.
var p plugin.Plugin
_ = p
type nameVersioner interface {
Name() string
Version() string
}
var _ nameVersioner = p // compile-time check
var _ nameVersioner = p
}
func TestRegistryExists(t *testing.T) {
// A registry holds loaded plugins and dispatches extension points.
r := &Registry{}
r := &plugin.Registry{}
if r == nil {
t.Error("Gap: Registry type is not defined. " +
"Define a Registry that holds loaded plugins and dispatches " +
"extension calls (forge, ticketing, storage, secrets, notify).")
t.Fatal("Registry type is not defined")
}
}
func TestLoadPluginsFromConfigDir(t *testing.T) {
// The daemon loads .go files from ~/.config/archie/plugins/ at
// startup. Failed plugins are logged and skipped.
plugins, err := LoadDir("/nonexistent/path")
if err != nil {
t.Error("Gap: LoadDir function is not defined. " +
"Define LoadDir(path string) ([]Plugin, error) that discovers " +
"and evaluates .go files from ~/.config/archie/plugins/.")
// ── LoadDir behavioral tests ─────────────────────────────────────────
var symbols = pluginextract.Symbols
func TestLoadDirLoadsValidPlugin(t *testing.T) {
// LoadDir must Yaegi-interpret .go files and return Plugins whose
// exported "Plugin" variable satisfies the Plugin interface.
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "hello.go"), []byte(`package main
import "github.com/samcharles93/archie-core/internal/plugin"
var Plugin = plugin._Plugin{
WName: func() string { return "hello" },
WVersion: func() string { return "1.0.0" },
}
`), 0o644); err != nil {
t.Fatal(err)
}
plugins, err := plugin.LoadDir(dir, symbols)
if err != nil {
t.Fatal(err)
}
if len(plugins) != 1 {
t.Fatalf("LoadDir returned %d plugins, want 1", len(plugins))
}
if plugins[0].Name() != "hello" {
t.Errorf("plugin.Name() = %q, want hello", plugins[0].Name())
}
if plugins[0].Version() != "1.0.0" {
t.Errorf("plugin.Version() = %q, want 1.0.0", plugins[0].Version())
}
}
func TestLoadDirSkipsInvalidPlugin(t *testing.T) {
// A .go file that doesn't compile must be skipped — not crash the loader.
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "broken.go"), []byte(`package main
this is not valid Go syntax @@@@
`), 0o644); err != nil {
t.Fatal(err)
}
plugins, err := plugin.LoadDir(dir, symbols)
if err != nil {
t.Fatal(err)
}
if len(plugins) != 0 {
t.Errorf("LoadDir returned %d plugins from broken source, want 0", len(plugins))
}
}
func TestLoadDirSkipsFileWithoutPluginExport(t *testing.T) {
// A valid Go file without a "Plugin" export must be skipped.
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "notaplugin.go"), []byte(`package main
func Something() string { return "nope" }
`), 0o644); err != nil {
t.Fatal(err)
}
plugins, err := plugin.LoadDir(dir, symbols)
if err != nil {
t.Fatal(err)
}
if len(plugins) != 0 {
t.Errorf("LoadDir returned %d plugins from file without Plugin export, want 0", len(plugins))
}
}
func TestLoadDirSkipsNonGoFiles(t *testing.T) {
// Only .go files are loaded. Other files are ignored.
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "readme.md"), []byte(`# Not a plugin`), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "hello.go"), []byte(`package main
import "github.com/samcharles93/archie-core/internal/plugin"
var Plugin = plugin._Plugin{
WName: func() string { return "hello" },
WVersion: func() string { return "1.0.0" },
}
`), 0o644); err != nil {
t.Fatal(err)
}
plugins, err := plugin.LoadDir(dir, symbols)
if err != nil {
t.Fatal(err)
}
if len(plugins) != 1 {
t.Fatalf("LoadDir returned %d plugins, want 1 (only .go files count)", len(plugins))
}
}
func TestLoadDirMultiplePlugins(t *testing.T) {
// Multiple valid plugins must all be loaded, sorted by filename.
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "01-alpha.go"), []byte(`package main
import "github.com/samcharles93/archie-core/internal/plugin"
var Plugin = plugin._Plugin{
WName: func() string { return "alpha" },
WVersion: func() string { return "1.0.0" },
}
`), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "02-beta.go"), []byte(`package main
import "github.com/samcharles93/archie-core/internal/plugin"
var Plugin = plugin._Plugin{
WName: func() string { return "beta" },
WVersion: func() string { return "2.0.0" },
}
`), 0o644); err != nil {
t.Fatal(err)
}
plugins, err := plugin.LoadDir(dir, symbols)
if err != nil {
t.Fatal(err)
}
if len(plugins) != 2 {
t.Fatalf("LoadDir returned %d plugins, want 2", len(plugins))
}
if plugins[0].Name() != "alpha" {
t.Errorf("plugin[0].Name() = %q, want alpha (sorted by filename)", plugins[0].Name())
}
if plugins[1].Name() != "beta" {
t.Errorf("plugin[1].Name() = %q, want beta", plugins[1].Name())
}
}
func TestLoadDirNonexistentDirReturnsNil(t *testing.T) {
plugins, err := plugin.LoadDir("/nonexistent/path/for/plugins")
if err != nil {
t.Fatal(err)
}
if plugins != nil {
// Even an empty directory should return empty slice, not nil.
_ = plugins
t.Errorf("LoadDir returned %v, want nil for nonexistent directory", plugins)
}
}
@@ -0,0 +1,36 @@
// Code generated by 'yaegi extract github.com/samcharles93/archie-core/internal/plugin'. DO NOT EDIT.
package pluginextract
import (
"github.com/samcharles93/archie-core/internal/plugin"
"reflect"
)
func init() {
Symbols["github.com/samcharles93/archie-core/internal/plugin/plugin"] = map[string]reflect.Value{
// function, constant and variable definitions
"LoadDir": reflect.ValueOf(plugin.LoadDir),
// type definitions
"Plugin": reflect.ValueOf((*plugin.Plugin)(nil)),
"Registry": reflect.ValueOf((*plugin.Registry)(nil)),
// interface wrapper definitions
"_Plugin": reflect.ValueOf((*_github_com_samcharles93_archie_core_internal_plugin_Plugin)(nil)),
}
}
// _github_com_samcharles93_archie_core_internal_plugin_Plugin is an interface wrapper for Plugin type
type _github_com_samcharles93_archie_core_internal_plugin_Plugin struct {
IValue interface{}
WName func() string
WVersion func() string
}
func (W _github_com_samcharles93_archie_core_internal_plugin_Plugin) Name() string {
return W.WName()
}
func (W _github_com_samcharles93_archie_core_internal_plugin_Plugin) Version() string {
return W.WVersion()
}
@@ -0,0 +1,10 @@
// Package pluginextract holds the Yaegi symbol table for internal/plugin,
// generated by `go generate`. Kept separate from internal/plugin itself
// so the generated file can import it without a self-import cycle.
package pluginextract
import "reflect"
//go:generate go run github.com/traefik/yaegi/cmd/yaegi extract -name pluginextract github.com/samcharles93/archie-core/internal/plugin
var Symbols = map[string]map[string]reflect.Value{}