Commit Graph
32 Commits
Author SHA1 Message Date
sam 1f05dbb33c feat(deployment): default to managed NATS workers
Autonomous workflows now always hand a complete task to a scoped archie-agent container. Embedded NATS is the default and external NATS changes only broker placement; legacy agent and containers.enabled inputs decode for migration but cannot select an execution path. The daemon flushes task-event subscriptions before a worker can publish, and runtime updates snapshot and restore the configured worker image.
2026-08-24 14:56:57 +10:00
sam d3da153768 refactor(agentexec): remove legacy execution paths
Autonomous workflows now have exactly one production boundary: a task-scoped archie-agent receives a full-task core-NATS request and runs the worker-local stage loop. The legacy [agent] section remains decode-only for migration; it no longer selects runtime behavior.
2026-08-24 14:56:57 +10:00
sam 9d73bc7157 feat(nats): expose embedded broker to managed workers 2026-08-24 14:56:57 +10:00
sam cc53e5258d fix(telegram): compact tool output and drop turn caps
Render completed ToolCallEvents as Telegram-safe streaming blocks
with compact name/status and a short cleaned preview. Summarize
file contents and listings instead of dumping source or quoted
JSON. Collapse repeated legacy turn-budget failures.

Remove max_tokens turn gating and the 200000 turn_budget_chars
default so absent or commented config cannot restore an aggregate
cap. Keep per-result max_result_chars and spill.

Confine find/grep discovery to the active workspace/worktree.

Refs #604
2026-08-23 16:40:16 +00:00
sam 8a799cd8cd feat(forge): observable webhook receiver + dedup regression proof; decide adversarial review design
archie-core-7d5u.5 (idempotency and poll/webhook parity): the dedup
mechanism (TaskEnvelope.IdempotencyKey + PublishUnique + JetStream
Nats-Msg-Id dedup) and the shared dispatch predicate were already fully
wired by 7d5u.4. What was missing: proof the dedup actually holds, and
observability for a wedged receiver.

- internal/infrastructure/eventbus/nats/dedup_test.go: regression test
  proving PublishUnique suppresses a duplicate publish on the same
  WorkQueuePolicy stream ARCHIE_TASKS uses in production.
- internal/forge/webhook/receiver.go: Receiver.Status() tracks
  authenticated deliveries separately from successful publishes; a GET
  to the receiver's listen address (GitHub only ever POSTs) now returns
  it as JSON, so a wedged/non-delivering receiver is observable instead
  of indistinguishable from an idle repo.

Poll/webhook parity (disabling webhook still polls) needed no code
change: Cycle()/Run() never reference cfg.Forge.Intake anywhere in
internal/daemon, only setupForgeWebhook gates the webhook HTTP server.

archie-core-h019.1 (design: adversarial self-review stage): one-page
decision at docs/prds/adversarial-self-review.md. Reviewer runs in a
second workspace pointed at a .git-stripped snapshot of the reviewed
commit, never the implementer's worktree, so isolation is a property of
what's mounted rather than a prompt instruction. Findings contract is
workflow.ReviewFinding, owned by internal/domain/workflow. A surviving
error-level finding sets StatusParked, not WaitingHuman -- the epic's
informal wording named two different states in the actual machine; this
resolves which.

Both changes reviewed by a fresh subagent with no memory of the
implementation, per CLAUDE.md's adversarial-verification requirement;
zero blocking findings, two minor coverage gaps closed.
2026-08-22 19:45:31 +10:00
sam 3ade200ca7 feat(webui): payload field mapping -- bind JSON paths to named fields, preview before saving
Adds t2db.3: schema-by-example field mapping for the no-code playbook epic.
An operator picks a captured event, clicks through its payload to bind named
fields to JSON paths, and previews the resolution against a real event
before saving -- a missing, null, or type-changed path fails loudly (named
by field and event) rather than handing an agent a blank value.

- internal/domain/mapping: Mapping/Field types, path parsing/walking,
  Resolve (per-field failure reporting), Validate (incl. duplicate names)
- internal/store/mappings.go: SQLite persistence, mirrors captures.go
- internal/webui/api_mapping.go: CRUD + preview HTTP API
- ui/src/mappings/: dashboard editor (clickable payload tree, live preview)

Design decisions recorded in docs/prds/payload-field-mapping.md. t2db.4
(binding a mapping + matcher to a workflow) is out of scope here.
2026-08-22 19:18:15 +10:00
sam 51bb80f6a1 feat(nats): run an embedded server when no external NATS is configured
Wire github.com/nats-io/nats-server/v2 into production startup so a
single-process deployment gets reaction delivery and task distribution
over an in-process server rather than a separate one
(docs/prds/embedded-nats.md).

Meaning change: [nats].url empty used to mean "no NATS, use the SQLite
ClaimNext flow"; it now means "start an embedded NATS server". The old
behaviour is the explicit opt-out [nats].mode = "off". mode resolves from
url when unset (url set -> external, url empty -> embedded), and
agent.mode="nats" / [containers] now require mode="external" (an embedded
loopback server cannot be reached by agent workers or containers).

The daemon owns the embedded server's lifecycle: StartEmbedded before
dialing the client, Shutdown after the client closes (LIFO cleanups). The
embedded server binds loopback on a random port, enables JetStream in a
store dir under the data directory, and leaves its logger nil so a startup
failure surfaces as a Connect error rather than an os.Exit.

Retention becomes a nats.Config field (default WorkQueuePolicy) so a future
reaction stream can fan out under LimitsPolicy instead of the hardcoded
competing-consumer policy that starves a second consumer on overlapping
subjects.
2026-08-22 05:18:31 +10:00
sam 101c133239 feat(capture): store unbound webhook events, retention-bounded
Adds POST /webhooks/capture/{source}: an unauthenticated endpoint (mounted
alongside /healthz, bypassing dashboard token auth by design) that persists
any inbound webhook archie has no binding for yet, so it can be inspected
and mapped later (t2db.2/t2db.3).

docs/prds/event-capture-storage.md decides SQLite over NATS JetStream for
storage: NATS is not unconditionally available until 7d5u.2 ships embedded
support, and capture's single-writer/single-reader access pattern doesn't
need JetStream's fan-out/replay machinery anyway.

internal/store/captures.go adds a captured_events table with prune-on-write
(retention age + row-count cap, both configurable). The HTTP handler applies
webhookguard redaction before persisting, and enforces two more independent
disk-bound mechanics before that: a per-request body-size cap and a
per-remote-address rate limit. New [capture] config section with documented
defaults; all fields optional, capture is on by default.

Adversarial review (manual, after the usual fresh-subagent pass hit a
session API limit mid-run) caught and fixed three real defects before this
landed, each with a regression test confirmed to fail against the pre-fix
code:
- the body-size cap silently no-op'd when CaptureMaxBodyBytes was left
  unset, the one bound that limits a single payload's size rather than row
  count or write rate
- the rate limiter was keyed by the request's own source path segment --
  unregistered and attacker-chosen by design for this endpoint, violating
  webhookguard.RateLimiter's documented bounded-key-space assumption and
  letting a sender bypass the limit by rotating the segment; rekeyed to
  remote address
- the retention prune compared time.RFC3339Nano strings, whose trimmed
  variable-width fractional seconds break lexicographic ordering within the
  same second; switched to fixed-width RFC3339

Closes archie-core-t2db.1.
2026-08-22 04:51:25 +10:00
sam 75a9838d8b docs(prds): decide event sources and reaction delivery
archie-core-7d5u.1 -- closes with the decision recorded. Investigated three
mechanisms already in the tree rather than designing from scratch:

- No out-of-process extension boundary exists or is needed; in-process typed
  families throughout, and plugins-and-extensions.md does not need amending.
- Reaction delivery reuses internal/eventbus's Publisher/Consumer contract,
  but NOT the existing nats.Client unmodified: it hardcodes
  jetstream.WorkQueuePolicy (client.go:65), the competing-consumer policy
  correct for ARCHIE_TASKS and wrong for fan-out reactions -- two reactions on
  overlapping subjects would hit the exact silent-starvation trap CLAUDE.md
  already documents. Retention needs to become a Config field; reactions get
  a second stream.
- Reactions in these two epics are producer-only (curator's WakeOnPrimaryInput
  for cheap wake, or the existing pollNATS->publishTask shape for real work).
  No veto/mutate-in-flight capability is being built here.
- No single generic Source interface. workintake.TaskEnvelope already is
  forge intake's typed contract and needs no new type -- its IdempotencyKey is
  keyed on owner/repo/number, not delivery source, so a webhook-sourced
  envelope dedups against a poll-discovered one for free. What's genuinely
  shared is a small cross-cutting package for HMAC/rate-limit/capture/approval
  mechanics, owned by neither domain.

Retitled and rescoped archie-core-7d5u.3 to match: it was still asking for
'one registry for every kind of inbound caller,' which this decision
explicitly rejects. Removed a false blocking dependency of t2db.1 on 7d5u.4
(GitHub-specific receiver) -- they are siblings sharing the new cross-cutting
package, not a chain -- and linked them instead.
2026-08-22 01:01:52 +10:00
sam d046ad6859 docs(prds): decide webhook intake security ahead of the code it constrains
archie-core-t2db.5 -- decides the five non-negotiables its description named
before capture (t2db.1) or the intake family (7d5u.3) exist to build against
it: per-source HMAC auth reusing internal/channels/webhook's existing scheme,
a draft/pending_approval/armed binding state machine modeled on telegram's
dangerous-command approval flow, per-source rate limiting (precedent:
pairing.MaxPendingPerPlatform) that rejects rather than silently drops,
blast-radius bounding via the same gate/diff-cap/sandbox every task already
gets rather than a new relaxation, and 7-day retention with best-effort
redaction reusing gateway/stream.go's sensitiveParameterKey heuristic.

The bead stays open: its code-level acceptance criteria constrain beads that
haven't landed yet and cannot be satisfied by a document alone.
2026-08-22 00:54:00 +10:00
sam 69640b5216 docs: consolidate into a single docs/ tree
Fold the old docs/ into the VitePress site and rename docs-2 -> docs.
The 12 architecture PRDs become first-class pages at docs/architecture/
with sidebar nav, rather than being buried under an archive path.
Historical material stays at docs/archive/. Rewrites every docs-2 and
docs/prds/architecture reference across the repo. Site builds clean.
2026-07-29 03:17:42 +10:00
sam d1bbd37d95 chore: cleanup 2026-07-29 02:10:44 +10:00
sam 65aa5b462d chore: cleanup 2026-07-28 18:46:50 +10:00
sam 647537c00a refactor(telegram): keep one model selector command 2026-07-27 11:44:27 +10:00
sam f73b2d1c8c feat: memory manager sync/prefetch/scan/shutdown + broad in-flight epic work
Checkpoint commit covering concurrent work across several archie-core-abg
epics to avoid losing in-progress state:

- Memory manager (18.7-18.16): background sync pipeline, prefetch with
  timeout, memory-write notification, threat scanning, shutdown drain
  with abandoned-write tracking, frozen snapshot pattern. Wired
  ScanContent and NotifyMemoryToolWrite into HandleToolCall so scanning
  and notification actually apply to writes, not just exist as unused
  primitives.
- Fixed a literal unresolved merge-conflict that had been committed to
  internal/memory/scrubber/scrubber.go.
- Tool provider/registry, MCP client/transport (stdio, HTTP, SSE),
  multimodal results, disclosure.
- Pairing device auth store, rate limiting.
- Gateway compression, session store, tasks.
- Various workflow, plugin host, and agentexec updates.
2026-07-27 03:34:46 +10:00
sam f6d51416f0 chore: replace 607 em-dashes with double hyphens project-wide
Includes MCP crash-after-init test fix (crash moved from handle() to serve()
entry point so transport sees immediate subprocess death).
2026-07-26 09:53:33 +10:00
sam 5396436d35 docs: Hermes feature gap analysis — 60 gaps across 11 subsystems
7 subagents mapped the full Hermes codebase (~200 .py files, ~500K LOC).
Master gap analysis identifies 60 missing features across:
  P0 (7): adapter lifecycle, message model, sessions, platform config,
         tool registry, MCP client, memory provider ABC
  P1 (13): send result, streaming, dedup, media, guardrails,
           progressive disclosure, session history, pairing/auth,
           CLI, config subcommands, profiles, readiness
  P2 (22): interactive UX, subagents, browser/search/file tools,
           compression, cron, dashboard, API server, delivery
  P3 (3): billing, OAuth, desktop app

15 new beads issues created under the abg epic. NellDB identified
as the persistent memory/config backend (P0 integration issue).
2026-07-26 00:03:42 +10:00
sam e4c008af5e docs: multi-agent collaboration PRD from discovery interview
Discovery interview (archie-core-abg.1) confirmed scope: daemon-side
concurrency, cross-repo/config flexibility, agent-to-agent handoff,
chat channels with full pipeline parity, and general extensibility.
Design answer confirms per-identity goroutine with shared
infrastructure for failure isolation.
2026-07-25 23:50:45 +10:00
sam e4368f70af feat: persistent volumes with TTL for container/worktree storage (archie-core-j7v)
Add [containers].volume_ttl (default 72h). Persistent-storage repos get a
per-repo bare Git cache (--reference-if-able --dissociate) so task
worktrees clone fast without repeated full network clones, while staying
fully independent of the cache's lifecycle — cache expiry can never
corrupt an active or parked task's worktree. Docker repo volumes are
labeled as Archie-owned and swept non-forced (a volume still attached to a
running container survives and retries next cycle); the repo cache
directory is swept by mtime. Both cleanups run at daemon startup and each
poll cycle.

Filed archie-core-756 (repo-wide golangci-lint baseline) and archie-core-340
(intermittent JetStream TempDir teardown flake) as separate follow-ups
rather than scope-creeping them into this change.
2026-07-24 10:22:11 +10:00
sam 4be28e0e48 feat: optional NATS token authentication (archie-core-l1q)
Add token_env to [nats] config, resolved and validated at daemon startup
(fails early if configured but the named env var is unset). Container
workers get NATS_URL/NATS_TOKEN; archie-agent honors both while CLI-supplied
URL still takes precedence. Anonymous auth remains the default for
single-node deployments.

Unblocks archie-core-5zh (promote archie-agent to per-task process).
2026-07-24 07:51:06 +10:00
sam b6b23bbb59 feat: per-repo allow_concurrent override for same-repo dispatch (archie-core-g1l)
Add config.Repo.AllowConcurrent (allow_concurrent toml key). Repos opted in
skip the daemon's default FIFO same-repo task serialization while still
respecting [containers].max_concurrency globally. Worktrees are already keyed
per issue number, so opted-in repos don't collide on disk.
2026-07-24 07:47:10 +10:00
sam f8df051b19 feat: bounded concurrent task dispatch in daemon (archie-core-u64)
Daemon previously processed one task at a time. Add bounded concurrent
dispatch for SQLite and NATS agent modes, reusing [containers].max_concurrency
(0 = unlimited). Same-repository tasks stay FIFO-serialized to avoid worktree
conflicts; different repositories now run concurrently.

Unblocks archie-core-l1q (NATS transport) and archie-core-g1l
(per-repo allow_concurrent override).
2026-07-24 07:42:13 +10:00
sam f38ba5d515 feat: wire Layer 2 daemon plugin loader into archied startup
Daemon plugins (Yaegi-interpreted .go files) are now loaded from the
configured plugin_dir at startup. Each file must export a 'Plugin'
variable using the _Plugin wrapper struct, which bridges Yaegi types
to the Go plugin.Plugin interface.

Changes:
- internal/plugin/pluginextract/ — generated Yaegi symbol table for
  the plugin package (mirrors wfextract pattern), including the
  _Plugin interface wrapper that lets interpreted code implement
  Go interfaces
- LoadDir now accepts optional symbol tables via variadic parameter,
  avoiding an import cycle between plugin and pluginextract
- Plugin tests rewritten as behavioral (not structural), verifying
  actual Yaegi interpretation of valid, invalid, and missing plugins
- config.PluginDir field for specifying the plugin directory
- daemon.PluginRegistry field for holding loaded plugins
- main.go wires plugin loading: LoadDir at startup, register into
  daemon, log each loaded plugin

Why: the Plugin interface and LoadDir existed but weren't connected
to the daemon. Skills define workflows; daemon plugins are the
extension point for forge resolvers, storage backends, and other
core-daemon concerns.
2026-07-22 04:11:02 +10:00
sam 121dd7d815 docs: sync PRD status table and lifecycle section with post-de57f72 implementation 2026-07-22 02:45:48 +10:00
sam ac44d56965 docs: finalise v2 architecture PRD
Resolve all seven open questions with concrete answers:
- Agent image versioning: tagged versions, :latest dev-only
- NATS auth: anonymous single-node, credentials for multi-node
- Worktree ownership: answered by current impl (fresh clone per task)
- Grace period: reset on human reply, cap with max_total_uptime
- Concurrent agents: blocked by default, per-repo override
- Plugin architecture: extend existing Yaegi surfaces, don't replace
- Skill loading scope: parse frontmatter + inject into prompt for v2

Status: Draft -> Final.
2026-07-22 01:38:50 +10:00
sam 334cd78471 docs: replace all unicode characters with ASCII equivalents 2026-07-22 01:30:31 +10:00
sam adcdcb4b76 docs: resolve all seven open questions with concrete answers 2026-07-22 01:26:45 +10:00
sam c6372ef886 docs: reconcile v2 architecture PRD with actual implementation state
Adds an implementation-status section and annotates each section
(binary split, NATS, container sandbox, /data volume, plugin layers,
skills, task lifecycle, phases) with what's actually built today vs.
aspirational. Reconciles the plugin design with the sibling
yaegi-plugin-system.md PRD, whose three phases are already
implemented but scoped narrower than this doc's Layer 1/2 split.
2026-07-22 01:07:33 +10:00
sam c0d108a5d0 feat: add Yaegi-based plugin system (gates, custom stages, skill scripts)
Implements the full PRD (docs/prds/yaegi-plugin-system.md): repos can
now define .archie/gate.go for diff-inspecting quality rules shell
commands can't express, .archie/stages/*.go for custom pipeline steps,
and skills can ship Go scripts run via a new run_go_script agent tool —
all interpreted in-process with Yaegi, with panics recovered so a bad
script can't take down the daemon. Hot reload (phase 4) needed no code:
every loader reads from disk per call and every task gets a fresh
clone, so nothing is ever cached to go stale.
2026-07-22 00:54:46 +10:00
sam 276f96fec3 docs: archie-core v2 architecture design doc (daemon/agent split, NATS, plugins, skills) 2026-07-21 19:48:38 +10:00
sam 3080adf87e docs: add TypeScript plugin engine PRD (goja + esbuild) 2026-07-21 17:49:50 +10:00
sam 1e73507d80 docs: add architecture docs, skills, and Yaegi PRD 2026-07-21 17:34:38 +10:00