Commit Graph
273 Commits
Author SHA1 Message Date
sam 1024feda21 test(taskstate): seed fixture task states along the transition table
The execution transition table refuses most of the pairs shipped fixtures
wrote to build a finished task -- running -> merged, queued -> parked,
queued -> waiting_human, queued -> merged -- so those fixtures fail the
moment the store enforces it.

internal/taskstate/taskstatetest seeds a status by walking the table's
legal edges (shortest route, computed from taskstate.CanTransition), so a
fixture cannot write an illegal pair and fails loudly when no route
exists. The fixtures in internal/daemon, internal/app/archied,
internal/webui and internal/infrastructure/postgres go through it, and
the table's own test names running -> merged as refused.
2026-09-26 04:01:08 +10:00
sam b1db80088f fix(archied): resolve stored provider credentials at boot 2026-09-26 03:50:07 +10:00
sam 3443e6b3de fix(controlplane): reject stored boot-derived provider keys 2026-09-26 03:44:40 +10:00
sam f43f1b3a46 refactor(agentexec): own the capture file format in agentexec, not the app layer 2026-09-26 03:30:19 +10:00
sam 0b64130b6a feat(agentexec): serve capture tools to harnesses over MCP
archie-agent mcp is a stdio MCP server, on the official go-sdk, serving a
stage's capture tools and appending accepted calls to a captures file.
The harness runner writes the tool spec and a standard mcpServers config,
registers it through HarnessSpec.MCPConfig, and reads the captures back
into the stage result. The captures file is writable by the harness user,
so the worker validates every record again with the same acceptCapture
the built-in loop now shares, and forged, undeclared or over-cap records
are dropped. A stage with capture tools on a harness without MCP is
refused before it runs.
2026-09-26 03:28:58 +10:00
sam 763023a2fe feat(egress): relay forwards the proxy, NATS and the State Store
A Kit task's worker shares the sandbox network with its harness, so it
reaches NATS and the State Store through the relay like the harness
reaches the proxy. archie-agent relay takes repeated -forward
listen=target pairs, binds them all before serving so a port conflict
fails at start, and stops every forward when one fails. RelayArgs builds
the command from the fixed relay ports.
2026-09-26 03:10:11 +10:00
sam c98c9dcc22 feat(egress): relay forwarding a sandbox network to the egress proxy
archie-agent relay listens on the sandbox's isolated network and joins
every connection to one fixed target, the proxy. The relay holds no
secrets and can reach nothing else, so the host needs no address on a
sandbox network at all.
2026-09-26 02:33:41 +10:00
sam 2485e67102 fix(archied): the State Store seeds provider-settings from the operator's secret reference
configuredSecretRegistry resolved provider keys in place, and the State Store's
copy of the config shared the providers map with b.cfg, so ImportConfig seeded
the process-local ARCHIE_PROVIDER_<hash> env name. Provider resolution now
runs only in the daemon and gateway boot (openStores).
2026-09-25 22:37:27 +10:00
sam 230280a6e0 feat(webui): Models picks from the provider catalog, and enables available providers
archied already finds usable providers (key set in the environment, or
configured) through the model catalog; the published config view now
carries that catalog, providers with their model ids. Settings > Models
offers role models only from configured providers, lists the catalog's
unconfigured providers as available, and adding one carries its class, key
variable and endpoint. Roles and providers are two accordions, each ending
in an add row.
2026-09-25 18:04:46 +10:00
sam 9344647719 fix(webui): multiplex dashboard live updates over one stream 2026-09-25 14:05:34 +10:00
sam c5faad5915 feat(controlplane): field-level sys_audit table, audit view on settings pages, whole-task time limit 2026-09-25 09:54:30 +10:00
sam 2051ca94d2 chore(state-store): remove the one-time SQLite import and db_path
Every install serves from Postgres, so the legacyread/legacyimport packages,
the archie-state-store import command, the boot gate that refused to serve
over unimported SQLite files, and db_path (which only located those files)
go. Migration 0010 drops the import_completion table.
2026-09-25 00:40:48 +10:00
sam f624d23f3c chore(config): remove token_env aliases for forge, artifacts and telegram
Each credential is now only a {engine, key} secret reference. A stale
token_env in a file is reported at startup as an unrecognised key.
nats.token_env stays (no secret-ref form), and providers.api_key_env stays:
it is the carrier the resolver writes each resolved key's env name into.
2026-09-24 23:58:26 +10:00
sam 9da6108617 chore(config): remove dead [agent], [oidc], [image] and containers.enabled keys
None had a production consumer: [agent] and containers.enabled were decoded
only so old files kept loading, [oidc] was never read (archie-ui takes OIDC
from flags), and [image] was validated but no provider exists.
2026-09-24 23:52:06 +10:00
sam 6f3289fd33 docs(development): checklists for every kind of change
docs/development/ holds one page per kind of change (settings, State Store
and persistence, task lifecycle, workflows and steps, event automation,
dashboard, agent execution and containers, processes and wiring, merging
work), each naming every layer the change must reach and pointing at the
document that owns the design. The docs build puts them in the development
set; docs/index.md and CLAUDE.md link them.

Also adds the container-profile case to the validator agreement test, the
check the profiles work skipped.
2026-09-24 21:03:34 +10:00
sam f538f457ce fix(controlplane): stored container policies replace file profiles; add profile image and workflow examples
A stored container-runtime-policies document that carries Profiles now
replaces the file's profiles instead of merging into them, so a profile
deleted in the dashboard stays deleted; one stored before profiles existed
still inherits the file's. Profiles drop their json tags so the stored
document keys them like its other fields.

Adds examples/profiles/Dockerfile.agent-net (agent image plus network
lookup tools) with a build-only agent-net compose service,
examples/workflows/firewall-investigate.yaml with a test that parses every
example against the registered step vocabulary, and names the postgres
service in the compose header.
2026-09-24 20:58:08 +10:00
sam 47fae672f1 feat(eda): workflow inputs, optional repository and agent profiles
Workflow definitions declare inputs, repository (none, optional or
required; default required) and profile. A binding assigns each input
from a mapped parameter or a constant and may take its repository from a
parameter, which must name a configured repo; both are checked when the
binding is saved and again at dispatch. Inputs travel as structured task
data (tasks.inputs, the task brief, and a JSON block in the agent's
mission), never as body text.

A task with no repository runs in a per-task scratch workspace with no
clone and no publication grant. The agent.run step runs one mission
without a repository and finishes the task in the new completed status.

Agent profiles live under [containers.profiles.<name>]: an image and an
allowlist over the tools archie adds (MCP, scripts, plugins).
[containers].image is the default profile; an unconfigured profile parks
the task. The workflow is now pinned before the container starts so the
profile can choose its image.

Also stop two archied tests writing embedded NATS state into the package
directory, refuse an empty state_dir for embedded NATS, and drop the NATS
store files an earlier lane commit tracked.
2026-09-24 20:29:33 +10:00
sam 2393cade0f Merge branch 'pg/foundation' into worktree-agent-a96503bc8828fc104
# Conflicts:
#	docs/data/generated/dev-docs.json
#	internal/app/archied/state_store_test.go
#	internal/app/archied/state_store_toolcalls_test.go
#	internal/contracts/state/v1/state.pb.go
#	internal/daemon/dispatch_bindings_test.go
#	internal/infrastructure/bindingcipher/cipher_test.go
#	internal/infrastructure/captureintake/receiver_test.go
#	internal/infrastructure/edastore/contracts_binding.go
#	internal/infrastructure/edastore/testsupport_test.go
#	internal/infrastructure/postgres/edastore.go
#	internal/infrastructure/postgres/edastore_test.go
#	internal/infrastructure/staterpc/error_test.go
#	internal/store/store.go
#	internal/webui/api_binding_test.go
2026-09-24 19:47:00 +10:00
sam f4b8707ba9 docs(store): drop residual SQLite references; fix lane lint findings 2026-09-24 19:39:06 +10:00
sam bbaca2a39d Merge remote-tracking branch 'origin/pg/foundation' into worktree-agent-a62864fcfbdb43f5a
# Conflicts:
#	internal/app/archied/state_store.go
#	internal/contracts/state/v1/state.pb.go
#	internal/daemon/daemon.go
#	internal/infrastructure/captureintake/receiver.go
#	internal/infrastructure/captureintake/receiver_test.go
#	internal/infrastructure/legacyimport/load.go
#	internal/infrastructure/postgres/edastore_test.go
#	internal/infrastructure/postgres/postgresdb/edastore.sql.go
#	internal/infrastructure/postgres/queries/edastore.sql
#	internal/infrastructure/staterpc/client.go
#	internal/infrastructure/staterpc/values.go
#	internal/webui/api_binding.go
#	internal/webui/api_binding_test.go
#	ui/dist/assets/index.js
#	ui/src/bindings/BindingEditor.vue
#	ui/src/bindings/BindingRow.vue
2026-09-24 19:34:31 +10:00
sam ecf243fc19 feat(config): add state_dir for the daemon's host state
The embedded NATS store and endpoint file, the task-log registry and the
readiness disk probe resolve from state_dir (default: the data home's archie
directory) instead of the directory of db_path. db_path now only locates the
legacy SQLite files for the one-time import and its boot gate.
2026-09-24 14:47:21 +10:00
sam e3ee044c5e refactor(store)!: remove the SQLite task store, the PocketBase event store and the SQLite session store
The State Store, event capture and Gateway conversations already serve from
PostgreSQL. This deletes internal/store and internal/infrastructure/edastore,
the SQLite gateway session store and openProductionTaskStore, and the -db
recovery path on a legacy task file; backup, restore, validate and rollback
work on the database database_url names.

- The binding-secret cipher moves to internal/infrastructure/bindingcipher and
  ToolCall to internal/infrastructure/postgres.
- NewSessionStoreMemory is now a pure in-memory store that passes the shared
  SessionStore conformance suite.
- Tests run against PostgreSQL through internal/infrastructure/postgres/pgstore;
  legacyimport/legacyfixture writes the legacy files the import and the boot
  gate read.
- The cron dialect is owned in internal/domain/scheduling, so
  github.com/pocketbase/pocketbase and pocketbase/dbx leave go.mod.
  modernc.org/sqlite stays for the one-time import (legacyread).
- The UI and Messaging architecture gates ban the Postgres and legacy readers
  in place of internal/store.
2026-09-24 14:43:50 +10:00
sam a3b4ead6a6 refactor(scripts): drop the SQLite task-database backup from the installer and watchdog
The pre-update PostgreSQL snapshot is the only backup; the watchdog restores
binaries and never the database.
2026-09-24 14:43:25 +10:00
sam 879d87831c feat(eda): mappings per event type, match counts, several bindings per source
A mapping belongs to an event type and is checked against its inferred
schema on save. Each event a mapping resolves is counted once in
mapping_matches (migration 0008), read back as match_count and
last_matched_at. Bindings carry an optional CEL filter over the mapping's
parameters, checked on save. The per-source unique indexes are dropped:
every armed binding whose mapping matches the capture's event type and
whose filter admits it dispatches once, claimed in the per-binding ledger
before its task is enqueued. The State Store gains RecordMappingMatch and
the new mapping and binding fields.
2026-09-24 14:42:58 +10:00
sam 8a2b5b95f9 merge: t2db.40 sources and signing into pg/foundation
Resolves against t2db.36: captures carry both event_type and unsigned (proto field 10), the State Store serves sources and event types from the Postgres EDA store, the capture row shows event type and signature together, and the staterpc sentinels map both ways from one table.
2026-09-24 14:38:40 +10:00
sam 684ec1263a feat(eda): sources own signing, with UUIDv7 paths and approved unsigned option
A source is the capture endpoint: a UUIDv7 path by default or a unique
URL-safe custom path, signed by default with an Archie-generated secret
(encrypted via the bindings cipher), or unsigned after an approval.
Intake verifies X-Hub-Signature-256/X-Signature-256 against the source
secret before redaction; an unauthenticated event on a signed source is
captured but never dispatched, an event on an approved unsigned source
dispatches and is marked unsigned. The binding secret is retired; migration
0006 derives sources from existing binding and capture source strings,
moving each binding's secret onto its source. SourceStore crosses the
State Store gRPC contract. The dashboard gains a sources panel and flags
unsigned in the capture inspector, binding list and task timeline.
2026-09-24 14:26:21 +10:00
sam 9fac216c21 Merge branch 'worktree-agent-a88b86b51f454341b' into worktree-giggly-marinating-lamport 2026-09-24 14:21:53 +10:00
sam 152cdc6523 feat(webui): inspector shows proposed and unidentified events, names and pastes event types 2026-09-24 14:21:21 +10:00
sam 3878ce0ebb feat(staterpc): event type RPCs and capture event_type on the wire 2026-09-24 14:21:21 +10:00
sam 60b88ad6d3 Merge branch 'worktree-agent-a2b33953bdffebf56' into worktree-giggly-marinating-lamport
# Conflicts:
#	docs/data/generated/dev-docs.json
#	internal/app/archied/state_store_pool.go
2026-09-24 10:06:00 +10:00
sam cea88060b1 feat(archied): serve the State Store, event store and Gateway conversations from Postgres
The State Store's task and event-capture stores and the conversation store
read by archied and served by archie-gateway now open on database_url.
Every serving process refuses to start when legacy SQLite data exists and no
import completion is recorded, naming archie-state-store import; a fresh
install records completion itself. The Gateway holds its Postgres serve
claim for its lifetime and releases it before the pool closes.
2026-09-24 10:01:54 +10:00
sam 3fc710fbe7 Merge branch 'worktree-agent-a091696d945ed3377' into worktree-giggly-marinating-lamport
# Conflicts:
#	cmd/archie-state-store/recovery.go
2026-09-24 09:56:10 +10:00
sam d80c01b9fb feat(archied): recovery commands on PostgreSQL; fail-closed pre-update snapshot, no automatic schema rollback
Without -db, backup/restore/validate/rollback resolve database_url from the
config and work on the whole PostgreSQL database. Restore claims every service
role; rollback claims the State Store role; validate refuses a schema newer
than the binary. The installer refuses a daemon update it cannot snapshot
first; the watchdog never restores the database and prints the explicit
offline restore instead.
2026-09-24 09:50:10 +10:00
sam 823a7d0b40 feat(state-store): one-time import of the legacy SQLite stores into Postgres
archie-state-store import copies the State Store task database, the
PocketBase event store and the Gateway conversation store, all named from
db_path, into Postgres in one transaction. The bead asked for an archied
command; offline maintenance of a store belongs to the binary that owns it,
so it sits beside backup/restore/validate/rollback instead.

It refuses rather than guesses: while the state-store or gateway ownership
claim is held, when a domain it imports already holds rows (an allowlist of
that domain's tables, checked after Migrate), on a completed prior import,
on FK orphans, alias case collisions, duplicate bindings, events whose (at,
id) order microsecond storage would change, schema gaps, and any value the
target column cannot hold, naming table, row, value and rule. Historical ids
are kept and every identity sequence is set from AUTOINCREMENT's high-water
mark. After loading it compares key sets, event order, resource blobs and
binding secrets byte for byte and the next id of every sequence, checks the
source files were not written while read, and only then writes an
import_completion row (migration 0005) in the same transaction.

postgres.ImportComplete and legacyimport.HasLegacyData are the two halves
of the boot gate: legacy data without a completed import must not serve.
2026-09-24 09:45:26 +10:00
sam 8c21928140 feat(postgres): whole-database backup and offline restore on pg_dump/pg_restore
Restore claims every service role (state-store and gateway) non-blockingly
and refuses if any is held or the claim cannot be checked; it drops tables a
later migration added before pg_restore --clean so a restored schema matches
the snapshot. Backup validates the archive before replacing the previous one.
Ports the store's write-side clip, park-class and transition-audit coverage
from internal/store.
2026-09-24 09:32:39 +10:00
sam 539c07a5eb feat(postgres): State Store claims serve ownership with a session advisory lock 2026-09-24 02:57:38 +10:00
sam 39967f0ad9 feat(postgres): wire the State Store to Postgres — DB URL, fail-closed pool, composition seam
Wave 1 of the Postgres migration (archie-core-7r0x.9). This is the shared shape
every port lane builds on; nothing here switches a live path to Postgres yet.

- config: add `database_url` beside `db_path` (config.go), mark it bootstrap-only
  in denied_keys, expose it in the webui StorageView/config schema, and add it to
  config.example.toml plus all deployments/*.toml.
- docker-compose: add a postgres:18 service; `task dev` (dev:postgres) brings it
  up and waits for it before the State Store boots.
- archied: `openStateStore` now opens and migrates the process-scoped Postgres
  pool via `openStateStorePool`. The State Store fails closed (D1.2) when
  database_url is absent or invalid rather than fall back to SQLite; serving
  against Postgres later requires an import-completion record (D1.3), which is a
  later lane's gate, not this one.
- composition seam: task-store construction (state_store_task.go) and EDA-store
  construction (state_store_eda.go) are extracted from openStateStore, with the
  pool open in state_store_pool.go. This is a behaviour-preserving refactor only
  — no store, interface, error, ordering or logging change — so L1 and L2 build
  against separate files instead of sharing state_store.go.

Tests: fail-closed pool behaviour is pinned (absent/invalid URL); the State Store
boot/restart tests now run against a throwaway Postgres 18 container (pgtest
TestMain). Scoped test command and module-wide build pass; see lane report.
2026-09-24 02:18:56 +10:00
sam 8350caf6da refactor(state-store): remove the PocketBase admin view
stateadmin served read-only PocketBase views over the SQLite task file on -admin-addr. PocketBase is SQLite-only and the State Store moves to Postgres (archie-core-7r0x), so the view goes; operators use a Postgres tool of their choice.
2026-09-24 01:09:42 +10:00
sam aadd1ca1fe fix(build): task check runs in ~27s cold, ~12s warm
task test runs -short with the test cache on; task test:full keeps the
uncached run. The 50s keepalive proof skips under -short (the deploy
workflow's full go test still runs it). Sleep-bound archied watch tests
run in parallel, and the read-only gate steps run concurrently after fmt
and proto:check. ui:deps is run: once so the parallel UI steps install once.
2026-09-24 00:29:07 +10:00
sam 068d89ebb4 feat(eda): playbook.Load findings name the offending action, when or args line 2026-09-23 22:42:14 +10:00
sam 63be83e0cf refactor(playbooks): serve on go.lsp.dev/protocol instead of hand-written LSP types
Replaces sourcegraph/jsonrpc2 with the go.lsp.dev stack (protocol, jsonrpc2, uri). The exit notification now ends Serve; closing the connection from inside the exit handler deadlocked.
2026-09-23 22:34:33 +10:00
sam ee16b8a711 feat(playbooks): archie-playbooks serve, a language server publishing lint findings as diagnostics 2026-09-23 22:22:56 +10:00
sam 69e6d276e6 feat(playbooks): lint an EDA playbook directory with -eda-dir 2026-09-23 22:11:01 +10:00
sam e867760711 feat(eda): run action playbooks at pin time through the at-most-once dispatch ledger 2026-09-23 22:00:52 +10:00
sam 75454d22c1 feat(workflow): routing and playbook findings carry the key's file:line 2026-09-23 21:41:56 +10:00
sam 6046aaf4c6 feat(daemon,forge): the review-reaction producers
Sequencing steps 6 and 7 of pr-review-remediation.md: the consumer
landed in 33e915c5, and these are the two sources that feed it.

Webhook (decision 2, low latency): the GitHub receiver decodes
pull_request_review and pull_request_review_comment events behind the
existing HMAC verification and publishes the same typed reaction the
poller produces, keyed source-independently so PublishUnique collapses
the two sources. Only submitted/created carry a verdict to remediate;
edited and dismissed are acknowledged. Reaction deliveries are
rate-limited per source (decision 6's 'add when the receiver gains a
second event family'), and the reaction path carries no dispatch
predicate -- eligibility is the consumer's owned-PR lookup.

Poll (both forges, the no-webhook backstop): one shared scan over all
pr_open tasks, each with the forge client its own identity owns
(reconcilePRs's shape), listing reviews and comments since per-task
persisted high-water marks and skipping the owning identity's bot user
(decision 5's own-comment exclusion, which the poll is positioned to
make identity-correct).

The cursors are two, not one: review IDs and comment IDs are independent
forge sequences, and a combined high-water mark would silently skip
reviews once a larger comment ID landed. watch_comment_id -- dead since
the comment-watch design it was created for never shipped -- carries the
comment cursor; review_cursor is a new presence-gated migrator column,
and SetReviewCursors is its guarded RPC (pr_open only, so cursors never
move under a run that owns the task). forge.ReviewComment gains ReviewID
so a comment is collectable into its review's unit (decision 5) on both
forges, and forge.Review gains Body so poll-delivered reviews carry
their summary.

Note: the store/proto/facade half of this work landed early in aac31b69
under a release session's message -- a shared-checkout collision, kept
rather than rewritten; that commit carries the review_cursor migration
and the SetReviewCursors contract, this one carries the producers.
2026-09-23 12:41:53 +10:00
sam aac31b69ad docs(agents): a bead never carries production instance detail
Beads sync to a Dolt remote, so anything written into one is published.
Two beads filed earlier today named an operator host, its internal
address and its filesystem paths -- the tracker was treated as a private
store, which it is not.

The rule is stated where the tracker is described, next to the existing
one about .beads/ never reaching a commit, because it is the same class:
the tracker is not a private store. Host-specific facts belong in the
operator's memos instance, which a bead may point at.
2026-09-23 12:29:29 +10:00
sam 33e915c5be feat(daemon): wire the review-reaction consumer to remediate runs
The remediate workflow was registered but had no trigger: the reaction
stream was provisioned, the envelope, resolver and owned-PR lookup all
landed, yet nothing consumed the stream (pr-review-remediation.md's
sequencing left the consumer for this step). The daemon now drains the
reaction stream each cycle, before the task drain, so a reaction can be
claimed in the same pass that carried it.

Per decision 3, the store's owned-PR lookup is the authorization
boundary: an unmatched, merged or closed PR drops with a counter, and a
public reaction can never start an agent against a foreign repository.
Per decision 5, a review is the unit of remediation -- its comment
events are appended to the unit while it is queued and dropped once the
run is claimed (the round cap bounds the exchange); a standalone comment
is its own unit of one; approved reviews never remediate; and archie's
own comments, matched on the owning identity's bot user, are excluded so
its replies cannot retrigger it.

The guarded BeginRemediation/UpdateReviewPayload store methods are the
dedup: the pr_open -> queued transition carries the review unit in the
same write, so re-delivered reactions fail the status guard instead of
starting a second round. Both are new StateStoreService RPCs following
the Requeue shape, covered by the local/grpc conformance battery.
2026-09-23 12:08:44 +10:00
sam b995eac1e1 merge(run/pb-toolcalls): project tool_call events into the tool_calls collection
Lane D. The collection existed in the event-capture schema with nothing
writing it; tool calls reached only the on-disk task log, readable as text
and never queryable.

The projection is a decorator on the State Store's task store, wired in
stateStoreDeps. InsertEvent persists the event first and only then, for a
tool_call event, writes one row. A projection failure is counted and logged
but never returned: event persistence is load-bearing, the transcript is
observability, and failing a task write because its shadow failed trades
the fact for the record of it. With no event-capture store the task store
passes through unwrapped.

The event adapts to the existing collection rather than the collection to
the event; args and duration_ms stay reserved. InsertToolCall stays silent
on the notify hook, matching the dispatch tables rather than the
operator-editable ones -- tool calls are high-frequency telemetry, not
dashboard activity.
2026-09-23 03:28:39 +10:00
sam a0d6cfbb28 feat(edastore): project tool_call events into the tool_calls collection
The tool_calls collection existed in the event-capture schema but nothing
wrote it. Give it its write path:

- edastore gains InsertToolCall and the minimal TaskToolCalls read, writing
  only the existing collection. No new index: the collection is an
  append-only transcript and the SQLite predecessor had no unique index.
- internal/app/archied composes toolCallProjectingTaskStore, a decorator on
  the task store that persists the event first, then -- only for tool_call
  events -- writes one row. A projection failure is counted and logged but
  never returned from InsertEvent: event persistence is load-bearing,
  tool_calls is observability.
- stateStoreDeps wires the decorator around deps.Tasks when both stores are
  open, and passes Tasks through untouched otherwise.

The event adapts to the existing schema, not the other way round: the tool
name comes from data, a succeeded call's detail is the row's result, a
failed call's detail its error; args and duration_ms stay reserved for a
producer that reports them.
2026-09-23 03:26:14 +10:00