Commit Graph
1754 Commits
Author SHA1 Message Date
sam 5e6c3fa3d3 Merge pull request #992 from samcharles93/feat/865-dashboard-vocabulary
feat(webui): serve change statuses and the config schema through /api/task-meta
2026-09-24 09:50:19 +10:00
sam 1b5ba30113 feat(archied): recovery commands on PostgreSQL; fail-closed pre-update snapshot, no automatic schema rollback
Without -db, backup/restore/validate/rollback resolve database_url from the
config and work on the whole PostgreSQL database. Restore claims every service
role; rollback claims the State Store role; validate refuses a schema newer
than the binary. The installer refuses a daemon update it cannot snapshot
first; the watchdog never restores the database and prints the explicit
offline restore instead.
2026-09-24 09:50:10 +10:00
sam 5009288729 fix(state-store): real-process tests give the State Store a database_url 2026-09-24 09:48:58 +10:00
sam d2b55e941a Merge branch 'worktree-agent-a7131b2ebdff5f551' into worktree-giggly-marinating-lamport 2026-09-24 09:46:02 +10:00
sam 35b0a305eb feat(state-store): one-time import of the legacy SQLite stores into Postgres
archie-state-store import copies the State Store task database, the
PocketBase event store and the Gateway conversation store, all named from
db_path, into Postgres in one transaction. The bead asked for an archied
command; offline maintenance of a store belongs to the binary that owns it,
so it sits beside backup/restore/validate/rollback instead.

It refuses rather than guesses: while the state-store or gateway ownership
claim is held, when a domain it imports already holds rows (an allowlist of
that domain's tables, checked after Migrate), on a completed prior import,
on FK orphans, alias case collisions, duplicate bindings, events whose (at,
id) order microsecond storage would change, schema gaps, and any value the
target column cannot hold, naming table, row, value and rule. Historical ids
are kept and every identity sequence is set from AUTOINCREMENT's high-water
mark. After loading it compares key sets, event order, resource blobs and
binding secrets byte for byte and the next id of every sequence, checks the
source files were not written while read, and only then writes an
import_completion row (migration 0005) in the same transaction.

postgres.ImportComplete and legacyimport.HasLegacyData are the two halves
of the boot gate: legacy data without a completed import must not serve.
2026-09-24 09:45:26 +10:00
sam 651021130d feat(postgres): whole-database backup and offline restore on pg_dump/pg_restore
Restore claims every service role (state-store and gateway) non-blockingly
and refuses if any is held or the claim cannot be checked; it drops tables a
later migration added before pg_restore --clean so a restored schema matches
the snapshot. Backup validates the archive before replacing the previous one.
Ports the store's write-side clip, park-class and transition-audit coverage
from internal/store.
2026-09-24 09:32:39 +10:00
sam 0f88e08913 Merge remote-tracking branch 'origin/main' into worktree-giggly-marinating-lamport 2026-09-24 09:31:28 +10:00
sam 5687d1b04e feat(webui): serve change statuses and the config schema through /api/task-meta
Closes #865. The dashboard hand-copied the Change* file-status ids and events.ConfigCapturedSchema. /api/task-meta now serves both; the Vue dashboard renders ui/src/lib/task-meta-snapshot.ts until the served catalog lands. The Go handler is pinned to internal/webui/testdata/task_meta.json and the snapshot is pinned to that same fixture, so a one-sided rename fails a test.
2026-09-24 09:31:09 +10:00
sam 671441ac4b feat(legacyread): read-only readers and import verification for the legacy SQLite stores 2026-09-24 03:28:42 +10:00
sam 8b006b35aa fix(postgres): lock keys follow D3.7/D4.3; event reader never sees a later id before an earlier one 2026-09-24 03:23:38 +10:00
sam c367a9e606 Merge pull request #991 from samcharles93/eda/event-automation
Eda/event automation
2026-09-24 03:10:54 +10:00
sam 88030e32e9 Merge pull request #990 from samcharles93/sams-cleanup
Sams cleanup
2026-09-24 03:08:53 +10:00
sam 5cbc9f9d6c feat(postgres): event inserts take a transaction advisory lock so ids commit in order 2026-09-24 03:01:41 +10:00
sam 20929cd444 feat(postgres): State Store claims serve ownership with a session advisory lock 2026-09-24 02:57:38 +10:00
sam fe73025d47 refactor(postgres): resources and statestore tests use storecontract, not internal/store 2026-09-24 02:47:06 +10:00
sam 4ce4b46977 style(postgres,gateway): adopt task fmt rewrites from the lane merge 2026-09-24 02:41:34 +10:00
sam f3a1ef6de1 fix(postgres): one isUniqueViolation after merging the EDA and status lanes 2026-09-24 02:33:52 +10:00
sam ecc1997384 Merge branch 'run/pg-w2-5' into worktree-giggly-marinating-lamport
# Conflicts:
#	internal/infrastructure/postgres/postgresdb/models.go
2026-09-24 02:31:06 +10:00
sam 5e1c43f1c5 Merge branch 'run/pg-w2-3' into worktree-giggly-marinating-lamport 2026-09-24 02:30:51 +10:00
sam dfed51547c Merge branch 'run/pg-w2-2' into worktree-giggly-marinating-lamport 2026-09-24 02:30:48 +10:00
sam cb1cd62eea Merge branch 'run/pg-w1' into worktree-giggly-marinating-lamport 2026-09-24 02:30:47 +10:00
sam feeaa16c7b feat(postgres): wire the State Store to Postgres — DB URL, fail-closed pool, composition seam
Wave 1 of the Postgres migration (archie-core-7r0x.9). This is the shared shape
every port lane builds on; nothing here switches a live path to Postgres yet.

- config: add `database_url` beside `db_path` (config.go), mark it bootstrap-only
  in denied_keys, expose it in the webui StorageView/config schema, and add it to
  config.example.toml plus all deployments/*.toml.
- docker-compose: add a postgres:18 service; `task dev` (dev:postgres) brings it
  up and waits for it before the State Store boots.
- archied: `openStateStore` now opens and migrates the process-scoped Postgres
  pool via `openStateStorePool`. The State Store fails closed (D1.2) when
  database_url is absent or invalid rather than fall back to SQLite; serving
  against Postgres later requires an import-completion record (D1.3), which is a
  later lane's gate, not this one.
- composition seam: task-store construction (state_store_task.go) and EDA-store
  construction (state_store_eda.go) are extracted from openStateStore, with the
  pool open in state_store_pool.go. This is a behaviour-preserving refactor only
  — no store, interface, error, ordering or logging change — so L1 and L2 build
  against separate files instead of sharing state_store.go.

Tests: fail-closed pool behaviour is pinned (absent/invalid URL); the State Store
boot/restart tests now run against a throwaway Postgres 18 container (pgtest
TestMain). Scoped test command and module-wide build pass; see lane report.
2026-09-24 02:18:56 +10:00
sam cadf248b1f feat(postgres): port events/identities/status surfaces, fix shipped defects
CHECKPOINT -- the full State Store port is in progress; this lands the
compile-clean core so the review pass has a diff to work against. Remaining:
advisory-lock ownership, event-insert serialization, maintenance, and the
ported test battery.

Two corrections to behaviour that already shipped, not new features:
- UpdateTask wrote park_reason raw where SQLite applies clip(park_reason,
  4000); the PostgreSQL adapter now clips before binding (clip.go).
- tasks.review_cursor existed on the protobuf wire and in workflow.Task but
  not in the PostgreSQL schema; 0004 adds it as a NEW migration rather than
  amending 0001 because a dev database may already have applied 0001 and
  goose tracks version numbers, not checksums.

Also moves the control-plane Resource/ResourceWrite types and their sentinels
into storecontract so the postgres package can stop importing internal/store
(they stay aliased there for daemon callers).
2026-09-24 02:17:44 +10:00
sam f3471131ff feat(postgres): port EDA store to Postgres (captures, mappings, bindings, ledgers, tool_calls)
L2 of the PocketBase->Postgres migration (bead archie-core-7r0x.3): a new
Postgres-backed EDA adapter (postgres.EDA) plus its goose migration
(0002_edastore.sql) and sqlc queries (queries/edastore.sql), satisfying the
same storecontract surfaces as internal/infrastructure/edastore.Store so the
cutover wave can swap them behind the contract. The PocketBase opener in
internal/app/archied/state_store.go is untouched: production still opens
SQLite, and the cutover waits for the one-time import.

Three recon-found hazards, each pinned by a test, not a comment:

1. Empty-secret early return. applyBinding deliberately never writes the
   secret column when the incoming secret is empty, so an edit form that does
   not echo the secret must not blank an armed binding's HMAC key. The
   UpdateBinding query keeps the stored secret via
   `secret = CASE WHEN $secret = '' THEN secret ELSE $secret END`; pinned by
   TestUpdateBindingEmptySecretPreservesStored (raw column unchanged across an
   empty-secret edit).

2. Unique-violation detection by SQLSTATE. isUniqueViolation previously grepped
   the driver's error text for "unique"/"already exists"; Postgres reports
   SQLSTATE 23505 and the text match would not fire, losing ErrAlreadyDispatched.
   The adapter detects by pgconn.PgError.Code == "23505"; pinned by
   TestIsUniqueViolationDetectsBySQLState (23505 with a message lacking those
   words is detected; a non-23505 whose text says "unique" is not).

3. One-binding-per-source / one-armed-binding-per-source were read-then-refuse
   with no transaction and a non-unique idx_bindings_source_status. Made them
   constraints: UNIQUE(source) plus a partial UNIQUE(source) WHERE
   status='armed' (the full unique subsumes the armed rule today, kept explicit
   for a future matcher extension). A concurrent insert for one source yields
   exactly one success and one ErrBindingOverlap; pinned by
   TestConcurrentInsertBindingSameSource.

Dispatch ledgers kept exactly: UNIQUE(binding, capture) and
UNIQUE(playbook_id, playbook_version, event_id, action_id); binding_version is
stored but deliberately not part of the key (pinned by
TestBindingDispatchVersionBumpDoesNotPermitRedispatch). The cipher is
unchanged: AES-256-GCM, key = SHA-256(material), decrypt stays in the store and
the gRPC wire still carries plaintext.

Notify is left as an injected callback (EDA.SetNotify) and is NOT wired to
LISTEN/NOTIFY. It is dead code today: edastore.Config.Notify is omitted at its
only production construction and no consumer maps binding_changed/
mapping_changed, so building a LISTEN/NOTIFY path would be building for a
consumer that does not exist. Wired or not is a separate decision.
2026-09-24 02:17:31 +10:00
sam 710fbab094 feat(gateway): port session store to Postgres
Move sessions, messages, turns and full-text search off the per-process
SQLite file onto Postgres behind the existing messaging.SessionStore
contract. New adapter internal/gateway/session_store_postgres.go, sqlc
queries/gateway.sql, migration 0003_gateway.sql, regenerated postgresdb.

FTS5 becomes a STORED generated tsvector column over (sender, text) plus
GIN. plainto_tsquery is the tsvector analog of ftsMatchQuery's
quote-escaping: query text is parsed as plain text, never query syntax, and
the surviving terms are ANDed. MessagePage.Truncated stays always-false on
Postgres too: the result set is counted exactly.

On internal/domain/messaging/session_store.go: this is a comment-only edit
to the MessagePage.Truncated doc comment (the brief's own "change the
comment" instruction). It changes no type, method or value, so L2 and the
import lane still build against an unchanged contract and the wire is
unaffected. No interface widened or altered.

Semantics preserved verbatim:
- The timestamp clamp is SQL-atomic (INSERT ... SELECT MAX(ts)+1). Appends to
  one session are serialised by a transaction-scoped advisory lock keyed on
  the session so two concurrent appends cannot read the same MAX(ts); a Go
  clamp would race (proven by the new cross-handle monotonicity test).
- Session ordering is GREATEST(last_active_at, created_at) with a matching
  expression index.
- ClaimTurn is insert-first (ON CONFLICT DO NOTHING), with a race test across
  two handles.

State outside the store is NOT made shared by this change, and none of it is
silently improved or worsened:
- sessionTracker.active, Router.titling and per-session persona selection are
  in-memory, per-process caches; a second Gateway still diverges from the
  first.
- messaging.Turns lanes and the per-process recovery latch stay per-process.
- RecoverTurns still fails every turn owned by another process, so a second
  Gateway starting would still fail the first's in-flight turns. This lane
  does NOT make that safe and does not try to: the safeguard is D4.1's
  single-Gateway advisory serve-ownership lock (not implemented here), which
  prevents a second Gateway from serving at all. The turn-recovery semantics
  are left exactly as they were.

Production openers still return SQLite; this lane adds the adapter and tests
only. The cutover and the state-sharing questions remain follow-up work.
2026-09-24 02:17:23 +10:00
sam 038b8035ff feat(postgres): preserve resource write semantics 2026-09-24 01:38:40 +10:00
sam a69f9c81c2 feat(postgres): State Store schema and sqlc queries, with the claim path proven on PG 18
goose migration 0001 lands the twelve State Store tables on Postgres, and sqlc generates the task queries from it. Two behaviours that genuinely differ from SQLite are pinned by tests against a real PG 18: claiming takes the oldest queued row with FOR UPDATE SKIP LOCKED (without the clause a second claimer blocks, which the test shows by timing out), and every column behind a Go int is bigint -- SQLite's INTEGER is 64-bit, and the chat issue-number allocator seeds at 10^15, which does not fit an int4.

task db:generate writes the committed postgresdb package; task db:check regenerates it in the gate, as proto:check and ui:check do.

The remaining store methods (events, resources, identities, the status tables) still run on SQLite: porting them is the rest of archie-core-7r0x.2.
2026-09-24 01:18:19 +10:00
sam c6df4b4a54 refactor(state-store): remove the PocketBase admin view
stateadmin served read-only PocketBase views over the SQLite task file on -admin-addr. PocketBase is SQLite-only and the State Store moves to Postgres (archie-core-7r0x), so the view goes; operators use a Postgres tool of their choice.
2026-09-24 01:09:42 +10:00
sam c620e00396 feat(postgres): pool that refuses servers before PG 18, goose migrations under an advisory lock, testcontainers harness
internal/infrastructure/postgres opens a pgx pool, checks server_version_num >= 180000, and applies goose migrations with a Postgres session advisory lock so processes starting together apply each migration once. internal/infrastructure/postgres/pgtest starts postgres:18 through testcontainers per test binary and gives each test its own database. Adds jackc/pgx/v5 v5.11.0, pressly/goose/v3 v3.28.0 and testcontainers-go v0.44.0. The database URL config, sqlc, the Compose service and the ownership lock move to archie-core-7r0x.2, where they have consumers.
2026-09-24 01:03:08 +10:00
sam 01586f1630 docs(prds): orgs, workspaces and access control
Approved design for orgs and workspaces, people and agents as identities, the Cedar policy chain (instance with network rules, org, workspace, identity and workflow, object), server-derived credential scope, and one run credential per agent run. Secrets move from agent profiles to identities; event-automation.md is marked superseded where it said otherwise.
2026-09-24 00:34:14 +10:00
sam d03e7fdef4 docs: regenerate dev-docs.json for the event automation PRD 2026-09-24 00:30:22 +10:00
sam f4d20cfe0a fix(build): task check runs in ~27s cold, ~12s warm
task test runs -short with the test cache on; task test:full keeps the
uncached run. The 50s keepalive proof skips under -short (the deploy
workflow's full go test still runs it). Sleep-bound archied watch tests
run in parallel, and the read-only gate steps run concurrently after fmt
and proto:check. ui:deps is run: once so the parallel UI steps install once.
2026-09-24 00:29:07 +10:00
sam 94ca015070 docs(prds): event automation — identified events, typed parameters, purpose-built agents 2026-09-23 23:50:27 +10:00
sam 7c6e7d54ad fix(webui): binding ids are strings and a new binding gets a generated signing secret
bindingPayload sent mapping_id as Number(id), which is 0 for every current mapping id, so every binding created or edited from the dashboard was refused. A new binding now starts with a generated secret the operator copies to the sender; an edit keeps the stored one unless replaced. The dev proxy keeps the browser's Host and forwards /webhooks. Checked with typecheck, task ui and task test:ui; the full gate was not run.
2026-09-23 23:50:27 +10:00
sam 86690ef73e fix(bindings): a new binding is created pending approval
Nothing moved a binding out of draft except an unrelated edit, so a newly created binding could not be approved and the dashboard showed no Approve button. Draft is removed; existing draft rows are promoted to pending_approval when the EDA store opens.
2026-09-23 23:13:49 +10:00
sam 528d6c4be2 fix(webui): dev proxy keeps the browser's Host, and editor errors show beside Save
Vite's proxy shorthand sets changeOrigin, so every mutation through the dev server failed archie-ui's same-origin check. The mapping and binding editors rendered their refusal at the top of a scrolled dialog, out of view of the Save button.
2026-09-23 22:58:45 +10:00
sam 891970f1ad feat(eda): playbook.Load findings name the offending action, when or args line 2026-09-23 22:42:14 +10:00
sam 3b303b353f refactor(playbooks): serve on go.lsp.dev/protocol instead of hand-written LSP types
Replaces sourcegraph/jsonrpc2 with the go.lsp.dev stack (protocol, jsonrpc2, uri). The exit notification now ends Serve; closing the connection from inside the exit handler deadlocked.
2026-09-23 22:34:33 +10:00
sam fda2c51f6d feat(playbooks): archie-playbooks serve, a language server publishing lint findings as diagnostics 2026-09-23 22:22:56 +10:00
sam 61dd33edce feat(playbooks): lint an EDA playbook directory with -eda-dir 2026-09-23 22:11:01 +10:00
sam 534153d50a feat(eda): reject an args value whose checked CEL type cannot fill its Args field 2026-09-23 22:05:50 +10:00
sam 2bc4b8db32 feat(eda): run action playbooks at pin time through the at-most-once dispatch ledger 2026-09-23 22:00:52 +10:00
sam 6046948ff0 docs(eda): playbook dispatch rows are retained until an explicit delete 2026-09-23 21:52:10 +10:00
sam 4237b560b0 fix(eda): type-aware actions classifier, one stable-identifier grammar, loader tests that fail for the named reason 2026-09-23 21:48:52 +10:00
sam 3a4af23caa feat(workflow): routing and playbook findings carry the key's file:line 2026-09-23 21:41:56 +10:00
sam 288e9ea370 fix(webui): move the live stream to /api/stream so reloading /events serves the page 2026-09-23 21:16:29 +10:00
sam aa054f8109 feat(webui): two-section quality gates, sweep the restating card prose
The quality gates card stacked its number and rows across the full row
width -- the rate pinned to the left edge, the counts to the right --
which reads as two places at once. The card now has two internal
sections, left to right: the rate, and the per-workflow rows that
explain it; the row collapses under 720px.

Also swept the restating descriptions: 'Work that passed its quality
gates' under Quality gates, 'Archie needs these before it can work on
its own' under Finish setting up, 'Next 30 days at the current rate'
under Token outlook, 'Last 50, newest first' under Live activity. A
title and a sentence that restate each other are noise; the table or
chart says the thing. CapturesCard's redaction note is a security fact,
not restatement, and stays. The unconfigured versions state is one
quiet line, not a card with a title, a description, and a sentence.
2026-09-23 15:29:28 +10:00
sam 5e82a18697 feat(webui): compact quality gates card, drop the setup celebration, hover nav
Three dashboard refinements from operator feedback:

The setup panel's completed state was celebration prose -- a full card
saying setup was done, occupying a grid cell until dismissed. A
configured daemon is the absence of a problem: the state machine now
only distinguishes incomplete from omit, the checklist returns on its
own if setup regresses, and the dismissal machinery is deleted with it.

'Gate pulse' led with a donut whose entire payload was one number, and
the name was decoration too. The card is 'Quality gates' now: the pass
rate as the number itself beside its run count, per-workflow rows as
tight hairline rows instead of muted pills, and the parked line is gone
-- Throughput's Needs-you tile already speaks for parked tasks, from the
statuses themselves. base/Gauge.vue had no other consumer and is deleted.

Topbar nav groups open on hover and close on leave, with a short grace
period bridging the trigger-to-content gap; click and keyboard
activation keep working through the same controlled open state.

The live activity feed scrolls within its card (previous commit); the
health row's cards are now compact enough to match it.
2026-09-23 14:53:40 +10:00
sam d76bf2be67 fix(webui): bound the live activity feed to the card
Group rows multiply with the number of interleaved tasks -- one group
per task per detail shape -- and nothing capped the card's height, so a
live instance rendered the dashboard ~4,700px tall with a void beside
the token outlook card. The feed now scrolls inside a 28rem region with
a sticky header: newest events at the top, the rest in place, and the
RIGHT NOW section's two cards stay comparable in height.
2026-09-23 14:42:43 +10:00
sam 475dc7c097 docs(releasing): record that main and the branch diverge on every release
The release commit lands on main, because tools/release.sh refuses to run
anywhere else, while the working branch keeps advancing. So after every release
neither tip is an ancestor of the other and --ff-only fails in both directions
-- a refusal that reads as "the tree is not ready" when the tree is fine. It
cost confusion on both v1.38.0 and v1.40.0, and both times the correct answer
had to be re-derived from the commit graph.

Documents the order that keeps a conflict off main (merge main into the branch
first, then fast-forward), and the trap that catches people looking for main: a
detached worktree sitting at the same commit accepts --ff-only and moves
nothing, which looks like success.
2026-09-23 14:42:32 +10:00