Commit Graph
2710 Commits
Author SHA1 Message Date
sam 052e719080 feat(webui): inspector shows proposed and unidentified events, names and pastes event types 2026-09-24 14:21:21 +10:00
sam 0ea59bb211 feat(staterpc): event type RPCs and capture event_type on the wire 2026-09-24 14:21:21 +10:00
sam 79b352dc44 feat(eventtype): infer event types from captures and gate dispatch on identification 2026-09-24 14:21:20 +10:00
sam a31ac8f675 Merge branch 'worktree-agent-a2b33953bdffebf56' into worktree-giggly-marinating-lamport
# Conflicts:
#	docs/data/generated/dev-docs.json
#	internal/app/archied/state_store_pool.go
2026-09-24 10:06:00 +10:00
sam 970a65fe36 feat(archied): serve the State Store, event store and Gateway conversations from Postgres
The State Store's task and event-capture stores and the conversation store
read by archied and served by archie-gateway now open on database_url.
Every serving process refuses to start when legacy SQLite data exists and no
import completion is recorded, naming archie-state-store import; a fresh
install records completion itself. The Gateway holds its Postgres serve
claim for its lifetime and releases it before the pool closes.
2026-09-24 10:01:54 +10:00
sam 9f5f072fe7 Merge branch 'worktree-agent-a091696d945ed3377' into worktree-giggly-marinating-lamport
# Conflicts:
#	cmd/archie-state-store/recovery.go
2026-09-24 09:56:10 +10:00
sam b90e65c041 Merge pull request #992 from samcharles93/feat/865-dashboard-vocabulary
feat(webui): serve change statuses and the config schema through /api/task-meta
2026-09-24 09:50:19 +10:00
sam fe819d489a feat(archied): recovery commands on PostgreSQL; fail-closed pre-update snapshot, no automatic schema rollback
Without -db, backup/restore/validate/rollback resolve database_url from the
config and work on the whole PostgreSQL database. Restore claims every service
role; rollback claims the State Store role; validate refuses a schema newer
than the binary. The installer refuses a daemon update it cannot snapshot
first; the watchdog never restores the database and prints the explicit
offline restore instead.
2026-09-24 09:50:10 +10:00
sam cdb34e2295 fix(state-store): real-process tests give the State Store a database_url 2026-09-24 09:48:58 +10:00
sam f970a7e5ce Merge branch 'worktree-agent-a7131b2ebdff5f551' into worktree-giggly-marinating-lamport 2026-09-24 09:46:02 +10:00
sam 42dcc80754 feat(state-store): one-time import of the legacy SQLite stores into Postgres
archie-state-store import copies the State Store task database, the
PocketBase event store and the Gateway conversation store, all named from
db_path, into Postgres in one transaction. The bead asked for an archied
command; offline maintenance of a store belongs to the binary that owns it,
so it sits beside backup/restore/validate/rollback instead.

It refuses rather than guesses: while the state-store or gateway ownership
claim is held, when a domain it imports already holds rows (an allowlist of
that domain's tables, checked after Migrate), on a completed prior import,
on FK orphans, alias case collisions, duplicate bindings, events whose (at,
id) order microsecond storage would change, schema gaps, and any value the
target column cannot hold, naming table, row, value and rule. Historical ids
are kept and every identity sequence is set from AUTOINCREMENT's high-water
mark. After loading it compares key sets, event order, resource blobs and
binding secrets byte for byte and the next id of every sequence, checks the
source files were not written while read, and only then writes an
import_completion row (migration 0005) in the same transaction.

postgres.ImportComplete and legacyimport.HasLegacyData are the two halves
of the boot gate: legacy data without a completed import must not serve.
2026-09-24 09:45:26 +10:00
sam b39f7c682f feat(postgres): whole-database backup and offline restore on pg_dump/pg_restore
Restore claims every service role (state-store and gateway) non-blockingly
and refuses if any is held or the claim cannot be checked; it drops tables a
later migration added before pg_restore --clean so a restored schema matches
the snapshot. Backup validates the archive before replacing the previous one.
Ports the store's write-side clip, park-class and transition-audit coverage
from internal/store.
2026-09-24 09:32:39 +10:00
sam f0464e5cf4 Merge remote-tracking branch 'origin/main' into worktree-giggly-marinating-lamport 2026-09-24 09:31:28 +10:00
sam 6195c05777 feat(webui): serve change statuses and the config schema through /api/task-meta
Closes #865. The dashboard hand-copied the Change* file-status ids and events.ConfigCapturedSchema. /api/task-meta now serves both; the Vue dashboard renders ui/src/lib/task-meta-snapshot.ts until the served catalog lands. The Go handler is pinned to internal/webui/testdata/task_meta.json and the snapshot is pinned to that same fixture, so a one-sided rename fails a test.
2026-09-24 09:31:09 +10:00
sam 201b96e6fe feat(legacyread): read-only readers and import verification for the legacy SQLite stores 2026-09-24 03:28:42 +10:00
sam 922fbf5696 fix(postgres): lock keys follow D3.7/D4.3; event reader never sees a later id before an earlier one 2026-09-24 03:23:38 +10:00
sam 423fa8aa52 Merge pull request #991 from samcharles93/eda/event-automation
Eda/event automation
2026-09-24 03:10:54 +10:00
sam fb9f1f8b1a Merge pull request #990 from samcharles93/sams-cleanup
Sams cleanup
2026-09-24 03:08:53 +10:00
sam 51836297b4 feat(postgres): event inserts take a transaction advisory lock so ids commit in order 2026-09-24 03:01:41 +10:00
sam a71d062b4a feat(postgres): State Store claims serve ownership with a session advisory lock 2026-09-24 02:57:38 +10:00
sam 972355fc63 refactor(postgres): resources and statestore tests use storecontract, not internal/store 2026-09-24 02:47:06 +10:00
sam 850b588777 style(postgres,gateway): adopt task fmt rewrites from the lane merge 2026-09-24 02:41:34 +10:00
sam b41161e0df fix(postgres): one isUniqueViolation after merging the EDA and status lanes 2026-09-24 02:33:52 +10:00
sam a5cc0acb86 Merge branch 'run/pg-w2-5' into worktree-giggly-marinating-lamport
# Conflicts:
#	internal/infrastructure/postgres/postgresdb/models.go
2026-09-24 02:31:06 +10:00
sam 0335beabae Merge branch 'run/pg-w2-3' into worktree-giggly-marinating-lamport 2026-09-24 02:30:51 +10:00
sam 6b6791416e Merge branch 'run/pg-w2-2' into worktree-giggly-marinating-lamport 2026-09-24 02:30:48 +10:00
sam f2b7f7cfbe Merge branch 'run/pg-w1' into worktree-giggly-marinating-lamport 2026-09-24 02:30:47 +10:00
sam 757b75de94 feat(postgres): wire the State Store to Postgres — DB URL, fail-closed pool, composition seam
Wave 1 of the Postgres migration (archie-core-7r0x.9). This is the shared shape
every port lane builds on; nothing here switches a live path to Postgres yet.

- config: add `database_url` beside `db_path` (config.go), mark it bootstrap-only
  in denied_keys, expose it in the webui StorageView/config schema, and add it to
  config.example.toml plus all deployments/*.toml.
- docker-compose: add a postgres:18 service; `task dev` (dev:postgres) brings it
  up and waits for it before the State Store boots.
- archied: `openStateStore` now opens and migrates the process-scoped Postgres
  pool via `openStateStorePool`. The State Store fails closed (D1.2) when
  database_url is absent or invalid rather than fall back to SQLite; serving
  against Postgres later requires an import-completion record (D1.3), which is a
  later lane's gate, not this one.
- composition seam: task-store construction (state_store_task.go) and EDA-store
  construction (state_store_eda.go) are extracted from openStateStore, with the
  pool open in state_store_pool.go. This is a behaviour-preserving refactor only
  — no store, interface, error, ordering or logging change — so L1 and L2 build
  against separate files instead of sharing state_store.go.

Tests: fail-closed pool behaviour is pinned (absent/invalid URL); the State Store
boot/restart tests now run against a throwaway Postgres 18 container (pgtest
TestMain). Scoped test command and module-wide build pass; see lane report.
2026-09-24 02:18:56 +10:00
sam c833081544 feat(postgres): port events/identities/status surfaces, fix shipped defects
CHECKPOINT -- the full State Store port is in progress; this lands the
compile-clean core so the review pass has a diff to work against. Remaining:
advisory-lock ownership, event-insert serialization, maintenance, and the
ported test battery.

Two corrections to behaviour that already shipped, not new features:
- UpdateTask wrote park_reason raw where SQLite applies clip(park_reason,
  4000); the PostgreSQL adapter now clips before binding (clip.go).
- tasks.review_cursor existed on the protobuf wire and in workflow.Task but
  not in the PostgreSQL schema; 0004 adds it as a NEW migration rather than
  amending 0001 because a dev database may already have applied 0001 and
  goose tracks version numbers, not checksums.

Also moves the control-plane Resource/ResourceWrite types and their sentinels
into storecontract so the postgres package can stop importing internal/store
(they stay aliased there for daemon callers).
2026-09-24 02:17:44 +10:00
sam 6d0b817b2c feat(postgres): port EDA store to Postgres (captures, mappings, bindings, ledgers, tool_calls)
L2 of the PocketBase->Postgres migration (bead archie-core-7r0x.3): a new
Postgres-backed EDA adapter (postgres.EDA) plus its goose migration
(0002_edastore.sql) and sqlc queries (queries/edastore.sql), satisfying the
same storecontract surfaces as internal/infrastructure/edastore.Store so the
cutover wave can swap them behind the contract. The PocketBase opener in
internal/app/archied/state_store.go is untouched: production still opens
SQLite, and the cutover waits for the one-time import.

Three recon-found hazards, each pinned by a test, not a comment:

1. Empty-secret early return. applyBinding deliberately never writes the
   secret column when the incoming secret is empty, so an edit form that does
   not echo the secret must not blank an armed binding's HMAC key. The
   UpdateBinding query keeps the stored secret via
   `secret = CASE WHEN $secret = '' THEN secret ELSE $secret END`; pinned by
   TestUpdateBindingEmptySecretPreservesStored (raw column unchanged across an
   empty-secret edit).

2. Unique-violation detection by SQLSTATE. isUniqueViolation previously grepped
   the driver's error text for "unique"/"already exists"; Postgres reports
   SQLSTATE 23505 and the text match would not fire, losing ErrAlreadyDispatched.
   The adapter detects by pgconn.PgError.Code == "23505"; pinned by
   TestIsUniqueViolationDetectsBySQLState (23505 with a message lacking those
   words is detected; a non-23505 whose text says "unique" is not).

3. One-binding-per-source / one-armed-binding-per-source were read-then-refuse
   with no transaction and a non-unique idx_bindings_source_status. Made them
   constraints: UNIQUE(source) plus a partial UNIQUE(source) WHERE
   status='armed' (the full unique subsumes the armed rule today, kept explicit
   for a future matcher extension). A concurrent insert for one source yields
   exactly one success and one ErrBindingOverlap; pinned by
   TestConcurrentInsertBindingSameSource.

Dispatch ledgers kept exactly: UNIQUE(binding, capture) and
UNIQUE(playbook_id, playbook_version, event_id, action_id); binding_version is
stored but deliberately not part of the key (pinned by
TestBindingDispatchVersionBumpDoesNotPermitRedispatch). The cipher is
unchanged: AES-256-GCM, key = SHA-256(material), decrypt stays in the store and
the gRPC wire still carries plaintext.

Notify is left as an injected callback (EDA.SetNotify) and is NOT wired to
LISTEN/NOTIFY. It is dead code today: edastore.Config.Notify is omitted at its
only production construction and no consumer maps binding_changed/
mapping_changed, so building a LISTEN/NOTIFY path would be building for a
consumer that does not exist. Wired or not is a separate decision.
2026-09-24 02:17:31 +10:00
sam de6431a50d feat(gateway): port session store to Postgres
Move sessions, messages, turns and full-text search off the per-process
SQLite file onto Postgres behind the existing messaging.SessionStore
contract. New adapter internal/gateway/session_store_postgres.go, sqlc
queries/gateway.sql, migration 0003_gateway.sql, regenerated postgresdb.

FTS5 becomes a STORED generated tsvector column over (sender, text) plus
GIN. plainto_tsquery is the tsvector analog of ftsMatchQuery's
quote-escaping: query text is parsed as plain text, never query syntax, and
the surviving terms are ANDed. MessagePage.Truncated stays always-false on
Postgres too: the result set is counted exactly.

On internal/domain/messaging/session_store.go: this is a comment-only edit
to the MessagePage.Truncated doc comment (the brief's own "change the
comment" instruction). It changes no type, method or value, so L2 and the
import lane still build against an unchanged contract and the wire is
unaffected. No interface widened or altered.

Semantics preserved verbatim:
- The timestamp clamp is SQL-atomic (INSERT ... SELECT MAX(ts)+1). Appends to
  one session are serialised by a transaction-scoped advisory lock keyed on
  the session so two concurrent appends cannot read the same MAX(ts); a Go
  clamp would race (proven by the new cross-handle monotonicity test).
- Session ordering is GREATEST(last_active_at, created_at) with a matching
  expression index.
- ClaimTurn is insert-first (ON CONFLICT DO NOTHING), with a race test across
  two handles.

State outside the store is NOT made shared by this change, and none of it is
silently improved or worsened:
- sessionTracker.active, Router.titling and per-session persona selection are
  in-memory, per-process caches; a second Gateway still diverges from the
  first.
- messaging.Turns lanes and the per-process recovery latch stay per-process.
- RecoverTurns still fails every turn owned by another process, so a second
  Gateway starting would still fail the first's in-flight turns. This lane
  does NOT make that safe and does not try to: the safeguard is D4.1's
  single-Gateway advisory serve-ownership lock (not implemented here), which
  prevents a second Gateway from serving at all. The turn-recovery semantics
  are left exactly as they were.

Production openers still return SQLite; this lane adds the adapter and tests
only. The cutover and the state-sharing questions remain follow-up work.
2026-09-24 02:17:23 +10:00
sam 05ee5eff31 feat(postgres): preserve resource write semantics 2026-09-24 01:38:40 +10:00
sam d1ed835030 feat(postgres): State Store schema and sqlc queries, with the claim path proven on PG 18
goose migration 0001 lands the twelve State Store tables on Postgres, and sqlc generates the task queries from it. Two behaviours that genuinely differ from SQLite are pinned by tests against a real PG 18: claiming takes the oldest queued row with FOR UPDATE SKIP LOCKED (without the clause a second claimer blocks, which the test shows by timing out), and every column behind a Go int is bigint -- SQLite's INTEGER is 64-bit, and the chat issue-number allocator seeds at 10^15, which does not fit an int4.

task db:generate writes the committed postgresdb package; task db:check regenerates it in the gate, as proto:check and ui:check do.

The remaining store methods (events, resources, identities, the status tables) still run on SQLite: porting them is the rest of archie-core-7r0x.2.
2026-09-24 01:18:19 +10:00
sam 65697bd4ae refactor(state-store): remove the PocketBase admin view
stateadmin served read-only PocketBase views over the SQLite task file on -admin-addr. PocketBase is SQLite-only and the State Store moves to Postgres (archie-core-7r0x), so the view goes; operators use a Postgres tool of their choice.
2026-09-24 01:09:42 +10:00
sam 4dc04deb7a feat(postgres): pool that refuses servers before PG 18, goose migrations under an advisory lock, testcontainers harness
internal/infrastructure/postgres opens a pgx pool, checks server_version_num >= 180000, and applies goose migrations with a Postgres session advisory lock so processes starting together apply each migration once. internal/infrastructure/postgres/pgtest starts postgres:18 through testcontainers per test binary and gives each test its own database. Adds jackc/pgx/v5 v5.11.0, pressly/goose/v3 v3.28.0 and testcontainers-go v0.44.0. The database URL config, sqlc, the Compose service and the ownership lock move to archie-core-7r0x.2, where they have consumers.
2026-09-24 01:03:08 +10:00
sam fca12939a3 docs(prds): orgs, workspaces and access control
Approved design for orgs and workspaces, people and agents as identities, the Cedar policy chain (instance with network rules, org, workspace, identity and workflow, object), server-derived credential scope, and one run credential per agent run. Secrets move from agent profiles to identities; event-automation.md is marked superseded where it said otherwise.
2026-09-24 00:34:14 +10:00
sam 580c12f9d9 docs: regenerate dev-docs.json for the event automation PRD 2026-09-24 00:30:22 +10:00
sam 90fc5d507c fix(build): task check runs in ~27s cold, ~12s warm
task test runs -short with the test cache on; task test:full keeps the
uncached run. The 50s keepalive proof skips under -short (the deploy
workflow's full go test still runs it). Sleep-bound archied watch tests
run in parallel, and the read-only gate steps run concurrently after fmt
and proto:check. ui:deps is run: once so the parallel UI steps install once.
2026-09-24 00:29:07 +10:00
sam f8fa8e2e1e docs(prds): event automation — identified events, typed parameters, purpose-built agents 2026-09-23 23:50:27 +10:00
sam 7e6eab1fed fix(webui): binding ids are strings and a new binding gets a generated signing secret
bindingPayload sent mapping_id as Number(id), which is 0 for every current mapping id, so every binding created or edited from the dashboard was refused. A new binding now starts with a generated secret the operator copies to the sender; an edit keeps the stored one unless replaced. The dev proxy keeps the browser's Host and forwards /webhooks. Checked with typecheck, task ui and task test:ui; the full gate was not run.
2026-09-23 23:50:27 +10:00
sam 2cca93b65f fix(bindings): a new binding is created pending approval
Nothing moved a binding out of draft except an unrelated edit, so a newly created binding could not be approved and the dashboard showed no Approve button. Draft is removed; existing draft rows are promoted to pending_approval when the EDA store opens.
2026-09-23 23:13:49 +10:00
sam 21ff46d3a7 fix(webui): dev proxy keeps the browser's Host, and editor errors show beside Save
Vite's proxy shorthand sets changeOrigin, so every mutation through the dev server failed archie-ui's same-origin check. The mapping and binding editors rendered their refusal at the top of a scrolled dialog, out of view of the Save button.
2026-09-23 22:58:45 +10:00
sam 46584da0d1 feat(eda): playbook.Load findings name the offending action, when or args line 2026-09-23 22:42:14 +10:00
sam 8c23797ea1 refactor(playbooks): serve on go.lsp.dev/protocol instead of hand-written LSP types
Replaces sourcegraph/jsonrpc2 with the go.lsp.dev stack (protocol, jsonrpc2, uri). The exit notification now ends Serve; closing the connection from inside the exit handler deadlocked.
2026-09-23 22:34:33 +10:00
sam 62645d1a93 feat(playbooks): archie-playbooks serve, a language server publishing lint findings as diagnostics 2026-09-23 22:22:56 +10:00
sam 0ad5d2acd9 feat(playbooks): lint an EDA playbook directory with -eda-dir 2026-09-23 22:11:01 +10:00
sam 8d6ddaa138 feat(eda): reject an args value whose checked CEL type cannot fill its Args field 2026-09-23 22:05:50 +10:00
sam 34112ac033 feat(eda): run action playbooks at pin time through the at-most-once dispatch ledger 2026-09-23 22:00:52 +10:00
sam 65b0a16a2a docs(eda): playbook dispatch rows are retained until an explicit delete 2026-09-23 21:52:10 +10:00
sam 60c19f450d fix(eda): type-aware actions classifier, one stable-identifier grammar, loader tests that fail for the named reason 2026-09-23 21:48:52 +10:00