Commit Graph
22 Commits
Author SHA1 Message Date
sam e16ba86b30 feat(harness): ingest OAuth storage and recover workflow write handling
Recover the OAuth storage slice from f6079e9d and the missing workflow persistence error handling from f07c314e, adapted to the current step contract. Require encryption for OAuth token storage and verify RPC round trips and task-grant denial.
2026-09-27 13:09:41 +10:00
sam e7b21b9e5d feat(state-store): ListSteps, dashboard run detail reads it; drop tasks.stage
docs/prds/execution-tree-state-machine.md: step_executions (0017) is now the
durable per-step record; the dashboard's run detail (GET /api/tasks/{id}/
attempts) still folded stage_start/stage_finish events into a rail by hand,
and tasks.stage duplicated what StartStep/FinishStep already record properly.

Added the missing read:
- ListSteps(execution_id, attempt) -- attempt 0 lists every attempt of the
  execution, oldest first, so the rail renders every attempt from one call.
  New storecontract.StepReader (separate from StepRecorder: this is a
  dashboard/API/dispatch action per the PRD's wire contract, never
  task-scoped-grant callable), postgresdb.ListStepExecutions, staterpc
  client/server/values.go plumbing, wired through state_store.go's Deps and
  archieui/compose.go's Server.Steps the same optional-capability pattern
  every other dashboard read uses.
- internal/webui/api_tasks_attempts.go: stageViewsFromSteps replaces
  foldStages/finishStage/stageOutcome. Attempt bounds, event_count and which
  attempt numbers exist stay sourced from events (an attempt can carry
  events -- agent calls -- with no step), only the stage list itself moves.
  The wire JSON contract is unchanged, so no frontend change was needed;
  `task ui` confirms ui/dist still matches a fresh build.
- A running step past the task's own current attempt reads interrupted, not
  running: RecoverStale (archie-core, still open per #1214) is what corrects
  a crashed run's stale "running" row, and until it does this is the honest
  read rather than trusting a row that may be stale.

Dropped tasks.stage and every reader, no fallback:
- Migration 0018 drops the column; tasks.sql's INSERT/UPDATE/RETURNING lists
  and the sqlc-generated code no longer carry it.
- workflow.Task.Stage is gone; workflow.Run's in-flight "which stage is this"
  bookkeeping moves to TaskContext.Stage (execution-time only, never
  persisted -- park/finish's event tagging is its only reader).
- pb.Task.stage (wire field 10) is `[deprecated = true]`, not deleted (never
  reuse a field number); staterpc's taskProto/taskValue no longer map it.
- taskactions.Task.Stage and the retry event's "previous_stage" data field
  are gone: the operator-action package had no reader for the column beyond
  echoing it.
- The chat /tasks command's per-task "workflow/stage" display drops to
  workflow-only (ChatTaskSummary.Stage removed, workflowStage() deleted):
  it was a genuine reader with no replacement in scope for this change.

Read the archie-web-ui skill and the state-store/daemon docs/development
pages first, per instruction. Stayed out of cancel/stop code throughout
(Pi's concurrent CancelExecution work); the only shared files touched
(workflow.go, daemon-adjacent SQL) got the smallest possible diff.
2026-09-27 06:57:47 +10:00
sam 3f0071e27b feat(statestore): step_executions, StartStep/FinishStep under the run credential
Plan item 3 of docs/prds/execution-tree-state-machine.md (archie-core-1220), on
top of the transition tables and ErrIllegalTransition enforcement (1214/1215,
archie-core-qdbo.5):

- Migration 0017 adds step_executions -- id, org_id, workspace_id, execution_id,
  attempt, parent_id, depth, kind, name, called_execution_id, status, detail,
  tokens_used, started_at, finished_at, indexed on (execution_id, attempt, id).
  Ownership fields are stamped from the execution's row; depth is derived from
  the parent, never supplied by a caller. Existing rows gain no backfilled
  steps.
- task.Store (the workflow engine's mid-run facade) gains StartStep/FinishStep
  (5 methods, within the 8 cap), with storecontract.StepRecorder fronting them
  on the server; deadlineStore wraps both with its per-call timeout.
- The store enforces each write as one transaction: the guarded row write
  under the shared step transition table plus the transition's own
  stage_start/stage_finish event row. Staleness is decided before legality,
  the way guardTransition decides it, and the store returns the persisted
  event for the caller's post-commit publish -- the EmitDurable convention
  (the daemon's event sink skips already-persisted rows by assigned ID), which
  is why workflow.Run stops emitting stage events beside the store and no
  longer discards step-write errors: a failed start or finish parks the
  execution, and the stage does not run unrecorded. An interrupted stage
  records interrupted rather than failed; a recording failure on that
  shutdown path is logged and left to RecoverStale, which owns the
  interrupted-execution sweep in plan item 6.
- The run credential of docs/prds/orgs-and-access.md authorises StartStep and
  FinishStep on its own execution only: both requests carry the execution ID
  the grant check verifies (FinishStep carries it because the interceptor
  never reads the database), and FinishStep verifies handler-side that the
  step really is that execution's. authorizesTaskScopedCall's sanctioned
  widenings are now held as a lookup table so the next one stops growing a
  switch.
- The PRD's wire contract is StartStep(execution, attempt, parent, kind, name)
  -> step id; the responses also carry the persisted event (needed for the
  post-commit publish) and FinishStep's request carries execution_id (needed
  for the grant check) -- both noted here as deliberate additions to the PRD's
  shorthand. called_execution_id exists in the table but not yet on the wire:
  its named consumer is the workflow-call recording of plan item 4 (#1216).

Conformance drives both adapters; grant tests pin the own-execution scoping;
the wire sentinel set gains ErrInvalidStep (InvalidArgument, "step execution
invalid") for a step kind no consumer names.
2026-09-27 05:42:32 +10:00
sam e6a5a9d3d8 feat(access): policy, denial and principal persistence on the State Store contract (archie-core-qc8p) 2026-09-26 15:23:32 +10:00
sam 17465f240b feat(store): workflow.call contract — callee enqueue, parent-scoped status read, task-scoped grant widening (archie-core-t2db.41) 2026-09-26 13:36:41 +10:00
sam 643d2ed665 feat(org): Postgres tenant tables, org-aware store and resumable default upgrade (archie-core-e5fn) 2026-09-26 11:43:10 +10:00
sam 93e96b8de5 feat(storepkg): install digest-pinned packages 2026-09-26 10:48:23 +10:00
sam 8fea8e8686 feat(controlplane): field-level sys_audit table, audit view on settings pages, whole-task time limit 2026-09-25 09:54:30 +10:00
sam c1cd5c8f5b chore(state-store): remove the one-time SQLite import and db_path
Every install serves from Postgres, so the legacyread/legacyimport packages,
the archie-state-store import command, the boot gate that refused to serve
over unimported SQLite files, and db_path (which only located those files)
go. Migration 0010 drops the import_completion table.
2026-09-25 00:40:48 +10:00
sam d320a1a01a feat(eda): workflow inputs, optional repository and agent profiles
Workflow definitions declare inputs, repository (none, optional or
required; default required) and profile. A binding assigns each input
from a mapped parameter or a constant and may take its repository from a
parameter, which must name a configured repo; both are checked when the
binding is saved and again at dispatch. Inputs travel as structured task
data (tasks.inputs, the task brief, and a JSON block in the agent's
mission), never as body text.

A task with no repository runs in a per-task scratch workspace with no
clone and no publication grant. The agent.run step runs one mission
without a repository and finishes the task in the new completed status.

Agent profiles live under [containers.profiles.<name>]: an image and an
allowlist over the tools archie adds (MCP, scripts, plugins).
[containers].image is the default profile; an unconfigured profile parks
the task. The workflow is now pinned before the container starts so the
profile can choose its image.

Also stop two archied tests writing embedded NATS state into the package
directory, refuse an empty state_dir for embedded NATS, and drop the NATS
store files an earlier lane commit tracked.
2026-09-24 20:29:33 +10:00
sam 30834223dc Merge remote-tracking branch 'origin/pg/foundation' into worktree-agent-a62864fcfbdb43f5a
# Conflicts:
#	internal/app/archied/state_store.go
#	internal/contracts/state/v1/state.pb.go
#	internal/daemon/daemon.go
#	internal/infrastructure/captureintake/receiver.go
#	internal/infrastructure/captureintake/receiver_test.go
#	internal/infrastructure/legacyimport/load.go
#	internal/infrastructure/postgres/edastore_test.go
#	internal/infrastructure/postgres/postgresdb/edastore.sql.go
#	internal/infrastructure/postgres/queries/edastore.sql
#	internal/infrastructure/staterpc/client.go
#	internal/infrastructure/staterpc/values.go
#	internal/webui/api_binding.go
#	internal/webui/api_binding_test.go
#	ui/dist/assets/index.js
#	ui/src/bindings/BindingEditor.vue
#	ui/src/bindings/BindingRow.vue
2026-09-24 19:34:31 +10:00
sam fec0ea1f4e feat(eda): mappings per event type, match counts, several bindings per source
A mapping belongs to an event type and is checked against its inferred
schema on save. Each event a mapping resolves is counted once in
mapping_matches (migration 0008), read back as match_count and
last_matched_at. Bindings carry an optional CEL filter over the mapping's
parameters, checked on save. The per-source unique indexes are dropped:
every armed binding whose mapping matches the capture's event type and
whose filter admits it dispatches once, claimed in the per-binding ledger
before its task is enqueued. The State Store gains RecordMappingMatch and
the new mapping and binding fields.
2026-09-24 14:42:58 +10:00
sam 0cabe5fd65 merge: t2db.40 sources and signing into pg/foundation
Resolves against t2db.36: captures carry both event_type and unsigned (proto field 10), the State Store serves sources and event types from the Postgres EDA store, the capture row shows event type and signature together, and the staterpc sentinels map both ways from one table.
2026-09-24 14:38:40 +10:00
sam 6adbdc55a9 feat(eda): sources own signing, with UUIDv7 paths and approved unsigned option
A source is the capture endpoint: a UUIDv7 path by default or a unique
URL-safe custom path, signed by default with an Archie-generated secret
(encrypted via the bindings cipher), or unsigned after an approval.
Intake verifies X-Hub-Signature-256/X-Signature-256 against the source
secret before redaction; an unauthenticated event on a signed source is
captured but never dispatched, an event on an approved unsigned source
dispatches and is marked unsigned. The binding secret is retired; migration
0006 derives sources from existing binding and capture source strings,
moving each binding's secret onto its source. SourceStore crosses the
State Store gRPC contract. The dashboard gains a sources panel and flags
unsigned in the capture inspector, binding list and task timeline.
2026-09-24 14:26:21 +10:00
sam 80b0d1f78d feat(eventtype): infer event types from captures and gate dispatch on identification 2026-09-24 14:21:20 +10:00
sam 35b0a305eb feat(state-store): one-time import of the legacy SQLite stores into Postgres
archie-state-store import copies the State Store task database, the
PocketBase event store and the Gateway conversation store, all named from
db_path, into Postgres in one transaction. The bead asked for an archied
command; offline maintenance of a store belongs to the binary that owns it,
so it sits beside backup/restore/validate/rollback instead.

It refuses rather than guesses: while the state-store or gateway ownership
claim is held, when a domain it imports already holds rows (an allowlist of
that domain's tables, checked after Migrate), on a completed prior import,
on FK orphans, alias case collisions, duplicate bindings, events whose (at,
id) order microsecond storage would change, schema gaps, and any value the
target column cannot hold, naming table, row, value and rule. Historical ids
are kept and every identity sequence is set from AUTOINCREMENT's high-water
mark. After loading it compares key sets, event order, resource blobs and
binding secrets byte for byte and the next id of every sequence, checks the
source files were not written while read, and only then writes an
import_completion row (migration 0005) in the same transaction.

postgres.ImportComplete and legacyimport.HasLegacyData are the two halves
of the boot gate: legacy data without a completed import must not serve.
2026-09-24 09:45:26 +10:00
sam ecc1997384 Merge branch 'run/pg-w2-5' into worktree-giggly-marinating-lamport
# Conflicts:
#	internal/infrastructure/postgres/postgresdb/models.go
2026-09-24 02:31:06 +10:00
sam 5e1c43f1c5 Merge branch 'run/pg-w2-3' into worktree-giggly-marinating-lamport 2026-09-24 02:30:51 +10:00
sam cadf248b1f feat(postgres): port events/identities/status surfaces, fix shipped defects
CHECKPOINT -- the full State Store port is in progress; this lands the
compile-clean core so the review pass has a diff to work against. Remaining:
advisory-lock ownership, event-insert serialization, maintenance, and the
ported test battery.

Two corrections to behaviour that already shipped, not new features:
- UpdateTask wrote park_reason raw where SQLite applies clip(park_reason,
  4000); the PostgreSQL adapter now clips before binding (clip.go).
- tasks.review_cursor existed on the protobuf wire and in workflow.Task but
  not in the PostgreSQL schema; 0004 adds it as a NEW migration rather than
  amending 0001 because a dev database may already have applied 0001 and
  goose tracks version numbers, not checksums.

Also moves the control-plane Resource/ResourceWrite types and their sentinels
into storecontract so the postgres package can stop importing internal/store
(they stay aliased there for daemon callers).
2026-09-24 02:17:44 +10:00
sam f3471131ff feat(postgres): port EDA store to Postgres (captures, mappings, bindings, ledgers, tool_calls)
L2 of the PocketBase->Postgres migration (bead archie-core-7r0x.3): a new
Postgres-backed EDA adapter (postgres.EDA) plus its goose migration
(0002_edastore.sql) and sqlc queries (queries/edastore.sql), satisfying the
same storecontract surfaces as internal/infrastructure/edastore.Store so the
cutover wave can swap them behind the contract. The PocketBase opener in
internal/app/archied/state_store.go is untouched: production still opens
SQLite, and the cutover waits for the one-time import.

Three recon-found hazards, each pinned by a test, not a comment:

1. Empty-secret early return. applyBinding deliberately never writes the
   secret column when the incoming secret is empty, so an edit form that does
   not echo the secret must not blank an armed binding's HMAC key. The
   UpdateBinding query keeps the stored secret via
   `secret = CASE WHEN $secret = '' THEN secret ELSE $secret END`; pinned by
   TestUpdateBindingEmptySecretPreservesStored (raw column unchanged across an
   empty-secret edit).

2. Unique-violation detection by SQLSTATE. isUniqueViolation previously grepped
   the driver's error text for "unique"/"already exists"; Postgres reports
   SQLSTATE 23505 and the text match would not fire, losing ErrAlreadyDispatched.
   The adapter detects by pgconn.PgError.Code == "23505"; pinned by
   TestIsUniqueViolationDetectsBySQLState (23505 with a message lacking those
   words is detected; a non-23505 whose text says "unique" is not).

3. One-binding-per-source / one-armed-binding-per-source were read-then-refuse
   with no transaction and a non-unique idx_bindings_source_status. Made them
   constraints: UNIQUE(source) plus a partial UNIQUE(source) WHERE
   status='armed' (the full unique subsumes the armed rule today, kept explicit
   for a future matcher extension). A concurrent insert for one source yields
   exactly one success and one ErrBindingOverlap; pinned by
   TestConcurrentInsertBindingSameSource.

Dispatch ledgers kept exactly: UNIQUE(binding, capture) and
UNIQUE(playbook_id, playbook_version, event_id, action_id); binding_version is
stored but deliberately not part of the key (pinned by
TestBindingDispatchVersionBumpDoesNotPermitRedispatch). The cipher is
unchanged: AES-256-GCM, key = SHA-256(material), decrypt stays in the store and
the gRPC wire still carries plaintext.

Notify is left as an injected callback (EDA.SetNotify) and is NOT wired to
LISTEN/NOTIFY. It is dead code today: edastore.Config.Notify is omitted at its
only production construction and no consumer maps binding_changed/
mapping_changed, so building a LISTEN/NOTIFY path would be building for a
consumer that does not exist. Wired or not is a separate decision.
2026-09-24 02:17:31 +10:00
sam 710fbab094 feat(gateway): port session store to Postgres
Move sessions, messages, turns and full-text search off the per-process
SQLite file onto Postgres behind the existing messaging.SessionStore
contract. New adapter internal/gateway/session_store_postgres.go, sqlc
queries/gateway.sql, migration 0003_gateway.sql, regenerated postgresdb.

FTS5 becomes a STORED generated tsvector column over (sender, text) plus
GIN. plainto_tsquery is the tsvector analog of ftsMatchQuery's
quote-escaping: query text is parsed as plain text, never query syntax, and
the surviving terms are ANDed. MessagePage.Truncated stays always-false on
Postgres too: the result set is counted exactly.

On internal/domain/messaging/session_store.go: this is a comment-only edit
to the MessagePage.Truncated doc comment (the brief's own "change the
comment" instruction). It changes no type, method or value, so L2 and the
import lane still build against an unchanged contract and the wire is
unaffected. No interface widened or altered.

Semantics preserved verbatim:
- The timestamp clamp is SQL-atomic (INSERT ... SELECT MAX(ts)+1). Appends to
  one session are serialised by a transaction-scoped advisory lock keyed on
  the session so two concurrent appends cannot read the same MAX(ts); a Go
  clamp would race (proven by the new cross-handle monotonicity test).
- Session ordering is GREATEST(last_active_at, created_at) with a matching
  expression index.
- ClaimTurn is insert-first (ON CONFLICT DO NOTHING), with a race test across
  two handles.

State outside the store is NOT made shared by this change, and none of it is
silently improved or worsened:
- sessionTracker.active, Router.titling and per-session persona selection are
  in-memory, per-process caches; a second Gateway still diverges from the
  first.
- messaging.Turns lanes and the per-process recovery latch stay per-process.
- RecoverTurns still fails every turn owned by another process, so a second
  Gateway starting would still fail the first's in-flight turns. This lane
  does NOT make that safe and does not try to: the safeguard is D4.1's
  single-Gateway advisory serve-ownership lock (not implemented here), which
  prevents a second Gateway from serving at all. The turn-recovery semantics
  are left exactly as they were.

Production openers still return SQLite; this lane adds the adapter and tests
only. The cutover and the state-sharing questions remain follow-up work.
2026-09-24 02:17:23 +10:00
sam a69f9c81c2 feat(postgres): State Store schema and sqlc queries, with the claim path proven on PG 18
goose migration 0001 lands the twelve State Store tables on Postgres, and sqlc generates the task queries from it. Two behaviours that genuinely differ from SQLite are pinned by tests against a real PG 18: claiming takes the oldest queued row with FOR UPDATE SKIP LOCKED (without the clause a second claimer blocks, which the test shows by timing out), and every column behind a Go int is bigint -- SQLite's INTEGER is 64-bit, and the chat issue-number allocator seeds at 10^15, which does not fit an int4.

task db:generate writes the committed postgresdb package; task db:check regenerates it in the gate, as proto:check and ui:check do.

The remaining store methods (events, resources, identities, the status tables) still run on SQLite: porting them is the rest of archie-core-7r0x.2.
2026-09-24 01:18:19 +10:00