Files
archie/docs/prds/event-capture-storage.md
T
sam 65e6b2f65b docs(prds): normalise statuses and remove writing dated to its own day
Three defects the first sweep left behind, each caught by a rule the guard
did not yet have.

Statuses. RULES.md gives three; the tree had 25 values, and the free-text ones
were carrying progress. "Implemented (wiring landed)" and "Ratified (rev. 2e).
Rev. 2e adds the OpenTaskByPR lookup. Rev. 2d revised section 10..." were both
changelogs living in the one field the other rules exempt. Every PRD now
declares Draft, Approved or Finalised and nothing else; five that declared
nothing now declare one. A revision number moves to **Revision:**, a pointer
to the authoritative document to **Authority:**, and what a past revision
changed to git. CLAUDE.md's citation of state-store-contract rev. 2e still
resolves.

Stale cross-references. "Supersedes the 2026-09-03 open question", "this
bullet previously said", "an earlier draft of this document claimed" all point
at text the reader cannot see, because it was replaced. A supersede marker
survives only when it names a file, a section or a symbol that can be found.

Temporal hedges. 100 uses of today, currently and right now across 31 files.
Each dates its sentence to the day it was written: "[notify] today is one
webhook URL, read in one place" stops being true the moment this design lands,
while still reading as the premise it rests on.

The header-field exemption was also too wide. "**Date:** 2026-09-03, Channel
resolved 2026-09-22" smuggled a dated status into the one line the rules did
not read; the exemption is now a Date holding only a date and a Status holding
only one of the three.
2026-09-22 00:38:35 +10:00

7.4 KiB

Event capture storage -- decision

Status: Finalised Date: 2026-08-22 Beads issue: archie-core-t2db.1

webhook-intake-security.md decided points 1-5 (auth split, approval gate, rate limits, blast radius, retention/redaction) and explicitly left the storage mechanism open, forward-pointing at "7d5u.2, embedded NATS." That pointer predates 7d5u.3's 2026-08-22 narrowing, which assigned capture storage to this bead by name and told it to decide NATS-vs-SQLite itself rather than inherit an answer. This document is that decision.

Decision: SQLite, not NATS JetStream

Captured events are stored in archied's existing SQLite database, in a new captured_events table following the same shape as internal/store/events.go (internal/store, *Store, SQL migrations inline in Go). No JetStream stream is introduced for capture.

Why

NATS availability does not make capture a broker concern. NATS is now unconditional for autonomous workflow handoff, including embedded mode, but capture is also used by deployments with no repository workflow configured. SQLite is guaranteed present because it backs daemon state, and keeps capture inspection independent of broker retention and consumer lifecycle. The original version of this decision cited optional NATS as an additional reason; that premise was superseded by archie-core-q15y, while the storage fit and independence arguments remain.

Capture's access pattern doesn't need what JetStream is for. JetStream earns its complexity for reaction delivery (7d5u.2) because multiple independent consumers need fan-out, at-least-once redelivery, and Ack/Nak semantics -- exactly the trap CLAUDE.md documents about WorkQueuePolicy and overlapping consumers. Capture has exactly one writer (the HTTP handler) and one reader (a human via the dashboard inspector, t2db.2). There is no competing-consumer problem to solve and no redelivery semantics to get right, so JetStream's core value proposition doesn't apply here. A bounded SQL table is a better fit on the merits, not merely as a fallback.

Reusing 7d5u.2's stream, once it exists, would still be wrong. That stream is provisioned for fan-out reaction delivery with its own retention policy decision. Capture's retention need -- age- and count-bounded, prune on write, no replay -- is a different shape. Building a second stream just for capture, gated on an unrelated bead landing first, would trade a small amount of duplication now for a hard dependency on 7d5u.2's timeline that this bead's own acceptance criteria say not to accrue by accident.

This does not foreclose NATS later. If cross-daemon capture sharing or durable replay becomes a real requirement, that is a new decision made against a real need, not now against a speculative one.

Storage shape

captured_events table (internal/store/captures.go, mirrors internal/store/events.go's pattern):

column type notes
id INTEGER primary key, autoincrement
received_at TEXT RFC3339Nano
source TEXT opaque path segment chosen by the sender's operator
remote_addr TEXT RemoteAddr at capture time
content_type TEXT Content-Type header
headers TEXT JSON, redacted
body TEXT JSON, redacted (webhookguard.RedactPayload)
authenticated INTEGER 0/1; always 0 until a source registers a secret (future: t2db.4)

Indexed on (source, id) for per-source listing and pruning, and id alone for the global cap.

Disk-bound mechanics

Three independent bounds, composed, per webhook-intake-security.md point 5 and this bead's own "capture cannot exhaust disk" criterion:

  1. Per-request size cap. The HTTP handler reads the body through http.MaxBytesReader (256 KiB default, configurable). A single oversized POST is rejected (413) before redaction or storage sees it.
  2. Per-remote-address rate limit. webhookguard.RateLimiter, keyed by the request's remote address, applied before the body is even read. Not keyed by the source path segment: RateLimiter's own doc comment states its bucket map assumes a bounded, operator-registered key space, never evicted. source is exactly the opposite here -- an unregistered, attacker-chosen URL segment, since capturing senders with no registration yet is this endpoint's entire purpose. Keying on it would let one sender bypass the limit by rotating the segment every request (a fresh key always starts with a full burst) and grow the bucket map without bound. Remote address is the bounded identity actually available pre-auth. Over-budget requests get 429, observable per point 3 of the security decision.
  3. Retention prune-on-write. Every insert prunes, in the same transaction: rows older than retention_days (default 7, per the security decision) OR beyond max_events total rows (default 5000, this bead's own addition -- age alone doesn't bound disk under a sustained flood inside the retention window; a count cap does). No background sweep goroutine; the daemon doesn't need one for this.

Together these give a deterministic worst-case: max_events * max_body_bytes regardless of sender behaviour, independent of whether the age-based or count-based bound is what actually fires first.

HTTP surface

Mounted on the existing internal/webui dashboard server (Server.Handler's top mux), not a new listener. POST /webhooks/capture/{source} bypasses requireToken the same way /healthz already does -- capture must accept unauthenticated senders by design (that is the entire point: you cannot HMAC-verify a source you haven't configured a secret for yet). Reachability is therefore whatever already exposes the dashboard in a given deployment; no new port/host config surface is added, and no new claim about public reachability is made beyond what already exists for the dashboard.

Amended 2026-09-10 by the UI cutover (archie-core-8cda.5.4): the one process serving the dashboard is now the standalone archie-ui service, so the receiver mounts there, composed from the UI process's own State Store client. The binding-dispatch loop that turns captures into tasks stays in the daemon, consuming the same store, so the HTTP front door moved without moving work intake.

GET listing/inspection endpoints (t2db.2) stay behind requireToken like every other /api/* route -- captured payloads are visible only to an authenticated operator, per the security decision's point 5 visibility rule.

Config

New [capture] section in config.example.toml / internal/config:

[capture]
retention_days = 7
max_events     = 5000
max_body_bytes = 262144
rate_per_second = 1.0
rate_burst      = 5

All fields have defaults; an empty [capture] section (or its absence) is valid and produces the defaults above -- capture is on by default, since an unbound POST being captured rather than rejected is this bead's acceptance criterion, not an opt-in.

What this does not decide

Binding a captured event to a workflow, the draft/pending_approval/armed state machine, and per-source HMAC secret registration are t2db.3/t2db.4. The dashboard inspector UI is t2db.2. This bead ships capture and storage only.