Three defects the first sweep left behind, each caught by a rule the guard did not yet have. Statuses. RULES.md gives three; the tree had 25 values, and the free-text ones were carrying progress. "Implemented (wiring landed)" and "Ratified (rev. 2e). Rev. 2e adds the OpenTaskByPR lookup. Rev. 2d revised section 10..." were both changelogs living in the one field the other rules exempt. Every PRD now declares Draft, Approved or Finalised and nothing else; five that declared nothing now declare one. A revision number moves to **Revision:**, a pointer to the authoritative document to **Authority:**, and what a past revision changed to git. CLAUDE.md's citation of state-store-contract rev. 2e still resolves. Stale cross-references. "Supersedes the 2026-09-03 open question", "this bullet previously said", "an earlier draft of this document claimed" all point at text the reader cannot see, because it was replaced. A supersede marker survives only when it names a file, a section or a symbol that can be found. Temporal hedges. 100 uses of today, currently and right now across 31 files. Each dates its sentence to the day it was written: "[notify] today is one webhook URL, read in one place" stops being true the moment this design lands, while still reading as the premise it rests on. The header-field exemption was also too wide. "**Date:** 2026-09-03, Channel resolved 2026-09-22" smuggled a dated status into the one line the rules did not read; the exemption is now a Date holding only a date and a Status holding only one of the three.
151 lines
7.4 KiB
Markdown
151 lines
7.4 KiB
Markdown
# Event capture storage -- decision
|
|
|
|
**Status:** Finalised
|
|
**Date:** 2026-08-22
|
|
**Beads issue:** `archie-core-t2db.1`
|
|
|
|
`webhook-intake-security.md` decided points 1-5 (auth split, approval gate,
|
|
rate limits, blast radius, retention/redaction) and explicitly left the
|
|
storage mechanism open, forward-pointing at "`7d5u.2`, embedded NATS." That
|
|
pointer predates `7d5u.3`'s 2026-08-22 narrowing, which assigned capture
|
|
storage to this bead by name and told it to decide NATS-vs-SQLite itself
|
|
rather than inherit an answer. This document is that decision.
|
|
|
|
## Decision: SQLite, not NATS JetStream
|
|
|
|
Captured events are stored in archied's existing SQLite database, in a new
|
|
`captured_events` table following the same shape as `internal/store/events.go`
|
|
(`internal/store`, `*Store`, SQL migrations inline in Go). No JetStream
|
|
stream is introduced for capture.
|
|
|
|
### Why
|
|
|
|
**NATS availability does not make capture a broker concern.** NATS is now
|
|
unconditional for autonomous workflow handoff, including embedded mode, but
|
|
capture is also used by deployments with no repository workflow configured.
|
|
SQLite is guaranteed present because it backs daemon state, and keeps
|
|
capture inspection independent of broker retention and consumer lifecycle.
|
|
The original version of this decision cited optional NATS as an additional
|
|
reason; that premise was superseded by `archie-core-q15y`, while the storage
|
|
fit and independence arguments remain.
|
|
|
|
**Capture's access pattern doesn't need what JetStream is for.** JetStream
|
|
earns its complexity for reaction delivery (`7d5u.2`) because multiple
|
|
independent consumers need fan-out, at-least-once redelivery, and Ack/Nak
|
|
semantics -- exactly the trap `CLAUDE.md` documents about `WorkQueuePolicy`
|
|
and overlapping consumers. Capture has exactly one writer (the HTTP handler)
|
|
and one reader (a human via the dashboard inspector, `t2db.2`). There is no
|
|
competing-consumer problem to solve and no redelivery semantics to get right,
|
|
so JetStream's core value proposition doesn't apply here. A bounded SQL
|
|
table is a better fit on the merits, not merely as a fallback.
|
|
|
|
**Reusing `7d5u.2`'s stream, once it exists, would still be wrong.** That
|
|
stream is provisioned for fan-out reaction delivery with its own retention
|
|
policy decision. Capture's retention need -- age- and count-bounded, prune
|
|
on write, no replay -- is a different shape. Building a second stream just
|
|
for capture, gated on an unrelated bead landing first, would trade a small
|
|
amount of duplication now for a hard dependency on `7d5u.2`'s timeline that
|
|
this bead's own acceptance criteria say not to accrue by accident.
|
|
|
|
This does not foreclose NATS later. If cross-daemon capture sharing or
|
|
durable replay becomes a real requirement, that is a new decision made
|
|
against a real need, not now against a speculative one.
|
|
|
|
## Storage shape
|
|
|
|
`captured_events` table (`internal/store/captures.go`, mirrors
|
|
`internal/store/events.go`'s pattern):
|
|
|
|
| column | type | notes |
|
|
|-----------------|---------|-------------------------------------------------|
|
|
| id | INTEGER | primary key, autoincrement |
|
|
| received_at | TEXT | RFC3339Nano |
|
|
| source | TEXT | opaque path segment chosen by the sender's operator |
|
|
| remote_addr | TEXT | `RemoteAddr` at capture time |
|
|
| content_type | TEXT | `Content-Type` header |
|
|
| headers | TEXT | JSON, redacted |
|
|
| body | TEXT | JSON, redacted (`webhookguard.RedactPayload`) |
|
|
| authenticated | INTEGER | 0/1; always 0 until a source registers a secret (future: `t2db.4`) |
|
|
|
|
Indexed on `(source, id)` for per-source listing and pruning, and `id` alone
|
|
for the global cap.
|
|
|
|
## Disk-bound mechanics
|
|
|
|
Three independent bounds, composed, per `webhook-intake-security.md` point 5
|
|
and this bead's own "capture cannot exhaust disk" criterion:
|
|
|
|
1. **Per-request size cap.** The HTTP handler reads the body through
|
|
`http.MaxBytesReader` (256 KiB default, configurable). A single oversized
|
|
POST is rejected (413) before redaction or storage sees it.
|
|
2. **Per-remote-address rate limit.** `webhookguard.RateLimiter`, keyed by
|
|
the request's remote address, applied before the body is even read.
|
|
*Not* keyed by the `source` path segment: `RateLimiter`'s own doc comment
|
|
states its bucket map assumes a bounded, operator-registered key space,
|
|
never evicted. `source` is exactly the opposite here -- an unregistered,
|
|
attacker-chosen URL segment, since capturing senders with no registration
|
|
yet is this endpoint's entire purpose. Keying on it would let one sender
|
|
bypass the limit by rotating the segment every request (a fresh key
|
|
always starts with a full burst) and grow the bucket map without bound.
|
|
Remote address is the bounded identity actually available pre-auth.
|
|
Over-budget requests get 429, observable per point 3 of the security
|
|
decision.
|
|
3. **Retention prune-on-write.** Every insert prunes, in the same
|
|
transaction: rows older than `retention_days` (default 7, per the
|
|
security decision) OR beyond `max_events` total rows (default 5000, this
|
|
bead's own addition -- age alone doesn't bound disk under a sustained
|
|
flood inside the retention window; a count cap does). No background
|
|
sweep goroutine; the daemon doesn't need one for this.
|
|
|
|
Together these give a deterministic worst-case: `max_events * max_body_bytes`
|
|
regardless of sender behaviour, independent of whether the age-based or
|
|
count-based bound is what actually fires first.
|
|
|
|
## HTTP surface
|
|
|
|
Mounted on the existing `internal/webui` dashboard server
|
|
(`Server.Handler`'s `top` mux), not a new listener. `POST
|
|
/webhooks/capture/{source}` bypasses `requireToken` the same way `/healthz`
|
|
already does -- capture must accept unauthenticated senders by design (that
|
|
is the entire point: you cannot HMAC-verify a source you haven't configured
|
|
a secret for yet). Reachability is therefore whatever already exposes the
|
|
dashboard in a given deployment; no new port/host config surface is added,
|
|
and no new claim about public reachability is made beyond what already
|
|
exists for the dashboard.
|
|
|
|
Amended 2026-09-10 by the UI cutover (`archie-core-8cda.5.4`): the one
|
|
process serving the dashboard is now the standalone archie-ui service, so
|
|
the receiver mounts there, composed from the UI process's own State Store
|
|
client. The binding-dispatch loop that turns captures into tasks stays in
|
|
the daemon, consuming the same store, so the HTTP front door moved without
|
|
moving work intake.
|
|
|
|
`GET` listing/inspection endpoints (`t2db.2`) stay behind `requireToken` like
|
|
every other `/api/*` route -- captured payloads are visible only to an
|
|
authenticated operator, per the security decision's point 5 visibility rule.
|
|
|
|
## Config
|
|
|
|
New `[capture]` section in `config.example.toml` / `internal/config`:
|
|
|
|
```toml
|
|
[capture]
|
|
retention_days = 7
|
|
max_events = 5000
|
|
max_body_bytes = 262144
|
|
rate_per_second = 1.0
|
|
rate_burst = 5
|
|
```
|
|
|
|
All fields have defaults; an empty `[capture]` section (or its absence) is
|
|
valid and produces the defaults above -- capture is on by default, since an
|
|
unbound POST being captured rather than rejected is this bead's acceptance
|
|
criterion, not an opt-in.
|
|
|
|
## What this does not decide
|
|
|
|
Binding a captured event to a workflow, the draft/pending_approval/armed
|
|
state machine, and per-source HMAC secret registration are `t2db.3`/`t2db.4`.
|
|
The dashboard inspector UI is `t2db.2`. This bead ships capture and storage
|
|
only.
|