Capture inbound events archie does not yet understand #432

Closed
opened 2026-10-27 13:36:18 +00:00 by sam · 0 comments
Owner

The precondition for everything else. Today an unrecognised inbound request is rejected; for schema-by-example it must be RECORDED, because you cannot map a payload you have never seen.

Accept, store and retain inbound events with no binding: raw body, headers, source, arrival time, and the identity that sent it. Retention must be bounded -- an open capture endpoint is an unbounded disk write, and this is exactly the kind of surface an attacker fills.

Strong argument for NATS JetStream as the store rather than the in-process events.Bus: durable, replayable, retention-bounded natively, already deployed. Note the constraint though -- internal/infrastructure/eventbus/nats/doc.go states 'when no NATS URL is configured the daemon uses its local SQLite claim flow instead; nothing here is required for single-process operation'. NATS is currently OPTIONAL. Decide deliberately whether capture requires it (making this feature unavailable in single-process deployments) or degrades to SQLite, and record which -- do not let it become a hard dependency by accident.

Acceptance criteria

An unbound webhook POST is captured and visible rather than rejected; capture is retention-bounded with a documented limit; the single-process (no NATS) behaviour is a deliberate, recorded decision; capture cannot exhaust disk.

Notes

2026-08-22: capture-storage ownership confirmed here (not in 7d5u.3, which was narrowed to storage-agnostic HMAC/rate-limit/redaction mechanics only). This bead's capture path should call webhookguard's redaction before persisting and its rate limiter at the HTTP boundary, but owns its own storage engine and the NATS-vs-SQLite decision.

Close reason

Built capture storage + endpoint: docs/prds/event-capture-storage.md decides SQLite over NATS JetStream (NATS not unconditionally available pre-7d5u.2, capture's single-writer/single-reader access pattern doesn't need JetStream fan-out/replay). internal/store/captures.go adds captured_events table + InsertCapture (prune-on-write by retention age and row count) + ListCaptures. internal/webui/api_capture.go adds POST /webhooks/capture/{source}, unauthenticated (bypasses requireToken like /healthz), applying webhookguard redaction before persisting and a per-remote-address rate limit + hard body-size cap before that. New [capture] config section with documented defaults. Adversarial self-review (the usual fresh subagent pass hit a session API limit mid-run, so this was a manual re-read with the same checklist) caught and fixed three real bugs before landing: (1) body-size cap silently unenforced when unset -- added a hardcoded fallback floor since it's the only bound on a SINGLE payload's size; (2) rate limiter was keyed by the request's own unregistered source path segment, violating webhookguard.RateLimiter's documented bounded-key-space assumption -- an attacker could rotate the segment to bypass the limit and grow the bucket map unbounded; rekeyed to remote address; (3) retention prune used time.RFC3339Nano, whose variable-width trimmed fractional seconds break lexicographic string comparison in WHERE received_at < ? (an exact-second timestamp can sort after a later fractional one) -- switched to fixed-width RFC3339. All three have regression tests verified to fail against the pre-fix code. task check clean, golangci-lint zero new findings repo-wide.


Imported from beads archie-core-t2db.1 · GitHub #538 · type feature · priority P1 · created 2026-08-21 · closed 2026-08-21

The precondition for everything else. Today an unrecognised inbound request is rejected; for schema-by-example it must be RECORDED, because you cannot map a payload you have never seen. Accept, store and retain inbound events with no binding: raw body, headers, source, arrival time, and the identity that sent it. Retention must be bounded -- an open capture endpoint is an unbounded disk write, and this is exactly the kind of surface an attacker fills. Strong argument for NATS JetStream as the store rather than the in-process events.Bus: durable, replayable, retention-bounded natively, already deployed. Note the constraint though -- internal/infrastructure/eventbus/nats/doc.go states 'when no NATS URL is configured the daemon uses its local SQLite claim flow instead; nothing here is required for single-process operation'. NATS is currently OPTIONAL. Decide deliberately whether capture requires it (making this feature unavailable in single-process deployments) or degrades to SQLite, and record which -- do not let it become a hard dependency by accident. ### Acceptance criteria An unbound webhook POST is captured and visible rather than rejected; capture is retention-bounded with a documented limit; the single-process (no NATS) behaviour is a deliberate, recorded decision; capture cannot exhaust disk. ### Notes 2026-08-22: capture-storage ownership confirmed here (not in 7d5u.3, which was narrowed to storage-agnostic HMAC/rate-limit/redaction mechanics only). This bead's capture path should call webhookguard's redaction before persisting and its rate limiter at the HTTP boundary, but owns its own storage engine and the NATS-vs-SQLite decision. ### Close reason Built capture storage + endpoint: docs/prds/event-capture-storage.md decides SQLite over NATS JetStream (NATS not unconditionally available pre-7d5u.2, capture's single-writer/single-reader access pattern doesn't need JetStream fan-out/replay). internal/store/captures.go adds captured_events table + InsertCapture (prune-on-write by retention age and row count) + ListCaptures. internal/webui/api_capture.go adds POST /webhooks/capture/{source}, unauthenticated (bypasses requireToken like /healthz), applying webhookguard redaction before persisting and a per-remote-address rate limit + hard body-size cap before that. New [capture] config section with documented defaults. Adversarial self-review (the usual fresh subagent pass hit a session API limit mid-run, so this was a manual re-read with the same checklist) caught and fixed three real bugs before landing: (1) body-size cap silently unenforced when unset -- added a hardcoded fallback floor since it's the only bound on a SINGLE payload's size; (2) rate limiter was keyed by the request's own unregistered source path segment, violating webhookguard.RateLimiter's documented bounded-key-space assumption -- an attacker could rotate the segment to bypass the limit and grow the bucket map unbounded; rekeyed to remote address; (3) retention prune used time.RFC3339Nano, whose variable-width trimmed fractional seconds break lexicographic string comparison in `WHERE received_at < ?` (an exact-second timestamp can sort after a later fractional one) -- switched to fixed-width RFC3339. All three have regression tests verified to fail against the pre-fix code. task check clean, golangci-lint zero new findings repo-wide. --- _Imported from beads `archie-core-t2db.1` · GitHub #538 · type feature · priority P1 · created 2026-08-21 · closed 2026-08-21_
sam added the highfeature labels 2026-10-27 13:36:18 +00:00
sam self-assigned this 2026-10-27 13:36:18 +00:00
sam closed this issue 2026-10-27 13:36:18 +00:00
sam added a new dependency 2026-10-27 13:41:42 +00:00
sam added a new dependency 2026-10-27 13:42:02 +00:00
sam added a new dependency 2026-10-27 13:42:04 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#432