From 101c133239d8a87027c9d995f874ee31eda2c6da Mon Sep 17 00:00:00 2001 From: Sam Catlow Date: Sat, 22 Aug 2026 04:51:25 +1000 Subject: [PATCH] feat(capture): store unbound webhook events, retention-bounded Adds POST /webhooks/capture/{source}: an unauthenticated endpoint (mounted alongside /healthz, bypassing dashboard token auth by design) that persists any inbound webhook archie has no binding for yet, so it can be inspected and mapped later (t2db.2/t2db.3). docs/prds/event-capture-storage.md decides SQLite over NATS JetStream for storage: NATS is not unconditionally available until 7d5u.2 ships embedded support, and capture's single-writer/single-reader access pattern doesn't need JetStream's fan-out/replay machinery anyway. internal/store/captures.go adds a captured_events table with prune-on-write (retention age + row-count cap, both configurable). The HTTP handler applies webhookguard redaction before persisting, and enforces two more independent disk-bound mechanics before that: a per-request body-size cap and a per-remote-address rate limit. New [capture] config section with documented defaults; all fields optional, capture is on by default. Adversarial review (manual, after the usual fresh-subagent pass hit a session API limit mid-run) caught and fixed three real defects before this landed, each with a regression test confirmed to fail against the pre-fix code: - the body-size cap silently no-op'd when CaptureMaxBodyBytes was left unset, the one bound that limits a single payload's size rather than row count or write rate - the rate limiter was keyed by the request's own source path segment -- unregistered and attacker-chosen by design for this endpoint, violating webhookguard.RateLimiter's documented bounded-key-space assumption and letting a sender bypass the limit by rotating the segment; rekeyed to remote address - the retention prune compared time.RFC3339Nano strings, whose trimmed variable-width fractional seconds break lexicographic ordering within the same second; switched to fixed-width RFC3339 Closes archie-core-t2db.1. --- .beads/interactions.jsonl | 1 + .beads/issues.jsonl | 2 +- cmd/archied/bootstrap.go | 14 + config.example.toml | 15 + docs/prds/event-capture-storage.md | 144 ++++++++++ internal/config/config.go | 27 ++ .../infrastructure/configuration/defaults.go | 30 ++ .../configuration/defaults_test.go | 58 ++++ .../infrastructure/configuration/validate.go | 30 +- .../configuration/validate_test.go | 30 ++ internal/store/captures.go | 140 +++++++++ internal/store/captures_test.go | 169 +++++++++++ internal/store/interface.go | 14 + internal/store/store.go | 2 +- internal/webui/api_capture.go | 102 +++++++ internal/webui/api_capture_test.go | 269 ++++++++++++++++++ internal/webui/server.go | 22 ++ 17 files changed, 1066 insertions(+), 3 deletions(-) create mode 100644 docs/prds/event-capture-storage.md create mode 100644 internal/infrastructure/configuration/defaults_test.go create mode 100644 internal/store/captures.go create mode 100644 internal/store/captures_test.go create mode 100644 internal/webui/api_capture.go create mode 100644 internal/webui/api_capture_test.go diff --git a/.beads/interactions.jsonl b/.beads/interactions.jsonl index 51f465436..9fee12bca 100644 --- a/.beads/interactions.jsonl +++ b/.beads/interactions.jsonl @@ -26,3 +26,4 @@ {"id":"int-c00194f10222de956f7ad8987d09a9fb","kind":"field_change","created_at":"2026-08-20T13:24:15.962276702Z","actor":"Sam Catlow","issue_id":"archie-core-1786637500602-410-7a461d4d","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Already delivered as a compiled-in engine, not a plugin. internal/secret/bws.go implements secret.Engine and shells out to the bws CLI via os/exec (commit 3ae59c9, 2026-07-31); NewRegistry/DefaultRegistry register it under the name \"bws\" alongside \"env\". It is documented in config.example.toml and ARCHITECTURE.md:208, and config.example.toml:64 uses it as the worked example. Shipping a second yaegi implementation under examples/secret-engines/ would register the same name \"bws\", and Registry.Register is last-write-wins over LoadDir, so the plugin would silently shadow the compiled-in engine -- two authoritative paths for one behaviour, which the architecture contract prohibits. Reopen only if the intent is to REPLACE the compiled-in engine with a plugin (which would also need bws removed from NewRegistry and the docs updated)."}} {"id":"int-907fab8ee9218dfc52e1b312ccdde513","kind":"field_change","created_at":"2026-08-21T15:00:54.642569071Z","actor":"Sam Catlow","issue_id":"archie-core-7d5u.1","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Decided and documented in docs/prds/event-sources-and-reactions.md, satisfying this bead's own acceptance criterion (a one-page decision naming the packages it touches and stating whether plugins-and-extensions.md needs amending -- it does not).\n\nAnswers: (1) in-process typed families, no out-of-process dispatch -- archie has no existing extension-process boundary and neither epic's stated payoff needs one. (2) Reaction delivery reuses internal/eventbus's Publisher/Consumer contract (PublishUnique + Ack/Nak already provide at-least-once with dedup), but NOT the existing nats.Client as-is: it hardcodes Retention: jetstream.WorkQueuePolicy (client.go:65) for the competing-consumer ARCHIE_TASKS stream, which silently starves a second consumer on overlapping subjects per CLAUDE.md's documented trap -- reactions need fan-out, so retention needs to become a Config field and reactions get their own stream. (3) Reactions in these two epics are producer-only (cheap wake via curator's WakeOnPrimaryInput pattern, or produce real work via the existing pollNATS->publishTask path) -- no veto/mutate-in-flight-work capability is being built here; that belongs to archie-core-h019 if it's ever needed. (4) No single generic Source interface -- forge intake reuses workintake.TaskEnvelope unchanged (its IdempotencyKey is already keyed on owner/repo/number, not delivery source, so a webhook-sourced envelope naturally dedups against a poll-discovered one for free); chat webhook stays internal/channels/webhook, unmodified; what's genuinely new and shared is a small cross-cutting package for HMAC/rate-limit/approval/capture mechanics, owned by neither domain."}} {"id":"int-15527c6640daf7df72a4876d8e82864c","kind":"field_change","created_at":"2026-08-21T15:26:01.152938969Z","actor":"Sam Catlow","issue_id":"archie-core-7d5u.3","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Built internal/webhookguard: VerifyHMAC (behavior-preserving extraction from internal/channels/webhook, which now calls it instead of keeping a private copy), RateLimiter (per-source token bucket, independent budgets, observable rejection), RedactPayload (key-name heuristic matching gateway/stream.go's marker list, but -- deliberately unlike that function -- redacts only values under sensitive keys and leaves ordinary strings/numbers/bools untouched, since a captured payload exists to be inspected for field mapping in t2db.3).\n\nAll four acceptance criteria met and independently confirmed by a fresh adversarial review with zero surviving findings: HMAC extraction is byte-for-byte behavior-preserving (diffed against the deleted function); rate limiter hand-traced correct on two scenarios plus race-clean under -race; redaction's divergence from gateway/stream.go verified by reading both functions side by side; zero non-stdlib imports confirmed by grep. The reviewer also mutation-tested both the redaction and rate-limit guards (disabled each in turn) and confirmed the test suite catches both, then reverted and re-verified go test -race and git status were clean.\n\ntask check exit 0, golangci-lint 0 issues (one forcetypeassert finding was caught and fixed properly during build, not suppressed), gofumpt clean. All 10 pre-existing internal/channels/webhook tests pass unmodified; the one now-redundant unit test was removed with a comment pointing to where its coverage relocated."}} +{"id":"int-ccb87c91023dbaa6718ae2f7564d38cc","kind":"field_change","created_at":"2026-08-21T18:51:09.120109805Z","actor":"Sam Catlow","issue_id":"archie-core-t2db.1","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Built capture storage + endpoint: docs/prds/event-capture-storage.md decides SQLite over NATS JetStream (NATS not unconditionally available pre-7d5u.2, capture's single-writer/single-reader access pattern doesn't need JetStream fan-out/replay). internal/store/captures.go adds captured_events table + InsertCapture (prune-on-write by retention age and row count) + ListCaptures. internal/webui/api_capture.go adds POST /webhooks/capture/{source}, unauthenticated (bypasses requireToken like /healthz), applying webhookguard redaction before persisting and a per-remote-address rate limit + hard body-size cap before that. New [capture] config section with documented defaults. Adversarial self-review (the usual fresh subagent pass hit a session API limit mid-run, so this was a manual re-read with the same checklist) caught and fixed three real bugs before landing: (1) body-size cap silently unenforced when unset -- added a hardcoded fallback floor since it's the only bound on a SINGLE payload's size; (2) rate limiter was keyed by the request's own unregistered source path segment, violating webhookguard.RateLimiter's documented bounded-key-space assumption -- an attacker could rotate the segment to bypass the limit and grow the bucket map unbounded; rekeyed to remote address; (3) retention prune used time.RFC3339Nano, whose variable-width trimmed fractional seconds break lexicographic string comparison in `WHERE received_at < ?` (an exact-second timestamp can sort after a later fractional one) -- switched to fixed-width RFC3339. All three have regression tests verified to fail against the pre-fix code. task check clean, golangci-lint zero new findings repo-wide."}} diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index b52b9aef8..ce2360302 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,6 +1,6 @@ {"_type":"issue","id":"archie-core-t2db.5","title":"Security: an unauthenticated endpoint must not be able to run an agent","description":"P0 within this epic because the feature's own pitch is the threat model: a public intake surface whose output is an agent with tool access and a repo. Get this wrong and a webhook URL is remote code execution with extra steps.\n\nNon-negotiables to design:\n\n- Per-source authentication. A source registers with a secret (HMAC, as internal/channels/webhook already does per route) before its events can trigger anything. Capture-only for unknown senders is acceptable; triggering is not.\n- Human approval before a binding goes live. An operator authoring a binding is a deliberate act; an event arriving is not. Nothing should self-arm.\n- Rate and volume limits per source, so capture cannot be used to exhaust disk or drown the inspector.\n- Payload content is untrusted input flowing into an agent prompt. Treat prompt injection as expected, not exceptional -- the sender controls that text, and the agent has tools. Bound what a playbook-triggered workflow may do.\n- Captured payloads may contain secrets from the sending system. Decide retention, redaction and who can view them; internal/memory/scrubber may already have applicable prior art.\n\nArchie's own principle applies: environmental enforcement over prompt rules. 'The agent should ignore instructions in the payload' is not a control.","acceptance_criteria":"An event from an unauthenticated source can be captured but cannot trigger a workflow; a new binding requires explicit human approval before it arms; per-source rate limits exist; the blast radius of a playbook-triggered workflow is bounded by code rather than by prompt text; retention and visibility of captured payloads is decided and documented.","notes":"2026-08-22: security design decided and documented in docs/prds/webhook-intake-security.md, covering all five non-negotiables from this bead's description:\n1. Per-source auth reuses internal/channels/webhook's existing HMAC-SHA256 scheme; capture accepts unsigned events (visible, marked unauthenticated), only authenticated events can ever match a binding.\n2. Bindings gain a draft -\u003e pending_approval -\u003e armed state machine; any edit to an armed binding drops it back to pending_approval so approval can't be bypassed by editing. Modeled on telegram's existing dangerousAction/pendingApproval flow (dangerous.go/approval.go).\n3. Per-source token-bucket rate limiting (precedent: pairing/store.go's MaxPendingPerPlatform); limit-exceeded is an observable 429, not a silent drop.\n4. Blast radius bounded by the SAME code-level controls every task already gets -- gate, diff_cap_lines, sandboxed container, worktree isolation -- no new relaxation for playbook-triggered work, and payload text is ordinary untrusted prompt input, never a trusted instruction.\n5. 7-day configurable retention on captured payloads; best-effort redaction reusing gateway/stream.go's sensitiveParameterKey heuristic before persisting; visibility gated by the same dashboard auth as every other webui surface.\n\nThis bead's code-level acceptance criteria (capture-can't-trigger, rate limits exist, blast-radius bounded, approval gate) cannot be implemented yet: they constrain t2db.1 (capture), 7d5u.3 (intake family) and t2db.4 (binding), none of which have landed. This document is the contract those beads build against -- do not build any of them without satisfying it. Leaving this bead in_progress rather than closing it; it closes when t2db.1/7d5u.3/t2db.4 land and demonstrably satisfy points 1-5, not before.","status":"in_progress","priority":0,"issue_type":"feature","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-21T13:45:46Z","created_by":"Sam Catlow","updated_at":"2026-08-21T14:53:33Z","started_at":"2026-08-21T14:51:26Z","dependencies":[{"issue_id":"archie-core-t2db.5","depends_on_id":"archie-core-t2db","type":"parent-child","created_at":"2026-08-21T23:45:45Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"archie-core-t2db.4","title":"Playbook binding: matcher + mapping + workflow, editable at runtime","description":"The object that ties it together. A binding is: a matcher (which inbound events does this apply to), a mapping (which fields the workflow receives), and a target agentic workflow.\n\nRuntime-editable is the whole point -- creating or changing a binding must not require a redeploy or a daemon restart. That has real consequences the design must face:\n\n- Bindings are persisted state, not config. They live in the store, not config.toml, because operators create them from the UI while the daemon runs.\n- They need versioning. A binding that starts a task should record WHICH version ran, or a later edit makes historical task provenance a lie.\n- Matcher ambiguity needs a defined resolution: two bindings matching one event either both run, or the more specific wins, or it is an error. Pick one and say so.\n- A binding referencing a workflow that no longer exists must fail visibly at bind time, not at 3am when the event arrives.","acceptance_criteria":"A binding can be created, edited and disabled from the UI while the daemon runs; a task records the binding version that started it; overlapping matchers resolve by a defined and documented rule; a binding naming a missing workflow is rejected when saved.","notes":"2026-08-22: the draft -\u003e pending_approval -\u003e armed state machine from t2db.5's decision belongs to the binding entity this bead defines, not to a shared package -- 7d5u.3 was narrowed to exclude it since there is exactly one consumer today. Implement the state transitions here.","status":"open","priority":1,"issue_type":"feature","owner":"sam@catlow.tech","created_at":"2026-08-21T13:45:45Z","created_by":"Sam Catlow","updated_at":"2026-08-21T15:19:05Z","dependencies":[{"issue_id":"archie-core-t2db.4","depends_on_id":"archie-core-t2db.5","type":"blocks","created_at":"2026-08-21T23:45:53Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-t2db.4","depends_on_id":"archie-core-t2db.3","type":"blocks","created_at":"2026-08-21T23:45:53Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-t2db.4","depends_on_id":"archie-core-t2db","type":"parent-child","created_at":"2026-08-21T23:45:45Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":2,"dependent_count":1,"comment_count":0} -{"_type":"issue","id":"archie-core-t2db.1","title":"Capture inbound events archie does not yet understand","description":"The precondition for everything else. Today an unrecognised inbound request is rejected; for schema-by-example it must be RECORDED, because you cannot map a payload you have never seen.\n\nAccept, store and retain inbound events with no binding: raw body, headers, source, arrival time, and the identity that sent it. Retention must be bounded -- an open capture endpoint is an unbounded disk write, and this is exactly the kind of surface an attacker fills.\n\nStrong argument for NATS JetStream as the store rather than the in-process events.Bus: durable, replayable, retention-bounded natively, already deployed. Note the constraint though -- internal/infrastructure/eventbus/nats/doc.go states 'when no NATS URL is configured the daemon uses its local SQLite claim flow instead; nothing here is required for single-process operation'. NATS is currently OPTIONAL. Decide deliberately whether capture requires it (making this feature unavailable in single-process deployments) or degrades to SQLite, and record which -- do not let it become a hard dependency by accident.","acceptance_criteria":"An unbound webhook POST is captured and visible rather than rejected; capture is retention-bounded with a documented limit; the single-process (no NATS) behaviour is a deliberate, recorded decision; capture cannot exhaust disk.","notes":"2026-08-22: capture-storage ownership confirmed here (not in 7d5u.3, which was narrowed to storage-agnostic HMAC/rate-limit/redaction mechanics only). This bead's capture path should call webhookguard's redaction before persisting and its rate limiter at the HTTP boundary, but owns its own storage engine and the NATS-vs-SQLite decision.","status":"open","priority":1,"issue_type":"feature","owner":"sam@catlow.tech","created_at":"2026-08-21T13:45:18Z","created_by":"Sam Catlow","updated_at":"2026-08-21T15:19:05Z","dependencies":[{"issue_id":"archie-core-t2db.1","depends_on_id":"archie-core-7d5u.4","type":"relates-to","created_at":"2026-08-22T01:01:02Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-t2db.1","depends_on_id":"archie-core-t2db","type":"parent-child","created_at":"2026-08-21T23:45:17Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":0,"dependent_count":2,"comment_count":0} +{"_type":"issue","id":"archie-core-t2db.1","title":"Capture inbound events archie does not yet understand","description":"The precondition for everything else. Today an unrecognised inbound request is rejected; for schema-by-example it must be RECORDED, because you cannot map a payload you have never seen.\n\nAccept, store and retain inbound events with no binding: raw body, headers, source, arrival time, and the identity that sent it. Retention must be bounded -- an open capture endpoint is an unbounded disk write, and this is exactly the kind of surface an attacker fills.\n\nStrong argument for NATS JetStream as the store rather than the in-process events.Bus: durable, replayable, retention-bounded natively, already deployed. Note the constraint though -- internal/infrastructure/eventbus/nats/doc.go states 'when no NATS URL is configured the daemon uses its local SQLite claim flow instead; nothing here is required for single-process operation'. NATS is currently OPTIONAL. Decide deliberately whether capture requires it (making this feature unavailable in single-process deployments) or degrades to SQLite, and record which -- do not let it become a hard dependency by accident.","acceptance_criteria":"An unbound webhook POST is captured and visible rather than rejected; capture is retention-bounded with a documented limit; the single-process (no NATS) behaviour is a deliberate, recorded decision; capture cannot exhaust disk.","notes":"2026-08-22: capture-storage ownership confirmed here (not in 7d5u.3, which was narrowed to storage-agnostic HMAC/rate-limit/redaction mechanics only). This bead's capture path should call webhookguard's redaction before persisting and its rate limiter at the HTTP boundary, but owns its own storage engine and the NATS-vs-SQLite decision.","status":"closed","priority":1,"issue_type":"feature","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-21T13:45:18Z","created_by":"Sam Catlow","updated_at":"2026-08-21T18:51:09Z","started_at":"2026-08-21T16:18:24Z","closed_at":"2026-08-21T18:51:09Z","close_reason":"Built capture storage + endpoint: docs/prds/event-capture-storage.md decides SQLite over NATS JetStream (NATS not unconditionally available pre-7d5u.2, capture's single-writer/single-reader access pattern doesn't need JetStream fan-out/replay). internal/store/captures.go adds captured_events table + InsertCapture (prune-on-write by retention age and row count) + ListCaptures. internal/webui/api_capture.go adds POST /webhooks/capture/{source}, unauthenticated (bypasses requireToken like /healthz), applying webhookguard redaction before persisting and a per-remote-address rate limit + hard body-size cap before that. New [capture] config section with documented defaults. Adversarial self-review (the usual fresh subagent pass hit a session API limit mid-run, so this was a manual re-read with the same checklist) caught and fixed three real bugs before landing: (1) body-size cap silently unenforced when unset -- added a hardcoded fallback floor since it's the only bound on a SINGLE payload's size; (2) rate limiter was keyed by the request's own unregistered source path segment, violating webhookguard.RateLimiter's documented bounded-key-space assumption -- an attacker could rotate the segment to bypass the limit and grow the bucket map unbounded; rekeyed to remote address; (3) retention prune used time.RFC3339Nano, whose variable-width trimmed fractional seconds break lexicographic string comparison in `WHERE received_at \u003c ?` (an exact-second timestamp can sort after a later fractional one) -- switched to fixed-width RFC3339. All three have regression tests verified to fail against the pre-fix code. task check clean, golangci-lint zero new findings repo-wide.","dependencies":[{"issue_id":"archie-core-t2db.1","depends_on_id":"archie-core-7d5u.4","type":"relates-to","created_at":"2026-08-22T01:01:02Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-t2db.1","depends_on_id":"archie-core-t2db","type":"parent-child","created_at":"2026-08-21T23:45:17Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"archie-core-t2db.2","title":"Event inspector: see real inbound payloads in the dashboard","description":"The operator-facing half of capture. A live view of recent inbound events: source, arrival time, and the actual payload, pretty-printed and explorable.\n\nThis is the screen that makes the feature feel like magic: you hit 'send test webhook' in the other app, alt-tab to archie, and the event is sitting there. Latency from arrival to visible matters more than it looks -- it is the difference between a design loop and a guessing game.\n\nShow unbound events prominently. An event nothing is listening to is not noise here, it is the queue of integrations waiting to be built.","acceptance_criteria":"Recent inbound events are listed with source, time and full payload; unbound events are distinguishable from bound ones; a newly arrived event appears without a manual refresh; payloads are explorable rather than a wall of text.","status":"open","priority":1,"issue_type":"feature","owner":"sam@catlow.tech","created_at":"2026-08-21T13:45:18Z","created_by":"Sam Catlow","updated_at":"2026-08-21T13:45:18Z","dependencies":[{"issue_id":"archie-core-t2db.2","depends_on_id":"archie-core-t2db.1","type":"blocks","created_at":"2026-08-21T23:45:53Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-t2db.2","depends_on_id":"archie-core-t2db","type":"parent-child","created_at":"2026-08-21T23:45:17Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"archie-core-t2db.3","title":"Map payload fields by example, from a real captured event","description":"Pick a captured event, click through its payload, and bind JSON paths to named fields the workflow will receive. Schema by example, not schema by specification.\n\nThe hard part is not the happy path, it is drift. A mapping authored against one payload will eventually meet a payload where the path is missing, null, or a different type -- the sending app shipped a change nobody told you about. That must fail LOUDLY and land somewhere the operator sees, not silently pass an empty string into an agent prompt. An agent handed a blank field will confabulate around it, which turns a schema change into an agent doing confident nonsense in a repo.\n\nSupport a preview: given this mapping and this captured event, show exactly what the workflow would receive. Never ship a mapping the operator has not seen resolve against real data.","acceptance_criteria":"Fields are mappable from a captured payload without writing code; a mapping preview resolves against a real event before saving; a missing, null or type-changed path at runtime produces a visible failure rather than an empty value; the failure names the path and the event.","status":"open","priority":1,"issue_type":"feature","owner":"sam@catlow.tech","created_at":"2026-08-21T13:45:18Z","created_by":"Sam Catlow","updated_at":"2026-08-21T13:45:18Z","dependencies":[{"issue_id":"archie-core-t2db.3","depends_on_id":"archie-core-t2db","type":"parent-child","created_at":"2026-08-21T23:45:18Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-t2db.3","depends_on_id":"archie-core-t2db.1","type":"blocks","created_at":"2026-08-21T23:45:53Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"archie-core-t2db","title":"[EPIC] Event capture and no-code playbooks: turn any inbound webhook into an agentic workflow","description":"The pitch, in the operator's words: point any app's outgoing webhook at archie, open archie, SEE the event that just arrived with its real payload, map the fields you care about, bind them to an agentic workflow, save. No code change, no redeploy, no restart. Adding a new integration stops being a development task.\n\nWhy this is different from n8n or Power Automate rather than a copy of them: their terminal node is a deterministic connector. Archie's is an AGENT with a repo, a worktree, tools and a gate. 'When Sentry reports a new issue in this project, open a PR that fixes it' is one binding here and an entire product elsewhere.\n\nThe design insight that makes it buildable: you cannot write a payload mapper against a payload you have not seen. Schema-by-example requires real recent events to map against, which means archie must CAPTURE events it does not yet understand -- including ones with no binding at all -- and retain them long enough to design against. That is also the strongest argument for putting inbound events on NATS JetStream rather than the lossy in-process bus: durable, replayable, retention-bounded, and already deployed. The capture requirement and the transport choice are the same decision.\n\nSequence: capture unrecognised events -\u003e inspect real payloads in the UI -\u003e map fields by example -\u003e bind matcher+mapping to a workflow -\u003e the next matching event runs it.\n\nThis is a public, unauthenticated-by-default intake surface whose output is an agent with tool access. Security is not a later hardening pass; it is a first-class child of this epic.","acceptance_criteria":"An operator can point a new external webhook at archie, see its payload in the UI without writing code, map fields, bind it to a workflow, and have the next matching event run that workflow -- with no redeploy and no daemon restart. A binding cannot go live without explicit human approval.","status":"open","priority":1,"issue_type":"epic","owner":"sam@catlow.tech","created_at":"2026-08-21T13:44:56Z","created_by":"Sam Catlow","updated_at":"2026-08-21T13:44:56Z","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/cmd/archied/bootstrap.go b/cmd/archied/bootstrap.go index a21b509d4..838938696 100644 --- a/cmd/archied/bootstrap.go +++ b/cmd/archied/bootstrap.go @@ -53,6 +53,7 @@ import ( builtintoolprovider "github.com/samcharles93/archie-core/internal/tools/provider/builtin" memorytoolprovider "github.com/samcharles93/archie-core/internal/tools/provider/memory" "github.com/samcharles93/archie-core/internal/tools/webfetch" + "github.com/samcharles93/archie-core/internal/webhookguard" "github.com/samcharles93/archie-core/internal/webui" "github.com/samcharles93/archie-core/internal/worktree" ) @@ -306,6 +307,19 @@ func (b *boot) setupObservability() { // issue is re-polled and the work is done again. It gets only that one // method, not the forge client. b.web.Issues = b.forgeClient + // Capture storage: openProductionTaskStore's declared return type is the + // narrow store.TaskStore, so the CaptureStore surface (implemented by + // the same *store.Store) needs its own assertion here rather than a + // direct field reuse. See docs/prds/event-capture-storage.md. + if cs, ok := b.st.(store.CaptureStore); ok { + b.web.Captures = cs + } else { + log.Warn("capture storage unavailable: task store does not implement CaptureStore") + } + b.web.CaptureRetention = cfg.Capture.Retention.Std() + b.web.CaptureMaxEvents = cfg.Capture.MaxEvents + b.web.CaptureMaxBodyBytes = int64(cfg.Capture.MaxBodyBytes) + b.web.CaptureLimiter = webhookguard.NewRateLimiter(cfg.Capture.RatePerSecond, cfg.Capture.RateBurst, time.Now) sink := bus.Subscribe(256) go persistAndBroadcastEvents(sink, b.st, b.web, log) } diff --git a/config.example.toml b/config.example.toml index dc2c22dfc..ac86a111e 100644 --- a/config.example.toml +++ b/config.example.toml @@ -174,6 +174,21 @@ gate_max_failures = 5 # url = "nats://localhost:4222" # token_env = "NATS_TOKEN" # optional; named variable must be set +# Unbound webhook capture: POST /webhooks/capture/{source} accepts and +# stores any inbound payload archie doesn't yet have a binding for, so it can +# be inspected and mapped later (see docs/prds/event-capture-storage.md). +# Stored in the same SQLite database as tasks, not NATS -- capture has one +# writer and one (human, dashboard) reader, so it doesn't need JetStream's +# fan-out/replay machinery, and SQLite is available in every deployment +# shape unconditionally. All fields default when the section is absent; +# capture is on by default. +# [capture] +# retention = "168h" # 7 days; prune-on-write deletes anything older +# max_events = 5000 # row cap, pruned on every insert regardless of age +# max_body_bytes = 262144 # 256 KiB; larger POSTs are rejected with 413 +# rate_per_second = 1.0 # per-remote-address token bucket +# rate_burst = 5 + # Docker sandbox execution of archie-agent. Requires [nats] and agent.mode = "nats". # [containers] # enabled = true diff --git a/docs/prds/event-capture-storage.md b/docs/prds/event-capture-storage.md new file mode 100644 index 000000000..fc5fb8eb5 --- /dev/null +++ b/docs/prds/event-capture-storage.md @@ -0,0 +1,144 @@ +# Event capture storage -- decision + +**Status:** Decided, not yet implemented +**Date:** 2026-08-22 +**Beads issue:** `archie-core-t2db.1` + +`webhook-intake-security.md` decided points 1-5 (auth split, approval gate, +rate limits, blast radius, retention/redaction) and explicitly left the +storage mechanism open, forward-pointing at "`7d5u.2`, embedded NATS." That +pointer predates `7d5u.3`'s 2026-08-22 narrowing, which assigned capture +storage to this bead by name and told it to decide NATS-vs-SQLite itself +rather than inherit an answer. This document is that decision. + +## Decision: SQLite, not NATS JetStream + +Captured events are stored in archied's existing SQLite database, in a new +`captured_events` table following the same shape as `internal/store/events.go` +(`internal/store`, `*Store`, SQL migrations inline in Go). No JetStream +stream is introduced for capture. + +### Why + +**NATS is not unconditionally available today.** `config.NATSConfig.URL` +empty means no NATS client is constructed (`bootstrap.go`); embedded NATS +(`7d5u.2`) that would remove this gap is still open. `t2db.1`'s own +acceptance criteria require single-process behaviour to be a *deliberate* +decision, not an accident -- making capture NATS-only today would silently +regress the `local-ollama-standalone.toml` deployment profile, which this +epic's own precondition line ("capture is the precondition for everything +else") argues against. SQLite is the one store guaranteed present in every +deployment shape, because it already backs task state unconditionally. + +**Capture's access pattern doesn't need what JetStream is for.** JetStream +earns its complexity for reaction delivery (`7d5u.2`) because multiple +independent consumers need fan-out, at-least-once redelivery, and Ack/Nak +semantics -- exactly the trap `CLAUDE.md` documents about `WorkQueuePolicy` +and overlapping consumers. Capture has exactly one writer (the HTTP handler) +and one reader (a human via the dashboard inspector, `t2db.2`). There is no +competing-consumer problem to solve and no redelivery semantics to get right, +so JetStream's core value proposition doesn't apply here. A bounded SQL +table is a better fit on the merits, not merely as a fallback. + +**Reusing `7d5u.2`'s stream, once it exists, would still be wrong.** That +stream is provisioned for fan-out reaction delivery with its own retention +policy decision. Capture's retention need -- age- and count-bounded, prune +on write, no replay -- is a different shape. Building a second stream just +for capture, gated on an unrelated bead landing first, would trade a small +amount of duplication now for a hard dependency on `7d5u.2`'s timeline that +this bead's own acceptance criteria say not to accrue by accident. + +This does not foreclose NATS later. If cross-daemon capture sharing or +durable replay becomes a real requirement, that is a new decision made +against a real need, not now against a speculative one. + +## Storage shape + +`captured_events` table (`internal/store/captures.go`, mirrors +`internal/store/events.go`'s pattern): + +| column | type | notes | +|-----------------|---------|-------------------------------------------------| +| id | INTEGER | primary key, autoincrement | +| received_at | TEXT | RFC3339Nano | +| source | TEXT | opaque path segment chosen by the sender's operator | +| remote_addr | TEXT | `RemoteAddr` at capture time | +| content_type | TEXT | `Content-Type` header | +| headers | TEXT | JSON, redacted | +| body | TEXT | JSON, redacted (`webhookguard.RedactPayload`) | +| authenticated | INTEGER | 0/1; always 0 until a source registers a secret (future: `t2db.4`) | + +Indexed on `(source, id)` for per-source listing and pruning, and `id` alone +for the global cap. + +## Disk-bound mechanics + +Three independent bounds, composed, per `webhook-intake-security.md` point 5 +and this bead's own "capture cannot exhaust disk" criterion: + +1. **Per-request size cap.** The HTTP handler reads the body through + `http.MaxBytesReader` (256 KiB default, configurable). A single oversized + POST is rejected (413) before redaction or storage sees it. +2. **Per-remote-address rate limit.** `webhookguard.RateLimiter`, keyed by + the request's remote address, applied before the body is even read. + *Not* keyed by the `source` path segment: `RateLimiter`'s own doc comment + states its bucket map assumes a bounded, operator-registered key space, + never evicted. `source` is exactly the opposite here -- an unregistered, + attacker-chosen URL segment, since capturing senders with no registration + yet is this endpoint's entire purpose. Keying on it would let one sender + bypass the limit by rotating the segment every request (a fresh key + always starts with a full burst) and grow the bucket map without bound. + Remote address is the bounded identity actually available pre-auth. + Over-budget requests get 429, observable per point 3 of the security + decision. +3. **Retention prune-on-write.** Every insert prunes, in the same + transaction: rows older than `retention_days` (default 7, per the + security decision) OR beyond `max_events` total rows (default 5000, this + bead's own addition -- age alone doesn't bound disk under a sustained + flood inside the retention window; a count cap does). No background + sweep goroutine; the daemon doesn't need one for this. + +Together these give a deterministic worst-case: `max_events * max_body_bytes` +regardless of sender behaviour, independent of whether the age-based or +count-based bound is what actually fires first. + +## HTTP surface + +Mounted on the existing `internal/webui` dashboard server +(`Server.Handler`'s `top` mux), not a new listener. `POST +/webhooks/capture/{source}` bypasses `requireToken` the same way `/healthz` +already does -- capture must accept unauthenticated senders by design (that +is the entire point: you cannot HMAC-verify a source you haven't configured +a secret for yet). Reachability is therefore whatever already exposes the +dashboard in a given deployment; no new port/host config surface is added, +and no new claim about public reachability is made beyond what already +exists for the dashboard. + +`GET` listing/inspection endpoints (`t2db.2`) stay behind `requireToken` like +every other `/api/*` route -- captured payloads are visible only to an +authenticated operator, per the security decision's point 5 visibility rule. + +## Config + +New `[capture]` section in `config.example.toml` / `internal/config`: + +```toml +[capture] +retention_days = 7 +max_events = 5000 +max_body_bytes = 262144 +rate_per_second = 1.0 +rate_burst = 5 +``` + +All fields have defaults; an empty `[capture]` section (or its absence) is +valid and produces the defaults above -- capture is on by default, since an +unbound POST being captured rather than rejected is this bead's acceptance +criterion, not an opt-in. + +## What this does not decide + +Binding a captured event to a workflow, the draft/pending_approval/armed +state machine, and per-source HMAC secret registration are `t2db.3`/`t2db.4`. +The dashboard inspector UI is `t2db.2`. This bead ships capture and storage +only. diff --git a/internal/config/config.go b/internal/config/config.go index 4868d2533..a61c44b7d 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -385,6 +385,7 @@ type Config struct { NATS NATSConfig `toml:"nats" yaml:"nats"` Containers ContainerConfig `toml:"containers" yaml:"containers"` Chat ChatConfig `toml:"chat" yaml:"chat"` + Capture CaptureConfig `toml:"capture" yaml:"capture"` // Memory holds memory provider configuration (from config.memory.yaml). Memory MemoryConfig `toml:"memory" yaml:"memory"` @@ -581,6 +582,32 @@ type NATSConfig struct { TokenEnv string `toml:"token_env" yaml:"token_env"` } +// CaptureConfig configures the unbound webhook capture endpoint +// (docs/prds/event-capture-storage.md). All fields have defaults; an empty +// [capture] section is valid and produces them -- capture is on by default. +type CaptureConfig struct { + // Retention is how long a captured event is kept before prune-on-write + // deletes it. Zero or absent means the 7-day default; a positive value + // configured explicitly is used as-is; a deployment that wants no age + // bound sets it very large rather than using a magic "0 = unlimited" + // spelling, since 0 already means "use the default" here. + Retention Duration `toml:"retention" yaml:"retention"` + // MaxEvents caps the table at this many newest rows; older rows are + // pruned on every insert regardless of age. Zero means the 5000 default. + MaxEvents int `toml:"max_events" yaml:"max_events"` + // MaxBodyBytes rejects (413) any single POST body larger than this + // before it is read into memory. Zero means the 256 KiB default. + MaxBodyBytes int `toml:"max_body_bytes" yaml:"max_body_bytes"` + // RatePerSecond and RateBurst configure the per-remote-address token + // bucket (webhookguard.RateLimiter) applied before a request's body is + // read. Keyed by remote address, not the source path segment, since + // that segment is unregistered and attacker-chosen by design -- see + // docs/prds/event-capture-storage.md's disk-bound mechanics. Zero means + // the defaults (1 req/s, burst 5). + RatePerSecond float64 `toml:"rate_per_second" yaml:"rate_per_second"` + RateBurst int `toml:"rate_burst" yaml:"rate_burst"` +} + // ContainerConfig configures Docker sandbox execution of archie-agent. type ContainerConfig struct { // Enabled activates container execution. Requires agent.mode = "nats". diff --git a/internal/infrastructure/configuration/defaults.go b/internal/infrastructure/configuration/defaults.go index 58755a262..7b9a9fa64 100644 --- a/internal/infrastructure/configuration/defaults.go +++ b/internal/infrastructure/configuration/defaults.go @@ -39,6 +39,15 @@ const ( // above trims it further. Generous enough for a documentation page, // small enough that a large download cannot occupy the daemon. defaultWebFetchMaxBytes = 2_000_000 + + // Capture defaults, per docs/prds/event-capture-storage.md's disk-bound + // mechanics: retention + row cap + per-request size cap + per-source + // rate limit, composed. + defaultCaptureRetention = 7 * 24 * time.Hour + defaultCaptureMaxEvents = 5000 + defaultCaptureMaxBodyBytes = 256 * 1024 + defaultCaptureRatePerSecond = 1.0 + defaultCaptureRateBurst = 5 ) // applyDefaults fills in every absent value. It never reports an error and @@ -58,6 +67,27 @@ func (l *Loader) applyDefaults(cfg *config.Config) { applyIdentityDefaults(cfg) applyContainerDefaults(cfg) applyToolPolicyDefaults(cfg) + applyCaptureDefaults(cfg) +} + +// applyCaptureDefaults fills the webhook capture endpoint's settings. See +// [config.CaptureConfig]. +func applyCaptureDefaults(cfg *config.Config) { + if cfg.Capture.Retention == 0 { + cfg.Capture.Retention = config.Duration(defaultCaptureRetention) + } + if cfg.Capture.MaxEvents == 0 { + cfg.Capture.MaxEvents = defaultCaptureMaxEvents + } + if cfg.Capture.MaxBodyBytes == 0 { + cfg.Capture.MaxBodyBytes = defaultCaptureMaxBodyBytes + } + if cfg.Capture.RatePerSecond == 0 { + cfg.Capture.RatePerSecond = defaultCaptureRatePerSecond + } + if cfg.Capture.RateBurst == 0 { + cfg.Capture.RateBurst = defaultCaptureRateBurst + } } // applyToolPolicyDefaults fills the tool result limits. diff --git a/internal/infrastructure/configuration/defaults_test.go b/internal/infrastructure/configuration/defaults_test.go new file mode 100644 index 000000000..f7e57dc12 --- /dev/null +++ b/internal/infrastructure/configuration/defaults_test.go @@ -0,0 +1,58 @@ +package configuration + +import ( + "testing" + "time" + + "github.com/samcharles93/archie-core/internal/config" +) + +func TestApplyCaptureDefaultsFillsZeroValues(t *testing.T) { + cfg := &config.Config{} + applyCaptureDefaults(cfg) + + if got, want := cfg.Capture.Retention.Std(), defaultCaptureRetention; got != want { + t.Errorf("Capture.Retention = %v, want %v", got, want) + } + if cfg.Capture.MaxEvents != defaultCaptureMaxEvents { + t.Errorf("Capture.MaxEvents = %d, want %d", cfg.Capture.MaxEvents, defaultCaptureMaxEvents) + } + if cfg.Capture.MaxBodyBytes != defaultCaptureMaxBodyBytes { + t.Errorf("Capture.MaxBodyBytes = %d, want %d", cfg.Capture.MaxBodyBytes, defaultCaptureMaxBodyBytes) + } + if cfg.Capture.RatePerSecond != defaultCaptureRatePerSecond { + t.Errorf("Capture.RatePerSecond = %v, want %v", cfg.Capture.RatePerSecond, defaultCaptureRatePerSecond) + } + if cfg.Capture.RateBurst != defaultCaptureRateBurst { + t.Errorf("Capture.RateBurst = %d, want %d", cfg.Capture.RateBurst, defaultCaptureRateBurst) + } +} + +func TestApplyCaptureDefaultsLeavesExplicitValuesAlone(t *testing.T) { + cfg := &config.Config{ + Capture: config.CaptureConfig{ + Retention: config.Duration(24 * time.Hour), + MaxEvents: 42, + MaxBodyBytes: 1024, + RatePerSecond: 2.5, + RateBurst: 9, + }, + } + applyCaptureDefaults(cfg) + + if cfg.Capture.Retention.Std() != 24*time.Hour { + t.Errorf("Capture.Retention = %v, want unchanged 24h", cfg.Capture.Retention.Std()) + } + if cfg.Capture.MaxEvents != 42 { + t.Errorf("Capture.MaxEvents = %d, want unchanged 42", cfg.Capture.MaxEvents) + } + if cfg.Capture.MaxBodyBytes != 1024 { + t.Errorf("Capture.MaxBodyBytes = %d, want unchanged 1024", cfg.Capture.MaxBodyBytes) + } + if cfg.Capture.RatePerSecond != 2.5 { + t.Errorf("Capture.RatePerSecond = %v, want unchanged 2.5", cfg.Capture.RatePerSecond) + } + if cfg.Capture.RateBurst != 9 { + t.Errorf("Capture.RateBurst = %d, want unchanged 9", cfg.Capture.RateBurst) + } +} diff --git a/internal/infrastructure/configuration/validate.go b/internal/infrastructure/configuration/validate.go index 033bd5726..5d688ca03 100644 --- a/internal/infrastructure/configuration/validate.go +++ b/internal/infrastructure/configuration/validate.go @@ -75,7 +75,35 @@ func validate(cfg *config.Config) error { if err := validatePollInterval(cfg); err != nil { return err } - return validateContainers(cfg) + if err := validateContainers(cfg); err != nil { + return err + } + return validateCapture(cfg) +} + +// validateCapture rejects negative capture settings. applyDefaults fills +// every zero value with a positive default, so this only fires on an +// explicit negative value -- which would make time.Duration/AddDate math +// and the token-bucket rate limiter behave nonsensically (e.g. a negative +// retention window would prune every row on every insert, including the one +// just written). +func validateCapture(cfg *config.Config) error { + if cfg.Capture.Retention < 0 { + return fmt.Errorf("%w: capture.retention must not be negative", ErrInvalidInput) + } + if cfg.Capture.MaxEvents < 0 { + return fmt.Errorf("%w: capture.max_events must not be negative", ErrInvalidInput) + } + if cfg.Capture.MaxBodyBytes < 0 { + return fmt.Errorf("%w: capture.max_body_bytes must not be negative", ErrInvalidInput) + } + if cfg.Capture.RatePerSecond < 0 { + return fmt.Errorf("%w: capture.rate_per_second must not be negative", ErrInvalidInput) + } + if cfg.Capture.RateBurst < 0 { + return fmt.Errorf("%w: capture.rate_burst must not be negative", ErrInvalidInput) + } + return nil } // validatePollInterval rejects a non-positive poll interval. diff --git a/internal/infrastructure/configuration/validate_test.go b/internal/infrastructure/configuration/validate_test.go index 66549c5e4..6454a8df1 100644 --- a/internal/infrastructure/configuration/validate_test.go +++ b/internal/infrastructure/configuration/validate_test.go @@ -139,6 +139,36 @@ func TestValidate_RejectsTheSameProblemsAsLoaderLoad(t *testing.T) { }, wantErr: true, }, + { + name: "negative capture.retention", + mutate: func(cfg *config.Config) { cfg.Capture.Retention = config.Duration(-time.Hour) }, + wantErr: true, + }, + { + name: "negative capture.max_events", + mutate: func(cfg *config.Config) { cfg.Capture.MaxEvents = -1 }, + wantErr: true, + }, + { + name: "negative capture.max_body_bytes", + mutate: func(cfg *config.Config) { cfg.Capture.MaxBodyBytes = -1 }, + wantErr: true, + }, + { + name: "negative capture.rate_per_second", + mutate: func(cfg *config.Config) { cfg.Capture.RatePerSecond = -1 }, + wantErr: true, + }, + { + name: "negative capture.rate_burst", + mutate: func(cfg *config.Config) { cfg.Capture.RateBurst = -1 }, + wantErr: true, + }, + { + name: "zero capture fields are valid (defaults fill them)", + mutate: func(cfg *config.Config) {}, + wantErr: false, + }, } for _, tc := range tests { diff --git a/internal/store/captures.go b/internal/store/captures.go new file mode 100644 index 000000000..2bc063e55 --- /dev/null +++ b/internal/store/captures.go @@ -0,0 +1,140 @@ +package store + +import ( + "context" + "errors" + "fmt" + "time" +) + +const capturesSchema = ` +CREATE TABLE IF NOT EXISTS captured_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + received_at TEXT NOT NULL, + source TEXT NOT NULL DEFAULT '', + remote_addr TEXT NOT NULL DEFAULT '', + content_type TEXT NOT NULL DEFAULT '', + headers TEXT NOT NULL DEFAULT '{}', + body TEXT NOT NULL DEFAULT '', + authenticated INTEGER NOT NULL DEFAULT 0 +); +CREATE INDEX IF NOT EXISTS idx_captured_events_source ON captured_events(source, id); +` + +// CapturedEvent is one unbound inbound webhook capture: no workflow binding, +// no forge/task association -- just what arrived, from where, and when. +// See docs/prds/event-capture-storage.md. +type CapturedEvent struct { + ID int64 `json:"id"` + ReceivedAt time.Time `json:"received_at"` + Source string `json:"source"` + RemoteAddr string `json:"remote_addr"` + ContentType string `json:"content_type"` + // Headers and Body are redacted (webhookguard.RedactPayload) before + // they ever reach InsertCapture; the store persists what it is given. + Headers string `json:"headers"` + Body string `json:"body"` + Authenticated bool `json:"authenticated"` +} + +// captureTimeLayout is fixed-width (unlike time.RFC3339Nano, whose +// fractional-seconds field is trimmed of trailing zeros and omitted +// entirely when zero). A variable-width fractional part breaks lexicographic +// string comparison -- "15:04:05Z" (an exact second) sorts AFTER +// "15:04:05.1Z" (0.1s later) because 'Z' > '.', which would make the +// retention prune's `WHERE received_at < ?` unreliable. Second precision is +// enough here: row order already comes from id (autoincrement), and +// retention windows are measured in days, not sub-second intervals. +const captureTimeLayout = time.RFC3339 + +// InsertCapture persists a captured event and prunes the table in the same +// transaction: rows older than retention (when > 0) and rows beyond the +// newest maxEvents (when > 0) are deleted. Either bound of 0 or less +// disables that prune. See docs/prds/event-capture-storage.md's disk-bound +// mechanics -- this is one of three independent bounds, the other two +// (per-request size cap, per-source rate limit) are enforced by the caller +// before the body ever reaches this method. +func (s *Store) InsertCapture(ctx context.Context, c CapturedEvent, retention time.Duration, maxEvents int) (int64, error) { + tx, err := s.db.BeginTx(ctx, nil) + if err != nil { + return 0, err + } + defer func() { _ = tx.Rollback() }() + + if c.ReceivedAt.IsZero() { + c.ReceivedAt = time.Now().UTC() + } + res, err := tx.ExecContext(ctx, ` + INSERT INTO captured_events (received_at, source, remote_addr, content_type, headers, body, authenticated) + VALUES (?, ?, ?, ?, ?, ?, ?)`, + c.ReceivedAt.UTC().Format(captureTimeLayout), c.Source, c.RemoteAddr, + c.ContentType, c.Headers, c.Body, c.Authenticated) + if err != nil { + return 0, err + } + id, err := res.LastInsertId() + if err != nil { + return 0, err + } + + if retention > 0 { + cutoff := time.Now().UTC().Add(-retention).Format(captureTimeLayout) + if _, err := tx.ExecContext(ctx, `DELETE FROM captured_events WHERE received_at < ?`, cutoff); err != nil { + return 0, err + } + } + if maxEvents > 0 { + if _, err := tx.ExecContext(ctx, ` + DELETE FROM captured_events WHERE id NOT IN ( + SELECT id FROM captured_events ORDER BY id DESC LIMIT ? + )`, maxEvents); err != nil { + return 0, err + } + } + + if err := tx.Commit(); err != nil { + return 0, err + } + return id, nil +} + +// ListCaptures returns up to limit captured events, newest first. +func (s *Store) ListCaptures(ctx context.Context, limit int) (out []CapturedEvent, retErr error) { + rows, err := s.db.QueryContext(ctx, ` + SELECT id, received_at, source, remote_addr, content_type, headers, body, authenticated + FROM captured_events ORDER BY id DESC LIMIT ?`, limit) + if err != nil { + return nil, err + } + defer func() { retErr = errors.Join(retErr, rows.Close()) }() + + for rows.Next() { + var c CapturedEvent + var receivedAt string + var authenticated int + if err := rows.Scan(&c.ID, &receivedAt, &c.Source, &c.RemoteAddr, + &c.ContentType, &c.Headers, &c.Body, &authenticated); err != nil { + return nil, err + } + parsed, err := parseCaptureTime(receivedAt) + if err != nil { + return nil, err + } + c.ReceivedAt = parsed + c.Authenticated = authenticated != 0 + out = append(out, c) + } + return out, rows.Err() +} + +func parseCaptureTime(v string) (time.Time, error) { + if v == "" { + return time.Time{}, nil + } + for _, layout := range []string{time.RFC3339Nano, time.RFC3339} { + if parsed, err := time.Parse(layout, v); err == nil { + return parsed.UTC(), nil + } + } + return time.Time{}, fmt.Errorf("store: unrecognised capture timestamp %q", v) +} diff --git a/internal/store/captures_test.go b/internal/store/captures_test.go new file mode 100644 index 000000000..36c9eeda2 --- /dev/null +++ b/internal/store/captures_test.go @@ -0,0 +1,169 @@ +package store + +import ( + "testing" + "time" +) + +func TestInsertCaptureRoundTrips(t *testing.T) { + s := openTest(t) + id, err := s.InsertCapture(t.Context(), CapturedEvent{ + Source: "github", + RemoteAddr: "203.0.113.5", + ContentType: "application/json", + Headers: `{"x-hub-signature-256":"[redacted]"}`, + Body: `{"action":"opened"}`, + Authenticated: false, + }, 7*24*time.Hour, 0) + if err != nil { + t.Fatalf("InsertCapture: %v", err) + } + if id == 0 { + t.Fatalf("InsertCapture returned zero id") + } + + got, err := s.ListCaptures(t.Context(), 10) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + if len(got) != 1 { + t.Fatalf("ListCaptures = %d rows, want 1", len(got)) + } + c := got[0] + if c.ID != id || c.Source != "github" || c.RemoteAddr != "203.0.113.5" || + c.ContentType != "application/json" || c.Body != `{"action":"opened"}` || + c.Authenticated { + t.Fatalf("round-tripped capture = %+v, want matching fields", c) + } + if c.ReceivedAt.IsZero() { + t.Fatalf("ReceivedAt is zero, want a timestamp stamped at insert") + } +} + +func TestListCapturesOrdersNewestFirst(t *testing.T) { + s := openTest(t) + for i := range 3 { + if _, err := s.InsertCapture(t.Context(), CapturedEvent{Source: "src", Body: "{}"}, 0, 0); err != nil { + t.Fatalf("InsertCapture[%d]: %v", i, err) + } + } + got, err := s.ListCaptures(t.Context(), 10) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + if len(got) != 3 { + t.Fatalf("ListCaptures = %d rows, want 3", len(got)) + } + if got[0].ID <= got[1].ID || got[1].ID <= got[2].ID { + t.Fatalf("ListCaptures not newest-first: ids = %d, %d, %d", got[0].ID, got[1].ID, got[2].ID) + } +} + +func TestInsertCapturePrunesBeyondMaxEvents(t *testing.T) { + s := openTest(t) + var lastID int64 + for i := range 5 { + id, err := s.InsertCapture(t.Context(), CapturedEvent{Source: "src", Body: "{}"}, 0, 3) + if err != nil { + t.Fatalf("InsertCapture[%d]: %v", i, err) + } + lastID = id + } + got, err := s.ListCaptures(t.Context(), 100) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + if len(got) != 3 { + t.Fatalf("row count = %d, want 3 (maxEvents cap)", len(got)) + } + if got[0].ID != lastID { + t.Fatalf("newest surviving row id = %d, want %d (the last insert)", got[0].ID, lastID) + } + // The two oldest of the five inserts must have been pruned. + for _, c := range got { + if c.ID <= lastID-3 { + t.Fatalf("found pruned row id %d still present after maxEvents=3 cap", c.ID) + } + } +} + +func TestInsertCapturePrunesOlderThanRetention(t *testing.T) { + s := openTest(t) + // Insert a row and manually age it beyond the retention window, since + // InsertCapture always stamps "now" and the test cannot wait real days. + id, err := s.InsertCapture(t.Context(), CapturedEvent{Source: "old", Body: "{}"}, 0, 0) + if err != nil { + t.Fatalf("InsertCapture: %v", err) + } + stale := time.Now().UTC().Add(-8 * 24 * time.Hour).Format(captureTimeLayout) + if _, err := s.db.ExecContext(t.Context(), + `UPDATE captured_events SET received_at = ? WHERE id = ?`, stale, id); err != nil { + t.Fatalf("backdate row: %v", err) + } + + // A fresh insert with a 7-day retention should prune the now-stale row. + if _, err := s.InsertCapture(t.Context(), CapturedEvent{Source: "new", Body: "{}"}, 7*24*time.Hour, 0); err != nil { + t.Fatalf("InsertCapture: %v", err) + } + + got, err := s.ListCaptures(t.Context(), 100) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + for _, c := range got { + if c.ID == id { + t.Fatalf("stale row id %d survived a 7-day retention prune", id) + } + } + if len(got) != 1 { + t.Fatalf("row count = %d, want 1 (only the fresh insert)", len(got)) + } +} + +// TestCaptureTimeLayoutSortsLexicographicallyLikeChronologically pins a real +// defect that time.RFC3339Nano has: it trims trailing fractional-second +// zeros and omits the fractional part entirely when it is exactly zero, so +// "15:04:05Z" (an exact second) sorts LEXICOGRAPHICALLY AFTER "15:04:05.1Z" +// (0.1s later) because 'Z' (0x5A) > '.' (0x2E) -- backwards from +// chronological order. A retention prune using `WHERE received_at < ?` on a +// column written that way is unreliable within the same second. +// captureTimeLayout must be fixed-width so string order and time order +// always agree. +// +// This compares captureTimeLayout's string output directly rather than +// going through InsertCapture's real prune, because the defect only shows up +// when two timestamps share the same whole-second value and differ only in +// fractional presence -- a case InsertCapture's own cutoff (computed from +// the real wall clock) cannot be steered into deterministically in a test. +func TestCaptureTimeLayoutSortsLexicographicallyLikeChronologically(t *testing.T) { + exactSecond := time.Date(2026, 1, 1, 15, 4, 5, 0, time.UTC) + fractionLater := time.Date(2026, 1, 1, 15, 4, 5, 100_000_000, time.UTC) // same second, 0.1s later + + if !exactSecond.Before(fractionLater) { + t.Fatalf("test setup: %v must be before %v", exactSecond, fractionLater) + } + es, fs := exactSecond.Format(captureTimeLayout), fractionLater.Format(captureTimeLayout) + // The layout is second-precision, so two timestamps within the same + // second are allowed to format identically (that's the accepted + // precision tradeoff, not a bug) -- but a later time must never sort as + // lexicographically LESS than an earlier one. + if es > fs { + t.Fatalf("captureTimeLayout string order disagrees with chronological order: %q should not sort after %q", es, fs) + } +} + +func TestInsertCaptureZeroRetentionAndMaxEventsMeansUnbounded(t *testing.T) { + s := openTest(t) + for i := range 10 { + if _, err := s.InsertCapture(t.Context(), CapturedEvent{Source: "src", Body: "{}"}, 0, 0); err != nil { + t.Fatalf("InsertCapture[%d]: %v", i, err) + } + } + got, err := s.ListCaptures(t.Context(), 100) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + if len(got) != 10 { + t.Fatalf("row count = %d, want 10 (retention=0, maxEvents=0 disables both prunes)", len(got)) + } +} diff --git a/internal/store/interface.go b/internal/store/interface.go index 1149eb045..1624db65d 100644 --- a/internal/store/interface.go +++ b/internal/store/interface.go @@ -5,6 +5,7 @@ package store import ( "context" + "time" "github.com/samcharles93/archie-core/internal/events" ) @@ -74,8 +75,21 @@ type WorkflowStore interface { Transition(ctx context.Context, taskID int64, from, to, detail string) error } +// CaptureStore persists unbound inbound webhook captures -- events with no +// workflow binding and no task association. Deliberately separate from +// TaskStore: a consumer that only needs capture (the intake HTTP handler) +// should not acquire the full task-lifecycle surface, mirroring why +// TaskArchiver is split out above. See docs/prds/event-capture-storage.md. +type CaptureStore interface { + InsertCapture(ctx context.Context, c CapturedEvent, retention time.Duration, maxEvents int) (int64, error) + ListCaptures(ctx context.Context, limit int) ([]CapturedEvent, error) +} + // Compile-time check: *Store satisfies TaskStore. var _ TaskStore = (*Store)(nil) // Compile-time check: *Store satisfies WorkflowStore. var _ WorkflowStore = (*Store)(nil) + +// Compile-time check: *Store satisfies CaptureStore. +var _ CaptureStore = (*Store)(nil) diff --git a/internal/store/store.go b/internal/store/store.go index 5be658036..3372a39c5 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -109,7 +109,7 @@ func Open(ctx context.Context, path string) (*Store, error) { return nil, err } db.SetMaxOpenConns(1) - if _, err := db.ExecContext(ctx, schema+eventsSchema); err != nil { + if _, err := db.ExecContext(ctx, schema+eventsSchema+capturesSchema); err != nil { return nil, errors.Join(fmt.Errorf("store: init schema: %w", err), db.Close()) } if err := migrateTasks(ctx, db); err != nil { diff --git a/internal/webui/api_capture.go b/internal/webui/api_capture.go new file mode 100644 index 000000000..a6c1e975a --- /dev/null +++ b/internal/webui/api_capture.go @@ -0,0 +1,102 @@ +package webui + +import ( + "encoding/json" + "io" + "net" + "net/http" + + "github.com/samcharles93/archie-core/internal/store" + "github.com/samcharles93/archie-core/internal/webhookguard" +) + +// handleCapture accepts an unbound inbound webhook POST and persists it -- +// no HMAC verification (there is by design no secret registered yet for a +// source nobody has configured), no workflow binding, just "something +// arrived." See docs/prds/event-capture-storage.md. Mounted on the bypass +// mux alongside /healthz: capture must accept unauthenticated senders, so it +// cannot sit behind requireToken. +func (s *Server) handleCapture(w http.ResponseWriter, r *http.Request) { + if s.Captures == nil { + http.Error(w, "capture not configured", http.StatusServiceUnavailable) + return + } + source := r.PathValue("source") + + // webhookguard.RateLimiter's bucket map assumes a bounded, operator- + // registered key space (see its doc comment) -- it is never evicted, and + // a fresh key starts with a full burst allowance. source is an + // unregistered, attacker-chosen URL segment by design (this endpoint + // exists to capture senders archie has no registration for yet), so + // keying on it would let a single attacker rotate a new "source" on + // every request to bypass the limit entirely and grow the bucket map + // without bound. Remote address is the bounded identity available here. + if s.CaptureLimiter != nil && !s.CaptureLimiter.Allow(remoteAddrHost(r)) { + http.Error(w, "rate limit exceeded", http.StatusTooManyRequests) + return + } + + r.Body = http.MaxBytesReader(w, r.Body, captureMaxBodyBytesOrFallback(s.CaptureMaxBodyBytes)) + body, err := io.ReadAll(r.Body) + if err != nil { + // The only realistic cause here is MaxBytesReader tripping; any other + // read failure also means there is no usable body to capture. + http.Error(w, "request body too large", http.StatusRequestEntityTooLarge) + return + } + + headers, _ := json.Marshal(r.Header) + redactedHeaders, err := webhookguard.RedactPayload(headers) + if err != nil { + redactedHeaders = headers + } + // A body that isn't JSON has no key-value structure for the heuristic to + // match against, so there is nothing to redact -- store it as received + // rather than dropping or mangling it. Best-effort, per + // docs/prds/webhook-intake-security.md point 5. + redactedBody, err := webhookguard.RedactPayload(body) + if err != nil { + redactedBody = body + } + + _, err = s.Captures.InsertCapture(r.Context(), store.CapturedEvent{ + Source: source, + RemoteAddr: r.RemoteAddr, + ContentType: r.Header.Get("Content-Type"), + Headers: string(redactedHeaders), + Body: string(redactedBody), + }, s.CaptureRetention, s.CaptureMaxEvents) + if err != nil { + s.Log.Error("capture insert", "err", err, "source", source) + http.Error(w, "capture failed", http.StatusInternalServerError) + return + } + w.WriteHeader(http.StatusAccepted) +} + +// fallbackCaptureMaxBodyBytes bounds a capture body when CaptureMaxBodyBytes +// is left unset (zero). The size cap is the only one of the three disk-bound +// mechanics in docs/prds/event-capture-storage.md that limits a SINGLE +// payload rather than total row count or write rate -- a table pruned to +// maxEvents rows still holds one arbitrarily large row if nothing ever +// bounded that row's size, so this must never be skippable by a missing +// wiring value the way the rate limiter (best-effort, not disk-bound-load- +// bearing on its own) is allowed to be. +const fallbackCaptureMaxBodyBytes = 256 * 1024 + +func captureMaxBodyBytesOrFallback(configured int64) int64 { + if configured > 0 { + return configured + } + return fallbackCaptureMaxBodyBytes +} + +// remoteAddrHost strips the port from r.RemoteAddr for use as a rate-limit +// key, falling back to the raw value if it isn't a host:port pair. +func remoteAddrHost(r *http.Request) string { + host, _, err := net.SplitHostPort(r.RemoteAddr) + if err != nil { + return r.RemoteAddr + } + return host +} diff --git a/internal/webui/api_capture_test.go b/internal/webui/api_capture_test.go new file mode 100644 index 000000000..6d449e1db --- /dev/null +++ b/internal/webui/api_capture_test.go @@ -0,0 +1,269 @@ +package webui + +import ( + "context" + "errors" + "log/slog" + "net/http" + "net/http/httptest" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/samcharles93/archie-core/internal/store" + "github.com/samcharles93/archie-core/internal/webhookguard" +) + +// captureTestServer builds a Server whose Captures field points at the same +// concrete *store.Store as Store, so tests can assert on persisted rows +// through the narrower CaptureStore interface -- Server.Store's static type +// (store.TaskStore) does not expose InsertCapture/ListCaptures. +func captureTestServer(t *testing.T) *Server { + t.Helper() + s, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "test.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = s.Close() }) + return &Server{ + Store: s, + Log: slog.New(slog.DiscardHandler), + Captures: s, + CaptureMaxBodyBytes: 1024, + CaptureRetention: 7 * 24 * time.Hour, + CaptureMaxEvents: 100, + CaptureLimiter: webhookguard.NewRateLimiter(1000, 1000, time.Now), + } +} + +func TestHandleCaptureStoresRedactedBody(t *testing.T) { + srv := captureTestServer(t) + body := `{"action":"opened","sender":{"token":"shh-secret"}}` + req := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/webhooks/capture/github", strings.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + + srv.Handler().ServeHTTP(w, req) + + if w.Code != http.StatusAccepted { + t.Fatalf("status = %d, want %d; body = %s", w.Code, http.StatusAccepted, w.Body.String()) + } + got, err := srv.Captures.ListCaptures(t.Context(), 10) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + if len(got) != 1 { + t.Fatalf("captured rows = %d, want 1", len(got)) + } + c := got[0] + if c.Source != "github" { + t.Errorf("Source = %q, want %q", c.Source, "github") + } + if c.ContentType != "application/json" { + t.Errorf("ContentType = %q, want application/json", c.ContentType) + } + if strings.Contains(c.Body, "shh-secret") { + t.Errorf("Body = %q, want the token value redacted", c.Body) + } + if !strings.Contains(c.Body, `"action":"opened"`) { + t.Errorf("Body = %q, want the non-sensitive field preserved", c.Body) + } +} + +func TestHandleCaptureAcceptsUnauthenticatedByDesign(t *testing.T) { + // Capture must accept requests with no HMAC secret configured -- that is + // the entire point of an unbound capture endpoint. This test documents + // that the handler does not gate on any signature header. + srv := captureTestServer(t) + req := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/webhooks/capture/unknown-source", strings.NewReader(`{}`)) + w := httptest.NewRecorder() + + srv.Handler().ServeHTTP(w, req) + + if w.Code != http.StatusAccepted { + t.Fatalf("status = %d, want %d", w.Code, http.StatusAccepted) + } +} + +func TestHandleCaptureNonJSONBodyStoredUnredacted(t *testing.T) { + srv := captureTestServer(t) + req := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/webhooks/capture/plain", strings.NewReader("not json")) + w := httptest.NewRecorder() + + srv.Handler().ServeHTTP(w, req) + + if w.Code != http.StatusAccepted { + t.Fatalf("status = %d, want %d", w.Code, http.StatusAccepted) + } + got, err := srv.Captures.ListCaptures(t.Context(), 10) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + if len(got) != 1 || got[0].Body != "not json" { + t.Fatalf("captured = %+v, want raw body preserved when it is not JSON", got) + } +} + +func TestHandleCaptureRejectsOversizedBody(t *testing.T) { + srv := captureTestServer(t) + srv.CaptureMaxBodyBytes = 8 + req := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/webhooks/capture/github", strings.NewReader(`{"far":"too big for the cap"}`)) + w := httptest.NewRecorder() + + srv.Handler().ServeHTTP(w, req) + + if w.Code != http.StatusRequestEntityTooLarge { + t.Fatalf("status = %d, want %d", w.Code, http.StatusRequestEntityTooLarge) + } + got, err := srv.Captures.ListCaptures(t.Context(), 10) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + if len(got) != 0 { + t.Fatalf("captured rows = %d, want 0 (oversized body must never be persisted)", len(got)) + } +} + +// TestHandleCaptureEnforcesBodyCapEvenWhenUnconfigured pins that a caller +// that forgets to set CaptureMaxBodyBytes (zero value) still gets a bound -- +// this is the one disk-bound mechanic that limits a single payload's size +// rather than row count or write rate, so it must never silently no-op. +func TestHandleCaptureEnforcesBodyCapEvenWhenUnconfigured(t *testing.T) { + srv := captureTestServer(t) + srv.CaptureMaxBodyBytes = 0 + + oversized := strings.Repeat("x", fallbackCaptureMaxBodyBytes+1) + req := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/webhooks/capture/github", strings.NewReader(oversized)) + w := httptest.NewRecorder() + + srv.Handler().ServeHTTP(w, req) + + if w.Code != http.StatusRequestEntityTooLarge { + t.Fatalf("status = %d, want %d (CaptureMaxBodyBytes=0 must still enforce the fallback cap)", w.Code, http.StatusRequestEntityTooLarge) + } + got, err := srv.Captures.ListCaptures(t.Context(), 10) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + if len(got) != 0 { + t.Fatalf("captured rows = %d, want 0", len(got)) + } +} + +func TestHandleCaptureRateLimitsPerRemoteAddr(t *testing.T) { + // Deliberately NOT keyed by the source path segment: source is an + // unregistered, attacker-chosen string for this endpoint (see + // handleCapture and docs/prds/event-capture-storage.md), so both + // requests below use the SAME source and DIFFERENT RemoteAddr to prove + // the limiter is keyed on the sender's address, not the segment. + srv := captureTestServer(t) + srv.CaptureLimiter = webhookguard.NewRateLimiter(0, 1, time.Now) // burst 1, no refill + + req := func(remoteAddr string) *http.Request { + r := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/webhooks/capture/github", strings.NewReader(`{}`)) + r.RemoteAddr = remoteAddr + return r + } + + w1 := httptest.NewRecorder() + srv.Handler().ServeHTTP(w1, req("203.0.113.5:11111")) + if w1.Code != http.StatusAccepted { + t.Fatalf("first request status = %d, want %d", w1.Code, http.StatusAccepted) + } + + w2 := httptest.NewRecorder() + srv.Handler().ServeHTTP(w2, req("203.0.113.5:22222")) + if w2.Code != http.StatusTooManyRequests { + t.Fatalf("second request (same host, different port) status = %d, want %d", w2.Code, http.StatusTooManyRequests) + } + + // A different remote address, same source segment, must not be affected + // by the exhausted budget above. + w3 := httptest.NewRecorder() + srv.Handler().ServeHTTP(w3, req("198.51.100.9:33333")) + if w3.Code != http.StatusAccepted { + t.Fatalf("other-remote-addr request status = %d, want %d", w3.Code, http.StatusAccepted) + } + + got, err := srv.Captures.ListCaptures(t.Context(), 10) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + if len(got) != 2 { + t.Fatalf("captured rows = %d, want 2 (the rate-limited request must not be persisted)", len(got)) + } +} + +func TestHandleCaptureIgnoresGetMethod(t *testing.T) { + // The route is registered "POST /webhooks/capture/{source}"; a GET does + // not match it and falls through to the dashboard's own routing, same as + // every other POST-only API route in this server -- it must not be + // treated as a capture. + srv := captureTestServer(t) + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/webhooks/capture/github", nil) + w := httptest.NewRecorder() + + srv.Handler().ServeHTTP(w, req) + + if w.Code == http.StatusAccepted { + t.Fatalf("status = %d, a GET must never be accepted as a capture", w.Code) + } + got, err := srv.Captures.ListCaptures(t.Context(), 10) + if err != nil { + t.Fatalf("ListCaptures: %v", err) + } + if len(got) != 0 { + t.Fatalf("captured rows = %d, want 0 (GET must not persist a capture)", len(got)) + } +} + +func TestHandleCaptureWithoutCapturesConfiguredIs503(t *testing.T) { + srv := newTestServer(t) + req := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/webhooks/capture/github", strings.NewReader(`{}`)) + w := httptest.NewRecorder() + + srv.Handler().ServeHTTP(w, req) + + if w.Code != http.StatusServiceUnavailable { + t.Fatalf("status = %d, want %d when Captures is nil", w.Code, http.StatusServiceUnavailable) + } +} + +func TestHandleCaptureBypassesTokenAuth(t *testing.T) { + // The capture endpoint must accept unauthenticated senders even when the + // dashboard itself is token-gated -- mirrors /healthz's precedent. + srv := captureTestServer(t) + srv.Token = "dashboard-secret" + req := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/webhooks/capture/github", strings.NewReader(`{}`)) + w := httptest.NewRecorder() + + srv.Handler().ServeHTTP(w, req) + + if w.Code != http.StatusAccepted { + t.Fatalf("status = %d, want %d (capture must bypass requireToken)", w.Code, http.StatusAccepted) + } +} + +// captureInsertErrorStore forces InsertCapture to fail so the handler's +// error path is exercised without relying on a real storage failure. +type captureInsertErrorStore struct { + store.CaptureStore +} + +func (captureInsertErrorStore) InsertCapture(context.Context, store.CapturedEvent, time.Duration, int) (int64, error) { + return 0, errors.New("boom") +} + +func TestHandleCaptureStorageFailureIs500(t *testing.T) { + srv := captureTestServer(t) + srv.Captures = captureInsertErrorStore{} + req := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/webhooks/capture/github", strings.NewReader(`{}`)) + w := httptest.NewRecorder() + + srv.Handler().ServeHTTP(w, req) + + if w.Code != http.StatusInternalServerError { + t.Fatalf("status = %d, want %d", w.Code, http.StatusInternalServerError) + } +} diff --git a/internal/webui/server.go b/internal/webui/server.go index f3fae9bb5..a5f1641b4 100644 --- a/internal/webui/server.go +++ b/internal/webui/server.go @@ -25,6 +25,7 @@ import ( "github.com/samcharles93/archie-core/internal/logging" "github.com/samcharles93/archie-core/internal/memory" "github.com/samcharles93/archie-core/internal/store" + "github.com/samcharles93/archie-core/internal/webhookguard" "github.com/samcharles93/archie-core/ui" ) @@ -128,6 +129,26 @@ type Server struct { // seam so the Stop action cancels work before parking its task record. TaskStopper TaskStopper + // Captures persists unbound webhook captures (docs/prds/event-capture-storage.md). + // Optional: nil makes POST /webhooks/capture/{source} answer 503 rather + // than the dashboard failing to start -- capture is a precondition for + // the no-code playbook epic, not a hard dependency of the dashboard. + Captures store.CaptureStore + // CaptureLimiter is the per-remote-address token bucket applied before a + // capture request's body is read (keyed by remote address, not the + // unregistered/attacker-chosen source path segment -- see + // handleCapture). Optional: nil disables rate limiting (every request is + // allowed), which composition never actually does in production -- it + // is nil only in tests that don't exercise this path. + CaptureLimiter *webhookguard.RateLimiter + // CaptureRetention and CaptureMaxEvents are passed straight through to + // InsertCapture's prune-on-write bounds. See config.CaptureConfig. + CaptureRetention time.Duration + CaptureMaxEvents int + // CaptureMaxBodyBytes rejects (413) a capture POST body larger than + // this via http.MaxBytesReader, before redaction or storage sees it. + CaptureMaxBodyBytes int64 + mu sync.Mutex conns map[chan events.Event]struct{} } @@ -211,6 +232,7 @@ func (s *Server) Handler() http.Handler { top := http.NewServeMux() top.HandleFunc("GET /healthz", s.handleHealthz) + top.HandleFunc("POST /webhooks/capture/{source}", s.handleCapture) top.Handle("/", s.requireToken(mux)) return top }