Audit retention: let the instance set per-event-kind retention #1856

Closed
opened 2026-11-10 15:26:23 +00:00 by sam · 2 comments
Owner

The audit-retention setting is a single window applied to a code-fixed allowlist of observability event kinds. The biggest kind by far — agent_transcript, every agent conversation and the majority of the events table — is outside that allowlist, so it is never pruned and the events table grows without bound whatever retention the instance sets.

Wanted: the instance's audit-retention document gains per-kind retention overrides (for example keep transcripts a week while everything else stays at the default), editable from Audit → Retention in the dashboard. Security-relevant event kinds stay impossible to prune: the write path validates against the same code-owned policy the sweep uses. An omitted kind inherits the default window; 0 keeps a kind forever.

Done when:

  • audit-retention accepts event_retention: {"<kind>": "<hours>h"} alongside runtime_event_retention; a kind that cannot be pruned, a non-duration or a window under a week is refused where it is written.
  • The sweep applies each kind's own cutoff in one bounded, locked transaction and records the rules it applied.
  • The Audit → Retention dialog edits the default and per-kind values, and the kind list comes from the server rather than a second hand-maintained list.
  • The security-boundary test covers agent transcripts joining the prunable set and every protected kind staying unprunable.
  • A release note is in place and the gate passes.
The audit-retention setting is a single window applied to a code-fixed allowlist of observability event kinds. The biggest kind by far — `agent_transcript`, every agent conversation and the majority of the events table — is outside that allowlist, so it is never pruned and the events table grows without bound whatever retention the instance sets. Wanted: the instance's audit-retention document gains per-kind retention overrides (for example keep transcripts a week while everything else stays at the default), editable from Audit → Retention in the dashboard. Security-relevant event kinds stay impossible to prune: the write path validates against the same code-owned policy the sweep uses. An omitted kind inherits the default window; `0` keeps a kind forever. Done when: - `audit-retention` accepts `event_retention: {"<kind>": "<hours>h"}` alongside `runtime_event_retention`; a kind that cannot be pruned, a non-duration or a window under a week is refused where it is written. - The sweep applies each kind's own cutoff in one bounded, locked transaction and records the rules it applied. - The Audit → Retention dialog edits the default and per-kind values, and the kind list comes from the server rather than a second hand-maintained list. - The security-boundary test covers agent transcripts joining the prunable set and every protected kind staying unprunable. - A release note is in place and the gate passes.
Author
Owner

Implemented in dcc12582c (local main, not deployed).

The audit-retention document now takes event_retention: {"<kind>": "<hours>h"} next to runtime_event_retention. An omitted kind inherits the default and 0 keeps that kind forever. agent_transcript joined the prunable kinds, so a transcript window is what keeps old agent conversations out of the events table. Security kinds are refused where the document is written, against the same policy the sweep uses (the policy now lives in the audit domain and is exposed through the audit facets). The sweep cuts each kind at its own cutoff in one bounded, locked transaction and records the rules it applied in its audit row.

Audit → Retention in the dashboard has a default field and one row per prunable kind; the kind list comes from a new retention_kinds field on the audit facets, so the UI cannot drift from the policy.

Verified on the commit: go fix/gofumpt, vet, golangci-lint, full go test -short ./..., dashboard build, typecheck and unit tests, and the protobuf/sqlc/news checks. task check itself could not run its pnpm install step in this sandbox, so its remaining steps were run individually. New tests cover the write-path refusals, normalization and the per-kind cutoffs.

Not deployed, so nothing prunes on the instance yet; once it is, the default window applies to transcripts too. Leave open until it runs there.

Implemented in dcc12582c (local main, not deployed). The audit-retention document now takes `event_retention: {"<kind>": "<hours>h"}` next to `runtime_event_retention`. An omitted kind inherits the default and `0` keeps that kind forever. `agent_transcript` joined the prunable kinds, so a transcript window is what keeps old agent conversations out of the events table. Security kinds are refused where the document is written, against the same policy the sweep uses (the policy now lives in the audit domain and is exposed through the audit facets). The sweep cuts each kind at its own cutoff in one bounded, locked transaction and records the rules it applied in its audit row. Audit → Retention in the dashboard has a default field and one row per prunable kind; the kind list comes from a new `retention_kinds` field on the audit facets, so the UI cannot drift from the policy. Verified on the commit: go fix/gofumpt, vet, golangci-lint, full `go test -short ./...`, dashboard build, typecheck and unit tests, and the protobuf/sqlc/news checks. `task check` itself could not run its pnpm install step in this sandbox, so its remaining steps were run individually. New tests cover the write-path refusals, normalization and the per-kind cutoffs. Not deployed, so nothing prunes on the instance yet; once it is, the default window applies to transcripts too. Leave open until it runs there.
Author
Owner

Closing as complete: dcc12582c on main. Not deployed yet, so nothing prunes per kind on the instance until it is.

Closing as complete: dcc12582c on main. Not deployed yet, so nothing prunes per kind on the instance until it is.
sam closed this issue 2026-11-10 15:45:50 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1856