Operator notifications: persistent dashboard alerts for conditions that need a human #1493

Closed
opened 2026-10-27 13:40:37 +00:00 by sam · 0 comments
Owner

Nothing tells the operator that something needs a human. The only outbound signal is notify.webhook, a fire-and-forget POST with no record, no retry and no way to see what was sent. Parked tasks, a missing default model alias, a provider with no credential binding or an unhealthy service are found only by browsing to the right page.

Wanted: durable alerts that the State Store owns, that services raise and clear through the state contract, and that the dashboard and outbound deliveries read.

Alert model:

  • Fields: org, kind, subject (what it is about: a task, provider, service or setting), severity (info, warning, action), title, body, link to the page or setting that fixes it, first raised, last seen, occurrence count, state (open, resolved, dismissed), resolved or dismissed by and when.
  • Dedup key is org + kind + subject. Raising an open alert again bumps last seen and count; it never creates a second row.
  • Raise and resolve are idempotent and safe to call from a periodic check.
  • Condition-backed alerts resolve themselves when the condition clears. Event alerts (a task failed) stay until dismissed.
  • Dismissing a condition-backed alert snoozes it; it comes back only if the condition clears and recurs.
  • Alerts are scoped to an org and visible only to members with access to it. Body text never contains secrets or credential values.
  • Resolved and dismissed alerts are kept for a retention window, then pruned.

Raising: a small Raise/Resolve call in the state contract. Services reach it only through that contract. A raiser that cannot reach the State Store logs and retries on its next check; it never blocks or fails its own work.

First conditions, each raised where it is detected and resolved when fixed:

  • A task parks waiting for a human (approval, review, feasibility hand-off): action, links to the task.
  • A task or workflow run fails after its retries: warning, links to the run; dismiss-only.
  • No default model alias is set, or an alias points at a model that no longer exists: action, links to model settings.
  • A provider or tool is configured with no credential binding, or its credential is rejected: action, links to credential bindings.
  • A service reports unhealthy for longer than a grace period: warning, links to Status.
  • A newer release is available: info, links to Status.

Delivery:

  • Dashboard: a bell in the top bar with the open count for the operator's org, a list grouped by severity with dismiss, and a link per alert. Updates live over the existing event stream.
  • Outbound: notify.webhook becomes a delivery of alerts (raised and resolved), with a severity filter, signed payloads and bounded retry, and each attempt recorded on the alert. Chat, email and push deliveries are extensions in archipelago, not in-tree.

Out of scope: per-user preferences and digests.

Acceptance criteria

  1. A parked task and a missing default model alias each appear once as an open alert, however often their checks run, and resolve on their own when fixed.
  2. A failed run appears and stays until dismissed.
  3. Alerts from one org are invisible to members of another.
  4. The dashboard bell shows the open count and updates without a reload.
  5. notify.webhook receives raised and resolved alerts above its severity threshold, and failed deliveries are retried and recorded.
  6. A table-driven test with fakes covers dedup, auto-resolve, dismiss-then-recur and org scoping. No test opens a real database.

Notes

Implemented on feat/operator-alerts (PR #1635). Open until checked on a running stack: migration applied, a parked task and a missing alias raise and resolve, webhook delivery recorded.

Close reason

Delivered by operator alerts: parked/needs-human tasks (postgres/task_alerts.go), missing or retired model aliases, providers without credentials, service.unhealthy. Per-org gaps tracked in f167i.


Imported from beads archie-core-om7vj · GitHub #1531 · type feature · priority P2 · created 2026-10-10 · closed 2026-10-12

Nothing tells the operator that something needs a human. The only outbound signal is notify.webhook, a fire-and-forget POST with no record, no retry and no way to see what was sent. Parked tasks, a missing default model alias, a provider with no credential binding or an unhealthy service are found only by browsing to the right page. Wanted: durable alerts that the State Store owns, that services raise and clear through the state contract, and that the dashboard and outbound deliveries read. Alert model: - Fields: org, kind, subject (what it is about: a task, provider, service or setting), severity (info, warning, action), title, body, link to the page or setting that fixes it, first raised, last seen, occurrence count, state (open, resolved, dismissed), resolved or dismissed by and when. - Dedup key is org + kind + subject. Raising an open alert again bumps last seen and count; it never creates a second row. - Raise and resolve are idempotent and safe to call from a periodic check. - Condition-backed alerts resolve themselves when the condition clears. Event alerts (a task failed) stay until dismissed. - Dismissing a condition-backed alert snoozes it; it comes back only if the condition clears and recurs. - Alerts are scoped to an org and visible only to members with access to it. Body text never contains secrets or credential values. - Resolved and dismissed alerts are kept for a retention window, then pruned. Raising: a small Raise/Resolve call in the state contract. Services reach it only through that contract. A raiser that cannot reach the State Store logs and retries on its next check; it never blocks or fails its own work. First conditions, each raised where it is detected and resolved when fixed: - A task parks waiting for a human (approval, review, feasibility hand-off): action, links to the task. - A task or workflow run fails after its retries: warning, links to the run; dismiss-only. - No default model alias is set, or an alias points at a model that no longer exists: action, links to model settings. - A provider or tool is configured with no credential binding, or its credential is rejected: action, links to credential bindings. - A service reports unhealthy for longer than a grace period: warning, links to Status. - A newer release is available: info, links to Status. Delivery: - Dashboard: a bell in the top bar with the open count for the operator's org, a list grouped by severity with dismiss, and a link per alert. Updates live over the existing event stream. - Outbound: notify.webhook becomes a delivery of alerts (raised and resolved), with a severity filter, signed payloads and bounded retry, and each attempt recorded on the alert. Chat, email and push deliveries are extensions in archipelago, not in-tree. Out of scope: per-user preferences and digests. ### Acceptance criteria 1. A parked task and a missing default model alias each appear once as an open alert, however often their checks run, and resolve on their own when fixed. 2. A failed run appears and stays until dismissed. 3. Alerts from one org are invisible to members of another. 4. The dashboard bell shows the open count and updates without a reload. 5. notify.webhook receives raised and resolved alerts above its severity threshold, and failed deliveries are retried and recorded. 6. A table-driven test with fakes covers dedup, auto-resolve, dismiss-then-recur and org scoping. No test opens a real database. ### Notes Implemented on feat/operator-alerts (PR #1635). Open until checked on a running stack: migration applied, a parked task and a missing alias raise and resolve, webhook delivery recorded. ### Close reason Delivered by operator alerts: parked/needs-human tasks (postgres/task_alerts.go), missing or retired model aliases, providers without credentials, service.unhealthy. Per-org gaps tracked in f167i. --- _Imported from beads `archie-core-om7vj` · GitHub #1531 · type feature · priority P2 · created 2026-10-10 · closed 2026-10-12_
sam added the mediumfeature labels 2026-10-27 13:40:37 +00:00
sam self-assigned this 2026-10-27 13:40:37 +00:00
sam closed this issue 2026-10-27 13:40:37 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1493