Files
archie/internal/domain/alert/delivery.go
T
sam 747a7cbfd7 Feat/operator alerts (#1635)
* feat(state): durable operator alerts owned by the State Store

Alerts dedup on org+kind+subject, resolve themselves when a condition
clears, and are raised, resolved, dismissed and listed through the state
contract. Task transitions raise needs-human and failed alerts in the same
transaction; the store checks service health and prunes closed alerts.
Each raise and resolve queues a webhook delivery whose attempts are kept.

* feat(notify): deliver operator alerts to the webhook; raise config and release alerts

notify.webhook now receives raised and resolved alerts above a severity
threshold with HMAC-signed payloads, bounded retry and every attempt recorded
on the alert. archied raises model alias, provider and credential-binding
alerts from the effective configuration, an update alert from the release
channel, and a credential-rejected alert from the egress resolver. The
dashboard API lists and dismisses the caller's org's alerts.

* feat(ui): alert bell in the top bar and webhook alert settings

The bell lists the org's open alerts by severity with dismiss and a link per
alert, and refreshes from the event stream. The task settings page gains the
webhook severity threshold and signing secret. ui/dist rebuilt.
2026-10-12 03:30:46 +11:00

86 lines
2.3 KiB
Go

package alert
import (
"context"
"time"
)
// Delivery states.
const (
DeliveryPending = "pending"
DeliveryDelivered = "delivered"
DeliveryFailed = "failed"
// DeliverySkipped is a notice nothing was configured to send, or whose
// severity was below the threshold.
DeliverySkipped = "skipped"
)
// What a delivery announces.
const (
EventRaised = "raised"
EventResolved = "resolved"
)
// MaxDeliveryAttempts bounds the retries of one notice. After the last the
// delivery is failed and kept on the alert.
const MaxDeliveryAttempts = 5
// RetryDelay is how long to wait after the attempt-th failed attempt.
func RetryDelay(attempt int64) time.Duration {
delays := [...]time.Duration{time.Minute, 5 * time.Minute, 15 * time.Minute, time.Hour}
return delays[min(max(attempt, 1), int64(len(delays)))-1]
}
// Delivery is one notice an alert owes: raised or resolved.
type Delivery struct {
ID int64
AlertID int64
Event string
State string
CreatedAt time.Time
// NextAttemptAt is when a pending delivery is next due.
NextAttemptAt time.Time
Attempts []Attempt
}
// Attempt is one try at a delivery.
type Attempt struct {
N int64
At time.Time
StatusCode int
Error string
}
// Due is a delivery with the alert it announces.
type Due struct {
Delivery
Alert Alert
// Attempt is the number of this try, counting from one.
Attempt int64
}
// Outcome is how one attempt went.
type Outcome struct {
DeliveryID int64
// Skipped closes the delivery without an attempt.
Skipped bool
StatusCode int
// Error is empty on success. It is stored on the alert and shown to the
// operator, so it names the failure and never the request.
Error string
}
// OK reports a successful attempt.
func (o Outcome) OK() bool { return !o.Skipped && o.Error == "" }
// Deliverer is the outbound half of the store: the due notices, and what
// became of each.
type Deliverer interface {
// ClaimDeliveries returns up to limit pending deliveries that are due and
// pushes each one's next attempt out, so a worker that dies mid-attempt
// does not strand it and two workers do not send it twice.
ClaimAlertDeliveries(ctx context.Context, limit int) ([]Due, error)
// RecordAttempt records an attempt and settles or reschedules the delivery.
RecordAlertAttempt(ctx context.Context, outcome Outcome) error
}