* feat(state): durable operator alerts owned by the State Store Alerts dedup on org+kind+subject, resolve themselves when a condition clears, and are raised, resolved, dismissed and listed through the state contract. Task transitions raise needs-human and failed alerts in the same transaction; the store checks service health and prunes closed alerts. Each raise and resolve queues a webhook delivery whose attempts are kept. * feat(notify): deliver operator alerts to the webhook; raise config and release alerts notify.webhook now receives raised and resolved alerts above a severity threshold with HMAC-signed payloads, bounded retry and every attempt recorded on the alert. archied raises model alias, provider and credential-binding alerts from the effective configuration, an update alert from the release channel, and a credential-rejected alert from the egress resolver. The dashboard API lists and dismisses the caller's org's alerts. * feat(ui): alert bell in the top bar and webhook alert settings The bell lists the org's open alerts by severity with dismiss and a link per alert, and refreshes from the event stream. The task settings page gains the webhook severity threshold and signing secret. ui/dist rebuilt.
86 lines
2.3 KiB
Go
86 lines
2.3 KiB
Go
package alert
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
)
|
|
|
|
// Delivery states.
|
|
const (
|
|
DeliveryPending = "pending"
|
|
DeliveryDelivered = "delivered"
|
|
DeliveryFailed = "failed"
|
|
// DeliverySkipped is a notice nothing was configured to send, or whose
|
|
// severity was below the threshold.
|
|
DeliverySkipped = "skipped"
|
|
)
|
|
|
|
// What a delivery announces.
|
|
const (
|
|
EventRaised = "raised"
|
|
EventResolved = "resolved"
|
|
)
|
|
|
|
// MaxDeliveryAttempts bounds the retries of one notice. After the last the
|
|
// delivery is failed and kept on the alert.
|
|
const MaxDeliveryAttempts = 5
|
|
|
|
// RetryDelay is how long to wait after the attempt-th failed attempt.
|
|
func RetryDelay(attempt int64) time.Duration {
|
|
delays := [...]time.Duration{time.Minute, 5 * time.Minute, 15 * time.Minute, time.Hour}
|
|
return delays[min(max(attempt, 1), int64(len(delays)))-1]
|
|
}
|
|
|
|
// Delivery is one notice an alert owes: raised or resolved.
|
|
type Delivery struct {
|
|
ID int64
|
|
AlertID int64
|
|
Event string
|
|
State string
|
|
CreatedAt time.Time
|
|
// NextAttemptAt is when a pending delivery is next due.
|
|
NextAttemptAt time.Time
|
|
Attempts []Attempt
|
|
}
|
|
|
|
// Attempt is one try at a delivery.
|
|
type Attempt struct {
|
|
N int64
|
|
At time.Time
|
|
StatusCode int
|
|
Error string
|
|
}
|
|
|
|
// Due is a delivery with the alert it announces.
|
|
type Due struct {
|
|
Delivery
|
|
Alert Alert
|
|
// Attempt is the number of this try, counting from one.
|
|
Attempt int64
|
|
}
|
|
|
|
// Outcome is how one attempt went.
|
|
type Outcome struct {
|
|
DeliveryID int64
|
|
// Skipped closes the delivery without an attempt.
|
|
Skipped bool
|
|
StatusCode int
|
|
// Error is empty on success. It is stored on the alert and shown to the
|
|
// operator, so it names the failure and never the request.
|
|
Error string
|
|
}
|
|
|
|
// OK reports a successful attempt.
|
|
func (o Outcome) OK() bool { return !o.Skipped && o.Error == "" }
|
|
|
|
// Deliverer is the outbound half of the store: the due notices, and what
|
|
// became of each.
|
|
type Deliverer interface {
|
|
// ClaimDeliveries returns up to limit pending deliveries that are due and
|
|
// pushes each one's next attempt out, so a worker that dies mid-attempt
|
|
// does not strand it and two workers do not send it twice.
|
|
ClaimAlertDeliveries(ctx context.Context, limit int) ([]Due, error)
|
|
// RecordAttempt records an attempt and settles or reschedules the delivery.
|
|
RecordAlertAttempt(ctx context.Context, outcome Outcome) error
|
|
}
|