package servicekit import ( "context" "log/slog" "time" ) // controlPlaneWatchRetry is how long a watch waits before re-establishing a // stream that ended. The minimum is the delay a healthy attempt leaves behind, // and the delay a failed attempt doubles; the cap bounds the delay, never the // retrying: a watch reconnects for as long as the process lives. // // A control plane that is down is retried slowly enough that it is not a // request loop, and the charge reaches the shapes of outage an attempt that // ENDS can take: a store that errors at once is charged, and so is one that // stalls for less than the window the attempt is judged against, because that // window is derived from the delay the attempt would otherwise be charged and // widens with the ladder instead of sitting at its floor (attemptHealthy). // // This loop charges only attempts that end, and imposes no timeout of its own: // nextUpdate waits on the stream and on the loop's own context, never on a // clock, so a peer that accepts the stream and then never answers it ends // nothing here and this code has nothing to charge. What ends that stream is // the transport's client keepalive, // internal/infrastructure/staterpc/dial.go (Time 10s, Timeout 5s): the // unacknowledged PING tears the connection down, the watch stream errors, the // attempt ends, and here it is charged like any other attempt that ended. A // hung peer is that keepalive's business, not this loop's. The watch rides // that connection because the daemon takes its control-plane client off the // *staterpc.Client the dial returned, on the same connection the keepalive was // installed on (internal/app/archied/bootstrap.go, openStateStoreAdapter, // which reads client.ControlPlane()). const ( controlPlaneWatchRetryMin = 250 * time.Millisecond controlPlaneWatchRetryMax = 30 * time.Second ) // controlPlaneWatchHealthyMultiple multiplies the delay an attempt would // otherwise be charged into the window that attempt must stay open to be read // as a store that answered and had nothing to say. It is strictly more than // one, so that window is always wider than the charge itself: a window equal to // the delay would read a store that fails as slowly as that delay as healthy on // every attempt, which is the fixed-rate loop this rule exists to charge. const controlPlaneWatchHealthyMultiple = 2 // KeepWatch delivers every update from first and, when that stream ends, // re-establishes it in this same goroutine until ctx ends. // // A control-plane watch stream is not permanent. The State Store restarts, the // connection drops, or the server closes the stream. KeepWatch reopens it in // place; workflow-execution-settings also reports the gap so apply status does // not keep re-stamping the previous successful version as current while no // live updates can arrive. // // One goroutine per kind, reconnecting in place: a long outage does not stack // watchers. Reopening resumes after the last version an update carried, so a // reconnect neither replays a document this process has already handled over a // newer one nor skips one it has not. A transport failure carries no version // and so leaves the resume point alone. A document the client refused to decode // does carry one -- controlplane.Client decodes what the store sends, and the // store answered with that version -- and the resume point moves past it // deliberately: that version is unusable to this build, and resuming before it // would re-offer it on every reconnect for the life of the process. Only a // cancellation stops the loop, and the loop watches for it itself, so shutdown // is prompt however long the current backoff is and whether or not the stream // is still open. // // The reconnect delay is decided by the attempt that just ended, and only by // that (attemptHealthy, retryDelay): a healthy attempt returns it to the // minimum, and a failed attempt doubles it up to the cap. An attempt is not // healthy because a reopen was accepted. A control plane accepts the stream // before it reads anything -- that is how gRPC hands a request to its handler // -- and controlplane.Server.Watch answers a store it cannot read by returning // an error, so during the outage this loop exists for every reopen is accepted // and every stream ends carrying nothing: judged on the acceptance alone the // delay stays at the minimum and the watch becomes a 4 RPCs/second request // loop. // // The rule is one sentence: an attempt is healthy iff it delivered an update // this watch had not handled, or it was still open when the window closed // (attemptHealthy, controlPlaneWatchHealthyWindow -- the delay the attempt would // otherwise be charged, multiplied by a factor strictly greater than one, so // the window is always wider than that charge). The window is derived from that // charge rather than fixed at the minimum, which is what carries the rule past // the shape an immediate error does not cover: a store that has stopped // answering does not error at once, it stalls, and an attempt that stalls for // less than the window is charged and the delay climbs a rung. // // The window is not a read timeout, and this loop imposes none of its own (see // controlPlaneWatchRetry for what ends an attempt against a peer that never // answers), so the reach of the rule stops at the window itself: a stall still // going when the window closes is read as a store that answered, and the delay // stays at the minimum for as long as every attempt takes that long. That is // deliberate rather than a gap to close with another rule. The stalled attempt // lasts the window, so the reopens it leaves behind are bounded by the attempt // and not by the delay -- about 0.8 a second at the minimum rung, against the // four a second this rule exists to charge -- and a shape that slow is not the // request loop it exists for. // // open, versionOf and deliver are the kind's halves. open re-establishes the // stream -- the caller opens the first one itself, so that a control plane // which cannot be watched at all still fails the boot that asked for it. // versionOf reads the resource version an update carries. deliver applies the // update, including reporting the error it carries, which is how a refused // document reaches the settings page. // // wait is the delay between a stream ending and the reopen that follows it, and // production passes WaitFor. It is injected so that the delay the loop hands it // is observable as a value: the retry_in log line states the delay the loop // charged, and a loop that charged one delay and waited another would leave // that attribute agreeing with itself. What the injected wait cannot show is a // multiple taken inside WaitFor itself, which is a timer for exactly the d it // is given. func KeepWatch[T any]( ctx context.Context, log *slog.Logger, kind string, after int64, first <-chan T, open func(ctx context.Context, afterVersion int64) (<-chan T, error), wait func(ctx context.Context, d time.Duration) bool, versionOf func(update T) int64, deliver func(update T), onEnded ...func(), ) { stream, backoff := first, controlPlaneWatchRetryMin for { started, progressed := time.Now(), false for { update, ok := nextUpdate(ctx, stream) if !ok { break } deliver(update) if version := versionOf(update); version > after { after, progressed = version, true } } if ctx.Err() != nil { return } if len(onEnded) > 0 { onEnded[0]() } backoff = retryDelay(backoff, attemptHealthy(progressed, time.Since(started), backoff)) log.WarnContext(ctx, "control plane watch stream ended; reconnecting", "kind", kind, "after_version", after, "retry_in", backoff) for { if !wait(ctx, backoff) { return } reopened, err := open(ctx, after) if err == nil { stream = reopened break } // A reopen the control plane refused is the store still not // answering, and is charged exactly as any other attempt that ends // that way. backoff = retryDelay(backoff, false) log.WarnContext(ctx, "control plane watch could not be re-established", "kind", kind, "after_version", after, "retry_in", backoff, "err", err) } } } // attemptHealthy is what one attempt that ended makes of the reconnect delay. // An attempt is healthy iff it delivered an update this watch had not handled // -- the store answered, whatever the stream did afterwards -- or it was still // open when controlPlaneWatchHealthyWindow closed, at that boundary and not // only past it, which is a store that had nothing to say and was still there. // An attempt that did neither delivered nothing and died inside the window it // would have been charged: the stream was accepted and the store was not // answering, which is a failed attempt and the only thing the delay grows on. func attemptHealthy(progressed bool, openFor, backoff time.Duration) bool { return progressed || openFor >= controlPlaneWatchHealthyWindow(backoff) } // controlPlaneWatchHealthyWindow is how long an attempt that delivered nothing // must stay open to count as a store that answered and had nothing to say: the // delay the attempt would otherwise be charged, multiplied by // controlPlaneWatchHealthyMultiple. The window moves with the ladder, so the // charge reaches a store that fails slowly at every rung of it, which a window // fixed at the minimum does not. func controlPlaneWatchHealthyWindow(backoff time.Duration) time.Duration { return controlPlaneWatchHealthyMultiple * retryDelay(backoff, false) } // retryDelay is the delay the attempt that just ended leaves behind: a healthy // attempt returns it to the minimum, so an outage is charged for itself and not // for the quiet time that preceded it, and a failed attempt doubles it up to // the cap, so a store that stays gone is retried ever more slowly. func retryDelay(current time.Duration, healthy bool) time.Duration { if healthy { return controlPlaneWatchRetryMin } return min(2*current, controlPlaneWatchRetryMax) } // nextUpdate receives the next update from stream, reporting false when the // stream ended or when ctx ended first. A watch's stream is closed by // controlplane.Client when the stream's own context ends, but shutdown must not // rest on that cross-package coupling: a stream that never closed would // otherwise hold the watch -- and the process that owns it -- past the // cancellation that asked it to stop. The coupling that is load-bearing runs // the other way: this watch returns on cancellation without draining what the // producer has left for it, so controlplane.Client's producer must stop on the // same context instead of parking on a send no reader will ever take. func nextUpdate[T any](ctx context.Context, stream <-chan T) (T, bool) { select { case <-ctx.Done(): var zero T return zero, false case update, ok := <-stream: return update, ok } } // WaitFor is the wait production injects into KeepWatch. It waits for d, // reporting false when ctx ended first: a watch's backoff must never delay the // shutdown of the process that owns it. func WaitFor(ctx context.Context, d time.Duration) bool { timer := time.NewTimer(d) defer timer.Stop() select { case <-ctx.Done(): return false case <-timer.C: return true } }