From b8ec491dd312de395637aba612a57024ae6e8de3 Mon Sep 17 00:00:00 2001 From: Sam Catlow Date: Mon, 21 Sep 2026 23:54:11 +1000 Subject: [PATCH] fix(archied): re-establish the control-plane watch streams after they end The settings and persona watches were launched once, at boot, and ended on the first stream failure: the goroutine returned, so every later stored change for that kind was silently never applied and the process recovered only by restarting (archie-core-yrmr). Both watches now run one keepWatch goroutine per kind, which re-establishes the stream in place after a bounded backoff, resuming after the last version an update carried and stopping on ctx cancellation -- which the loop watches for itself, so a stream that never closes cannot hold up shutdown. A first open that fails still fails the boot that asked for it. The reconnect delay is decided by the attempt that just ended, and only by that (attemptHealthy, retryDelay): an attempt is healthy iff it delivered an update this watch had not handled, or it outlived a window that is a strict multiple of the delay it would otherwise be charged. A healthy attempt returns the delay to the minimum, a failed one doubles it up to the cap, and the window moves with the ladder instead of sitting at its floor. The acceptance of a reopen is the wrong unit: gRPC takes the stream before the handler reads anything, and controlplane.Server.Watch answers a store it cannot read by ending the stream, so during the outage this reconnect exists for every reopen is accepted and every stream ends carrying nothing -- measured with the real server and client over bufconn, 4.0 watch RPCs a second across 2s of store outage with retry_in pinned at 250ms when that acceptance is read as health. A window fixed at the minimum is the same defect for the likelier shape of outage: a store that has stopped answering does not error at once, it stalls until its read times out, and an attempt that takes longer to fail than the minimum outlives that window on every attempt -- measured in the end-to-end outage test before the change, a 300ms stall left the ladder flat at 250ms/250ms/250ms/250ms while the store never answered a read. Both shapes are charged now, and the outage test drives three of them (an immediate error, a 300ms stall, and a stall as long as the delay that rung would be charged) up four rungs of the ladder, each logged retry held to the delay actually waited. The wait is injected (wait, production passes waitFor), because the ladder was only ever asserted through the retry_in attribute: a loop that logged one delay and waited another would have reported a growing ladder and reopened on a schedule of its own. TestKeepWatchWaitsTheDelayItCharged reads the delays back as values and asserts the ladder, and the outage test's gaps catch a wait that returns without waiting. The retargeted reset test is joined by TestKeepWatchChargesAReopenTheStoreMerelyAccepted: a delay reset by the acceptance of a reopen charges every attempt of the outage from the minimum, and nothing pinned that before. The watch returns on cancellation without draining what the producer left for it, so controlplane.Client's producers stop on the same context instead of parking on a send no reader will ever take -- reproduced 40/40 times as a goroutine still blocked on an undelivered send after keepWatch returned, one goroutine and one channel leaked per kind at every exit. That cross-package coupling is load-bearing in both directions and is stated as such; the two producers now share the one stream reader that carries it. A Recv failure is the store being unreachable, not a document this process refused, and docs/prds/control-plane-apply-status.md ("What can be reported, and what cannot") is what forbids reporting it as a failure of the process it affected. The record is now written only for an update that carries a version, which is what controlplane.Client emits for a document that would not validate and never for a transport failure. Reporting the transport failure would also overwrite the text of a standing refusal, replacing the one message that says what to fix with "control-plane unavailable". The record is not un-clearable of itself -- Report overwrites it on the next applied version -- but a store that never changes again leaves the one stamp standing, and the process must not write it at all. That emission is the whole of the failure path and now has the test it lacked, from the producer's side: TestWatchClientsDeliverARecvFailureAsAVersionlessUpdate pins that a Recv failure is handed over carrying the mapped error and no version, and that a stream the server closed and one that ended with the watch's context are not updates at all. The refused version is deliberately not re-offered: the resume point moves past it, so an edit only a newer build can decode cannot become a hot loop. Both ends of that contract are pinned now: the watch stub filters its scripted stream by AfterVersion as the server does, and controlplane.Server.Watch's own filter has a test for the first time. The apply-status reporter's 30s re-stamp is unchanged (TestRestampKeepsAReportedFailure). --- internal/app/archied/control_plane.go | 35 +- .../archied/control_plane_live_apply_test.go | 7 +- internal/app/archied/control_plane_watch.go | 203 +++ .../control_plane_watch_outage_test.go | 188 +++ .../app/archied/control_plane_watch_test.go | 1222 +++++++++++++++++ internal/app/archied/gateway_runtime.go | 40 +- internal/app/controlplane/client.go | 67 +- .../controlplane/controlplane_watch_test.go | 306 +++++ .../domain/applystatus/applystatus_test.go | 40 + 9 files changed, 2067 insertions(+), 41 deletions(-) create mode 100644 internal/app/archied/control_plane_watch.go create mode 100644 internal/app/archied/control_plane_watch_outage_test.go create mode 100644 internal/app/archied/control_plane_watch_test.go create mode 100644 internal/app/controlplane/controlplane_watch_test.go diff --git a/internal/app/archied/control_plane.go b/internal/app/archied/control_plane.go index 4ddff2564..7303eb276 100644 --- a/internal/app/archied/control_plane.go +++ b/internal/app/archied/control_plane.go @@ -85,6 +85,11 @@ func (b *boot) reloadConfig(ctx context.Context, doc *configuration.Document) er return nil } +// startWorkflowExecutionSettings loads the stored limits, applies them and +// keeps the stream that delivers later ones established. The first read and +// the first stream are synchronous, so a control plane that cannot be reached +// fails the boot that asked for it; after that the watch does not return, it +// reconnects (see keepWatch). func (b *boot) startWorkflowExecutionSettings(ctx context.Context) error { settings, version, err := b.controlPlane.WorkflowExecutionSettings(ctx) if err != nil { @@ -97,8 +102,11 @@ func (b *boot) startWorkflowExecutionSettings(ctx context.Context) error { if err != nil { return err } - go func() { - for update := range updates { + go keepWatch(ctx, b.log, controlplane.WorkflowExecutionSettingsKind, version, updates, + b.controlPlane.WatchWorkflowExecutionSettings, + waitFor, + func(update controlplane.AppliedSettings) int64 { return update.Version }, + func(update controlplane.AppliedSettings) { if update.Err != nil { // A refused update arrives here, not through // applyWorkflowExecutionSettings: controlplane.Client decodes every @@ -108,15 +116,28 @@ func (b *boot) startWorkflowExecutionSettings(ctx context.Context) error { // so the settings page shows the refusal instead of a version this // process never ran (docs/prds/control-plane-apply-status.md). // - // The stream is not re-established after this point; that gap is - // archie-core-yrmr, not this apply path. - b.applyStatus.Report(ctx, controlplane.WorkflowExecutionSettingsKind, update.Version, update.Err) + // A stream failure arrives the same way and must not be reported: + // it carries no version at all, because a Recv error is the store + // not answering rather than a document, and versions start at 1 + // (store.PutResource). The rule is + // docs/prds/control-plane-apply-status.md, "What can be reported, + // and what cannot": a process reaches its apply point only after + // the State Store has answered it, and an unreachable store is + // never reported as a failure by the process it affected. The + // record would not stick of itself -- Report overwrites it on the + // next applied version -- but a store that never changes again + // leaves the one stamp standing, and the process must not write it + // at all. Reporting it would also overwrite the text of a standing + // refusal, which is the one message that says what to fix. The + // reconnect is the loop's business, not the record's. + if update.Version > 0 { + b.applyStatus.Report(ctx, controlplane.WorkflowExecutionSettingsKind, update.Version, update.Err) + } b.log.Error("workflow execution settings watch failed", "err", update.Err) return } _ = b.applyWorkflowExecutionSettings(ctx, update.Settings, update.Version) - } - }() + }) return nil } diff --git a/internal/app/archied/control_plane_live_apply_test.go b/internal/app/archied/control_plane_live_apply_test.go index b11d319d1..7aad7c828 100644 --- a/internal/app/archied/control_plane_live_apply_test.go +++ b/internal/app/archied/control_plane_live_apply_test.go @@ -281,8 +281,11 @@ func TestLiveExecutionSettingsUpdateIsRefusedWhenTheSnapshotIsNotRunnable(t *tes // process cannot run arrives as a watch error, never as invalid settings: the // watch branch must report that refusal against the version still live instead // of only logging it, or the settings page shows a live resource as healthy -// while nothing has been applied. (That the stream is not re-established -// afterwards is archie-core-yrmr, not this apply path.) +// while nothing has been applied. (The stream does not stay ended: the watch +// re-establishes it, which control_plane_watch_test.go covers. This stub +// answers every re-established stream with the same documents, so this test +// reads the record the refusal wrote rather than counting how often it was +// written.) func TestLiveExecutionSettingsRefusalIsReportedThroughTheClient(t *testing.T) { tests := []struct { name string diff --git a/internal/app/archied/control_plane_watch.go b/internal/app/archied/control_plane_watch.go new file mode 100644 index 000000000..c50a53112 --- /dev/null +++ b/internal/app/archied/control_plane_watch.go @@ -0,0 +1,203 @@ +package archied + +import ( + "context" + "log/slog" + "time" +) + +// controlPlaneWatchRetry is how long a watch waits before re-establishing a +// stream that ended. The minimum is the delay a healthy attempt leaves behind, +// and the delay a failed attempt doubles; the cap bounds the delay, never the +// retrying: a watch reconnects for as long as the process lives. +// +// A control plane that is down is retried slowly enough that it is not a +// request loop, and that holds for every shape the outage takes because the +// window an attempt is judged against is derived from the delay it would +// otherwise be charged (attemptHealthy): a store that errors at once and a +// store that stalls until its read times out are both charged, so both climb +// the ladder instead of sitting at the minimum. +const ( + controlPlaneWatchRetryMin = 250 * time.Millisecond + controlPlaneWatchRetryMax = 30 * time.Second +) + +// controlPlaneWatchHealthyMultiple is how many times the delay an attempt would +// otherwise be charged an attempt that delivered nothing must outlive to be +// read as a store that answered and had nothing to say. It is strictly more +// than one, so that window is always wider than the charge itself: a window +// equal to the delay would read a store that fails as slowly as that delay as +// healthy on every attempt, which is the fixed-rate loop this rule exists to +// charge. +const controlPlaneWatchHealthyMultiple = 2 + +// keepWatch delivers every update from first and, when that stream ends, +// re-establishes it in this same goroutine until ctx ends. +// +// A control-plane watch stream is not permanent. The State Store restarts, the +// connection drops, the server closes the stream -- and nothing on the client +// side reopens it. A watch that ends on its first end stops applying that kind +// for the rest of the process's life: every later stored change is silently +// never applied while the last apply-status record keeps re-stamping, so the +// settings page reads the process as current, and the process recovers only by +// restarting (archie-core-yrmr). +// +// One goroutine per kind, reconnecting in place: a long outage does not stack +// watchers. Reopening resumes after the last version an update carried, so a +// reconnect neither replays a document this process has already handled over a +// newer one nor skips one it has not. A transport failure carries no version +// and so leaves the resume point alone. A document the client refused to decode +// does carry one -- controlplane.Client decodes what the store sends, and the +// store answered with that version -- and the resume point moves past it +// deliberately: that version is unusable to this build, and resuming before it +// would re-offer it on every reconnect for the life of the process. Only a +// cancellation stops the loop, and the loop watches for it itself, so shutdown +// is prompt however long the current backoff is and whether or not the stream +// is still open. +// +// The reconnect delay is decided by the attempt that just ended, and only by +// that (attemptHealthy, retryDelay): a healthy attempt returns it to the +// minimum, and a failed attempt doubles it up to the cap. An attempt is not +// healthy because a reopen was accepted. A control plane accepts the stream +// before it reads anything -- that is how gRPC hands a request to its handler +// -- and controlplane.Server.Watch answers a store it cannot read by returning +// an error, so during the outage this loop exists for every reopen is accepted +// and every stream ends carrying nothing: judged on the acceptance alone the +// delay stays at the minimum and the watch becomes a 4 RPCs/second request +// loop. +// +// The rule is one sentence: an attempt is healthy iff it delivered an update +// this watch had not handled, or it outlived a window that is a strict multiple +// of the delay it would otherwise be charged. The window is derived from that +// charge rather than fixed at the minimum, which is what makes the rule reach +// every shape of outage: a store that has stopped answering does not error at +// once, it stalls until its read times out, and an attempt that took longer to +// fail than a window fixed at the minimum would be read as a healthy store on +// every attempt, leaving the delay one minimum per timeout for as long as the +// store is gone. +// +// open, versionOf and deliver are the kind's halves. open re-establishes the +// stream -- the caller opens the first one itself, so that a control plane +// which cannot be watched at all still fails the boot that asked for it. +// versionOf reads the resource version an update carries. deliver applies the +// update, including reporting the error it carries, which is how a refused +// document reaches the settings page. +// +// wait is the delay between a stream ending and the reopen that follows it, and +// production passes waitFor. It is injected so that the delay the loop actually +// waited is observable: the retry_in log line states the delay the loop charged +// and would agree with a loop that waited some other multiple of it. +func keepWatch[T any]( + ctx context.Context, + log *slog.Logger, + kind string, + after int64, + first <-chan T, + open func(ctx context.Context, afterVersion int64) (<-chan T, error), + wait func(ctx context.Context, d time.Duration) bool, + versionOf func(update T) int64, + deliver func(update T), +) { + stream, backoff := first, controlPlaneWatchRetryMin + for { + started, progressed := time.Now(), false + for { + update, ok := nextUpdate(ctx, stream) + if !ok { + break + } + deliver(update) + if version := versionOf(update); version > after { + after, progressed = version, true + } + } + if ctx.Err() != nil { + return + } + backoff = retryDelay(backoff, attemptHealthy(progressed, time.Since(started), backoff)) + log.Warn("control plane watch stream ended; reconnecting", + "kind", kind, "after_version", after, "retry_in", backoff) + for { + if !wait(ctx, backoff) { + return + } + reopened, err := open(ctx, after) + if err == nil { + stream = reopened + break + } + // A reopen the control plane refused is the store still not + // answering, and is charged exactly as any other attempt that ends + // that way. + backoff = retryDelay(backoff, false) + log.Warn("control plane watch could not be re-established", + "kind", kind, "after_version", after, "retry_in", backoff, "err", err) + } + } +} + +// attemptHealthy is what one attempt that ended makes of the reconnect delay. +// An attempt is healthy iff it delivered an update this watch had not handled +// -- the store answered, whatever the stream did afterwards -- or it outlived +// controlPlaneWatchHealthyWindow, which is a store that had nothing to say and +// was still there. An attempt that did neither delivered nothing and died +// inside the window it would have been charged: the stream was accepted and the +// store was not answering, which is a failed attempt and the only thing the +// delay grows on. +func attemptHealthy(progressed bool, openFor, backoff time.Duration) bool { + return progressed || openFor >= controlPlaneWatchHealthyWindow(backoff) +} + +// controlPlaneWatchHealthyWindow is how long an attempt that delivered nothing +// must stay open to count as a store that answered and had nothing to say: a +// strict multiple of the delay the attempt would otherwise be charged. The +// window moves with the ladder, so the charge reaches a store that fails slowly +// at every rung of it, which a window fixed at the minimum does not. +func controlPlaneWatchHealthyWindow(backoff time.Duration) time.Duration { + return controlPlaneWatchHealthyMultiple * retryDelay(backoff, false) +} + +// retryDelay is the delay the attempt that just ended leaves behind: a healthy +// attempt returns it to the minimum, so an outage is charged for itself and not +// for the quiet time that preceded it, and a failed attempt doubles it up to +// the cap, so a store that stays gone is retried ever more slowly. +func retryDelay(current time.Duration, healthy bool) time.Duration { + if healthy { + return controlPlaneWatchRetryMin + } + return min(2*current, controlPlaneWatchRetryMax) +} + +// nextUpdate receives the next update from stream, reporting false when the +// stream ended or when ctx ended first. A watch's stream is closed by +// controlplane.Client when the stream's own context ends, but shutdown must not +// rest on that cross-package coupling: a stream that never closed would +// otherwise hold the watch -- and the process that owns it -- past the +// cancellation that asked it to stop. The coupling that is load-bearing runs +// the other way: this watch returns on cancellation without draining what the +// producer has left for it, so controlplane.Client's producer must stop on the +// same context instead of parking on a send no reader will ever take (its send +// guard is pinned by TestKeepWatchLeavesNoProducerParkedOnAnUndeliveredUpdate). +func nextUpdate[T any](ctx context.Context, stream <-chan T) (T, bool) { + select { + case <-ctx.Done(): + var zero T + return zero, false + case update, ok := <-stream: + return update, ok + } +} + +// waitFor is the wait production injects into keepWatch. It waits for d, +// reporting false when ctx ended first: a watch's backoff must never delay the +// shutdown of the process that owns it. +func waitFor(ctx context.Context, d time.Duration) bool { + timer := time.NewTimer(d) + defer timer.Stop() + select { + case <-ctx.Done(): + return false + case <-timer.C: + return true + } +} diff --git a/internal/app/archied/control_plane_watch_outage_test.go b/internal/app/archied/control_plane_watch_outage_test.go new file mode 100644 index 000000000..68ab01315 --- /dev/null +++ b/internal/app/archied/control_plane_watch_outage_test.go @@ -0,0 +1,188 @@ +package archied + +import ( + "context" + "errors" + "net" + "slices" + "sync" + "testing" + "time" + + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/test/bufconn" + + "github.com/samcharles93/archie-core/internal/app/controlplane" + pb "github.com/samcharles93/archie-core/internal/contracts/controlplane/v1" + "github.com/samcharles93/archie-core/internal/infrastructure/workflowsteps" + "github.com/samcharles93/archie-core/internal/store" +) + +// outageStore is the State Store's resource half with a switch that makes every +// read fail after taking stall to do it. It stands in for the outage the +// reconnect exists for: the control plane takes the stream and then answers its +// first read with mapError, so the client is handed a stream that ends carrying +// no document -- while every reopen is accepted, because gRPC accepts before +// the handler runs. +// +// The stall is the other shape that outage takes, and the shape a store that has +// stopped answering actually has: it does not error at once, it stops reading +// and the call times out. An attempt that ends that way delivered nothing and +// lasted longer than the minimum retry interval, which is a failed attempt +// whichever way the store failed it. +type outageStore struct { + *store.Store + + mu sync.Mutex + failing bool + stall time.Duration +} + +// down starts the outage, which takes stall to surface. +func (s *outageStore) down(stall time.Duration) { + s.mu.Lock() + defer s.mu.Unlock() + s.failing, s.stall = true, stall +} + +func (s *outageStore) Resource(ctx context.Context, kind string) (store.Resource, error) { + s.mu.Lock() + failing, stall := s.failing, s.stall + s.mu.Unlock() + if failing { + time.Sleep(stall) + return store.Resource{}, errors.New("state store unreachable") + } + return s.Store.Resource(ctx, kind) +} + +// TestWatchRetryGrowsWhileTheStateStoreIsDown is the outage this reconnect +// exists for, end to end: a real controlplane.Server over bufconn, a real +// controlplane.Client, a real store, and the store then stops answering -- in +// each of the shapes it stops answering in, from the store that errors at once +// to the one that takes as long to fail as the delay it would be charged. +// +// The server accepts every reopened stream and fails its first read, so the +// client is handed an update for that failure that carries no version at all: +// an attempt that ends this way delivered nothing, and the reconnect behind it +// doubles the delay. That charge has to reach a store that fails slowly too. +// The delay a slow failure is charged rests on the healthy window being derived +// from the delay the attempt would otherwise be charged, not on the minimum: a +// store that takes 300ms to time out outlived a window pinned at the 250ms +// minimum on every attempt, so the delay never grew -- measured across 3s of +// that shape, 1.7 watch RPCs a second with retry_in flat at 250ms, which is the +// request loop this reconnect exists to avoid, reached by the likelier shape. +// +// The first retry is the minimum in every shape: the boot stream is a healthy +// attempt, because the store answered it for longer than the window that +// attempt would have been charged with before the outage began. The ladder's +// top is reached by the delay alone and is pinned by +// TestReconnectDelayFollowsAttemptHealth, which does not have to wait 30s for it. +func TestWatchRetryGrowsWhileTheStateStoreIsDown(t *testing.T) { + tests := []struct { + name string + stall time.Duration + want []time.Duration + }{ + { + name: "the store errors at once", + want: []time.Duration{controlPlaneWatchRetryMin, 2 * controlPlaneWatchRetryMin, 4 * controlPlaneWatchRetryMin}, + }, + { + name: "the store stalls before it times out", + stall: 300 * time.Millisecond, + want: []time.Duration{ + controlPlaneWatchRetryMin, 2 * controlPlaneWatchRetryMin, + 4 * controlPlaneWatchRetryMin, 8 * controlPlaneWatchRetryMin, + }, + }, + { + // The window an attempt is judged against is strictly wider than the + // delay it would be charged with, and this is the shape that says so: + // a failure that takes exactly that delay to surface is still a store + // that did not answer the read. + name: "the store fails as slowly as the delay it would be charged", + stall: 2 * controlPlaneWatchRetryMin, + want: []time.Duration{ + controlPlaneWatchRetryMin, 2 * controlPlaneWatchRetryMin, + 4 * controlPlaneWatchRetryMin, 8 * controlPlaneWatchRetryMin, + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + database := &outageStore{Store: store.OpenTest(t)} + defer database.Close() + steps, err := workflowsteps.NewManager() + if err != nil { + t.Fatalf("build the workflow step vocabulary: %v", err) + } + server, err := controlplane.NewServer(database, steps) + if err != nil { + t.Fatalf("build the control plane: %v", err) + } + if _, err := server.ImportWorkflowExecutionSettings(t.Context(), bootedSettings); err != nil { + t.Fatalf("seed the stored settings: %v", err) + } + + log, retries := newRetryLog() + b, _ := newLiveApplyBoot(t, fileConfig()) + b.log = log + b.controlPlane = controlplane.NewClient(dialControlPlane(t, server)) + + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + if err := b.startWorkflowExecutionSettings(ctx); err != nil { + t.Fatalf("startWorkflowExecutionSettings: %v", err) + } + + // The store answers the boot stream for longer than the window that + // attempt would be charged with, which is what makes it the healthy + // attempt the first retry is not charged for. + time.Sleep(controlPlaneWatchHealthyWindow(controlPlaneWatchRetryMin)) + database.down(tt.stall) + + delays := awaitRetries(t, retries, len(tt.want)) + if got := delays[:len(tt.want)]; !slices.Equal(got, tt.want) { + t.Errorf("reconnects waited %v, want %v: the delay is decided by the attempt that just ended, so the healthy boot attempt leaves the %v minimum behind and every attempt of the outage doubles the delay before it. A delay that never grows is this watch back at a request loop: pinned at %v by a healthy window fixed at the minimum, which a store that fails as slowly as this one outlives on every attempt, and pinned at %v by a reset on every reopen the store accepted, which charges each attempt of the outage from the minimum again", + got, tt.want, controlPlaneWatchRetryMin, controlPlaneWatchRetryMin, 2*controlPlaneWatchRetryMin) + } + // Each logged retry is the wait the loop then took. A loop that logged + // one delay and waited another would reopen sooner than it said it + // would, and the attribute alone cannot show that. + for i := 0; i < len(tt.want)-1; i++ { + if gap := retries.gap(i); gap < tt.want[i] { + t.Errorf("the retry logged as %v was followed by the next stream ending %v later, want at least that delay: the loop waits the delay it charged", tt.want[i], gap) + } + } + }) + } +} + +// dialControlPlane serves server over bufconn and returns the client +// connection a watch dials it through. The transport is real: the stream's +// context is the RPC's, and an unaccepted stream is not something a stub can +// stand in for. +func dialControlPlane(t *testing.T, server pb.ControlPlaneServiceServer) *grpc.ClientConn { + t.Helper() + listener := bufconn.Listen(1 << 20) + grpcServer := grpc.NewServer() + pb.RegisterControlPlaneServiceServer(grpcServer, server) + go func() { _ = grpcServer.Serve(listener) }() + t.Cleanup(func() { + grpcServer.Stop() + _ = listener.Close() + }) + conn, err := grpc.NewClient("passthrough:///control-plane", + grpc.WithTransportCredentials(insecure.NewCredentials()), + grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) { + return listener.DialContext(ctx) + })) + if err != nil { + t.Fatalf("dial the control plane: %v", err) + } + t.Cleanup(func() { _ = conn.Close() }) + return conn +} diff --git a/internal/app/archied/control_plane_watch_test.go b/internal/app/archied/control_plane_watch_test.go new file mode 100644 index 000000000..ac523d665 --- /dev/null +++ b/internal/app/archied/control_plane_watch_test.go @@ -0,0 +1,1222 @@ +package archied + +import ( + "context" + "errors" + "go/ast" + "go/parser" + "go/token" + "io" + "log/slog" + "slices" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + + "github.com/samcharles93/archie-core/internal/app/controlplane" + pb "github.com/samcharles93/archie-core/internal/contracts/controlplane/v1" + "github.com/samcharles93/archie-core/internal/domain/workflow" + "github.com/samcharles93/archie-core/internal/gateway" +) + +// The stream ends these tests drive. A control-plane watch stream stops for +// both of them and neither is answered on the client side: a broken stream +// arrives as an error from Recv, and a stream the server closed arrives as +// io.EOF, which controlplane.Client emits no update for at all -- it just +// closes the channel. +var watchStreamEnds = []struct { + name string + end error +}{ + {"the stream fails", status.Error(codes.Unavailable, "state store restarting")}, + {"the server closes the stream", io.EOF}, +} + +// watchScript is one stream the stub hands a Watch call: the resources it +// sends, then what ends it. A nil end leaves the stream open until the caller +// goes away, which is what a healthy idle stream looks like. +type watchScript struct { + resources []*pb.Resource + end error +} + +// watchStub is the control-plane end of a watch that ends once and is then +// re-established. Query answers the boot read for each kind, Watch hands each +// successive call the next scripted stream -- filtered to the versions newer +// than the one the call resumed after, as the real server filters them -- and +// the version each call resumed after is recorded so a test can assert the +// reconnect resumes rather than replays. Calls past the script stay open. +type watchStub struct { + boot map[string]*pb.Resource + scripts map[string][]watchScript + + mu sync.Mutex + resumed map[string][]int64 + requests map[string][]*pb.WatchRequest +} + +func (s *watchStub) Catalog(context.Context, *pb.CatalogRequest, ...grpc.CallOption) (*pb.CatalogResponse, error) { + return &pb.CatalogResponse{}, nil +} + +func (s *watchStub) Query(_ context.Context, request *pb.QueryRequest, _ ...grpc.CallOption) (*pb.QueryResponse, error) { + resource, ok := s.boot[request.Kind] + if !ok { + return nil, status.Error(codes.NotFound, "resource not found") + } + return &pb.QueryResponse{Resource: resource}, nil +} + +func (*watchStub) History(context.Context, *pb.HistoryRequest, ...grpc.CallOption) (*pb.HistoryResponse, error) { + panic("unexpected History") +} + +func (*watchStub) Command(context.Context, *pb.CommandRequest, ...grpc.CallOption) (*pb.CommandResponse, error) { + panic("unexpected Command") +} + +func (s *watchStub) Watch(ctx context.Context, request *pb.WatchRequest, _ ...grpc.CallOption) (grpc.ServerStreamingClient[pb.WatchResponse], error) { + s.mu.Lock() + if s.resumed == nil { + s.resumed = map[string][]int64{} + } + if s.requests == nil { + s.requests = map[string][]*pb.WatchRequest{} + } + s.resumed[request.Kind] = append(s.resumed[request.Kind], request.AfterVersion) + s.requests[request.Kind] = append(s.requests[request.Kind], request) + scripts := s.scripts[request.Kind] + var next watchScript + if len(scripts) > 0 { + next = scripts[0] + s.scripts[request.Kind] = scripts[1:] + } + s.mu.Unlock() + // controlplane.Server.Watch serves only versions greater than the resume + // point, so the stub does too. A stub that replayed its whole script would + // hand a reconnect that asked from zero this process's own already-applied + // document, and the double apply would go unnoticed. + served := make([]*pb.Resource, 0, len(next.resources)) + for _, resource := range next.resources { + if resource.Version > request.AfterVersion { + served = append(served, resource) + } + } + return &scriptStream{done: ctx.Done(), resources: served, end: next.end}, nil +} + +// resumedAfter is the AfterVersion of every Watch call for a kind, in order. +func (s *watchStub) resumedAfter(kind string) []int64 { + s.mu.Lock() + defer s.mu.Unlock() + return append([]int64(nil), s.resumed[kind]...) +} + +// awaitWatchCalls waits until n Watch calls have been made for kind. The watch +// makes the second one only after it has delivered the update that ended the +// first stream, so a test asserting what that update did *not* report has +// something to synchronise on: the delivery precedes the reopen. +func (s *watchStub) awaitWatchCalls(t *testing.T, kind string, n int) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for len(s.resumedAfter(kind)) < n { + if time.Now().After(deadline) { + t.Fatalf("watch calls for %s = %d, want %d", kind, len(s.resumedAfter(kind)), n) + } + time.Sleep(5 * time.Millisecond) + } +} + +// scriptStream is the client end of one scripted Watch stream. Recv replays the +// resources, then the stream's end: io.EOF for a server that closed it, the +// scripted failure for one that broke, or a wait for the caller to go away for +// a stream that is simply idle. The embedded ClientStream is nil, so any other +// call panics rather than pretending to work, as in the settingsWatchStub of +// control_plane_live_apply_test.go. +type scriptStream struct { + grpc.ClientStream + // done is the stream context's Done channel, held rather than the context + // itself: an idle stream ends the way a real one does when its RPC is + // cancelled. + done <-chan struct{} + resources []*pb.Resource + end error + next int +} + +func (s *scriptStream) Recv() (*pb.WatchResponse, error) { + if s.next < len(s.resources) { + resource := s.resources[s.next] + s.next++ + return &pb.WatchResponse{Resource: resource}, nil + } + if s.end != nil { + end := s.end + s.end = nil + return nil, end + } + <-s.done + return nil, context.Canceled +} + +// TestWorkflowExecutionSettingsWatchIsReEstablishedAfterTheStreamEnds is +// archie-core-yrmr. A watch goroutine is launched once, at boot, and a +// control-plane stream is not permanent: the State Store restarts, the +// connection drops, or the server closes the stream. A watch that ends on the +// first stream failure stops applying that kind for the rest of the process's +// life -- every later stored change is silently never applied and the process +// only recovers by restarting. The stream must be re-established and the later +// update delivered on it must be applied. +func TestWorkflowExecutionSettingsWatchIsReEstablishedAfterTheStreamEnds(t *testing.T) { + // 25 steps and an hour, so an applied update is unmistakably not the value + // the daemon booted with. + updated := workflow.ExecutionSettings{MaxModelToolSteps: 25, MaxRuntime: time.Hour, MaxConsecutiveGateFailures: 4} + updatedDocument := `{"max_model_tool_steps": 25, "max_runtime_seconds": 3600, "max_consecutive_gate_failures": 4}` + + for _, tt := range watchStreamEnds { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + b, status := newLiveApplyBoot(t, fileConfig()) + stub := &watchStub{ + boot: map[string]*pb.Resource{ + controlplane.WorkflowExecutionSettingsKind: { + Kind: controlplane.WorkflowExecutionSettingsKind, Version: 1, ValueJson: []byte(bootedDocument), + }, + }, + scripts: map[string][]watchScript{ + controlplane.WorkflowExecutionSettingsKind: { + {end: tt.end}, + {resources: []*pb.Resource{{ + Kind: controlplane.WorkflowExecutionSettingsKind, Version: 2, ValueJson: []byte(updatedDocument), + }}}, + }, + }, + } + b.controlPlane = controlplane.NewRPCClient(stub) + + if err := b.startWorkflowExecutionSettings(t.Context()); err != nil { + t.Fatalf("startWorkflowExecutionSettings: %v", err) + } + + awaitRunningSettings(t, b, updated) + + // The reconnect resumes after the version this process last handled + // rather than from zero: the second stream is asked for everything + // after version 1, so a reconnect neither replays the document it + // already applied over a newer one nor skips an unseen one. + if got := stub.resumedAfter(controlplane.WorkflowExecutionSettingsKind); !slices.Equal(got, []int64{1, 1}) { + t.Errorf("watch resumed after %v, want the boot version then the last version handled", got) + } + + // Boot's own apply, then the update the re-established stream + // delivered. Neither way the stream ends writes a record of its own: + // a stream the server closed emits no update at all (controlplane + // .Client sends one only for a Recv error), and a Recv failure + // carries no version, which is the unreachable store the process must + // never report as a failure of its own + // (docs/prds/control-plane-apply-status.md, "What can be reported, and + // what cannot"). + records := status.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 2) + if len(records) != 2 { + t.Fatalf("apply status = %+v, want the boot apply and the recovered one alone", records) + } + if records[0].AppliedVersion != 1 || records[0].Error != "" { + t.Errorf("boot report = %+v, want a clean apply of version 1", records[0]) + } + last := records[len(records)-1] + if last.AppliedVersion != 2 || last.Error != "" { + t.Errorf("last apply status = %+v, want the recovered apply of version 2 with the failure cleared", last) + } + }) + } +} + +// TestWatchTransportFailureIsNotReportedAsAFailedApply is +// docs/prds/control-plane-apply-status.md, "What can be reported, and what +// cannot": "When the State Store is unreachable ... An unreachable store is +// never reported as a failure by the process it affected. A record carries the +// other failure: the store answered, and the value would not validate." +// +// A stream failure is that unreachable store: controlplane.Client emits it with +// no version, because a Recv error is the transport and not a document. An +// earlier revision reported it anyway, against the version still live, and the +// record then could not be cleared: a re-established stream over an unchanged +// store delivers nothing (the server sends only what is newer than the resume +// point), applystatus.Reporter keeps a standing error and re-stamps it forever, +// and the dashboard renders any non-empty error as failed. The process reads as +// failed for the rest of its life, after one blip. +func TestWatchTransportFailureIsNotReportedAsAFailedApply(t *testing.T) { + t.Parallel() + b, recorder := newLiveApplyBoot(t, fileConfig()) + stub := &watchStub{ + boot: map[string]*pb.Resource{ + controlplane.WorkflowExecutionSettingsKind: { + Kind: controlplane.WorkflowExecutionSettingsKind, Version: 1, ValueJson: []byte(bootedDocument), + }, + }, + scripts: map[string][]watchScript{ + controlplane.WorkflowExecutionSettingsKind: { + // The store goes away, and comes back unchanged: the second + // stream carries no document, so only a report written for the + // failure itself could ever be cleared. + {end: status.Error(codes.Unavailable, "state store restarting")}, + {}, + }, + }, + } + b.controlPlane = controlplane.NewRPCClient(stub) + + if err := b.startWorkflowExecutionSettings(t.Context()); err != nil { + t.Fatalf("startWorkflowExecutionSettings: %v", err) + } + + // The failure has been delivered by the time the second stream is opened: + // the loop delivers what ended the stream, then reopens. + stub.awaitWatchCalls(t, controlplane.WorkflowExecutionSettingsKind, 2) + + records := recorder.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 1) + if len(records) != 1 { + t.Fatalf("apply status = %+v, want the boot apply alone: an unreachable store writes no record", records) + } + if last := records[0]; last.AppliedVersion != 1 || last.Error != "" { + t.Errorf("apply status = %+v, want the clean apply of version 1 still standing", last) + } +} + +// TestWatchTransportFailureDoesNotOverwriteARefusal is the other half of the +// same rule, and the reason the branch is on the version rather than on the +// error: a refused document arrives carrying the version it came from, so it is +// the store answering and must be reported; a transport failure carries no +// version and must not. Reporting the transport failure against the version +// still live overwrites the refusal's text, so a validation error against +// version 2 reads "control-plane unavailable: ..." after any blip, and the +// operator loses the one message that says what to fix. +func TestWatchTransportFailureDoesNotOverwriteARefusal(t *testing.T) { + t.Parallel() + b, recorder := newLiveApplyBoot(t, fileConfig()) + stub := &watchStub{ + boot: map[string]*pb.Resource{ + controlplane.WorkflowExecutionSettingsKind: { + Kind: controlplane.WorkflowExecutionSettingsKind, Version: 1, ValueJson: []byte(bootedDocument), + }, + }, + scripts: map[string][]watchScript{ + controlplane.WorkflowExecutionSettingsKind: { + { + resources: []*pb.Resource{{ + Kind: controlplane.WorkflowExecutionSettingsKind, Version: 2, + ValueJson: []byte(`{"max_model_tool_steps": -1, "max_runtime_seconds": 3600, "max_consecutive_gate_failures": 4}`), + }}, + end: status.Error(codes.Unavailable, "state store restarting"), + }, + {}, + }, + }, + } + b.controlPlane = controlplane.NewRPCClient(stub) + + if err := b.startWorkflowExecutionSettings(t.Context()); err != nil { + t.Fatalf("startWorkflowExecutionSettings: %v", err) + } + + stub.awaitWatchCalls(t, controlplane.WorkflowExecutionSettingsKind, 2) + + records := recorder.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 2) + if len(records) != 2 { + t.Fatalf("apply status = %+v, want the boot apply and the refusal alone", records) + } + last := records[len(records)-1] + if last.AppliedVersion != 1 { + t.Errorf("refusal reported against version %d, want the version still live (1)", last.AppliedVersion) + } + if !strings.Contains(last.Error, "max model/tool steps must not be negative") { + t.Errorf("apply status error = %q, want the refusal's own text", last.Error) + } + if strings.Contains(last.Error, "control-plane unavailable") { + t.Errorf("apply status error = %q, want the transport blip not to have overwritten the refusal", last.Error) + } + // The refused version is unusable to this build, so the resume point moves + // past it rather than letting every reconnect re-offer it forever. + if got := stub.resumedAfter(controlplane.WorkflowExecutionSettingsKind); !slices.Equal(got, []int64{1, 2}) { + t.Errorf("watch resumed after %v, want the boot version then the refused one", got) + } +} + +// TestWatchReconnectDoesNotReplayWhatItAlreadyApplied ties the resume contract +// to both of its ends at once. The re-established stream is asked for +// everything after the last version this process handled, and the stub -- like +// controlplane.Server.Watch -- serves only what is newer than that, so a store +// offering its whole history back is filtered down to the one version this +// process has not seen. A replay would be applied over settings this process +// already runs, and reported a second time. +func TestWatchReconnectDoesNotReplayWhatItAlreadyApplied(t *testing.T) { + t.Parallel() + updated := workflow.ExecutionSettings{MaxModelToolSteps: 25, MaxRuntime: time.Hour, MaxConsecutiveGateFailures: 4} + updatedDocument := `{"max_model_tool_steps": 25, "max_runtime_seconds": 3600, "max_consecutive_gate_failures": 4}` + b, recorder := newLiveApplyBoot(t, fileConfig()) + stub := &watchStub{ + boot: map[string]*pb.Resource{ + controlplane.WorkflowExecutionSettingsKind: { + Kind: controlplane.WorkflowExecutionSettingsKind, Version: 1, ValueJson: []byte(bootedDocument), + }, + }, + scripts: map[string][]watchScript{ + controlplane.WorkflowExecutionSettingsKind: { + {end: status.Error(codes.Unavailable, "state store restarting")}, + // The version this process already ran, offered alongside the new + // one, as a server asked from zero would offer it. + {resources: []*pb.Resource{ + {Kind: controlplane.WorkflowExecutionSettingsKind, Version: 1, ValueJson: []byte(bootedDocument)}, + {Kind: controlplane.WorkflowExecutionSettingsKind, Version: 2, ValueJson: []byte(updatedDocument)}, + }}, + }, + }, + } + b.controlPlane = controlplane.NewRPCClient(stub) + + if err := b.startWorkflowExecutionSettings(t.Context()); err != nil { + t.Fatalf("startWorkflowExecutionSettings: %v", err) + } + + awaitRunningSettings(t, b, updated) + if got := stub.resumedAfter(controlplane.WorkflowExecutionSettingsKind); !slices.Equal(got, []int64{1, 1}) { + t.Errorf("watch resumed after %v, want the boot version then the last version handled", got) + } + + // One report for the boot apply and one for the update: a third would be + // the re-served version 1 applied over itself. + records := recorder.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 2) + if len(records) != 2 { + t.Fatalf("apply status = %+v, want the boot apply and the update alone", records) + } + last := records[len(records)-1] + if last.AppliedVersion != 2 || last.Error != "" { + t.Errorf("last apply status = %+v, want the recovered apply of version 2", last) + } +} + +// TestPersonaWatchIsReEstablishedAfterTheStreamEnds is the persona half of the +// same gap, over the seam the daemon meets persona edits through: the persona +// watch is launched the same way and ends the same way, so a persona edit +// stored after a stream failure must still reach the registry. +func TestPersonaWatchIsReEstablishedAfterTheStreamEnds(t *testing.T) { + booted := `{"default":"archie","personas":[{"name":"archie","prompt":"the baseline"}]}` + updated := `{"default":"plain","personas":[{"name":"plain","prompt":"no flourish"}]}` + + for _, tt := range watchStreamEnds { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + b := &boot{cfg: fileConfig(), log: slog.New(slog.DiscardHandler)} + b.personas = gateway.NewPersonaRegistry(nil) + stub := &watchStub{ + boot: map[string]*pb.Resource{ + controlplane.PersonasKind: {Kind: controlplane.PersonasKind, Version: 1, ValueJson: []byte(booted)}, + }, + scripts: map[string][]watchScript{ + controlplane.PersonasKind: { + {end: tt.end}, + {resources: []*pb.Resource{{ + Kind: controlplane.PersonasKind, Version: 2, ValueJson: []byte(updated), + }}}, + }, + }, + } + b.controlPlane = controlplane.NewRPCClient(stub) + + if err := b.watchPersonas(t.Context(), 1); err != nil { + t.Fatalf("watchPersonas: %v", err) + } + + deadline := time.Now().Add(5 * time.Second) + for { + if _, ok := b.personas.Get("plain"); ok { + break + } + if time.Now().After(deadline) { + t.Fatalf("the persona edit stored after the stream ended was never applied: personas = %v", b.personas.List()) + } + time.Sleep(5 * time.Millisecond) + } + if got := b.personas.List(); !slices.Equal(got, []string{"plain"}) { + t.Errorf("personas = %v, want the re-established stream's collection alone", got) + } + }) + } +} + +// TestKeepWatchGuardrails pins the properties the reconnect loop must not +// trade away for recovery: one stream at a time, a retry that is spaced rather +// than spun, a shutdown that does not wait out the backoff, and a resume point +// that only updates carrying a version can move. +func TestKeepWatchGuardrails(t *testing.T) { + log := slog.New(slog.DiscardHandler) + versionOf := func(version int64) int64 { return version } + t.Run("one stream is held at a time", func(t *testing.T) { + var opens atomic.Int64 + first := make(chan int64) + close(first) + // Every re-established stream stays open, so a reconnect that stacked + // watchers instead of replacing the stream would keep opening more. + open := func(ctx context.Context, _ int64) (<-chan int64, error) { + opens.Add(1) + return idleStream[int64](ctx), nil + } + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan struct{}) + go func() { + defer close(done) + keepWatch(ctx, log, "test", 0, first, open, waitFor, versionOf, func(int64) {}) + }() + + awaitOpenCount(t, &opens, 1) + time.Sleep(2 * controlPlaneWatchRetryMin) + if got := opens.Load(); got != 1 { + t.Errorf("streams opened = %d while one was still live, want 1: the watch reconnects in place", got) + } + cancel() + awaitStopped(t, done) + }) + + t.Run("a stream that cannot be re-established is retried with backoff", func(t *testing.T) { + var opens atomic.Int64 + first := make(chan int64) + close(first) + open := func(context.Context, int64) (<-chan int64, error) { + opens.Add(1) + return nil, errors.New("state store unreachable") + } + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan struct{}) + go func() { + defer close(done) + keepWatch(ctx, log, "test", 0, first, open, waitFor, versionOf, func(int64) {}) + }() + + firstOpen := awaitOpenCount(t, &opens, 1) + secondOpen := awaitOpenCount(t, &opens, 2) + thirdOpen := awaitOpenCount(t, &opens, 3) + previous := secondOpen.Sub(firstOpen) + if previous < controlPlaneWatchRetryMin { + t.Errorf("retried after %v, want at least the %v minimum backoff: a failing stream must not be reopened in a loop", previous, controlPlaneWatchRetryMin) + } + if grew := thirdOpen.Sub(secondOpen); grew <= previous { + t.Errorf("the retry delay went from %v to %v, want each attempt spaced further apart than the last", previous, grew) + } + cancel() + awaitStopped(t, done) + }) + + t.Run("a cancelled context ends the watch while it is backing off", func(t *testing.T) { + var opens atomic.Int64 + first := make(chan int64) + close(first) + open := func(context.Context, int64) (<-chan int64, error) { + opens.Add(1) + return nil, errors.New("state store unreachable") + } + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan struct{}) + go func() { + defer close(done) + keepWatch(ctx, log, "test", 0, first, open, waitFor, versionOf, func(int64) {}) + }() + + // One retry is in flight, so the loop is waiting out a backoff. + awaitOpenCount(t, &opens, 1) + start := time.Now() + cancel() + awaitStopped(t, done) + if stopped := time.Since(start); stopped > controlPlaneWatchRetryMin/2 { + t.Errorf("the watch stopped %v after the context ended, want it to stop without waiting out the backoff", stopped) + } + }) + + t.Run("the reconnect resumes after the last version delivered", func(t *testing.T) { + // A version this process applied, then the update a stream failure + // arrives as: the failure carries no version, so it must not move the + // resume point and the next stream must not be asked to replay history. + first := make(chan int64, 2) + first <- 7 + first <- 0 + close(first) + resumed := make(chan int64, 1) + open := func(ctx context.Context, afterVersion int64) (<-chan int64, error) { + resumed <- afterVersion + return idleStream[int64](ctx), nil + } + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan struct{}) + go func() { + defer close(done) + keepWatch(ctx, log, "test", 0, first, open, waitFor, versionOf, func(int64) {}) + }() + + select { + case after := <-resumed: + if after != 7 { + t.Errorf("re-established the stream after version %d, want the last version handled (7)", after) + } + case <-time.After(5 * time.Second): + t.Fatal("the stream was never re-established") + } + cancel() + awaitStopped(t, done) + }) +} + +// TestKeepWatchChargesAnAttemptThatEndedWithNothing pins the reconnect delay +// against the three ways one attempt can end, which is the whole of the +// decision: an attempt that delivered an update of the store's is healthy even +// though the stream ended the instant after it, an attempt that outlived the +// window it would have been charged is healthy even though the store had +// nothing to say, and an attempt that did neither failed -- the stream was +// accepted and then died without the store having answered -- so the reconnect +// behind it is charged. +// +// Every case reopens successfully, so a loop that reset the delay on the mere +// acceptance of a stream would report the minimum for the failed attempt too: +// gRPC takes the stream before the handler reads anything, so during a store +// outage every reopen is accepted and every stream ends carrying nothing, and a +// delay reset by that acceptance spins at the minimum -- the case this loop +// exists for. The delay is read off the watch's own log line, which is where +// the loop states the wait it is about to take. +func TestKeepWatchChargesAnAttemptThatEndedWithNothing(t *testing.T) { + tests := []struct { + name string + first func() <-chan int64 + want time.Duration + }{ + { + name: "an attempt that delivered nothing and died inside the window it would be charged failed", + first: func() <-chan int64 { stream := make(chan int64); close(stream); return stream }, + want: 2 * controlPlaneWatchRetryMin, + }, + { + name: "an attempt that carried an update is healthy however fast it ended", + first: func() <-chan int64 { stream := make(chan int64, 1); stream <- 8; close(stream); return stream }, + want: controlPlaneWatchRetryMin, + }, + { + name: "an attempt that outlived the window it would be charged is healthy", + first: func() <-chan int64 { + stream := make(chan int64) + go func() { + defer close(stream) + time.Sleep(controlPlaneWatchHealthyWindow(controlPlaneWatchRetryMin)) + }() + return stream + }, + want: controlPlaneWatchRetryMin, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + log, retries := newRetryLog() + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + open := func(ctx context.Context, _ int64) (<-chan int64, error) { return idleStream[int64](ctx), nil } + go keepWatch(ctx, log, "test", 0, tt.first(), open, waitFor, func(version int64) int64 { return version }, func(int64) {}) + + if got := awaitRetries(t, retries, 1)[0]; got != tt.want { + t.Errorf("the reconnect waited %v, want %v: %s", got, tt.want, tt.name) + } + cancel() + }) + } +} + +// TestKeepWatchBackoffResetsAfterASuccessfulReopen pins the retry delay's other +// half: it grows while attempts fail, and a reopen the store answers returns it +// to the minimum. Without the reset a store that is up and quiet ratchets the +// delay up to the cap -- it is only ever reset by a healthy attempt, and a quiet +// store sends nothing -- and the next outage is charged the whole of it. +// +// Successful means the store answered the reopened stream, which is what the +// window is for and not what the acceptance of a stream is: gRPC takes the +// stream before the handler reads anything, so a store that is down accepts +// every reopen. TestKeepWatchChargesAReopenTheStoreMerelyAccepted is the other +// half of that, and it is the half this test cannot pin: it reads a delay the +// log says was the minimum, and the minimum is also what a loop that reset on +// the acceptance would leave behind. The delay is read off the watch's own log +// line, which is where the loop states the wait it is about to take. +// +// The re-established stream is quiet for longer than the window that attempt +// would be charged with before this test ends it, which is what makes it +// healthy: a store that answers and then says nothing for a while is a store +// that is there. The first stream is the failed attempt at the other end of the +// same rule -- it ended at once having delivered nothing -- so the delay has +// something to reset from. +func TestKeepWatchBackoffResetsAfterASuccessfulReopen(t *testing.T) { + log, retries := newRetryLog() + first := make(chan int64) + close(first) + // The store is up and quiet once the first reopen succeeds: the stream stays + // open until this test ends it, so no update arrives to reset the delay. + quiet := make(chan int64) + var opens atomic.Int64 + open := func(context.Context, int64) (<-chan int64, error) { + opens.Add(1) + return quiet, nil + } + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan struct{}) + go func() { + defer close(done) + keepWatch(ctx, log, "test", 0, first, open, waitFor, func(version int64) int64 { return version }, func(int64) {}) + }() + + awaitOpenCount(t, &opens, 1) + // Quiet for longer than the window this attempt would be charged with, so + // this attempt is the healthy one. The next stream end is the outage that + // follows it. + time.Sleep(controlPlaneWatchHealthyWindow(2 * controlPlaneWatchRetryMin)) + close(quiet) + awaitOpenCount(t, &opens, 2) + + delays := awaitRetries(t, retries, 2) + if delays[0] != 2*controlPlaneWatchRetryMin { + t.Errorf("the retry after an attempt that ended at once waited %v, want %v: it delivered nothing", delays[0], 2*controlPlaneWatchRetryMin) + } + if delays[1] != controlPlaneWatchRetryMin { + t.Errorf("the retry after a reopened stream the store answered waited %v, want %v: the store answered, so the outage was not charged the whole of it", delays[1], controlPlaneWatchRetryMin) + } + cancel() + awaitStopped(t, done) +} + +// TestKeepWatchChargesAReopenTheStoreMerelyAccepted pins the unit the delay is +// decided on. The acceptance of a reopen is not it: a control plane takes the +// stream before the handler reads anything, so a store that is down accepts +// every reopen and every one of those streams then ends at once carrying +// nothing. A loop that reset the delay there -- the shape this loop was written +// to remove -- charges each attempt of the outage from the minimum and never +// climbs, which is retry_in flat at one charge per attempt while every reopen +// succeeded. These attempts are charged instead, one rung each, from the +// minimum they start at. The delay is read off the watch's own log line, which +// is where the loop states the wait it is about to take. +func TestKeepWatchChargesAReopenTheStoreMerelyAccepted(t *testing.T) { + log, retries := newRetryLog() + first := make(chan int64) + close(first) + // Accepted, over the moment it is read, and carrying nothing: what a reopen + // against a store that is down hands back, however many times it is asked. + flaky := make(chan int64) + close(flaky) + var opens atomic.Int64 + open := func(context.Context, int64) (<-chan int64, error) { + opens.Add(1) + return flaky, nil + } + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan struct{}) + go func() { + defer close(done) + keepWatch(ctx, log, "test", 0, first, open, waitFor, func(version int64) int64 { return version }, func(int64) {}) + }() + + want := []time.Duration{2 * controlPlaneWatchRetryMin, 4 * controlPlaneWatchRetryMin, 8 * controlPlaneWatchRetryMin} + delays := awaitRetries(t, retries, len(want)) + if got := delays[:len(want)]; !slices.Equal(got, want) { + t.Errorf("reconnects waited %v, want %v: every one of those reopens was accepted and the attempt that carried it delivered nothing, so each is charged the delay before it doubled rather than reset by the acceptance", got, want) + } + cancel() + awaitStopped(t, done) +} + +// TestKeepWatchWaitsTheDelayItCharged is the behavioural half of the ladder. +// The retry_in attribute states the delay the loop is about to take, so a loop +// that waited some other multiple of it -- twice what it logged, say -- would +// report a growing ladder to the operator and reopen on a schedule of its own, +// and the attribute agrees with itself whatever the loop waits. The wait is +// injected, so what the loop waited is read back as values and asserted against +// the ladder it charged: the same ladder TestReconnectDelayFollowsAttemptHealth +// pins for retryDelay, driven through the loop instead of the function. +func TestKeepWatchWaitsTheDelayItCharged(t *testing.T) { + log, retries := newRetryLog() + recorder := &waitRecorder{} + first := make(chan int64) + close(first) + // The store is gone: every reopen is refused, so nothing but the delay the + // loop charges decides when the next attempt starts. + open := func(context.Context, int64) (<-chan int64, error) { + return nil, errors.New("state store unreachable") + } + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan struct{}) + go func() { + defer close(done) + keepWatch(ctx, log, "test", 0, first, open, recorder.wait, func(version int64) int64 { return version }, func(int64) {}) + }() + + want := []time.Duration{ + 2 * controlPlaneWatchRetryMin, 4 * controlPlaneWatchRetryMin, + 8 * controlPlaneWatchRetryMin, 16 * controlPlaneWatchRetryMin, + } + if waited := recorder.await(t, len(want)); !slices.Equal(waited, want) { + t.Errorf("the watch waited %v, want %v: the delay it waits is the delay the attempt that just ended charged it", waited, want) + } + // What the operator reads has to be the delay the loop took, and not a + // figure of its own. + if logged := awaitRetries(t, retries, len(want)); !slices.Equal(logged[:len(want)], want) { + t.Errorf("the watch logged retries %v, want %v: retry_in is the delay the loop is about to wait", logged[:len(want)], want) + } + cancel() + awaitStopped(t, done) +} + +// TestReconnectDelayFollowsAttemptHealth pins the whole of the ladder the +// reconnect climbs, which no test of the loop can reach in reasonable time: the +// delay doubles on every failed attempt, stops at the cap instead of doubling +// for as long as the outage lasts, and returns to the minimum the moment an +// attempt is healthy. Attempt health itself is pinned at the loop, where it is +// read off the stream (TestKeepWatchChargesAnAttemptThatEndedWithNothing). +func TestReconnectDelayFollowsAttemptHealth(t *testing.T) { + tests := []struct { + name string + healthy []bool + wantWaits []time.Duration + }{ + { + name: "consecutive failed attempts double the delay and then hold at the cap", + healthy: []bool{false, false, false, false, false, false, false, false, false, false}, + wantWaits: []time.Duration{ + 500 * time.Millisecond, time.Second, 2 * time.Second, 4 * time.Second, 8 * time.Second, + 16 * time.Second, controlPlaneWatchRetryMax, controlPlaneWatchRetryMax, controlPlaneWatchRetryMax, controlPlaneWatchRetryMax, + }, + }, + { + name: "a healthy attempt returns the delay to the minimum at once", + healthy: []bool{false, false, false, true, false}, + wantWaits: []time.Duration{500 * time.Millisecond, time.Second, 2 * time.Second, controlPlaneWatchRetryMin, 500 * time.Millisecond}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + delay := controlPlaneWatchRetryMin + waits := make([]time.Duration, 0, len(tt.healthy)) + for _, healthy := range tt.healthy { + delay = retryDelay(delay, healthy) + waits = append(waits, delay) + } + if !slices.Equal(waits, tt.wantWaits) { + t.Errorf("retry waits = %v, want %v: %s", waits, tt.wantWaits, tt.name) + } + }) + } +} + +// TestKeepWatchResumesPastAVersionedFailure pins the resume point's behaviour +// on an update that carries a version and an error at once, which is how +// controlplane.Client delivers a document this process refuses to decode: the +// store answered, so the update carries the version it answered with. The +// resume point moves past it deliberately -- that version is unusable to this +// build, and resuming before it would re-offer it on every reconnect for the +// life of the process -- so the next reader must not "fix" this into a hot +// loop. +func TestKeepWatchResumesPastAVersionedFailure(t *testing.T) { + type refusal struct { + version int64 + err error + } + first := make(chan refusal, 2) + first <- refusal{version: 7} + first <- refusal{version: 8, err: errors.New("max model/tool steps must not be negative")} + close(first) + resumed := make(chan int64, 1) + open := func(ctx context.Context, afterVersion int64) (<-chan refusal, error) { + resumed <- afterVersion + return idleStream[refusal](ctx), nil + } + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan struct{}) + go func() { + defer close(done) + keepWatch(ctx, slog.New(slog.DiscardHandler), "test", 0, first, open, waitFor, + func(update refusal) int64 { return update.version }, func(refusal) {}) + }() + + select { + case after := <-resumed: + if after != 8 { + t.Errorf("re-established the stream after version %d, want the refused version's own (8)", after) + } + case <-time.After(5 * time.Second): + t.Fatal("the stream was never re-established") + } + cancel() + awaitStopped(t, done) +} + +// TestKeepWatchReturnsOnCancellationWithANeverClosingStream is the shutdown +// half of the loop. The range over the stream returns today only because +// controlplane.Client's producer closes its channel when the stream's context +// ends, so shutdown promptness rests on an untested cross-package coupling; a +// stream that never closes would otherwise hold the process up past the +// cancellation that asked it to stop, however long its backoff cap is. +func TestKeepWatchReturnsOnCancellationWithANeverClosingStream(t *testing.T) { + stream := make(chan int64) + open := func(context.Context, int64) (<-chan int64, error) { + t.Error("the stream never ended, so nothing should have been reopened") + return nil, errors.New("not reopened") + } + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan struct{}) + go func() { + defer close(done) + keepWatch(ctx, slog.New(slog.DiscardHandler), "test", 0, stream, open, waitFor, + func(version int64) int64 { return version }, func(int64) {}) + }() + + cancel() + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("the watch was still blocked on a stream that never closed 2s after its context ended") + } +} + +// TestKeepWatchLeavesNoProducerParkedOnAnUndeliveredUpdate is the producer half +// of the shutdown coupling nextUpdate names. The watch returns on cancellation +// without draining what the producer has left for it, so the producer must stop +// on the same context rather than park on a send no reader will ever take. A +// bare send does park: reproduced 40/40 times as a goroutine still blocked on an +// undelivered send after keepWatch returned, one goroutine and one channel +// leaked per kind at every exit. +func TestKeepWatchLeavesNoProducerParkedOnAnUndeliveredUpdate(t *testing.T) { + t.Run("workflow execution settings", func(t *testing.T) { + t.Parallel() + client := controlplane.NewRPCClient(busyWatch{resource: &pb.Resource{ + Kind: controlplane.WorkflowExecutionSettingsKind, Version: 2, ValueJson: []byte(bootedDocument), + }}) + assertProducerStopsWhenTheReadingStops(t, client.WatchWorkflowExecutionSettings, + func(update controlplane.AppliedSettings) int64 { return update.Version }, + func(update controlplane.AppliedSettings) { + if update.Err != nil { + t.Errorf("the watch was handed %v, want the document the store answers with", update.Err) + } + }) + }) + t.Run("personas", func(t *testing.T) { + t.Parallel() + client := controlplane.NewRPCClient(busyWatch{resource: &pb.Resource{ + Kind: controlplane.PersonasKind, Version: 2, + ValueJson: []byte(`{"default":"archie","personas":[{"name":"archie","prompt":"the baseline"}]}`), + }}) + assertProducerStopsWhenTheReadingStops(t, client.WatchPersonas, + func(update controlplane.AppliedPersonas) int64 { return update.Version }, + func(update controlplane.AppliedPersonas) { + if update.Err != nil { + t.Errorf("the watch was handed %v, want the document the store answers with", update.Err) + } + }) + }) +} + +// assertProducerStopsWhenTheReadingStops runs the watch against a producer that +// always has another document ready, stops reading it, and then holds the +// channel the producer writes to. An update that arrives once the watch has +// returned is a send the producer was parked on. +func assertProducerStopsWhenTheReadingStops[T any]( + t *testing.T, + watch func(ctx context.Context, afterVersion int64) (<-chan T, error), + versionOf func(update T) int64, + deliver func(update T), +) { + t.Helper() + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + updates, err := watch(ctx, 1) + if err != nil { + t.Fatalf("watching: %v", err) + } + done := make(chan struct{}) + go func() { + defer close(done) + keepWatch(ctx, slog.New(slog.DiscardHandler), "test", 1, updates, watch, waitFor, versionOf, deliver) + }() + + // A few rounds through Recv and a send, then the cancellation the watch + // answers by returning without draining what is left behind it. + time.Sleep(25 * time.Millisecond) + cancel() + awaitStopped(t, done) + + // The producer's select has only the context left to choose from now that + // the watch has gone, and this wait is what lets it take that case: a + // receive here would be the reader that unblocks a parked send, and would + // hide the leak as a delivered update. + time.Sleep(250 * time.Millisecond) + deadline := time.Now().Add(5 * time.Second) + for { + select { + case update, ok := <-updates: + if !ok { + return + } + t.Fatalf("the producer handed the watch %v after the watch had returned: it was parked on a send no reader will take, leaking a goroutine and a channel per kind at every exit", update) + default: + } + if time.Now().After(deadline) { + t.Fatal("the producer never closed the watch's stream after the watch returned") + } + time.Sleep(5 * time.Millisecond) + } +} + +// busyWatch is the rpc end of a watch that always has another document of one +// kind ready: Recv never blocks and never ends, so the client's producer is +// only ever waiting on its reader. That is the wait a watch that stops reading +// leaves it in, and the one a bare send parks on for the life of the process. +type busyWatch struct{ resource *pb.Resource } + +func (busyWatch) Catalog(context.Context, *pb.CatalogRequest, ...grpc.CallOption) (*pb.CatalogResponse, error) { + return &pb.CatalogResponse{}, nil +} + +func (b busyWatch) Query(context.Context, *pb.QueryRequest, ...grpc.CallOption) (*pb.QueryResponse, error) { + return &pb.QueryResponse{Resource: b.resource}, nil +} + +func (busyWatch) History(context.Context, *pb.HistoryRequest, ...grpc.CallOption) (*pb.HistoryResponse, error) { + panic("unexpected History") +} + +func (busyWatch) Command(context.Context, *pb.CommandRequest, ...grpc.CallOption) (*pb.CommandResponse, error) { + panic("unexpected Command") +} + +func (b busyWatch) Watch(context.Context, *pb.WatchRequest, ...grpc.CallOption) (grpc.ServerStreamingClient[pb.WatchResponse], error) { + return &busyStream{resource: b.resource}, nil +} + +// busyStream is the client end of that stream. +type busyStream struct { + grpc.ClientStream + resource *pb.Resource +} + +func (s *busyStream) Recv() (*pb.WatchResponse, error) { + return &pb.WatchResponse{Resource: s.resource}, nil +} + +// TestBothWatchStreamsRunThroughTheReconnectLoop pins the wiring the +// behavioural tests drive one step below: both launch sites must hand their +// stream to the one reconnect loop, or a kind keeps the single-shot goroutine +// that ends for the life of the process. Asserted on the source because +// setupChatRuntime needs a fully built boot (stores, runtime, memory engines) +// to run -- the technique composition_order_test.go and +// TestSetupGatewayChatCarriesStatusHealth already use for wiring nothing at +// runtime observes. +func TestBothWatchStreamsRunThroughTheReconnectLoop(t *testing.T) { + fileset := token.NewFileSet() + for _, tt := range []struct{ file, holder, call string }{ + {file: "control_plane.go", holder: "startWorkflowExecutionSettings", call: "keepWatch"}, + {file: "gateway_runtime.go", holder: "setupChatRuntime", call: "watchPersonas"}, + {file: "gateway_runtime.go", holder: "watchPersonas", call: "keepWatch"}, + } { + parsed, err := parser.ParseFile(fileset, tt.file, nil, 0) + if err != nil { + t.Fatalf("parse %s: %v", tt.file, err) + } + if !callsName(methodBody(t, parsed, tt.holder), tt.call) { + t.Errorf("%s never calls %s: that watch stream would not be re-established", tt.holder, tt.call) + } + } +} + +// callsName reports whether body calls name, as a plain function or as a +// method of anything. +func callsName(body *ast.BlockStmt, name string) bool { + called := false + ast.Inspect(body, func(node ast.Node) bool { + call, ok := node.(*ast.CallExpr) + if !ok { + return true + } + switch fun := call.Fun.(type) { + case *ast.Ident: + called = called || fun.Name == name + case *ast.SelectorExpr: + called = called || fun.Sel.Name == name + } + return true + }) + return called +} + +// retryLog is a slog.Handler that records the retry delay of each reconnect, +// and when it was logged. The delay is the reconnect loop's own state, and the +// log line is where it is observable without reading a clock. +type retryLog struct { + mu sync.Mutex + retries []time.Duration + at []time.Time +} + +func newRetryLog() (*slog.Logger, *retryLog) { + handler := &retryLog{} + return slog.New(handler), handler +} + +func (*retryLog) Enabled(context.Context, slog.Level) bool { return true } +func (h *retryLog) WithAttrs([]slog.Attr) slog.Handler { return h } +func (h *retryLog) WithGroup(string) slog.Handler { return h } + +func (h *retryLog) Handle(_ context.Context, record slog.Record) error { + record.Attrs(func(attr slog.Attr) bool { + if attr.Key == "retry_in" { + h.mu.Lock() + h.retries = append(h.retries, attr.Value.Duration()) + h.at = append(h.at, time.Now()) + h.mu.Unlock() + } + return true + }) + return nil +} + +// gap is how long passed between the retry logged at i and the one after it. +// The loop logs the delay it has charged and then waits it, so the gap is that +// wait plus the attempt it led to, and it can only be longer than the delay it +// followed. It is what makes the delay actually waited observable: retry_in +// states the intent, and a loop that waited some other multiple of it would +// never be caught by the attribute alone. +func (h *retryLog) gap(i int) time.Duration { + h.mu.Lock() + defer h.mu.Unlock() + return h.at[i+1].Sub(h.at[i]) +} + +func (h *retryLog) snapshot() []time.Duration { + h.mu.Lock() + defer h.mu.Unlock() + return slices.Clone(h.retries) +} + +// waitRecorder is the wait a test injects into keepWatch: it records each delay +// and reports that it passed at once, so the ladder the loop waited is a value a +// test asserts instead of a clock it reads, and climbing the ladder costs no +// time. It answers for the delay's purpose and not for a stub of it -- true is +// "the delay passed" -- so it still reports the cancellation waitFor reports, +// which is what returns the watch a finished test is holding. +type waitRecorder struct { + mu sync.Mutex + waits []time.Duration +} + +func (r *waitRecorder) wait(ctx context.Context, d time.Duration) bool { + r.mu.Lock() + r.waits = append(r.waits, d) + r.mu.Unlock() + return ctx.Err() == nil +} + +// await waits until n delays have been recorded and returns them. +func (r *waitRecorder) await(t *testing.T, n int) []time.Duration { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for { + r.mu.Lock() + recorded := slices.Clone(r.waits) + r.mu.Unlock() + if len(recorded) >= n { + return recorded[:n] + } + if time.Now().After(deadline) { + t.Fatalf("the watch waited %d times, want %d", len(recorded), n) + } + time.Sleep(time.Millisecond) + } +} + +// awaitRetries waits until the watch has logged n retries, returning what it +// logged. The retry delay is the reconnect loop's own state, and its log line +// is where it is observable without reading a clock: the loop is back in a wait +// by then, so a test can cancel it without racing the attempt. +func awaitRetries(t *testing.T, retries *retryLog, n int) []time.Duration { + t.Helper() + deadline := time.Now().Add(10 * time.Second) + for { + if got := retries.snapshot(); len(got) >= n { + return got + } + if time.Now().After(deadline) { + t.Fatalf("the watch logged %d retries, want %d", len(retries.snapshot()), n) + } + time.Sleep(5 * time.Millisecond) + } +} + +// idleStream is the client end of a stream that sends nothing: its channel +// stays open until ctx ends, the way the channel controlplane.Client returns +// closes when the stream's context ends. That is what lets a watch be +// cancelled while it is idle rather than blocked on a server that has gone +// quiet. +func idleStream[T any](ctx context.Context) <-chan T { + stream := make(chan T) + go func() { + defer close(stream) + <-ctx.Done() + }() + return stream +} + +// awaitOpenCount waits until at least n streams have been opened, returning +// when the nth was seen. A test synchronises on the stream count because the +// reconnect loop is what opens them. +func awaitOpenCount(t *testing.T, opens *atomic.Int64, n int64) time.Time { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for opens.Load() < n { + if time.Now().After(deadline) { + t.Fatalf("streams opened = %d, want at least %d", opens.Load(), n) + } + time.Sleep(5 * time.Millisecond) + } + return time.Now() +} + +// awaitStopped waits for the watch goroutine to return. +func awaitStopped(t *testing.T, done <-chan struct{}) { + t.Helper() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("the watch goroutine did not return") + } +} + +// awaitRunningSettings waits for the watch goroutine to apply want, which is +// the only evidence that it re-established the stream: the watch runs on its +// own goroutine, so a test driving it through the stream has nothing else to +// synchronise on. The snapshot every new task is built from has to move with +// the settings, so it is read too. +func awaitRunningSettings(t *testing.T, b *boot, want workflow.ExecutionSettings) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for { + if running := b.executionSettings.Load(); running != nil && *running == want { + if budgets := b.cfgHolder.Get().Budgets; budgets != budgetsFor(want) { + t.Fatalf("running budgets = %+v, want %+v", budgets, budgetsFor(want)) + } + return + } + if time.Now().After(deadline) { + t.Fatalf("the watch did not re-establish the stream, so the stored update was never applied: running settings = %v, want %+v", + b.executionSettings.Load(), want) + } + time.Sleep(5 * time.Millisecond) + } +} diff --git a/internal/app/archied/gateway_runtime.go b/internal/app/archied/gateway_runtime.go index 712daad17..f674cee6c 100644 --- a/internal/app/archied/gateway_runtime.go +++ b/internal/app/archied/gateway_runtime.go @@ -5,6 +5,7 @@ import ( "fmt" "github.com/samcharles93/archie-core/internal/agentexec" + "github.com/samcharles93/archie-core/internal/app/controlplane" "github.com/samcharles93/archie-core/internal/config" "github.com/samcharles93/archie-core/internal/domain/agent" "github.com/samcharles93/archie-core/internal/gateway" @@ -34,19 +35,9 @@ func (b *boot) setupChatRuntime(ctx context.Context, cfg config.Config) error { } b.personas = gateway.NewPersonaRegistry(nil) b.applyPersonas(personas, version) - updates, err := b.controlPlane.WatchPersonas(ctx, version) - if err != nil { + if err := b.watchPersonas(ctx, version); err != nil { return fmt.Errorf("watch personas: %w", err) } - go func() { - for update := range updates { - if update.Err != nil { - b.log.Error("persona watch failed", "err", update.Err) - return - } - b.applyPersonas(update.Collection, update.Version) - } - }() } // ── Operator health surface ────────────────────────────────────── @@ -75,6 +66,33 @@ func (b *boot) setupChatRuntime(ctx context.Context, cfg config.Config) error { return nil } +// watchPersonas keeps the persona stream established for the life of the +// process. The first stream is opened synchronously, so a control plane that +// cannot be watched at all fails the boot that asked for it rather than +// leaving the process running personas it can no longer update; after that the +// watch reconnects instead of ending (see keepWatch). +// +// Persona updates are not reported through apply status -- the personas kind +// has no apply-status wiring, which is archie-core-aj35, not this watch. +func (b *boot) watchPersonas(ctx context.Context, version int64) error { + updates, err := b.controlPlane.WatchPersonas(ctx, version) + if err != nil { + return err + } + go keepWatch(ctx, b.log, controlplane.PersonasKind, version, updates, + b.controlPlane.WatchPersonas, + waitFor, + func(update controlplane.AppliedPersonas) int64 { return update.Version }, + func(update controlplane.AppliedPersonas) { + if update.Err != nil { + b.log.Error("persona watch failed", "err", update.Err) + return + } + b.applyPersonas(update.Collection, update.Version) + }) + return nil +} + func (b *boot) applyPersonas(collection agent.PersonaCollection, version int64) { personas := make([]gateway.Persona, 0, len(collection.Personas)) for _, persona := range collection.Personas { diff --git a/internal/app/controlplane/client.go b/internal/app/controlplane/client.go index f1e1e1c0d..c0520aa92 100644 --- a/internal/app/controlplane/client.go +++ b/internal/app/controlplane/client.go @@ -115,22 +115,12 @@ func (c *Client) WatchWorkflowExecutionSettings(ctx context.Context, afterVersio if err != nil { return nil, clientError(err) } - out := make(chan AppliedSettings) - go func() { - defer close(out) - for { - response, err := stream.Recv() - if err != nil { - if !errors.Is(err, io.EOF) && ctx.Err() == nil { - out <- AppliedSettings{Err: clientError(err)} - } - return - } - settings, err := decodeSettings(response.Resource.ValueJson) - out <- AppliedSettings{Settings: settings, Version: response.Resource.Version, Err: err} - } - }() - return out, nil + return watchUpdates(ctx, stream, + func(resource *pb.Resource) AppliedSettings { + settings, err := decodeSettings(resource.ValueJson) + return AppliedSettings{Settings: settings, Version: resource.Version, Err: err} + }, + func(err error) AppliedSettings { return AppliedSettings{Err: err} }), nil } type AppliedSettings struct { @@ -159,22 +149,57 @@ func (c *Client) WatchPersonas(ctx context.Context, afterVersion int64) (<-chan if err != nil { return nil, clientError(err) } - out := make(chan AppliedPersonas) + return watchUpdates(ctx, stream, + func(resource *pb.Resource) AppliedPersonas { + collection, err := decodePersonas(resource.ValueJson) + return AppliedPersonas{Collection: collection, Version: resource.Version, Err: err} + }, + func(err error) AppliedPersonas { return AppliedPersonas{Err: err} }), nil +} + +// watchUpdates turns one Watch stream into the channel a watch loop reads: every +// document the stream carries, decoded, and then a single update carrying the +// error that ended it -- unless the stream ended cleanly or the context did, in +// which case the channel just closes. +func watchUpdates[T any]( + ctx context.Context, + stream grpc.ServerStreamingClient[pb.WatchResponse], + update func(resource *pb.Resource) T, + failed func(err error) T, +) <-chan T { + out := make(chan T) go func() { defer close(out) for { response, err := stream.Recv() if err != nil { if !errors.Is(err, io.EOF) && ctx.Err() == nil { - out <- AppliedPersonas{Err: clientError(err)} + sendUpdate(ctx, out, failed(clientError(err))) } return } - collection, err := decodePersonas(response.Resource.ValueJson) - out <- AppliedPersonas{Collection: collection, Version: response.Resource.Version, Err: err} + if !sendUpdate(ctx, out, update(response.Resource)) { + return + } } }() - return out, nil + return out +} + +// sendUpdate hands update to the reader of one Watch stream, reporting false +// when ctx ended first. A reader leaves without draining what is left for it: +// the watch that reads these streams returns on cancellation, so a bare send +// parks this producer -- and the stream it holds -- for the rest of the +// process's life, one goroutine and one channel per kind at every exit. That +// coupling is load-bearing: the reader's prompt shutdown rests on this side +// stopping on the same context, and neither half is safe alone. +func sendUpdate[T any](ctx context.Context, out chan<- T, update T) bool { + select { + case out <- update: + return true + case <-ctx.Done(): + return false + } } func (c *Client) Schedules(ctx context.Context) ([]cronstore.JobSpec, int64, error) { diff --git a/internal/app/controlplane/controlplane_watch_test.go b/internal/app/controlplane/controlplane_watch_test.go new file mode 100644 index 000000000..0cfbfedd4 --- /dev/null +++ b/internal/app/controlplane/controlplane_watch_test.go @@ -0,0 +1,306 @@ +package controlplane + +import ( + "context" + "errors" + "io" + "net" + "testing" + "time" + + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/status" + "google.golang.org/grpc/test/bufconn" + + pb "github.com/samcharles93/archie-core/internal/contracts/controlplane/v1" + "github.com/samcharles93/archie-core/internal/domain/workflow" + "github.com/samcharles93/archie-core/internal/store" +) + +// scriptedWatchClient is the rpc end of a Watch call whose stream is scripted +// rather than served. These tests are about what the client hands the watch that +// reads it, so the store stays out of them and the transport is a value. +type scriptedWatchClient struct { + streams func() grpc.ServerStreamingClient[pb.WatchResponse] +} + +func (scriptedWatchClient) Catalog(context.Context, *pb.CatalogRequest, ...grpc.CallOption) (*pb.CatalogResponse, error) { + panic("Catalog is not part of the watch path") +} + +func (scriptedWatchClient) Query(context.Context, *pb.QueryRequest, ...grpc.CallOption) (*pb.QueryResponse, error) { + panic("Query is not part of the watch path") +} + +func (scriptedWatchClient) History(context.Context, *pb.HistoryRequest, ...grpc.CallOption) (*pb.HistoryResponse, error) { + panic("History is not part of the watch path") +} + +func (scriptedWatchClient) Command(context.Context, *pb.CommandRequest, ...grpc.CallOption) (*pb.CommandResponse, error) { + panic("Command is not part of the watch path") +} + +func (c scriptedWatchClient) Watch(context.Context, *pb.WatchRequest, ...grpc.CallOption) (grpc.ServerStreamingClient[pb.WatchResponse], error) { + return c.streams(), nil +} + +// scriptedRecvStream is the client end of one Watch stream: Recv replays the +// resources it was given and then returns the scripted error. release, when set, +// is what that error waits behind, so a test can end the stream after the +// watch's own context has already ended. +type scriptedRecvStream struct { + grpc.ClientStream + resources []*pb.Resource + end error + release <-chan struct{} + next int +} + +func (s *scriptedRecvStream) Recv() (*pb.WatchResponse, error) { + if s.next < len(s.resources) { + resource := s.resources[s.next] + s.next++ + return &pb.WatchResponse{Resource: resource}, nil + } + if s.release != nil { + <-s.release + } + return nil, s.end +} + +// receiveUpdate reads the next update the watch was handed, reporting false when +// its stream closed without one. +func receiveUpdate[T any](t *testing.T, updates <-chan T) (T, bool) { + t.Helper() + select { + case update, ok := <-updates: + return update, ok + case <-time.After(5 * time.Second): + t.Fatal("the watch's stream neither delivered an update nor closed") + var zero T + return zero, false + } +} + +// assertDeliveredNothing asserts the watch was handed no update at all, which is +// what a stream that did not fail leaves behind: the reconnect is the loop's own +// business, and an update here would be a message about a store that is still +// there. +func assertDeliveredNothing[T any](t *testing.T, updates <-chan T, why string) { + t.Helper() + if update, ok := receiveUpdate(t, updates); ok { + t.Errorf("the watch was handed %+v, want nothing: %s", update, why) + } +} + +// assertDeliveredARecvFailure asserts the watch was handed the one update a Recv +// failure is -- carrying the mapped transport failure and no version at all -- +// and nothing after it. +func assertDeliveredARecvFailure[T any](t *testing.T, updates <-chan T, version func(T) int64, carried func(T) error) { + t.Helper() + update, ok := receiveUpdate(t, updates) + if !ok { + t.Fatal("the Recv failure was never handed to the watch: the stream just closed, so the reconnect behind it logs nothing and the operator is told nothing") + } + if got := version(update); got != 0 { + t.Errorf("the transport failure carried version %d, want none: a Recv error is the store not answering, not a document", got) + } + if err := carried(update); !errors.Is(err, ErrUnavailable) { + t.Errorf("the transport failure was handed over as %v, want %v", err, ErrUnavailable) + } + if extra, ok := receiveUpdate(t, updates); ok { + t.Errorf("a second update followed the transport failure: %+v", extra) + } +} + +// TestWatchClientsDeliverARecvFailureAsAVersionlessUpdate is the producer side +// of the rule every consumer of these streams keys on, which the daemon's own +// tests pin from the other end: a Recv failure is the State Store unreachable, +// so the update the client hands the watch for it carries that error and no +// version at all. Versions start at 1 (store.PutResource), so a watch that +// reports what it is handed reports nothing for a transport failure, and the +// settings page keeps the last apply instead of reading the process as failed +// for as long as the store is gone (docs/prds/control-plane-apply-status.md, +// "What can be reported, and what cannot"). +// +// That one update is the whole of the failure path. A producer that closed the +// stream instead would hand the watch a stream that simply ended, which is a +// reconnect with nothing to log and nothing to report -- and every test of the +// watch below this would still pass, because they are handed the update by a +// script rather than by the client. +func TestWatchClientsDeliverARecvFailureAsAVersionlessUpdate(t *testing.T) { + t.Parallel() + brokenStream := func() grpc.ServerStreamingClient[pb.WatchResponse] { + return &scriptedRecvStream{end: status.Error(codes.Unavailable, "state store restarting")} + } + t.Run("workflow execution settings", func(t *testing.T) { + t.Parallel() + client := NewRPCClient(scriptedWatchClient{streams: brokenStream}) + updates, err := client.WatchWorkflowExecutionSettings(t.Context(), 4) + if err != nil { + t.Fatalf("WatchWorkflowExecutionSettings: %v", err) + } + assertDeliveredARecvFailure(t, updates, + func(update AppliedSettings) int64 { return update.Version }, + func(update AppliedSettings) error { return update.Err }) + }) + t.Run("personas", func(t *testing.T) { + t.Parallel() + client := NewRPCClient(scriptedWatchClient{streams: brokenStream}) + updates, err := client.WatchPersonas(t.Context(), 4) + if err != nil { + t.Fatalf("WatchPersonas: %v", err) + } + assertDeliveredARecvFailure(t, updates, + func(update AppliedPersonas) int64 { return update.Version }, + func(update AppliedPersonas) error { return update.Err }) + }) +} + +// TestWatchClientsDeliverNothingForAStreamThatDidNotFail is the other half of +// that contract. A stream the server closed cleanly is io.EOF, which is not a +// failure of anything, and a stream that ended because the watch's own context +// ended is the process shutting down: neither may be handed over as an update, +// because the consumer of one reports it, and a process that reports a store for +// going away when it was the process that left is reporting the wrong side of +// the outage. +func TestWatchClientsDeliverNothingForAStreamThatDidNotFail(t *testing.T) { + t.Parallel() + tests := []struct { + name string + end error + cancel bool + }{ + {"the server closed the stream", io.EOF, false}, + {"the stream failed as the watch was cancelled", status.Error(codes.Unavailable, "state store restarting"), true}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + // The failure waits behind release, so a test that cancels first can + // end the stream after the context that ended the watch has ended. + var release chan struct{} + if tt.cancel { + release = make(chan struct{}) + } + client := NewRPCClient(scriptedWatchClient{streams: func() grpc.ServerStreamingClient[pb.WatchResponse] { + return &scriptedRecvStream{end: tt.end, release: release} + }}) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + settings, err := client.WatchWorkflowExecutionSettings(ctx, 4) + if err != nil { + t.Fatalf("WatchWorkflowExecutionSettings: %v", err) + } + personas, err := client.WatchPersonas(ctx, 4) + if err != nil { + t.Fatalf("WatchPersonas: %v", err) + } + if tt.cancel { + cancel() + close(release) + } + assertDeliveredNothing(t, settings, tt.name) + assertDeliveredNothing(t, personas, tt.name) + }) + } +} + +// TestWatchServesOnlyVersionsAfterTheRequestedOne is the producer side of the +// resume contract a reconnecting client depends on. A watch resumes after the +// last version it handled, so the server must treat AfterVersion as an +// exclusive lower bound: a stream that re-sent the version the client had +// already applied would have it applied again, over whatever is newer. The +// stored version and the requested one are equal here, which is the state a +// client resuming correctly always presents. +// +// The stream is a real one over bufconn rather than a hand-rolled WatchServer: +// its context is the RPC's, so the test ends the server's loop the way +// production does, by cancelling the client. +func TestWatchServesOnlyVersionsAfterTheRequestedOne(t *testing.T) { + t.Parallel() + + resources := store.OpenTest(t) + defer resources.Close() + server := testServer(t, resources) + version, err := server.ImportWorkflowExecutionSettings(t.Context(), workflow.ExecutionSettings{MaxModelToolSteps: 10, MaxRuntime: time.Minute, MaxConsecutiveGateFailures: 2}) + if err != nil { + t.Fatal(err) + } + + listener := bufconn.Listen(1 << 20) + grpcServer := grpc.NewServer() + pb.RegisterControlPlaneServiceServer(grpcServer, server) + go func() { _ = grpcServer.Serve(listener) }() + t.Cleanup(func() { grpcServer.Stop(); _ = listener.Close() }) + conn, err := grpc.NewClient("passthrough:///controlplane", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) { return listener.DialContext(ctx) })) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = conn.Close() }) + + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + stream, err := pb.NewControlPlaneServiceClient(conn).Watch(ctx, &pb.WatchRequest{Kind: WorkflowExecutionSettingsKind, AfterVersion: version}) + if err != nil { + t.Fatal(err) + } + // served carries each version the stream sends, and is closed when the + // stream ends. It is the only thing the test can read: what the server + // sends is the contract. + served := make(chan int64, 8) + go func() { + defer close(served) + for { + response, err := stream.Recv() + if err != nil { + return + } + served <- response.GetResource().GetVersion() + } + }() + + // The stored version is the one the stream was asked to resume after, so + // nothing is due. Two poll intervals is long enough to see a server that + // re-serves it. + select { + case servedVersion, ok := <-served: + if !ok { + t.Fatal("the stream ended before serving anything") + } + t.Fatalf("the stream served version %d, want only versions after %d", servedVersion, version) + case <-time.After(2 * 250 * time.Millisecond): + } + + // The next stored version is served. + next := workflow.ExecutionSettings{MaxModelToolSteps: 40, MaxRuntime: time.Hour, MaxConsecutiveGateFailures: 5} + value, err := encodeSettings(next) + if err != nil { + t.Fatal(err) + } + if _, err := resources.PutResource(ctx, store.ResourceWrite{Kind: WorkflowExecutionSettingsKind, Value: value, ExpectedVersion: version, Actor: "test", Source: "test", RequestID: "watch-next"}); err != nil { + t.Fatal(err) + } + select { + case servedVersion, ok := <-served: + if !ok { + t.Fatal("the stream ended before the stored version was served") + } + if servedVersion != version+1 { + t.Errorf("the stream served version %d, want %d", servedVersion, version+1) + } + case <-time.After(5 * time.Second): + t.Fatal("the stored version after the resume point was never served") + } + + // Cancelling the client ends the server's loop: Recv stops, which is what + // closes served. + cancel() + select { + case <-served: + case <-time.After(5 * time.Second): + t.Fatal("the stream did not end after its context was cancelled") + } +} diff --git a/internal/domain/applystatus/applystatus_test.go b/internal/domain/applystatus/applystatus_test.go index 6d3eace80..c96ac23dc 100644 --- a/internal/domain/applystatus/applystatus_test.go +++ b/internal/domain/applystatus/applystatus_test.go @@ -106,6 +106,46 @@ func TestRestampRewritesEveryKnownKind(t *testing.T) { } } +// TestRestampKeepsAReportedFailure answers the second half of +// archie-core-yrmr's question: the reporter rewrites every record it holds at +// every RestampInterval, so a re-stamp that replaced a reported failure with a +// clean record would leave the settings page reading a kind as current while +// the process applying it had stopped (internal/webui's applyState reads a +// record carrying an error as failed -- TestApplyStatusReportsAnError). A +// reported failure survives the re-stamp, error included. +func TestRestampKeepsAReportedFailure(t *testing.T) { + store := &recordingStore{} + at := time.Date(2026, 9, 21, 10, 0, 0, 0, time.UTC) + r := New(Daemon, store, nil) + r.now = func() time.Time { return at } + + r.Report(t.Context(), "workflow-execution-settings", 3, nil) + // A failure, reported without a version the process ever ran. The reporter + // keeps the version it last applied, which is the rule the watch leans on when + // it reports a document this process would not run: the watch has the refused + // version, and the record has to show the version still live. (An unreachable + // store writes no record at all, per docs/prds/control-plane-apply-status.md.) + r.Report(t.Context(), "workflow-execution-settings", 0, errors.New("max model/tool steps must not be negative")) + + at = at.Add(RestampInterval) + r.restamp(t.Context()) + + got := store.snapshot() + if len(got) != 3 { + t.Fatalf("wrote %d records, want the apply, the failure and the re-stamp", len(got)) + } + restamped := got[2] + if restamped.Error == "" { + t.Error("the re-stamp dropped the reported failure; the kind would read as current") + } + if restamped.AppliedVersion != 3 { + t.Errorf("re-stamped version = %d, want the version still live (3)", restamped.AppliedVersion) + } + if !restamped.ReportedAt.Equal(at) { + t.Errorf("re-stamped at %v, want %v", restamped.ReportedAt, at) + } +} + func TestStale(t *testing.T) { at := time.Date(2026, 9, 21, 10, 0, 0, 0, time.UTC) for _, tt := range []struct {