From b6357c8fd42515c3387f60d32ef742cc3a194e24 Mon Sep 17 00:00:00 2001 From: Sam Catlow Date: Sat, 26 Sep 2026 20:38:48 +1000 Subject: [PATCH] fix(archied): report workflow settings watch outages --- docs/data/generated/dev-docs.json | 2 +- docs/prds/control-plane-apply-status.md | 13 ++- internal/app/archied/control_plane.go | 52 ++++++------ .../archied/control_plane_live_apply_test.go | 12 ++- internal/app/archied/control_plane_watch.go | 14 ++-- .../app/archied/control_plane_watch_test.go | 79 ++++++++++--------- 6 files changed, 97 insertions(+), 75 deletions(-) diff --git a/docs/data/generated/dev-docs.json b/docs/data/generated/dev-docs.json index 7bb93d14f..b45385b4f 100644 --- a/docs/data/generated/dev-docs.json +++ b/docs/data/generated/dev-docs.json @@ -232,7 +232,7 @@ "url": "/dev/prds/control-plane-apply-status/", "title": "Control-plane apply status", "section": "PRDs", - "body": "# Control-plane apply status\n\n**Status:** Approved\n**Date:** 2026-09-21\n**Tracking:** `archie-core-pskb`, and `archie-core-nwa0` which depends on it\n\n## Outcome\n\nAn operator who edits a control-plane resource can see which processes are\nrunning that version, which are still on an older one, which rejected it, and\nwhich have stopped reporting.\n\n`docs/prds/runtime-control-plane.md` requires this and does not say where the\nreport goes. This decides that.\n\n## Where the report goes\n\nApply status is a State Store contract surface. `ControlPlaneService` is the\nsurface for editing a resource, and a status report is not an edit. The\nprecedent is the config snapshot pair: a process publishes what it knows, the\nUI process reads it, and both RPCs are administrative, so a task-scoped grant\nreaches neither. That denial needs no new rule: `authorizesTaskScopedCall`\nnames three methods and denies everything else.\n\n`docs/prds/state-store-contract.md` gains the pair in its inventory, and its\nRPC count at two places moves from 45 to 47.\n\nThe read returns all records as a list, including an empty one. The\n`found=false` convention for absent single rows does not apply.\n\n## What is reported\n\nOne record per process and resource kind, replaced in place. It holds the\nprocess name, the resource kind, the version applied, the error if the process\ncould not apply it, and the time it reported.\n\nEach process re-stamps its records every 30 seconds. A record is stale once it\nis older than 90 seconds, and a stale record means the process that wrote it is\nno longer running. The report time is therefore a liveness signal, not only a\nwrite timestamp.\n\nA process reports at the point it applies a resource, so the record says what\nhappened rather than what was intended. Nothing probes.\n\nProcess names come from one shared constant naming `archied`, `archie-gateway`\nand `archie-messaging`. A deployment runs at most one of each. Writer and\nreader are bound to that constant by test, because a drifted name is\nindistinguishable from a process that never reported.\n\n## Which processes report\n\n- `archied` and `archie-gateway` apply every restart-required kind through\n `boot.runtimeConfig`, at boot and on SIGHUP.\n- `archied` also applies the live kinds it watches, and reports each update it\n applies.\n- `archie-messaging` applies channel settings at startup and reports that kind.\n\n## What can be reported, and what cannot\n\nA process reaches its apply point only after the State Store has answered it.\nWhen the State Store is unreachable, the process exits or refuses to start and\nwrites no record. An unreachable store is never reported as a failure by the\nprocess it affected. A record carries the other failure: the store answered,\nand the value would not validate.\n\nAbsence has two meanings and the UI never collapses them: no process applies\nthis kind, or a process that applies other kinds has not reported this one.\n\n## Relationship to the reload status\n\nApply status covers control-plane resource kinds. `ReloadStatus` covers the\nfile document and stays the only report for a SIGHUP that failed to resolve or\nvalidate it. Neither is derived from the other.\n\nWriting a record is best-effort. On the SIGHUP path it takes its own bounded\ncontext, separate from the budget the reload spends reading the control plane,\nand a failed write is logged rather than turned into a failed reload.\n\n## What the UI shows\n\nThe settings page shows, per resource, the stored version beside each process's\napplied version, error, and report time. A process on an older version of a\nrestart-required resource is pending a restart. A process with an error is\nfailed, with the error beside it. A process whose record is stale is unknown,\nnever current.\n\n## A record outlives the process that wrote it\n\nRecords are replaced in place, so a process that applies a version and then\nstops would otherwise leave a record asserting that version indefinitely, and\nthe settings page would show a dead process as current.\n`docs/prds/status-health-surface.md` faced the same choice and stamped the start\nof a poll pass so a wedged poller reads as stale rather than healthy.\n\nSettled: the re-stamp above. The alternatives considered were\ncarrying the instance identity generated at process start, which detects a\nrestart but not a death, and accepting the limitation with the page saying so.\n\n## Verification\n\n- A process that applies a resource writes a record carrying that version.\n- A process whose apply fails validation writes a record carrying the error and\n no new version.\n- A reload that fails leaves the previously applied version recorded.\n- A record written under a process name outside the shared constant fails the\n test binding writer to reader.\n- A task-scoped credential is refused by both RPCs.\n- A failed report does not fail the reload that produced it.\n- A record that stops being re-stamped renders as unknown rather than current.\n\n## Not included\n\n- History of past applies. Records are current state; resource history already\n records what was written.\n- Checking a live replacement before switching to it. That is\n `archie-core-nwa0`, which reports its failures through this surface.\n- Any change to how a resource is applied.\n" + "body": "# Control-plane apply status\n\n**Status:** Approved\n**Date:** 2026-09-21\n**Tracking:** `archie-core-pskb`, and `archie-core-nwa0` which depends on it\n\n## Outcome\n\nAn operator who edits a control-plane resource can see which processes are\nrunning that version, which are still on an older one, which rejected it, and\nwhich have stopped reporting.\n\n`docs/prds/runtime-control-plane.md` requires this and does not say where the\nreport goes. This decides that.\n\n## Where the report goes\n\nApply status is a State Store contract surface. `ControlPlaneService` is the\nsurface for editing a resource, and a status report is not an edit. The\nprecedent is the config snapshot pair: a process publishes what it knows, the\nUI process reads it, and both RPCs are administrative, so a task-scoped grant\nreaches neither. That denial needs no new rule: `authorizesTaskScopedCall`\nnames three methods and denies everything else.\n\n`docs/prds/state-store-contract.md` gains the pair in its inventory, and its\nRPC count at two places moves from 45 to 47.\n\nThe read returns all records as a list, including an empty one. The\n`found=false` convention for absent single rows does not apply.\n\n## What is reported\n\nOne record per process and resource kind, replaced in place. It holds the\nprocess name, the resource kind, the version applied, the error if the process\ncould not apply it, and the time it reported.\n\nEach process re-stamps its records every 30 seconds. A record is stale once it\nis older than 90 seconds, and a stale record means the process that wrote it is\nno longer running. The report time is therefore a liveness signal, not only a\nwrite timestamp.\n\nA process reports at the point it applies a resource, so the record says what\nhappened rather than what was intended. Nothing probes.\n\nProcess names come from one shared constant naming `archied`, `archie-gateway`\nand `archie-messaging`. A deployment runs at most one of each. Writer and\nreader are bound to that constant by test, because a drifted name is\nindistinguishable from a process that never reported.\n\n## Which processes report\n\n- `archied` and `archie-gateway` apply every restart-required kind through\n `boot.runtimeConfig`, at boot and on SIGHUP.\n- `archied` also applies the live kinds it watches, and reports each update it\n applies.\n- `archie-messaging` applies channel settings at startup and reports that kind.\n\n## What can be reported, and what cannot\n\nA process reaches its apply point only after the State Store has answered it.\nAt startup, when the State Store is unreachable, the process exits or refuses\nto start and writes no record. A live watcher that loses its established\ncontrol-plane stream reports the connection failure against its last applied\nversion while it retries. Re-establishing the stream clears that outage record\nwithout clearing a versioned document rejection. A transport failure is not a\ndocument rejection; it reports that live delivery is unavailable.\n\nAbsence has two meanings and the UI never collapses them: no process applies\nthis kind, or a process that applies other kinds has not reported this one.\n\n## Relationship to the reload status\n\nApply status covers control-plane resource kinds. `ReloadStatus` covers the\nfile document and stays the only report for a SIGHUP that failed to resolve or\nvalidate it. Neither is derived from the other.\n\nWriting a record is best-effort. On the SIGHUP path it takes its own bounded\ncontext, separate from the budget the reload spends reading the control plane,\nand a failed write is logged rather than turned into a failed reload.\n\n## What the UI shows\n\nThe settings page shows, per resource, the stored version beside each process's\napplied version, error, and report time. A process on an older version of a\nrestart-required resource is pending a restart. A process with an error is\nfailed, with the error beside it. A process whose record is stale is unknown,\nnever current.\n\n## A record outlives the process that wrote it\n\nRecords are replaced in place, so a process that applies a version and then\nstops would otherwise leave a record asserting that version indefinitely, and\nthe settings page would show a dead process as current.\n`docs/prds/status-health-surface.md` faced the same choice and stamped the start\nof a poll pass so a wedged poller reads as stale rather than healthy.\n\nSettled: the re-stamp above. The alternatives considered were\ncarrying the instance identity generated at process start, which detects a\nrestart but not a death, and accepting the limitation with the page saying so.\n\n## Verification\n\n- A process that applies a resource writes a record carrying that version.\n- A process whose apply fails validation writes a record carrying the error and\n no new version.\n- A live watch outage is reported against the last applied version until the\n stream is re-established; re-establishment preserves any outstanding\n versioned rejection.\n- A reload that fails leaves the previously applied version recorded.\n- A record written under a process name outside the shared constant fails the\n test binding writer to reader.\n- A task-scoped credential is refused by both RPCs.\n- A failed report does not fail the reload that produced it.\n- A record that stops being re-stamped renders as unknown rather than current.\n\n## Not included\n\n- History of past applies. Records are current state; resource history already\n records what was written.\n- Checking a live replacement before switching to it. That is\n `archie-core-nwa0`, which reports its failures through this surface.\n- Any change to how a resource is applied.\n" }, { "url": "/dev/prds/curator-sampler-wave1/", diff --git a/docs/prds/control-plane-apply-status.md b/docs/prds/control-plane-apply-status.md index 5424b0e45..44d82bf06 100644 --- a/docs/prds/control-plane-apply-status.md +++ b/docs/prds/control-plane-apply-status.md @@ -58,10 +58,12 @@ indistinguishable from a process that never reported. ## What can be reported, and what cannot A process reaches its apply point only after the State Store has answered it. -When the State Store is unreachable, the process exits or refuses to start and -writes no record. An unreachable store is never reported as a failure by the -process it affected. A record carries the other failure: the store answered, -and the value would not validate. +At startup, when the State Store is unreachable, the process exits or refuses +to start and writes no record. A live watcher that loses its established +control-plane stream reports the connection failure against its last applied +version while it retries. Re-establishing the stream clears that outage record +without clearing a versioned document rejection. A transport failure is not a +document rejection; it reports that live delivery is unavailable. Absence has two meanings and the UI never collapses them: no process applies this kind, or a process that applies other kinds has not reported this one. @@ -101,6 +103,9 @@ restart but not a death, and accepting the limitation with the page saying so. - A process that applies a resource writes a record carrying that version. - A process whose apply fails validation writes a record carrying the error and no new version. +- A live watch outage is reported against the last applied version until the + stream is re-established; re-establishment preserves any outstanding + versioned rejection. - A reload that fails leaves the previously applied version recorded. - A record written under a process name outside the shared constant fails the test binding writer to reader. diff --git a/internal/app/archied/control_plane.go b/internal/app/archied/control_plane.go index 624333292..d182664d1 100644 --- a/internal/app/archied/control_plane.go +++ b/internal/app/archied/control_plane.go @@ -115,41 +115,45 @@ func (b *boot) startWorkflowExecutionSettings(ctx context.Context) error { if err != nil { return err } + var lastApplyErr, streamErr error + open := func(ctx context.Context, after int64) (<-chan controlplane.AppliedSettings, error) { + updates, err := b.controlPlane.WatchWorkflowExecutionSettings(ctx, after) + if err == nil { + b.applyStatus.Report(ctx, controlplane.WorkflowExecutionSettingsKind, after, lastApplyErr) + } + return updates, err + } go keepWatch(ctx, b.log, controlplane.WorkflowExecutionSettingsKind, version, updates, - b.controlPlane.WatchWorkflowExecutionSettings, + open, waitFor, func(update controlplane.AppliedSettings) int64 { return update.Version }, func(update controlplane.AppliedSettings) { if update.Err != nil { - // A refused update arrives here, not through - // applyWorkflowExecutionSettings: controlplane.Client decodes every - // document it is handed, so a stored value this process cannot run - // never reaches the apply call. Report it against the version it - // came from -- the reporter keeps the version it last applied -- - // so the settings page shows the refusal instead of a version this - // process never ran (docs/prds/control-plane-apply-status.md). - // - // A stream failure arrives the same way and must not be reported: - // it carries no version at all, because a Recv error is the store - // not answering rather than a document, and versions start at 1 - // (store.PutResource). The rule is - // docs/prds/control-plane-apply-status.md, "What can be reported, - // and what cannot": a process reaches its apply point only after - // the State Store has answered it, and an unreachable store is - // never reported as a failure by the process it affected. The - // record would not stick of itself -- Report overwrites it on the - // next applied version -- but a store that never changes again - // leaves the one stamp standing, and the process must not write it - // at all. Reporting it would also overwrite the text of a standing - // refusal, which is the one message that says what to fix. The - // reconnect is the loop's business, not the record's. + // A refused document carries its version, while a stream failure + // does not. Keep a refusal visible across a transport outage; when + // there is no outstanding refusal, the end callback records the + // outage until a watch is re-established. if update.Version > 0 { + lastApplyErr = update.Err b.applyStatus.Report(ctx, controlplane.WorkflowExecutionSettingsKind, update.Version, update.Err) + } else { + streamErr = update.Err } b.log.Error("workflow execution settings watch failed", "err", update.Err) return } - _ = b.applyWorkflowExecutionSettings(ctx, update.Settings, update.Version) + lastApplyErr = b.applyWorkflowExecutionSettings(ctx, update.Settings, update.Version) + }, + func() { + if lastApplyErr != nil { + return + } + if streamErr == nil { + streamErr = fmt.Errorf("control-plane watch stream ended") + } + b.applyStatus.Report(ctx, controlplane.WorkflowExecutionSettingsKind, 0, + fmt.Errorf("workflow execution settings watch unavailable: %w", streamErr)) + streamErr = nil }) return nil } diff --git a/internal/app/archied/control_plane_live_apply_test.go b/internal/app/archied/control_plane_live_apply_test.go index d97f89bbb..18aeb3fb0 100644 --- a/internal/app/archied/control_plane_live_apply_test.go +++ b/internal/app/archied/control_plane_live_apply_test.go @@ -132,7 +132,8 @@ func (*settingsWatchStub) Command(context.Context, *pb.CommandRequest, ...grpc.C panic("unexpected Command") } -func (s *settingsWatchStub) Watch(context.Context, *pb.WatchRequest, ...grpc.CallOption) (grpc.ServerStreamingClient[pb.WatchResponse], error) { +func (s *settingsWatchStub) Watch(ctx context.Context, request *pb.WatchRequest, _ ...grpc.CallOption) (grpc.ServerStreamingClient[pb.WatchResponse], error) { + _ = request responses := make([]*pb.WatchResponse, 0, len(s.documents)) for i, document := range s.documents { responses = append(responses, &pb.WatchResponse{Resource: &pb.Resource{ @@ -141,7 +142,7 @@ func (s *settingsWatchStub) Watch(context.Context, *pb.WatchRequest, ...grpc.Cal ValueJson: []byte(document), }}) } - return &watchStreamStub{responses: responses}, nil + return &watchStreamStub{responses: responses, done: ctx.Done()}, nil } // watchStreamStub is the client end of a Watch stream. Recv is the only method @@ -150,6 +151,7 @@ func (s *settingsWatchStub) Watch(context.Context, *pb.WatchRequest, ...grpc.Cal type watchStreamStub struct { grpc.ClientStream responses []*pb.WatchResponse + done <-chan struct{} next int // onDeliver, when set, runs as the stream delivers each response, so a // stub standing in for the store can store what it just delivered. @@ -158,7 +160,11 @@ type watchStreamStub struct { func (s *watchStreamStub) Recv() (*pb.WatchResponse, error) { if s.next >= len(s.responses) { - return nil, io.EOF + if s.done == nil { + return nil, io.EOF + } + <-s.done + return nil, context.Canceled } response := s.responses[s.next] s.next++ diff --git a/internal/app/archied/control_plane_watch.go b/internal/app/archied/control_plane_watch.go index e7b99136d..46118f1d8 100644 --- a/internal/app/archied/control_plane_watch.go +++ b/internal/app/archied/control_plane_watch.go @@ -52,12 +52,10 @@ const controlPlaneWatchHealthyMultiple = 2 // re-establishes it in this same goroutine until ctx ends. // // A control-plane watch stream is not permanent. The State Store restarts, the -// connection drops, the server closes the stream -- and nothing on the client -// side reopens it. A watch that ends on its first end stops applying that kind -// for the rest of the process's life: every later stored change is silently -// never applied while the last apply-status record keeps re-stamping, so the -// settings page reads the process as current, and the process recovers only by -// restarting (archie-core-yrmr). +// connection drops, or the server closes the stream. keepWatch reopens it in +// place; workflow-execution-settings also reports the gap so apply status does +// not keep re-stamping the previous successful version as current while no +// live updates can arrive (archie-core-1050). // // One goroutine per kind, reconnecting in place: a long outage does not stack // watchers. Reopening resumes after the last version an update carried, so a @@ -129,6 +127,7 @@ func keepWatch[T any]( wait func(ctx context.Context, d time.Duration) bool, versionOf func(update T) int64, deliver func(update T), + onEnded ...func(), ) { stream, backoff := first, controlPlaneWatchRetryMin for { @@ -146,6 +145,9 @@ func keepWatch[T any]( if ctx.Err() != nil { return } + if len(onEnded) > 0 { + onEnded[0]() + } backoff = retryDelay(backoff, attemptHealthy(progressed, time.Since(started), backoff)) log.Warn("control plane watch stream ended; reconnecting", "kind", kind, "after_version", after, "retry_in", backoff) diff --git a/internal/app/archied/control_plane_watch_test.go b/internal/app/archied/control_plane_watch_test.go index 41b53bb20..e324c3a1a 100644 --- a/internal/app/archied/control_plane_watch_test.go +++ b/internal/app/archied/control_plane_watch_test.go @@ -219,22 +219,22 @@ func TestWorkflowExecutionSettingsWatchIsReEstablishedAfterTheStreamEnds(t *test t.Errorf("watch resumed after %v, want the boot version then the last version handled", got) } - // Boot's own apply, then the update the re-established stream - // delivered. Neither way the stream ends writes a record of its own: - // a stream the server closed emits no update at all (controlplane - // .Client sends one only for a Recv error), and a Recv failure - // carries no version, which is the unreachable store the process must - // never report as a failure of its own - // (docs/prds/control-plane-apply-status.md, "What can be reported, and - // what cannot"). - records := status.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 2) - if len(records) != 2 { - t.Fatalf("apply status = %+v, want the boot apply and the recovered one alone", records) + // Boot applies version 1, the ended stream reports the outage, the + // reopened watch clears it, and the delivered version 2 is applied. + records := status.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 4) + if len(records) != 4 { + t.Fatalf("apply status = %+v, want boot, outage, reconnect, and update reports", records) } if records[0].AppliedVersion != 1 || records[0].Error != "" { t.Errorf("boot report = %+v, want a clean apply of version 1", records[0]) } - last := records[len(records)-1] + if records[1].AppliedVersion != 1 || !strings.Contains(records[1].Error, "watch unavailable") { + t.Errorf("outage report = %+v, want version 1 with a watch outage", records[1]) + } + if records[2].AppliedVersion != 1 || records[2].Error != "" { + t.Errorf("reconnect report = %+v, want the outage cleared at version 1", records[2]) + } + last := records[3] if last.AppliedVersion != 2 || last.Error != "" { t.Errorf("last apply status = %+v, want the recovered apply of version 2 with the failure cleared", last) } @@ -256,7 +256,7 @@ func TestWorkflowExecutionSettingsWatchIsReEstablishedAfterTheStreamEnds(t *test // point), applystatus.Reporter keeps a standing error and re-stamps it forever, // and the dashboard renders any non-empty error as failed. The process reads as // failed for the rest of its life, after one blip. -func TestWatchTransportFailureIsNotReportedAsAFailedApply(t *testing.T) { +func TestWatchTransportFailureIsReportedUntilTheWatchRecovers(t *testing.T) { t.Parallel() b, recorder := newLiveApplyBoot(t, fileConfig()) stub := &watchStub{ @@ -285,23 +285,25 @@ func TestWatchTransportFailureIsNotReportedAsAFailedApply(t *testing.T) { // the loop delivers what ended the stream, then reopens. stub.awaitWatchCalls(t, controlplane.WorkflowExecutionSettingsKind, 2) - records := recorder.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 1) - if len(records) != 1 { - t.Fatalf("apply status = %+v, want the boot apply alone: an unreachable store writes no record", records) + records := recorder.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 3) + if len(records) != 3 { + t.Fatalf("apply status = %+v, want boot, outage, and recovery reports", records) } - if last := records[0]; last.AppliedVersion != 1 || last.Error != "" { - t.Errorf("apply status = %+v, want the clean apply of version 1 still standing", last) + if records[0].AppliedVersion != 1 || records[0].Error != "" { + t.Errorf("boot apply status = %+v, want the clean apply of version 1", records[0]) + } + if records[1].AppliedVersion != 1 || !strings.Contains(records[1].Error, "watch unavailable") { + t.Errorf("outage apply status = %+v, want version 1 with a watch outage", records[1]) + } + if records[2].AppliedVersion != 1 || records[2].Error != "" { + t.Errorf("recovered apply status = %+v, want version 1 with the outage cleared", records[2]) } } -// TestWatchTransportFailureDoesNotOverwriteARefusal is the other half of the -// same rule, and the reason the branch is on the version rather than on the -// error: a refused document arrives carrying the version it came from, so it is -// the store answering and must be reported; a transport failure carries no -// version and must not. Reporting the transport failure against the version -// still live overwrites the refusal's text, so a validation error against -// version 2 reads "control-plane unavailable: ..." after any blip, and the -// operator loses the one message that says what to fix. +// TestWatchTransportFailureDoesNotOverwriteARefusal keeps the more useful +// versioned rejection visible when a stream drops afterward. Reconnection +// restores that rejection instead of clearing it as though a valid update had +// arrived. func TestWatchTransportFailureDoesNotOverwriteARefusal(t *testing.T) { t.Parallel() b, recorder := newLiveApplyBoot(t, fileConfig()) @@ -332,18 +334,18 @@ func TestWatchTransportFailureDoesNotOverwriteARefusal(t *testing.T) { stub.awaitWatchCalls(t, controlplane.WorkflowExecutionSettingsKind, 2) - records := recorder.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 2) - if len(records) != 2 { - t.Fatalf("apply status = %+v, want the boot apply and the refusal alone", records) + records := recorder.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 3) + if len(records) != 3 { + t.Fatalf("apply status = %+v, want boot, refusal, and the preserved refusal after reconnect", records) } - last := records[len(records)-1] + last := records[2] if last.AppliedVersion != 1 { t.Errorf("refusal reported against version %d, want the version still live (1)", last.AppliedVersion) } if !strings.Contains(last.Error, "max model/tool steps must not be negative") { t.Errorf("apply status error = %q, want the refusal's own text", last.Error) } - if strings.Contains(last.Error, "control-plane unavailable") { + if strings.Contains(last.Error, "watch unavailable") { t.Errorf("apply status error = %q, want the transport blip not to have overwritten the refusal", last.Error) } // The refused version is unusable to this build, so the resume point moves @@ -394,13 +396,16 @@ func TestWatchReconnectDoesNotReplayWhatItAlreadyApplied(t *testing.T) { t.Errorf("watch resumed after %v, want the boot version then the last version handled", got) } - // One report for the boot apply and one for the update: a third would be - // the re-served version 1 applied over itself. - records := recorder.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 2) - if len(records) != 2 { - t.Fatalf("apply status = %+v, want the boot apply and the update alone", records) + // The outage and reconnection records sit between boot and version 2; the + // re-served version 1 is still filtered out. + records := recorder.awaitCount(t, controlplane.WorkflowExecutionSettingsKind, 4) + if len(records) != 4 { + t.Fatalf("apply status = %+v, want boot, outage, reconnect, and version 2", records) } - last := records[len(records)-1] + if records[1].AppliedVersion != 1 || !strings.Contains(records[1].Error, "watch unavailable") { + t.Errorf("outage report = %+v, want an outage against version 1", records[1]) + } + last := records[3] if last.AppliedVersion != 2 || last.Error != "" { t.Errorf("last apply status = %+v, want the recovered apply of version 2", last) }