fix(archied): re-establish the control-plane watch streams after they end

The settings and persona watches were launched once, at boot, and ended on the
first stream failure: the goroutine returned, so every later stored change for
that kind was silently never applied and the process recovered only by
restarting (archie-core-yrmr).

Both watches now run one keepWatch goroutine per kind, which re-establishes the
stream in place after a bounded backoff, resuming after the last version an
update carried and stopping on ctx cancellation -- which the loop watches for
itself, so a stream that never closes cannot hold up shutdown. A first open that
fails still fails the boot that asked for it.

The reconnect delay is decided by the attempt that just ended, and only by that
(attemptHealthy, retryDelay): an attempt is healthy iff it delivered an update
this watch had not handled, or it outlived a window that is a strict multiple of
the delay it would otherwise be charged. A healthy attempt returns the delay to
the minimum, a failed one doubles it up to the cap, and the window moves with
the ladder instead of sitting at its floor. The acceptance of a reopen is the
wrong unit: gRPC takes the stream before the handler reads anything, and
controlplane.Server.Watch answers a store it cannot read by ending the stream,
so during the outage this reconnect exists for every reopen is accepted and
every stream ends carrying nothing -- measured with the real server and client
over bufconn, 4.0 watch RPCs a second across 2s of store outage with retry_in
pinned at 250ms when that acceptance is read as health. A window fixed at the
minimum is the same defect for the likelier shape of outage: a store that has
stopped answering does not error at once, it stalls until its read times out,
and an attempt that takes longer to fail than the minimum outlives that window
on every attempt -- measured in the end-to-end outage test before the change, a
300ms stall left the ladder flat at 250ms/250ms/250ms/250ms while the store
never answered a read. Both shapes are charged now, and the outage test drives
three of them (an immediate error, a 300ms stall, and a stall as long as the
delay that rung would be charged) up four rungs of the ladder, each logged retry
held to the delay actually waited.

The wait is injected (wait, production passes waitFor), because the ladder was
only ever asserted through the retry_in attribute: a loop that logged one delay
and waited another would have reported a growing ladder and reopened on a
schedule of its own. TestKeepWatchWaitsTheDelayItCharged reads the delays back
as values and asserts the ladder, and the outage test's gaps catch a wait that
returns without waiting. The retargeted reset test is joined by
TestKeepWatchChargesAReopenTheStoreMerelyAccepted: a delay reset by the
acceptance of a reopen charges every attempt of the outage from the minimum, and
nothing pinned that before.

The watch returns on cancellation without draining what the producer left for
it, so controlplane.Client's producers stop on the same context instead of
parking on a send no reader will ever take -- reproduced 40/40 times as a
goroutine still blocked on an undelivered send after keepWatch returned, one
goroutine and one channel leaked per kind at every exit. That cross-package
coupling is load-bearing in both directions and is stated as such; the two
producers now share the one stream reader that carries it.

A Recv failure is the store being unreachable, not a document this process
refused, and docs/prds/control-plane-apply-status.md ("What can be reported, and
what cannot") is what forbids reporting it as a failure of the process it
affected. The record is now written only for an update that carries a version,
which is what controlplane.Client emits for a document that would not validate
and never for a transport failure. Reporting the transport failure would also
overwrite the text of a standing refusal, replacing the one message that says
what to fix with "control-plane unavailable". The record is not un-clearable of
itself -- Report overwrites it on the next applied version -- but a store that
never changes again leaves the one stamp standing, and the process must not
write it at all. That emission is the whole of the failure path and now has the
test it lacked, from the producer's side:
TestWatchClientsDeliverARecvFailureAsAVersionlessUpdate pins that a Recv failure
is handed over carrying the mapped error and no version, and that a stream the
server closed and one that ended with the watch's context are not updates at all.

The refused version is deliberately not re-offered: the resume point moves past
it, so an edit only a newer build can decode cannot become a hot loop. Both ends
of that contract are pinned now: the watch stub filters its scripted stream by
AfterVersion as the server does, and controlplane.Server.Watch's own filter has
a test for the first time. The apply-status reporter's 30s re-stamp is unchanged
(TestRestampKeepsAReportedFailure).
This commit is contained in:
sam committed 2026-09-22 01:58:03 +10:00
1 parent 1a0fb39617
commit b8ec491dd3
9 files changed
+2067 -41

No files matched your search

+28 -7
View File
@@ -85,6 +85,11 @@ func (b *boot) reloadConfig(ctx context.Context, doc *configuration.Document) er
return nil
}
// startWorkflowExecutionSettings loads the stored limits, applies them and
// keeps the stream that delivers later ones established. The first read and
// the first stream are synchronous, so a control plane that cannot be reached
// fails the boot that asked for it; after that the watch does not return, it
// reconnects (see keepWatch).
func (b *boot) startWorkflowExecutionSettings(ctx context.Context) error {
settings, version, err := b.controlPlane.WorkflowExecutionSettings(ctx)
if err != nil {
@@ -97,8 +102,11 @@ func (b *boot) startWorkflowExecutionSettings(ctx context.Context) error {
if err != nil {
return err
}
go func() {
for update := range updates {
go keepWatch(ctx, b.log, controlplane.WorkflowExecutionSettingsKind, version, updates,
b.controlPlane.WatchWorkflowExecutionSettings,
waitFor,
func(update controlplane.AppliedSettings) int64 { return update.Version },
func(update controlplane.AppliedSettings) {
if update.Err != nil {
// A refused update arrives here, not through
// applyWorkflowExecutionSettings: controlplane.Client decodes every
@@ -108,15 +116,28 @@ func (b *boot) startWorkflowExecutionSettings(ctx context.Context) error {
// so the settings page shows the refusal instead of a version this
// process never ran (docs/prds/control-plane-apply-status.md).
//
// The stream is not re-established after this point; that gap is
// archie-core-yrmr, not this apply path.
b.applyStatus.Report(ctx, controlplane.WorkflowExecutionSettingsKind, update.Version, update.Err)
// A stream failure arrives the same way and must not be reported:
// it carries no version at all, because a Recv error is the store
// not answering rather than a document, and versions start at 1
// (store.PutResource). The rule is
// docs/prds/control-plane-apply-status.md, "What can be reported,
// and what cannot": a process reaches its apply point only after
// the State Store has answered it, and an unreachable store is
// never reported as a failure by the process it affected. The
// record would not stick of itself -- Report overwrites it on the
// next applied version -- but a store that never changes again
// leaves the one stamp standing, and the process must not write it
// at all. Reporting it would also overwrite the text of a standing
// refusal, which is the one message that says what to fix. The
// reconnect is the loop's business, not the record's.
if update.Version > 0 {
b.applyStatus.Report(ctx, controlplane.WorkflowExecutionSettingsKind, update.Version, update.Err)
}
b.log.Error("workflow execution settings watch failed", "err", update.Err)
return
}
_ = b.applyWorkflowExecutionSettings(ctx, update.Settings, update.Version)
}
}()
})
return nil
}
@@ -281,8 +281,11 @@ func TestLiveExecutionSettingsUpdateIsRefusedWhenTheSnapshotIsNotRunnable(t *tes
// process cannot run arrives as a watch error, never as invalid settings: the
// watch branch must report that refusal against the version still live instead
// of only logging it, or the settings page shows a live resource as healthy
// while nothing has been applied. (That the stream is not re-established
// afterwards is archie-core-yrmr, not this apply path.)
// while nothing has been applied. (The stream does not stay ended: the watch
// re-establishes it, which control_plane_watch_test.go covers. This stub
// answers every re-established stream with the same documents, so this test
// reads the record the refusal wrote rather than counting how often it was
// written.)
func TestLiveExecutionSettingsRefusalIsReportedThroughTheClient(t *testing.T) {
tests := []struct {
name string
+203
View File
@@ -0,0 +1,203 @@
package archied
import (
"context"
"log/slog"
"time"
)
// controlPlaneWatchRetry is how long a watch waits before re-establishing a
// stream that ended. The minimum is the delay a healthy attempt leaves behind,
// and the delay a failed attempt doubles; the cap bounds the delay, never the
// retrying: a watch reconnects for as long as the process lives.
//
// A control plane that is down is retried slowly enough that it is not a
// request loop, and that holds for every shape the outage takes because the
// window an attempt is judged against is derived from the delay it would
// otherwise be charged (attemptHealthy): a store that errors at once and a
// store that stalls until its read times out are both charged, so both climb
// the ladder instead of sitting at the minimum.
const (
controlPlaneWatchRetryMin = 250 * time.Millisecond
controlPlaneWatchRetryMax = 30 * time.Second
)
// controlPlaneWatchHealthyMultiple is how many times the delay an attempt would
// otherwise be charged an attempt that delivered nothing must outlive to be
// read as a store that answered and had nothing to say. It is strictly more
// than one, so that window is always wider than the charge itself: a window
// equal to the delay would read a store that fails as slowly as that delay as
// healthy on every attempt, which is the fixed-rate loop this rule exists to
// charge.
const controlPlaneWatchHealthyMultiple = 2
// keepWatch delivers every update from first and, when that stream ends,
// re-establishes it in this same goroutine until ctx ends.
//
// A control-plane watch stream is not permanent. The State Store restarts, the
// connection drops, the server closes the stream -- and nothing on the client
// side reopens it. A watch that ends on its first end stops applying that kind
// for the rest of the process's life: every later stored change is silently
// never applied while the last apply-status record keeps re-stamping, so the
// settings page reads the process as current, and the process recovers only by
// restarting (archie-core-yrmr).
//
// One goroutine per kind, reconnecting in place: a long outage does not stack
// watchers. Reopening resumes after the last version an update carried, so a
// reconnect neither replays a document this process has already handled over a
// newer one nor skips one it has not. A transport failure carries no version
// and so leaves the resume point alone. A document the client refused to decode
// does carry one -- controlplane.Client decodes what the store sends, and the
// store answered with that version -- and the resume point moves past it
// deliberately: that version is unusable to this build, and resuming before it
// would re-offer it on every reconnect for the life of the process. Only a
// cancellation stops the loop, and the loop watches for it itself, so shutdown
// is prompt however long the current backoff is and whether or not the stream
// is still open.
//
// The reconnect delay is decided by the attempt that just ended, and only by
// that (attemptHealthy, retryDelay): a healthy attempt returns it to the
// minimum, and a failed attempt doubles it up to the cap. An attempt is not
// healthy because a reopen was accepted. A control plane accepts the stream
// before it reads anything -- that is how gRPC hands a request to its handler
// -- and controlplane.Server.Watch answers a store it cannot read by returning
// an error, so during the outage this loop exists for every reopen is accepted
// and every stream ends carrying nothing: judged on the acceptance alone the
// delay stays at the minimum and the watch becomes a 4 RPCs/second request
// loop.
//
// The rule is one sentence: an attempt is healthy iff it delivered an update
// this watch had not handled, or it outlived a window that is a strict multiple
// of the delay it would otherwise be charged. The window is derived from that
// charge rather than fixed at the minimum, which is what makes the rule reach
// every shape of outage: a store that has stopped answering does not error at
// once, it stalls until its read times out, and an attempt that took longer to
// fail than a window fixed at the minimum would be read as a healthy store on
// every attempt, leaving the delay one minimum per timeout for as long as the
// store is gone.
//
// open, versionOf and deliver are the kind's halves. open re-establishes the
// stream -- the caller opens the first one itself, so that a control plane
// which cannot be watched at all still fails the boot that asked for it.
// versionOf reads the resource version an update carries. deliver applies the
// update, including reporting the error it carries, which is how a refused
// document reaches the settings page.
//
// wait is the delay between a stream ending and the reopen that follows it, and
// production passes waitFor. It is injected so that the delay the loop actually
// waited is observable: the retry_in log line states the delay the loop charged
// and would agree with a loop that waited some other multiple of it.
func keepWatch[T any](
ctx context.Context,
log *slog.Logger,
kind string,
after int64,
first <-chan T,
open func(ctx context.Context, afterVersion int64) (<-chan T, error),
wait func(ctx context.Context, d time.Duration) bool,
versionOf func(update T) int64,
deliver func(update T),
) {
stream, backoff := first, controlPlaneWatchRetryMin
for {
started, progressed := time.Now(), false
for {
update, ok := nextUpdate(ctx, stream)
if !ok {
break
}
deliver(update)
if version := versionOf(update); version > after {
after, progressed = version, true
}
}
if ctx.Err() != nil {
return
}
backoff = retryDelay(backoff, attemptHealthy(progressed, time.Since(started), backoff))
log.Warn("control plane watch stream ended; reconnecting",
"kind", kind, "after_version", after, "retry_in", backoff)
for {
if !wait(ctx, backoff) {
return
}
reopened, err := open(ctx, after)
if err == nil {
stream = reopened
break
}
// A reopen the control plane refused is the store still not
// answering, and is charged exactly as any other attempt that ends
// that way.
backoff = retryDelay(backoff, false)
log.Warn("control plane watch could not be re-established",
"kind", kind, "after_version", after, "retry_in", backoff, "err", err)
}
}
}
// attemptHealthy is what one attempt that ended makes of the reconnect delay.
// An attempt is healthy iff it delivered an update this watch had not handled
// -- the store answered, whatever the stream did afterwards -- or it outlived
// controlPlaneWatchHealthyWindow, which is a store that had nothing to say and
// was still there. An attempt that did neither delivered nothing and died
// inside the window it would have been charged: the stream was accepted and the
// store was not answering, which is a failed attempt and the only thing the
// delay grows on.
func attemptHealthy(progressed bool, openFor, backoff time.Duration) bool {
return progressed || openFor >= controlPlaneWatchHealthyWindow(backoff)
}
// controlPlaneWatchHealthyWindow is how long an attempt that delivered nothing
// must stay open to count as a store that answered and had nothing to say: a
// strict multiple of the delay the attempt would otherwise be charged. The
// window moves with the ladder, so the charge reaches a store that fails slowly
// at every rung of it, which a window fixed at the minimum does not.
func controlPlaneWatchHealthyWindow(backoff time.Duration) time.Duration {
return controlPlaneWatchHealthyMultiple * retryDelay(backoff, false)
}
// retryDelay is the delay the attempt that just ended leaves behind: a healthy
// attempt returns it to the minimum, so an outage is charged for itself and not
// for the quiet time that preceded it, and a failed attempt doubles it up to
// the cap, so a store that stays gone is retried ever more slowly.
func retryDelay(current time.Duration, healthy bool) time.Duration {
if healthy {
return controlPlaneWatchRetryMin
}
return min(2*current, controlPlaneWatchRetryMax)
}
// nextUpdate receives the next update from stream, reporting false when the
// stream ended or when ctx ended first. A watch's stream is closed by
// controlplane.Client when the stream's own context ends, but shutdown must not
// rest on that cross-package coupling: a stream that never closed would
// otherwise hold the watch -- and the process that owns it -- past the
// cancellation that asked it to stop. The coupling that is load-bearing runs
// the other way: this watch returns on cancellation without draining what the
// producer has left for it, so controlplane.Client's producer must stop on the
// same context instead of parking on a send no reader will ever take (its send
// guard is pinned by TestKeepWatchLeavesNoProducerParkedOnAnUndeliveredUpdate).
func nextUpdate[T any](ctx context.Context, stream <-chan T) (T, bool) {
select {
case <-ctx.Done():
var zero T
return zero, false
case update, ok := <-stream:
return update, ok
}
}
// waitFor is the wait production injects into keepWatch. It waits for d,
// reporting false when ctx ended first: a watch's backoff must never delay the
// shutdown of the process that owns it.
func waitFor(ctx context.Context, d time.Duration) bool {
timer := time.NewTimer(d)
defer timer.Stop()
select {
case <-ctx.Done():
return false
case <-timer.C:
return true
}
}
@@ -0,0 +1,188 @@
package archied
import (
"context"
"errors"
"net"
"slices"
"sync"
"testing"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/test/bufconn"
"github.com/samcharles93/archie-core/internal/app/controlplane"
pb "github.com/samcharles93/archie-core/internal/contracts/controlplane/v1"
"github.com/samcharles93/archie-core/internal/infrastructure/workflowsteps"
"github.com/samcharles93/archie-core/internal/store"
)
// outageStore is the State Store's resource half with a switch that makes every
// read fail after taking stall to do it. It stands in for the outage the
// reconnect exists for: the control plane takes the stream and then answers its
// first read with mapError, so the client is handed a stream that ends carrying
// no document -- while every reopen is accepted, because gRPC accepts before
// the handler runs.
//
// The stall is the other shape that outage takes, and the shape a store that has
// stopped answering actually has: it does not error at once, it stops reading
// and the call times out. An attempt that ends that way delivered nothing and
// lasted longer than the minimum retry interval, which is a failed attempt
// whichever way the store failed it.
type outageStore struct {
*store.Store
mu sync.Mutex
failing bool
stall time.Duration
}
// down starts the outage, which takes stall to surface.
func (s *outageStore) down(stall time.Duration) {
s.mu.Lock()
defer s.mu.Unlock()
s.failing, s.stall = true, stall
}
func (s *outageStore) Resource(ctx context.Context, kind string) (store.Resource, error) {
s.mu.Lock()
failing, stall := s.failing, s.stall
s.mu.Unlock()
if failing {
time.Sleep(stall)
return store.Resource{}, errors.New("state store unreachable")
}
return s.Store.Resource(ctx, kind)
}
// TestWatchRetryGrowsWhileTheStateStoreIsDown is the outage this reconnect
// exists for, end to end: a real controlplane.Server over bufconn, a real
// controlplane.Client, a real store, and the store then stops answering -- in
// each of the shapes it stops answering in, from the store that errors at once
// to the one that takes as long to fail as the delay it would be charged.
//
// The server accepts every reopened stream and fails its first read, so the
// client is handed an update for that failure that carries no version at all:
// an attempt that ends this way delivered nothing, and the reconnect behind it
// doubles the delay. That charge has to reach a store that fails slowly too.
// The delay a slow failure is charged rests on the healthy window being derived
// from the delay the attempt would otherwise be charged, not on the minimum: a
// store that takes 300ms to time out outlived a window pinned at the 250ms
// minimum on every attempt, so the delay never grew -- measured across 3s of
// that shape, 1.7 watch RPCs a second with retry_in flat at 250ms, which is the
// request loop this reconnect exists to avoid, reached by the likelier shape.
//
// The first retry is the minimum in every shape: the boot stream is a healthy
// attempt, because the store answered it for longer than the window that
// attempt would have been charged with before the outage began. The ladder's
// top is reached by the delay alone and is pinned by
// TestReconnectDelayFollowsAttemptHealth, which does not have to wait 30s for it.
func TestWatchRetryGrowsWhileTheStateStoreIsDown(t *testing.T) {
tests := []struct {
name string
stall time.Duration
want []time.Duration
}{
{
name: "the store errors at once",
want: []time.Duration{controlPlaneWatchRetryMin, 2 * controlPlaneWatchRetryMin, 4 * controlPlaneWatchRetryMin},
},
{
name: "the store stalls before it times out",
stall: 300 * time.Millisecond,
want: []time.Duration{
controlPlaneWatchRetryMin, 2 * controlPlaneWatchRetryMin,
4 * controlPlaneWatchRetryMin, 8 * controlPlaneWatchRetryMin,
},
},
{
// The window an attempt is judged against is strictly wider than the
// delay it would be charged with, and this is the shape that says so:
// a failure that takes exactly that delay to surface is still a store
// that did not answer the read.
name: "the store fails as slowly as the delay it would be charged",
stall: 2 * controlPlaneWatchRetryMin,
want: []time.Duration{
controlPlaneWatchRetryMin, 2 * controlPlaneWatchRetryMin,
4 * controlPlaneWatchRetryMin, 8 * controlPlaneWatchRetryMin,
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
database := &outageStore{Store: store.OpenTest(t)}
defer database.Close()
steps, err := workflowsteps.NewManager()
if err != nil {
t.Fatalf("build the workflow step vocabulary: %v", err)
}
server, err := controlplane.NewServer(database, steps)
if err != nil {
t.Fatalf("build the control plane: %v", err)
}
if _, err := server.ImportWorkflowExecutionSettings(t.Context(), bootedSettings); err != nil {
t.Fatalf("seed the stored settings: %v", err)
}
log, retries := newRetryLog()
b, _ := newLiveApplyBoot(t, fileConfig())
b.log = log
b.controlPlane = controlplane.NewClient(dialControlPlane(t, server))
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
if err := b.startWorkflowExecutionSettings(ctx); err != nil {
t.Fatalf("startWorkflowExecutionSettings: %v", err)
}
// The store answers the boot stream for longer than the window that
// attempt would be charged with, which is what makes it the healthy
// attempt the first retry is not charged for.
time.Sleep(controlPlaneWatchHealthyWindow(controlPlaneWatchRetryMin))
database.down(tt.stall)
delays := awaitRetries(t, retries, len(tt.want))
if got := delays[:len(tt.want)]; !slices.Equal(got, tt.want) {
t.Errorf("reconnects waited %v, want %v: the delay is decided by the attempt that just ended, so the healthy boot attempt leaves the %v minimum behind and every attempt of the outage doubles the delay before it. A delay that never grows is this watch back at a request loop: pinned at %v by a healthy window fixed at the minimum, which a store that fails as slowly as this one outlives on every attempt, and pinned at %v by a reset on every reopen the store accepted, which charges each attempt of the outage from the minimum again",
got, tt.want, controlPlaneWatchRetryMin, controlPlaneWatchRetryMin, 2*controlPlaneWatchRetryMin)
}
// Each logged retry is the wait the loop then took. A loop that logged
// one delay and waited another would reopen sooner than it said it
// would, and the attribute alone cannot show that.
for i := 0; i < len(tt.want)-1; i++ {
if gap := retries.gap(i); gap < tt.want[i] {
t.Errorf("the retry logged as %v was followed by the next stream ending %v later, want at least that delay: the loop waits the delay it charged", tt.want[i], gap)
}
}
})
}
}
// dialControlPlane serves server over bufconn and returns the client
// connection a watch dials it through. The transport is real: the stream's
// context is the RPC's, and an unaccepted stream is not something a stub can
// stand in for.
func dialControlPlane(t *testing.T, server pb.ControlPlaneServiceServer) *grpc.ClientConn {
t.Helper()
listener := bufconn.Listen(1 << 20)
grpcServer := grpc.NewServer()
pb.RegisterControlPlaneServiceServer(grpcServer, server)
go func() { _ = grpcServer.Serve(listener) }()
t.Cleanup(func() {
grpcServer.Stop()
_ = listener.Close()
})
conn, err := grpc.NewClient("passthrough:///control-plane",
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) {
return listener.DialContext(ctx)
}))
if err != nil {
t.Fatalf("dial the control plane: %v", err)
}
t.Cleanup(func() { _ = conn.Close() })
return conn
}
File diff suppressed because it is too large. Load diff
+29 -11
View File
@@ -5,6 +5,7 @@ import (
"fmt"
"github.com/samcharles93/archie-core/internal/agentexec"
"github.com/samcharles93/archie-core/internal/app/controlplane"
"github.com/samcharles93/archie-core/internal/config"
"github.com/samcharles93/archie-core/internal/domain/agent"
"github.com/samcharles93/archie-core/internal/gateway"
@@ -34,19 +35,9 @@ func (b *boot) setupChatRuntime(ctx context.Context, cfg config.Config) error {
}
b.personas = gateway.NewPersonaRegistry(nil)
b.applyPersonas(personas, version)
updates, err := b.controlPlane.WatchPersonas(ctx, version)
if err != nil {
if err := b.watchPersonas(ctx, version); err != nil {
return fmt.Errorf("watch personas: %w", err)
}
go func() {
for update := range updates {
if update.Err != nil {
b.log.Error("persona watch failed", "err", update.Err)
return
}
b.applyPersonas(update.Collection, update.Version)
}
}()
}
// ── Operator health surface ──────────────────────────────────────
@@ -75,6 +66,33 @@ func (b *boot) setupChatRuntime(ctx context.Context, cfg config.Config) error {
return nil
}
// watchPersonas keeps the persona stream established for the life of the
// process. The first stream is opened synchronously, so a control plane that
// cannot be watched at all fails the boot that asked for it rather than
// leaving the process running personas it can no longer update; after that the
// watch reconnects instead of ending (see keepWatch).
//
// Persona updates are not reported through apply status -- the personas kind
// has no apply-status wiring, which is archie-core-aj35, not this watch.
func (b *boot) watchPersonas(ctx context.Context, version int64) error {
updates, err := b.controlPlane.WatchPersonas(ctx, version)
if err != nil {
return err
}
go keepWatch(ctx, b.log, controlplane.PersonasKind, version, updates,
b.controlPlane.WatchPersonas,
waitFor,
func(update controlplane.AppliedPersonas) int64 { return update.Version },
func(update controlplane.AppliedPersonas) {
if update.Err != nil {
b.log.Error("persona watch failed", "err", update.Err)
return
}
b.applyPersonas(update.Collection, update.Version)
})
return nil
}
func (b *boot) applyPersonas(collection agent.PersonaCollection, version int64) {
personas := make([]gateway.Persona, 0, len(collection.Personas))
for _, persona := range collection.Personas {
+46 -21
View File
@@ -115,22 +115,12 @@ func (c *Client) WatchWorkflowExecutionSettings(ctx context.Context, afterVersio
if err != nil {
return nil, clientError(err)
}
out := make(chan AppliedSettings)
go func() {
defer close(out)
for {
response, err := stream.Recv()
if err != nil {
if !errors.Is(err, io.EOF) && ctx.Err() == nil {
out <- AppliedSettings{Err: clientError(err)}
}
return
}
settings, err := decodeSettings(response.Resource.ValueJson)
out <- AppliedSettings{Settings: settings, Version: response.Resource.Version, Err: err}
}
}()
return out, nil
return watchUpdates(ctx, stream,
func(resource *pb.Resource) AppliedSettings {
settings, err := decodeSettings(resource.ValueJson)
return AppliedSettings{Settings: settings, Version: resource.Version, Err: err}
},
func(err error) AppliedSettings { return AppliedSettings{Err: err} }), nil
}
type AppliedSettings struct {
@@ -159,22 +149,57 @@ func (c *Client) WatchPersonas(ctx context.Context, afterVersion int64) (<-chan
if err != nil {
return nil, clientError(err)
}
out := make(chan AppliedPersonas)
return watchUpdates(ctx, stream,
func(resource *pb.Resource) AppliedPersonas {
collection, err := decodePersonas(resource.ValueJson)
return AppliedPersonas{Collection: collection, Version: resource.Version, Err: err}
},
func(err error) AppliedPersonas { return AppliedPersonas{Err: err} }), nil
}
// watchUpdates turns one Watch stream into the channel a watch loop reads: every
// document the stream carries, decoded, and then a single update carrying the
// error that ended it -- unless the stream ended cleanly or the context did, in
// which case the channel just closes.
func watchUpdates[T any](
ctx context.Context,
stream grpc.ServerStreamingClient[pb.WatchResponse],
update func(resource *pb.Resource) T,
failed func(err error) T,
) <-chan T {
out := make(chan T)
go func() {
defer close(out)
for {
response, err := stream.Recv()
if err != nil {
if !errors.Is(err, io.EOF) && ctx.Err() == nil {
out <- AppliedPersonas{Err: clientError(err)}
sendUpdate(ctx, out, failed(clientError(err)))
}
return
}
collection, err := decodePersonas(response.Resource.ValueJson)
out <- AppliedPersonas{Collection: collection, Version: response.Resource.Version, Err: err}
if !sendUpdate(ctx, out, update(response.Resource)) {
return
}
}
}()
return out, nil
return out
}
// sendUpdate hands update to the reader of one Watch stream, reporting false
// when ctx ended first. A reader leaves without draining what is left for it:
// the watch that reads these streams returns on cancellation, so a bare send
// parks this producer -- and the stream it holds -- for the rest of the
// process's life, one goroutine and one channel per kind at every exit. That
// coupling is load-bearing: the reader's prompt shutdown rests on this side
// stopping on the same context, and neither half is safe alone.
func sendUpdate[T any](ctx context.Context, out chan<- T, update T) bool {
select {
case out <- update:
return true
case <-ctx.Done():
return false
}
}
func (c *Client) Schedules(ctx context.Context) ([]cronstore.JobSpec, int64, error) {
@@ -0,0 +1,306 @@
package controlplane
import (
"context"
"errors"
"io"
"net"
"testing"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/status"
"google.golang.org/grpc/test/bufconn"
pb "github.com/samcharles93/archie-core/internal/contracts/controlplane/v1"
"github.com/samcharles93/archie-core/internal/domain/workflow"
"github.com/samcharles93/archie-core/internal/store"
)
// scriptedWatchClient is the rpc end of a Watch call whose stream is scripted
// rather than served. These tests are about what the client hands the watch that
// reads it, so the store stays out of them and the transport is a value.
type scriptedWatchClient struct {
streams func() grpc.ServerStreamingClient[pb.WatchResponse]
}
func (scriptedWatchClient) Catalog(context.Context, *pb.CatalogRequest, ...grpc.CallOption) (*pb.CatalogResponse, error) {
panic("Catalog is not part of the watch path")
}
func (scriptedWatchClient) Query(context.Context, *pb.QueryRequest, ...grpc.CallOption) (*pb.QueryResponse, error) {
panic("Query is not part of the watch path")
}
func (scriptedWatchClient) History(context.Context, *pb.HistoryRequest, ...grpc.CallOption) (*pb.HistoryResponse, error) {
panic("History is not part of the watch path")
}
func (scriptedWatchClient) Command(context.Context, *pb.CommandRequest, ...grpc.CallOption) (*pb.CommandResponse, error) {
panic("Command is not part of the watch path")
}
func (c scriptedWatchClient) Watch(context.Context, *pb.WatchRequest, ...grpc.CallOption) (grpc.ServerStreamingClient[pb.WatchResponse], error) {
return c.streams(), nil
}
// scriptedRecvStream is the client end of one Watch stream: Recv replays the
// resources it was given and then returns the scripted error. release, when set,
// is what that error waits behind, so a test can end the stream after the
// watch's own context has already ended.
type scriptedRecvStream struct {
grpc.ClientStream
resources []*pb.Resource
end error
release <-chan struct{}
next int
}
func (s *scriptedRecvStream) Recv() (*pb.WatchResponse, error) {
if s.next < len(s.resources) {
resource := s.resources[s.next]
s.next++
return &pb.WatchResponse{Resource: resource}, nil
}
if s.release != nil {
<-s.release
}
return nil, s.end
}
// receiveUpdate reads the next update the watch was handed, reporting false when
// its stream closed without one.
func receiveUpdate[T any](t *testing.T, updates <-chan T) (T, bool) {
t.Helper()
select {
case update, ok := <-updates:
return update, ok
case <-time.After(5 * time.Second):
t.Fatal("the watch's stream neither delivered an update nor closed")
var zero T
return zero, false
}
}
// assertDeliveredNothing asserts the watch was handed no update at all, which is
// what a stream that did not fail leaves behind: the reconnect is the loop's own
// business, and an update here would be a message about a store that is still
// there.
func assertDeliveredNothing[T any](t *testing.T, updates <-chan T, why string) {
t.Helper()
if update, ok := receiveUpdate(t, updates); ok {
t.Errorf("the watch was handed %+v, want nothing: %s", update, why)
}
}
// assertDeliveredARecvFailure asserts the watch was handed the one update a Recv
// failure is -- carrying the mapped transport failure and no version at all --
// and nothing after it.
func assertDeliveredARecvFailure[T any](t *testing.T, updates <-chan T, version func(T) int64, carried func(T) error) {
t.Helper()
update, ok := receiveUpdate(t, updates)
if !ok {
t.Fatal("the Recv failure was never handed to the watch: the stream just closed, so the reconnect behind it logs nothing and the operator is told nothing")
}
if got := version(update); got != 0 {
t.Errorf("the transport failure carried version %d, want none: a Recv error is the store not answering, not a document", got)
}
if err := carried(update); !errors.Is(err, ErrUnavailable) {
t.Errorf("the transport failure was handed over as %v, want %v", err, ErrUnavailable)
}
if extra, ok := receiveUpdate(t, updates); ok {
t.Errorf("a second update followed the transport failure: %+v", extra)
}
}
// TestWatchClientsDeliverARecvFailureAsAVersionlessUpdate is the producer side
// of the rule every consumer of these streams keys on, which the daemon's own
// tests pin from the other end: a Recv failure is the State Store unreachable,
// so the update the client hands the watch for it carries that error and no
// version at all. Versions start at 1 (store.PutResource), so a watch that
// reports what it is handed reports nothing for a transport failure, and the
// settings page keeps the last apply instead of reading the process as failed
// for as long as the store is gone (docs/prds/control-plane-apply-status.md,
// "What can be reported, and what cannot").
//
// That one update is the whole of the failure path. A producer that closed the
// stream instead would hand the watch a stream that simply ended, which is a
// reconnect with nothing to log and nothing to report -- and every test of the
// watch below this would still pass, because they are handed the update by a
// script rather than by the client.
func TestWatchClientsDeliverARecvFailureAsAVersionlessUpdate(t *testing.T) {
t.Parallel()
brokenStream := func() grpc.ServerStreamingClient[pb.WatchResponse] {
return &scriptedRecvStream{end: status.Error(codes.Unavailable, "state store restarting")}
}
t.Run("workflow execution settings", func(t *testing.T) {
t.Parallel()
client := NewRPCClient(scriptedWatchClient{streams: brokenStream})
updates, err := client.WatchWorkflowExecutionSettings(t.Context(), 4)
if err != nil {
t.Fatalf("WatchWorkflowExecutionSettings: %v", err)
}
assertDeliveredARecvFailure(t, updates,
func(update AppliedSettings) int64 { return update.Version },
func(update AppliedSettings) error { return update.Err })
})
t.Run("personas", func(t *testing.T) {
t.Parallel()
client := NewRPCClient(scriptedWatchClient{streams: brokenStream})
updates, err := client.WatchPersonas(t.Context(), 4)
if err != nil {
t.Fatalf("WatchPersonas: %v", err)
}
assertDeliveredARecvFailure(t, updates,
func(update AppliedPersonas) int64 { return update.Version },
func(update AppliedPersonas) error { return update.Err })
})
}
// TestWatchClientsDeliverNothingForAStreamThatDidNotFail is the other half of
// that contract. A stream the server closed cleanly is io.EOF, which is not a
// failure of anything, and a stream that ended because the watch's own context
// ended is the process shutting down: neither may be handed over as an update,
// because the consumer of one reports it, and a process that reports a store for
// going away when it was the process that left is reporting the wrong side of
// the outage.
func TestWatchClientsDeliverNothingForAStreamThatDidNotFail(t *testing.T) {
t.Parallel()
tests := []struct {
name string
end error
cancel bool
}{
{"the server closed the stream", io.EOF, false},
{"the stream failed as the watch was cancelled", status.Error(codes.Unavailable, "state store restarting"), true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
// The failure waits behind release, so a test that cancels first can
// end the stream after the context that ended the watch has ended.
var release chan struct{}
if tt.cancel {
release = make(chan struct{})
}
client := NewRPCClient(scriptedWatchClient{streams: func() grpc.ServerStreamingClient[pb.WatchResponse] {
return &scriptedRecvStream{end: tt.end, release: release}
}})
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
settings, err := client.WatchWorkflowExecutionSettings(ctx, 4)
if err != nil {
t.Fatalf("WatchWorkflowExecutionSettings: %v", err)
}
personas, err := client.WatchPersonas(ctx, 4)
if err != nil {
t.Fatalf("WatchPersonas: %v", err)
}
if tt.cancel {
cancel()
close(release)
}
assertDeliveredNothing(t, settings, tt.name)
assertDeliveredNothing(t, personas, tt.name)
})
}
}
// TestWatchServesOnlyVersionsAfterTheRequestedOne is the producer side of the
// resume contract a reconnecting client depends on. A watch resumes after the
// last version it handled, so the server must treat AfterVersion as an
// exclusive lower bound: a stream that re-sent the version the client had
// already applied would have it applied again, over whatever is newer. The
// stored version and the requested one are equal here, which is the state a
// client resuming correctly always presents.
//
// The stream is a real one over bufconn rather than a hand-rolled WatchServer:
// its context is the RPC's, so the test ends the server's loop the way
// production does, by cancelling the client.
func TestWatchServesOnlyVersionsAfterTheRequestedOne(t *testing.T) {
t.Parallel()
resources := store.OpenTest(t)
defer resources.Close()
server := testServer(t, resources)
version, err := server.ImportWorkflowExecutionSettings(t.Context(), workflow.ExecutionSettings{MaxModelToolSteps: 10, MaxRuntime: time.Minute, MaxConsecutiveGateFailures: 2})
if err != nil {
t.Fatal(err)
}
listener := bufconn.Listen(1 << 20)
grpcServer := grpc.NewServer()
pb.RegisterControlPlaneServiceServer(grpcServer, server)
go func() { _ = grpcServer.Serve(listener) }()
t.Cleanup(func() { grpcServer.Stop(); _ = listener.Close() })
conn, err := grpc.NewClient("passthrough:///controlplane", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) { return listener.DialContext(ctx) }))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = conn.Close() })
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
stream, err := pb.NewControlPlaneServiceClient(conn).Watch(ctx, &pb.WatchRequest{Kind: WorkflowExecutionSettingsKind, AfterVersion: version})
if err != nil {
t.Fatal(err)
}
// served carries each version the stream sends, and is closed when the
// stream ends. It is the only thing the test can read: what the server
// sends is the contract.
served := make(chan int64, 8)
go func() {
defer close(served)
for {
response, err := stream.Recv()
if err != nil {
return
}
served <- response.GetResource().GetVersion()
}
}()
// The stored version is the one the stream was asked to resume after, so
// nothing is due. Two poll intervals is long enough to see a server that
// re-serves it.
select {
case servedVersion, ok := <-served:
if !ok {
t.Fatal("the stream ended before serving anything")
}
t.Fatalf("the stream served version %d, want only versions after %d", servedVersion, version)
case <-time.After(2 * 250 * time.Millisecond):
}
// The next stored version is served.
next := workflow.ExecutionSettings{MaxModelToolSteps: 40, MaxRuntime: time.Hour, MaxConsecutiveGateFailures: 5}
value, err := encodeSettings(next)
if err != nil {
t.Fatal(err)
}
if _, err := resources.PutResource(ctx, store.ResourceWrite{Kind: WorkflowExecutionSettingsKind, Value: value, ExpectedVersion: version, Actor: "test", Source: "test", RequestID: "watch-next"}); err != nil {
t.Fatal(err)
}
select {
case servedVersion, ok := <-served:
if !ok {
t.Fatal("the stream ended before the stored version was served")
}
if servedVersion != version+1 {
t.Errorf("the stream served version %d, want %d", servedVersion, version+1)
}
case <-time.After(5 * time.Second):
t.Fatal("the stored version after the resume point was never served")
}
// Cancelling the client ends the server's loop: Recv stops, which is what
// closes served.
cancel()
select {
case <-served:
case <-time.After(5 * time.Second):
t.Fatal("the stream did not end after its context was cancelled")
}
}
@@ -106,6 +106,46 @@ func TestRestampRewritesEveryKnownKind(t *testing.T) {
}
}
// TestRestampKeepsAReportedFailure answers the second half of
// archie-core-yrmr's question: the reporter rewrites every record it holds at
// every RestampInterval, so a re-stamp that replaced a reported failure with a
// clean record would leave the settings page reading a kind as current while
// the process applying it had stopped (internal/webui's applyState reads a
// record carrying an error as failed -- TestApplyStatusReportsAnError). A
// reported failure survives the re-stamp, error included.
func TestRestampKeepsAReportedFailure(t *testing.T) {
store := &recordingStore{}
at := time.Date(2026, 9, 21, 10, 0, 0, 0, time.UTC)
r := New(Daemon, store, nil)
r.now = func() time.Time { return at }
r.Report(t.Context(), "workflow-execution-settings", 3, nil)
// A failure, reported without a version the process ever ran. The reporter
// keeps the version it last applied, which is the rule the watch leans on when
// it reports a document this process would not run: the watch has the refused
// version, and the record has to show the version still live. (An unreachable
// store writes no record at all, per docs/prds/control-plane-apply-status.md.)
r.Report(t.Context(), "workflow-execution-settings", 0, errors.New("max model/tool steps must not be negative"))
at = at.Add(RestampInterval)
r.restamp(t.Context())
got := store.snapshot()
if len(got) != 3 {
t.Fatalf("wrote %d records, want the apply, the failure and the re-stamp", len(got))
}
restamped := got[2]
if restamped.Error == "" {
t.Error("the re-stamp dropped the reported failure; the kind would read as current")
}
if restamped.AppliedVersion != 3 {
t.Errorf("re-stamped version = %d, want the version still live (3)", restamped.AppliedVersion)
}
if !restamped.ReportedAt.Equal(at) {
t.Errorf("re-stamped at %v, want %v", restamped.ReportedAt, at)
}
}
func TestStale(t *testing.T) {
at := time.Date(2026, 9, 21, 10, 0, 0, 0, time.UTC)
for _, tt := range []struct {