Files
archie/internal/app/controlplane/client.go
T
sam 4c77a20fa8 fix(archied): re-establish the control-plane watch streams after they end
The settings and persona watches were launched once, at boot, and ended on the
first stream failure: the goroutine returned, so every later stored change for
that kind was silently never applied and the process recovered only by
restarting (archie-core-yrmr).

Both watches now run one keepWatch goroutine per kind, which re-establishes the
stream in place after a bounded backoff, resuming after the last version an
update carried and stopping on ctx cancellation -- which the loop watches for
itself, so a stream that never closes cannot hold up shutdown. A first open that
fails still fails the boot that asked for it.

The reconnect delay is decided by the attempt that just ended, and only by that
(attemptHealthy, retryDelay): an attempt is healthy iff it delivered an update
this watch had not handled, or it outlived a window that is a strict multiple of
the delay it would otherwise be charged. A healthy attempt returns the delay to
the minimum, a failed one doubles it up to the cap, and the window moves with
the ladder instead of sitting at its floor. The acceptance of a reopen is the
wrong unit: gRPC takes the stream before the handler reads anything, and
controlplane.Server.Watch answers a store it cannot read by ending the stream,
so during the outage this reconnect exists for every reopen is accepted and
every stream ends carrying nothing -- measured with the real server and client
over bufconn, 4.0 watch RPCs a second across 2s of store outage with retry_in
pinned at 250ms when that acceptance is read as health. A window fixed at the
minimum is the same defect for the likelier shape of outage: a store that has
stopped answering does not error at once, it stalls until its read times out,
and an attempt that takes longer to fail than the minimum outlives that window
on every attempt -- measured in the end-to-end outage test before the change, a
300ms stall left the ladder flat at 250ms/250ms/250ms/250ms while the store
never answered a read. Both shapes are charged now, and the outage test drives
three of them (an immediate error, a 300ms stall, and a stall as long as the
delay that rung would be charged) up four rungs of the ladder, each logged retry
held to the delay actually waited.

The wait is injected (wait, production passes waitFor), because the ladder was
only ever asserted through the retry_in attribute: a loop that logged one delay
and waited another would have reported a growing ladder and reopened on a
schedule of its own. TestKeepWatchWaitsTheDelayItCharged reads the delays back
as values and asserts the ladder, and the outage test's gaps catch a wait that
returns without waiting. The retargeted reset test is joined by
TestKeepWatchChargesAReopenTheStoreMerelyAccepted: a delay reset by the
acceptance of a reopen charges every attempt of the outage from the minimum, and
nothing pinned that before.

The watch returns on cancellation without draining what the producer left for
it, so controlplane.Client's producers stop on the same context instead of
parking on a send no reader will ever take -- reproduced 40/40 times as a
goroutine still blocked on an undelivered send after keepWatch returned, one
goroutine and one channel leaked per kind at every exit. That cross-package
coupling is load-bearing in both directions and is stated as such; the two
producers now share the one stream reader that carries it.

A Recv failure is the store being unreachable, not a document this process
refused, and docs/prds/control-plane-apply-status.md ("What can be reported, and
what cannot") is what forbids reporting it as a failure of the process it
affected. The record is now written only for an update that carries a version,
which is what controlplane.Client emits for a document that would not validate
and never for a transport failure. Reporting the transport failure would also
overwrite the text of a standing refusal, replacing the one message that says
what to fix with "control-plane unavailable". The record is not un-clearable of
itself -- Report overwrites it on the next applied version -- but a store that
never changes again leaves the one stamp standing, and the process must not
write it at all. That emission is the whole of the failure path and now has the
test it lacked, from the producer's side:
TestWatchClientsDeliverARecvFailureAsAVersionlessUpdate pins that a Recv failure
is handed over carrying the mapped error and no version, and that a stream the
server closed and one that ended with the watch's context are not updates at all.

The refused version is deliberately not re-offered: the resume point moves past
it, so an edit only a newer build can decode cannot become a hot loop. Both ends
of that contract are pinned now: the watch stub filters its scripted stream by
AfterVersion as the server does, and controlplane.Server.Watch's own filter has
a test for the first time. The apply-status reporter's 30s re-stamp is unchanged
(TestRestampKeepsAReportedFailure).
2026-09-22 01:58:03 +10:00

243 lines
9.0 KiB
Go

package controlplane
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
pb "github.com/samcharles93/archie-core/internal/contracts/controlplane/v1"
"github.com/samcharles93/archie-core/internal/domain/agent"
"github.com/samcharles93/archie-core/internal/domain/workflow"
"github.com/samcharles93/archie-core/internal/infrastructure/cronstore"
)
// Client is the control-plane client for every resource that does not resolve
// a workflow step type: catalog, history, settings, personas, schedules. It
// carries no step vocabulary, so a process that never reads or writes a
// workflow definition -- archie-messaging, which only loads channel settings,
// and archie-gateway, which reads catalog, runtime settings and personas --
// needs no provider set and cannot be failed by one.
type Client struct{ rpc pb.ControlPlaneServiceClient }
func NewClient(conn grpc.ClientConnInterface) *Client {
return &Client{rpc: pb.NewControlPlaneServiceClient(conn)}
}
func NewRPCClient(client pb.ControlPlaneServiceClient) *Client { return &Client{rpc: client} }
// WorkflowDefinitionsClient reads and replaces the workflow-definitions
// resource, the one control-plane surface whose stored values name workflow
// step types. Resolving those names needs the step vocabulary the process
// registered at its composition root, so the vocabulary is a constructor
// dependency of this surface and of no other: a caller that cannot reach
// workflow definitions cannot ask for one.
type WorkflowDefinitionsClient struct {
rpc pb.ControlPlaneServiceClient
steps workflow.StepRegistry
}
func NewWorkflowDefinitionsClient(client pb.ControlPlaneServiceClient, steps *workflow.Manager) (*WorkflowDefinitionsClient, error) {
registry, err := stepRegistry(steps)
if err != nil {
return nil, err
}
return &WorkflowDefinitionsClient{rpc: client, steps: registry}, nil
}
func (c *Client) Catalog(ctx context.Context) ([]*pb.ResourceDescriptor, error) {
response, err := c.rpc.Catalog(ctx, &pb.CatalogRequest{})
if err != nil {
return nil, clientError(err)
}
return response.Resources, nil
}
func (c *WorkflowDefinitionsClient) WorkflowDefinitions(ctx context.Context) (workflow.WorkflowDefinitionCollection, int64, error) {
response, err := c.rpc.Query(ctx, &pb.QueryRequest{Kind: WorkflowDefinitionsKind})
if err != nil {
return workflow.WorkflowDefinitionCollection{}, 0, clientError(err)
}
definitions, err := workflow.DecodeDefinitionCollection(response.Resource.ValueJson, c.steps)
return definitions, response.Resource.Version, err
}
// ReplaceWorkflowDefinitions replaces the collection. Passing
// workflow.ShippedDefinitions() restores all shipped definitions while keeping
// the previous override in resource history.
//
// No production process calls it today: the daemon's only consumer of this
// surface reads (internal/daemon's WorkflowDefinitions interface), so this half
// is reached by tests until a writer exists.
func (c *WorkflowDefinitionsClient) ReplaceWorkflowDefinitions(ctx context.Context, definitions workflow.WorkflowDefinitionCollection, expectedVersion int64, actor, source, requestID string) (int64, error) {
value, err := encodeWorkflowDefinitions(definitions, c.steps)
if err != nil {
return 0, err
}
response, err := c.rpc.Command(ctx, &pb.CommandRequest{Kind: WorkflowDefinitionsKind, Command: "replace", ValueJson: value, ExpectedVersion: expectedVersion, Actor: actor, Source: source, RequestId: requestID})
if err != nil {
return 0, clientError(err)
}
return response.Resource.Version, nil
}
func (c *Client) WorkflowExecutionSettings(ctx context.Context) (workflow.ExecutionSettings, int64, error) {
response, err := c.rpc.Query(ctx, &pb.QueryRequest{Kind: WorkflowExecutionSettingsKind})
if err != nil {
return workflow.ExecutionSettings{}, 0, clientError(err)
}
settings, err := decodeSettings(response.Resource.ValueJson)
return settings, response.Resource.Version, err
}
func (c *Client) ReplaceWorkflowExecutionSettings(ctx context.Context, settings workflow.ExecutionSettings, expectedVersion int64, actor, source, requestID string) (int64, error) {
if err := settings.Validate(); err != nil {
return 0, fmt.Errorf("%w: %w", ErrValidation, err)
}
value, err := encodeSettings(settings)
if err != nil {
return 0, err
}
response, err := c.rpc.Command(ctx, &pb.CommandRequest{Kind: WorkflowExecutionSettingsKind, Command: "replace", ValueJson: value, ExpectedVersion: expectedVersion, Actor: actor, Source: source, RequestId: requestID})
if err != nil {
return 0, clientError(err)
}
return response.Resource.Version, nil
}
func (c *Client) WatchWorkflowExecutionSettings(ctx context.Context, afterVersion int64) (<-chan AppliedSettings, error) {
stream, err := c.rpc.Watch(ctx, &pb.WatchRequest{Kind: WorkflowExecutionSettingsKind, AfterVersion: afterVersion})
if err != nil {
return nil, clientError(err)
}
return watchUpdates(ctx, stream,
func(resource *pb.Resource) AppliedSettings {
settings, err := decodeSettings(resource.ValueJson)
return AppliedSettings{Settings: settings, Version: resource.Version, Err: err}
},
func(err error) AppliedSettings { return AppliedSettings{Err: err} }), nil
}
type AppliedSettings struct {
Settings workflow.ExecutionSettings
Version int64
Err error
}
func (c *Client) Personas(ctx context.Context) (agent.PersonaCollection, int64, error) {
response, err := c.rpc.Query(ctx, &pb.QueryRequest{Kind: PersonasKind})
if err != nil {
return agent.PersonaCollection{}, 0, clientError(err)
}
collection, err := decodePersonas(response.Resource.ValueJson)
return collection, response.Resource.Version, err
}
type AppliedPersonas struct {
Collection agent.PersonaCollection
Version int64
Err error
}
func (c *Client) WatchPersonas(ctx context.Context, afterVersion int64) (<-chan AppliedPersonas, error) {
stream, err := c.rpc.Watch(ctx, &pb.WatchRequest{Kind: PersonasKind, AfterVersion: afterVersion})
if err != nil {
return nil, clientError(err)
}
return watchUpdates(ctx, stream,
func(resource *pb.Resource) AppliedPersonas {
collection, err := decodePersonas(resource.ValueJson)
return AppliedPersonas{Collection: collection, Version: resource.Version, Err: err}
},
func(err error) AppliedPersonas { return AppliedPersonas{Err: err} }), nil
}
// watchUpdates turns one Watch stream into the channel a watch loop reads: every
// document the stream carries, decoded, and then a single update carrying the
// error that ended it -- unless the stream ended cleanly or the context did, in
// which case the channel just closes.
func watchUpdates[T any](
ctx context.Context,
stream grpc.ServerStreamingClient[pb.WatchResponse],
update func(resource *pb.Resource) T,
failed func(err error) T,
) <-chan T {
out := make(chan T)
go func() {
defer close(out)
for {
response, err := stream.Recv()
if err != nil {
if !errors.Is(err, io.EOF) && ctx.Err() == nil {
sendUpdate(ctx, out, failed(clientError(err)))
}
return
}
if !sendUpdate(ctx, out, update(response.Resource)) {
return
}
}
}()
return out
}
// sendUpdate hands update to the reader of one Watch stream, reporting false
// when ctx ended first. A reader leaves without draining what is left for it:
// the watch that reads these streams returns on cancellation, so a bare send
// parks this producer -- and the stream it holds -- for the rest of the
// process's life, one goroutine and one channel per kind at every exit. That
// coupling is load-bearing: the reader's prompt shutdown rests on this side
// stopping on the same context, and neither half is safe alone.
func sendUpdate[T any](ctx context.Context, out chan<- T, update T) bool {
select {
case out <- update:
return true
case <-ctx.Done():
return false
}
}
func (c *Client) Schedules(ctx context.Context) ([]cronstore.JobSpec, int64, error) {
response, err := c.rpc.Query(ctx, &pb.QueryRequest{Kind: SchedulesKind})
if err != nil {
return nil, 0, clientError(err)
}
var jobs []cronstore.JobSpec
if err := json.Unmarshal(response.Resource.ValueJson, &jobs); err != nil {
return nil, 0, err
}
return jobs, response.Resource.Version, nil
}
func (c *Client) ReplaceSchedules(ctx context.Context, jobs []cronstore.JobSpec, expectedVersion int64, actor, source, requestID string) (int64, error) {
value, err := json.Marshal(jobs)
if err != nil {
return 0, err
}
response, err := c.rpc.Command(ctx, &pb.CommandRequest{Kind: SchedulesKind, Command: "replace", ValueJson: value, ExpectedVersion: expectedVersion, Actor: actor, Source: source, RequestId: requestID})
if err != nil {
return 0, clientError(err)
}
return response.Resource.Version, nil
}
func clientError(err error) error {
switch status.Code(err) {
case codes.InvalidArgument:
return fmt.Errorf("%w: %w", ErrValidation, err)
case codes.NotFound:
return fmt.Errorf("%w: %w", ErrNotFound, err)
case codes.Aborted:
return fmt.Errorf("%w: %w", ErrVersionConflict, err)
case codes.Unavailable:
return fmt.Errorf("%w: %w", ErrUnavailable, err)
default:
return err
}
}