Files
archie/internal/app/archied/gateway_runtime.go
T
sam 189ac806dc fix(archied): re-establish the control-plane watch streams after they end
The settings and persona watches were launched once, at boot, and ended on the
first stream failure: the goroutine returned, so every later stored change for
that kind was silently never applied and the process recovered only by
restarting (archie-core-yrmr).

Both watches now run one keepWatch goroutine per kind, which re-establishes the
stream in place after a bounded backoff, resuming after the last version an
update carried and stopping on ctx cancellation -- which the loop watches for
itself, so a stream that never closes cannot hold up shutdown. A first open that
fails still fails the boot that asked for it.

The reconnect delay is decided by the attempt that just ended, and only by that
(attemptHealthy, retryDelay): an attempt is healthy iff it delivered an update
this watch had not handled, or it outlived a window that is a strict multiple of
the delay it would otherwise be charged. A healthy attempt returns the delay to
the minimum, a failed one doubles it up to the cap, and the window moves with
the ladder instead of sitting at its floor. The acceptance of a reopen is the
wrong unit: gRPC takes the stream before the handler reads anything, and
controlplane.Server.Watch answers a store it cannot read by ending the stream,
so during the outage this reconnect exists for every reopen is accepted and
every stream ends carrying nothing -- measured with the real server and client
over bufconn, 4.0 watch RPCs a second across 2s of store outage with retry_in
pinned at 250ms when that acceptance is read as health. A window fixed at the
minimum is the same defect for the likelier shape of outage: a store that has
stopped answering does not error at once, it stalls until its read times out,
and an attempt that takes longer to fail than the minimum outlives that window
on every attempt -- measured in the end-to-end outage test before the change, a
300ms stall left the ladder flat at 250ms/250ms/250ms/250ms while the store
never answered a read. Both shapes are charged now, and the outage test drives
three of them (an immediate error, a 300ms stall, and a stall as long as the
delay that rung would be charged) up four rungs of the ladder, each logged retry
held to the delay actually waited.

The wait is injected (wait, production passes waitFor), because the ladder was
only ever asserted through the retry_in attribute: a loop that logged one delay
and waited another would have reported a growing ladder and reopened on a
schedule of its own. TestKeepWatchWaitsTheDelayItCharged reads the delays back
as values and asserts the ladder, and the outage test's gaps catch a wait that
returns without waiting. The retargeted reset test is joined by
TestKeepWatchChargesAReopenTheStoreMerelyAccepted: a delay reset by the
acceptance of a reopen charges every attempt of the outage from the minimum, and
nothing pinned that before.

The watch returns on cancellation without draining what the producer left for
it, so controlplane.Client's producers stop on the same context instead of
parking on a send no reader will ever take -- reproduced 40/40 times as a
goroutine still blocked on an undelivered send after keepWatch returned, one
goroutine and one channel leaked per kind at every exit. That cross-package
coupling is load-bearing in both directions and is stated as such; the two
producers now share the one stream reader that carries it.

A Recv failure is the store being unreachable, not a document this process
refused, and docs/prds/control-plane-apply-status.md ("What can be reported, and
what cannot") is what forbids reporting it as a failure of the process it
affected. The record is now written only for an update that carries a version,
which is what controlplane.Client emits for a document that would not validate
and never for a transport failure. Reporting the transport failure would also
overwrite the text of a standing refusal, replacing the one message that says
what to fix with "control-plane unavailable". The record is not un-clearable of
itself -- Report overwrites it on the next applied version -- but a store that
never changes again leaves the one stamp standing, and the process must not
write it at all. That emission is the whole of the failure path and now has the
test it lacked, from the producer's side:
TestWatchClientsDeliverARecvFailureAsAVersionlessUpdate pins that a Recv failure
is handed over carrying the mapped error and no version, and that a stream the
server closed and one that ended with the watch's context are not updates at all.

The refused version is deliberately not re-offered: the resume point moves past
it, so an edit only a newer build can decode cannot become a hot loop. Both ends
of that contract are pinned now: the watch stub filters its scripted stream by
AfterVersion as the server does, and controlplane.Server.Watch's own filter has
a test for the first time. The apply-status reporter's 30s re-stamp is unchanged
(TestRestampKeepsAReportedFailure).
2026-09-22 01:58:03 +10:00

143 lines
6.1 KiB
Go

package archied
import (
"context"
"fmt"
"github.com/samcharles93/archie-core/internal/agentexec"
"github.com/samcharles93/archie-core/internal/app/controlplane"
"github.com/samcharles93/archie-core/internal/config"
"github.com/samcharles93/archie-core/internal/domain/agent"
"github.com/samcharles93/archie-core/internal/gateway"
"github.com/samcharles93/archie-core/internal/tools"
)
// setupChatRuntime wires the model runtime, tool registry, persona registry,
// chat task creator/controller, default chat identity and the release-update
// service. The daemon and the standalone Gateway build through this one
// constructor so the two processes cannot diverge.
func (b *boot) setupChatRuntime(ctx context.Context, cfg config.Config) error {
// ── LLM runtime ──────────────────────────────────────────────────
// Created before the router so the LLMResponder can be wired in for
// non-command message processing.
providers := executionProviders(cfg)
b.llm = agentexec.NewRuntime(providers)
b.toolReg = tools.NewRegistry()
chatModels := newChatModelManager(cfg.Models, cfg.Chat.Models, b.catalogModels)
chatModels.ApplyModelCatalog(b.catalog)
b.chatModels = chatModels
// ── Persona registry ─────────────────────────────────────────────
if b.personas == nil {
personas, version, err := b.controlPlane.Personas(ctx)
if err != nil {
return fmt.Errorf("load personas: %w", err)
}
b.personas = gateway.NewPersonaRegistry(nil)
b.applyPersonas(personas, version)
if err := b.watchPersonas(ctx, version); err != nil {
return fmt.Errorf("watch personas: %w", err)
}
}
// ── Operator health surface ──────────────────────────────────────
// Built before the gateways because every turn runner and router built
// later carries both: the recorder is written by sendChatTurn, the source
// is read by /status.
b.providerOutcomes = newProviderOutcomeRecorder()
b.statusHealth = newStatusHealth(b)
profiles, defaultChatIdentity := chatTaskProfiles(cfg)
var chatTasks gateway.TaskCreator
if len(profiles) > 0 {
chatTasks = gateway.NewStoreTaskCreatorForProfiles(
chatTaskWriterAdapter{enqueue: b.stateStore.EnqueueChatTask},
profiles,
)
}
b.chatTasks = chatTasks
b.defaultChatIdentity = defaultChatIdentity
b.chatController = gateway.NewStoreTaskController(chatTaskControllerAdapter{
taskByID: b.stateStore.TaskByID,
requeue: b.stateStore.Requeue,
transition: b.stateStore.Transition,
})
b.updateService = makeUpdateService(chatSetup{Cfg: config.NewHolder(cfg)})
return nil
}
// watchPersonas keeps the persona stream established for the life of the
// process. The first stream is opened synchronously, so a control plane that
// cannot be watched at all fails the boot that asked for it rather than
// leaving the process running personas it can no longer update; after that the
// watch reconnects instead of ending (see keepWatch).
//
// Persona updates are not reported through apply status -- the personas kind
// has no apply-status wiring, which is archie-core-aj35, not this watch.
func (b *boot) watchPersonas(ctx context.Context, version int64) error {
updates, err := b.controlPlane.WatchPersonas(ctx, version)
if err != nil {
return err
}
go keepWatch(ctx, b.log, controlplane.PersonasKind, version, updates,
b.controlPlane.WatchPersonas,
waitFor,
func(update controlplane.AppliedPersonas) int64 { return update.Version },
func(update controlplane.AppliedPersonas) {
if update.Err != nil {
b.log.Error("persona watch failed", "err", update.Err)
return
}
b.applyPersonas(update.Collection, update.Version)
})
return nil
}
func (b *boot) applyPersonas(collection agent.PersonaCollection, version int64) {
personas := make([]gateway.Persona, 0, len(collection.Personas))
for _, persona := range collection.Personas {
personas = append(personas, gateway.Persona{Name: persona.Name, Prompt: persona.Prompt})
}
b.personas.Replace(personas, collection.Default)
b.log.Info("personas applied", "version", version)
}
// setupGatewayChat is the sole production constructor of the local contract.
// Frontends use its gRPC representation; only this service owns the router.
func (b *boot) setupGatewayChat(ctx context.Context, actor gateway.ChatTaskActor) (gateway.ChatContract, error) {
if err := b.setupChatRuntime(ctx, b.cfg); err != nil {
return nil, err
}
cfg := b.cfg
router := gateway.NewRouter(b.stateStore, nil, "web")
router.Limiter = b.rateLimiter
router.Version = fmt.Sprintf("Archie\nGateway: %s\nRuntime: %s", gatewayVersion, runtimeVersion)
router.Models = b.chatModels
router.Personas = b.personas
router.Updates = b.updateService
router.InitSessions(b.chatSessionStore)
configureTaskCommands(router, b.chatTasks, b.chatController, chatTaskListerAdapter{tasks: b.stateStore.Tasks}, b.defaultChatIdentity)
router.Health = b.statusHealth
setup := chatSetup{
Cfg: config.NewHolder(cfg), LLM: b.llm, ChatModels: b.chatModels, ToolReg: b.toolReg,
Personas: b.personas, ChatTasks: b.chatTasks,
ChatTaskLister: chatTaskListerAdapter{tasks: b.stateStore.Tasks},
ChatTaskLogs: chatTaskLogReaderAdapter{tasks: b.stateStore.TaskByID, taskLogs: b.taskLogs},
ChatTaskActor: actor, ChatPRReviewer: b.prReviewer(),
DefaultChatIdentity: b.defaultChatIdentity,
Bus: b.bus, Log: b.log,
MemoryEngine: b.memoryStore(),
MemoryWriter: b.memoryWriter(),
// The Gateway executes the web chat's turns, so it is the process
// that must record their outcomes for /status.
ProviderOutcomes: b.providerOutcomes,
}
router.LLM, router.LLMStream = makeChatLLMResponder(ctx, "web", setup, b.chatSessionStore, router)
router.Titles = newChatTitleGenerator(setup)
router.Log = b.log
return &gateway.LocalChatAdapter{
Router: router, Sessions: b.chatSessionStore, Turns: gateway.NewTurns(b.log),
Models: b.chatModels, Personas: b.personas, TaskActor: actor,
}, nil
}