Commit Graph
6 Commits
Author SHA1 Message Date
sam b8ec491dd3 fix(archied): re-establish the control-plane watch streams after they end
The settings and persona watches were launched once, at boot, and ended on the
first stream failure: the goroutine returned, so every later stored change for
that kind was silently never applied and the process recovered only by
restarting (archie-core-yrmr).

Both watches now run one keepWatch goroutine per kind, which re-establishes the
stream in place after a bounded backoff, resuming after the last version an
update carried and stopping on ctx cancellation -- which the loop watches for
itself, so a stream that never closes cannot hold up shutdown. A first open that
fails still fails the boot that asked for it.

The reconnect delay is decided by the attempt that just ended, and only by that
(attemptHealthy, retryDelay): an attempt is healthy iff it delivered an update
this watch had not handled, or it outlived a window that is a strict multiple of
the delay it would otherwise be charged. A healthy attempt returns the delay to
the minimum, a failed one doubles it up to the cap, and the window moves with
the ladder instead of sitting at its floor. The acceptance of a reopen is the
wrong unit: gRPC takes the stream before the handler reads anything, and
controlplane.Server.Watch answers a store it cannot read by ending the stream,
so during the outage this reconnect exists for every reopen is accepted and
every stream ends carrying nothing -- measured with the real server and client
over bufconn, 4.0 watch RPCs a second across 2s of store outage with retry_in
pinned at 250ms when that acceptance is read as health. A window fixed at the
minimum is the same defect for the likelier shape of outage: a store that has
stopped answering does not error at once, it stalls until its read times out,
and an attempt that takes longer to fail than the minimum outlives that window
on every attempt -- measured in the end-to-end outage test before the change, a
300ms stall left the ladder flat at 250ms/250ms/250ms/250ms while the store
never answered a read. Both shapes are charged now, and the outage test drives
three of them (an immediate error, a 300ms stall, and a stall as long as the
delay that rung would be charged) up four rungs of the ladder, each logged retry
held to the delay actually waited.

The wait is injected (wait, production passes waitFor), because the ladder was
only ever asserted through the retry_in attribute: a loop that logged one delay
and waited another would have reported a growing ladder and reopened on a
schedule of its own. TestKeepWatchWaitsTheDelayItCharged reads the delays back
as values and asserts the ladder, and the outage test's gaps catch a wait that
returns without waiting. The retargeted reset test is joined by
TestKeepWatchChargesAReopenTheStoreMerelyAccepted: a delay reset by the
acceptance of a reopen charges every attempt of the outage from the minimum, and
nothing pinned that before.

The watch returns on cancellation without draining what the producer left for
it, so controlplane.Client's producers stop on the same context instead of
parking on a send no reader will ever take -- reproduced 40/40 times as a
goroutine still blocked on an undelivered send after keepWatch returned, one
goroutine and one channel leaked per kind at every exit. That cross-package
coupling is load-bearing in both directions and is stated as such; the two
producers now share the one stream reader that carries it.

A Recv failure is the store being unreachable, not a document this process
refused, and docs/prds/control-plane-apply-status.md ("What can be reported, and
what cannot") is what forbids reporting it as a failure of the process it
affected. The record is now written only for an update that carries a version,
which is what controlplane.Client emits for a document that would not validate
and never for a transport failure. Reporting the transport failure would also
overwrite the text of a standing refusal, replacing the one message that says
what to fix with "control-plane unavailable". The record is not un-clearable of
itself -- Report overwrites it on the next applied version -- but a store that
never changes again leaves the one stamp standing, and the process must not
write it at all. That emission is the whole of the failure path and now has the
test it lacked, from the producer's side:
TestWatchClientsDeliverARecvFailureAsAVersionlessUpdate pins that a Recv failure
is handed over carrying the mapped error and no version, and that a stream the
server closed and one that ended with the watch's context are not updates at all.

The refused version is deliberately not re-offered: the resume point moves past
it, so an edit only a newer build can decode cannot become a hot loop. Both ends
of that contract are pinned now: the watch stub filters its scripted stream by
AfterVersion as the server does, and controlplane.Server.Watch's own filter has
a test for the first time. The apply-status reporter's 30s re-stamp is unchanged
(TestRestampKeepsAReportedFailure).
2026-09-22 01:58:03 +10:00
sam 0070fba7a2 fix(archied): probe what the live settings update would actually run
Review of d7f3d0da found the build-and-probe step shape-only: the check it
added re-ran workflow.ExecutionSettings.Validate, which the store runs before
writing the document and controlplane.Client runs again on the way out, so no
producer could trip it. That review was right; this corrects it.

- The probe is now this process's own runnability check on the candidate it
  would switch to: configuration.Validate over the configuration snapshot the
  new limits would be published as, the same gate runtimeConfig applies before
  publishing a layered config and the readiness probe applies to the published
  one. The duplicated schema check is gone.
- The refusal the daemon really meets arrives as a watch error, because the
  client decodes every document before the apply path sees it. That branch
  logged and returned without writing an apply-status record, so the settings
  page showed a live resource as healthy while nothing was being applied. It
  now reports the refusal, against the version the reporter last applied.
  Whether the stream is re-established afterwards stays archie-core-yrmr.
- `Closes archie-core-nwa0` in d7f3d0da is withdrawn: the item stays open. For
  a limits-only kind no producer can make the owned check fail today, so the
  refusal path is exercised by test, not by traffic; the live kind whose
  candidate check has content is personas (archie-core-nw7t). Both callers are
  covered by tests now, through the client seam, instead of by calling the
  apply function directly.
- The comments no longer claim behaviour the code lacks: the boot abort is the
  fail-closed direction of a rule whose branch boot cannot reach, and the
  carried-forward version is the reporter's property, not this call's.

Out of scope and recorded instead: the live apply still publishes with
cfgHolder.Set rather than publishConfig, leaving the dashboard's config
snapshot stale after a live change (archie-core-75cc). workflow-definitions
gets no check because nothing swaps a running definition: the daemon reads the
collection per task pin (daemon.go:1479, pinWorkflowFromCollection) and the
only other reader is the dashboard's stream, so there is no live replacement to
probe.
2026-09-21 20:32:04 +10:00
sam c773dde98a feat(archied): check a live settings replacement before switching to it
docs/prds/runtime-control-plane.md ("API") requires a live change to start
and check the new component before switching to it, keeping the old one
running when the check fails. The live workflow-execution-settings apply
stored whatever arrived on the watch, republished the configuration snapshot
new tasks are built from, and reported success unconditionally.

Stage the update as a candidate first: validate the limits and build the
snapshot they produce without touching anything running, then switch both in
only if the check passes. A refused update leaves the running limits and the
snapshot in place and reports the failure through the apply-status surface,
which keeps the version that is still live and carries the error. Boot
propagates the same rejection, so a process is never started on limits it
cannot run.

personas is the other live-watched kind and gets no check or report here;
filed as archie-core-nw7t rather than half-wired.

Closes archie-core-nwa0.
2026-09-21 20:32:04 +10:00
sam 9accf112c6 feat(controlplane): report which settings version each process is running
Every process that layers database-owned settings now publishes the version it
applied, and re-stamps it every 30 seconds so the report time doubles as a
liveness signal. A record older than 90 seconds belongs to a process that has
stopped, which is the only way this surface can tell a live process from one
that applied a version and then died.

RuntimeConfig returned the merged config and discarded the versions it read,
so no caller could report what it had applied. It now returns them, and
archied and archie-gateway report each kind through boot.runtimeConfig at boot
and on SIGHUP. A validation failure reports every kind with the error, because
the layering is all-or-nothing, and leaves the recorded version alone so a
rejected edit does not erase the fact that an older one is live. The daemon
also reports workflow execution settings as the watch applies them, and
archie-messaging reports channel settings at startup.

Reporting is best-effort by construction: the configuration a report describes
has already been applied, so a failed write is logged and never returned. That
keeps it off the reload's error path, where it would have raised a banner
saying a reload failed when its config took effect.

applystatus.New refuses a process name Processes() does not list. The reader
renders known names, so an invented one would be invisible rather than wrong,
which is the same thing the page means by a process that never started.

Relates to archie-core-pskb.
2026-09-21 15:49:54 +10:00
sam 9af013eb3b fix(archied): re-apply database-owned settings on SIGHUP reload
The reload callback re-resolved the file document and republished it as the
running config. Every setting the control plane owns lives only in the
database, so a reload reverted providers, model roles, repository policies,
channel settings, scheduling policy, tool and plugin settings, container
policies and workflow execution budgets to their file values until the
process restarted, and logged "config reloaded" while doing it.

Boot's layering is now one call, boot.runtimeConfig, and the reload makes the
same call over the document it just resolved. Execution budgets arrive on a
watch rather than in a resource query, so boot records the last settings it
applied and the reload re-applies them from there.

apply gains an error return, which the controller records exactly like a bad
file: a control plane that cannot answer fails the reload and leaves the
running config alone, because it already holds the values the file does not.
The query is bounded at 10s so a State Store that stops answering surfaces as
a failed reload rather than a SIGHUP that never returns.

The callback also read the old config from b.d.Cfg, the same Holder as
b.cfgHolder; reading it from the Holder boot owns drops the daemon from this
path and makes it testable.

Closes archie-core-ju85.
2026-09-21 14:41:50 +10:00
sam 924b02a1b0 feat(controlplane): runtime administration over the State Store's gRPC server
The Web UI and messaging channels now read and write Archie's runtime
settings through one versioned, audited control-plane API hosted on the
State Store's existing gRPC server: catalog, query, command, watch. Each
feature owns its resource, validation and commands; storage only writes
versions and audit records.

Workflows move with it. Definitions become portable YAML compiled through
a closed, typed step registry and stored as a control-plane resource, so
the yaegi interpretation paths (internal/gate, wfeval, wfextract,
gateeval, gateextract, skillbuild) are deleted rather than carried.

The UI process reads definitions as data: the JSON projection lives in
internal/domain/workflow/task so a dashboard decodes the stored
collection without linking the workflow engine, which cmd/archie-ui's
deletion gate forbids. Shipped definitions ride the catalog descriptor
rather than a bespoke REST route, keeping the API to its four operations.

archie-messaging gains -state-store-target/-state-store-token and reads
channel settings from the database, failing closed when the State Store
is unreachable: the database is authoritative for those settings.
2026-09-21 08:17:42 +10:00