Files
archie/internal/gateway/health.go
T
sam 16e3b84048 fix: address the Greptile findings on the merged PRs (#902)
* fix: address the Greptile findings on the merged PRs

Four valid review findings from PRs #892, #893, #898 and #899, which were
merged without the comments being read.

- Config.Clone left the new diff-cap pointers aliased (#899, mine). Clone
  deep-copies every reference field, including the *bool beside it, so a
  clone could write the cap through to the published snapshot. Both the
  top-level field and the identity overrides are copied, with the clone test
  extended to cover them.
- providerOutcomeRecorder.record could report an older call as the latest
  (#892). The completion time is taken before the lock, so two concurrent
  calls can reach the lock in the opposite order to their completion and the
  unconditional store overwrote the newer outcome. It now keeps the newer,
  via a recordAt seam that makes the reordering testable.
- Curator model calls bypassed the recorder (#892). curatorLLMRunner calls
  the runtime directly, so a daemon whose only recent model traffic was
  curator work reported "no calls attempted yet" while the provider was
  demonstrably reachable. It records at its own adapter, and sendChatTurn's
  "single choke point" comment is corrected to say there are two.
- A provider error reached /status raw (#892), unlike channel details, so a
  multiline or oversized error broke the one-line-per-fact layout. Both now
  share one bounded formatter.

Found while fixing the curator path: agentexec.NewRuntime returns nil when no
provider is configured, and a nil *runtime.Runtime panics on its first method
call, so a curator asking for a completion took the daemon down. It now
returns an error, and records nothing -- a local misconfiguration is not a
provider call.

Not fixed, filed instead: the shared lastPollAt masking a stuck identity, the
container max-uptime callback resurrecting teardown bookkeeping, the
dashboard dropping a served-only status filter on cold load, and the route
registry test's line-by-line parser.

* fix(build): drop goimports from the gate's formatter set

`task check` spent 136 of its 167 seconds in `task fmt`, and 134 of those
136 in one formatter: goimports alone over the tree.

golangci-lint's goimports formatter calls imports.Process once per file,
which builds a fresh x/tools ProcessEnv each time, so
internal/imports.getFixesWithSource re-reads modindex.Read($GOMODCACHE) per
file. On this machine that index is 62.7 MiB / 2,081,070 lines (an 11 GiB
module cache), and modindex's in-process memo does not take effect: 3 files
read 187.9 MiB back, 15 files open the payload 15 times. That is ~53 GiB of
I/O and CSV parsing per run, and it is linear at ~155 ms per file (122 files
-> 18.8s, 864 -> 133.8s). Pointing the index cache elsewhere drops the same
command to 3.9s.

The same analyzer runs under `golangci-lint run`, so a cold cache pays it
again: 169s cold with goimports, 31s without -- which is what CI and archied's
own PR gate see, since only `fmt` never caches locally.

Removing it costs no enforcement. gofumpt and gci remain, `task lint` still
reports formatter drift as an issue, and the whole-tree fmt is now 2.2s with
the tree byte-identical. What is given up is goimports' convenience of
adding and removing import lines, which compile errors already cover.

The examples/secret-engines exclusion stays: gofumpt and gci leave those
Yaegi programs alone today, but the exclusion is the stated invariant that a
formatter must not rewrite the imports of files it cannot type-check, and
TestFormattersExcludeInterpretedSecretEngines holds it. Its comment no longer
pins the reason to the formatter that was dropped.

* fix(gateway): stop /status promising and mis-reporting health it cannot see

Two findings from the cloud review of #892.

Title generation no longer records chat-model health. It runs detached after
a first turn under its own 30s bound and its error is swallowed by the
caller, but it shared the process-wide outcome recorder, so a title that
merely timed out overwrote the health line with "failed" -- reporting a
broken provider on the strength of a call no user was waiting for.

The /status command description no longer claims "workers" and "channels".
Neither can render: newStatusHealth reads the container pool and the poll
loop from boot, but the only production router.Health assignment is in
setupGatewayChat, which runs solely in the archie-gateway process, where
neither exists; channels is nil since the Messaging Service took the
front-ends. Only the broker and chat-model lines actually appear, so that is
what the help text now says.

The underlying split -- daemon owns the facts, Gateway owns the Router -- is
filed rather than patched here: it needs either a cross-process health
contract or a ratified two-section /status, not a quick wire-up.
2026-09-20 02:42:36 +10:00

207 lines
7.4 KiB
Go

package gateway
import (
"fmt"
"strings"
"time"
)
// This file is /status' health contract: the facts the daemon reports about
// its own well-being, and the rules for rendering them. It is deliberately
// separate from the work surface (/tasks, and the identity/stage/age those
// summaries carry): /status answers "is archie well", /tasks answers "what is
// archie doing", and neither repeats the other.
//
// Every section below is optional, and a nil section means "this process has
// no truthful source for that fact". The renderer omits it. It must never
// print a zero, a dash or a probe result in its place: an operator cannot
// tell an invented "0 running" from a measured one, and a probe reports
// whether a fresh connection works rather than whether the daemon's own
// connections are alive.
// BrokerHealth is the state of this process's own task-broker connection.
type BrokerHealth struct {
// Connected is read from the connection the process already holds,
// never from a fresh dial (see BrokerHealth's package note above).
Connected bool
}
// ContainerHealth is the managed worker pool's occupancy. Capacity is the
// configured concurrency cap; zero means unlimited, so it is not a ratio.
type ContainerHealth struct {
Active int
Cap int
}
// ChatModelOutcome is the last-known result of one chat-model call. Err is
// empty when that call succeeded.
type ChatModelOutcome struct {
// Model is the model reference the call used, so a failure that
// happened before a /model switch is not read as the new model's.
Model string
At time.Time
Err string
}
// ChatModelHealth is what this process knows about its chat-model calls: the
// last one that actually happened, or Attempted=false, meaning none has been
// made since this process started. It is never a live probe -- see the file
// note above.
type ChatModelHealth struct {
Attempted bool
Outcome ChatModelOutcome
}
// ChatModelStatus reports the most recent chat-model call this process made,
// and whether any has been made at all. The daemon records outcomes at the
// one place every chat-model call passes through; nothing here dials a
// provider.
type ChatModelStatus interface {
LastChatModelOutcome() (ChatModelOutcome, bool)
}
// ChannelHealth is one channel adapter's lifecycle state, as the channel
// manager reports it. Detail carries the failure reason when the adapter
// itself recorded one.
type ChannelHealth struct {
Name string
State string
Detail string
}
// HealthReport is a point-in-time snapshot of the daemon's health. Every
// section is optional; see the file note above for what nil means.
type HealthReport struct {
Broker *BrokerHealth
Containers *ContainerHealth
ChatModel *ChatModelHealth
Channels []ChannelHealth
// LastPoll is when the most recent forge poll pass began. A nil
// LastPoll means this process does not poll at all (the standalone
// Gateway has no poll loop); a non-nil zero time means it does poll
// but no pass has started yet.
LastPoll *time.Time
}
// HealthSource reports daemon health for /status. The composition root
// supplies whichever facts the process actually owns -- the daemon's poll
// loop and container pool live in the daemon process, not in the standalone
// Gateway -- and leaves the rest nil.
type HealthSource interface {
Health() HealthReport
}
// maxChannelDetail caps one channel's detail as it appears in /status. The
// detail is an error string from an adapter and can be arbitrarily long; it
// is also already at its source (the adapter's log and the readiness
// endpoint), so truncating here costs nothing.
const maxChannelDetail = 200
// formatHealth renders the health section of /status, one "Key: value" line
// per reported fact, in a fixed order. It returns "" when nothing was
// reported, so an unwired source leaves the reply exactly as it was.
func formatHealth(report HealthReport, now time.Time) string {
lines := make([]string, 0, 5)
if b := report.Broker; b != nil {
lines = append(lines, "Broker: "+connectivity(b.Connected))
}
if c := report.Containers; c != nil {
lines = append(lines, "Containers: "+containerOccupancy(*c))
}
if len(report.Channels) > 0 {
lines = append(lines, "Channels: "+formatChannels(report.Channels))
}
if c := report.ChatModel; c != nil {
lines = append(lines, "Chat model: "+formatChatModel(*c, now))
}
if p := report.LastPoll; p != nil {
lines = append(lines, "Last poll: "+lastPollAge(*p, now))
}
// Joined with the report's hard line break, not "\n": every one of these
// is a line, and a Markdown-rendering channel soft-joins plain newlines
// into one run-on paragraph (see reportLineBreak).
return strings.Join(lines, reportLineBreak)
}
func connectivity(connected bool) string {
if connected {
return "connected"
}
return "disconnected"
}
// containerOccupancy renders active containers against the configured cap.
// An unlimited pool (cap <= 0) renders the count alone: "1/0" would read as
// "at capacity" when it means the opposite.
func containerOccupancy(c ContainerHealth) string {
if c.Cap <= 0 {
return fmt.Sprintf("%d active", c.Active)
}
return fmt.Sprintf("%d/%d active", c.Active, c.Cap)
}
func formatChannels(channels []ChannelHealth) string {
parts := make([]string, 0, len(channels))
for _, c := range channels {
part := strings.TrimSpace(c.Name + " " + c.State)
if detail := channelDetail(c.Detail); detail != "" {
part += " (" + detail + ")"
}
parts = append(parts, part)
}
return strings.Join(parts, " · ")
}
// channelDetail collapses an adapter's error detail onto one line, so a
// multi-line error cannot break the chat reply's layout, and bounds its
// length. A failed adapter is the case where an operator most needs the
// reason, so this trims rather than drops it.
// boundedDetail flattens and caps a detail so one line of /status stays one
// line. A provider error carries whatever the provider sent back, which for an
// HTML error page or a echoed request body is neither short nor single-line.
func boundedDetail(detail string) string {
flattened := strings.Join(strings.Fields(detail), " ")
runes := []rune(flattened)
if len(runes) <= maxChannelDetail {
return flattened
}
return string(runes[:maxChannelDetail]) + "…"
}
func channelDetail(detail string) string { return boundedDetail(detail) }
func formatChatModel(health ChatModelHealth, now time.Time) string {
if !health.Attempted {
// The decision behind this line: /status never probes the provider.
// Saying so plainly beats a guess, and beats omitting the line, which
// would leave an operator unable to tell "no call yet" from "the
// daemon cannot see the provider at all".
return "no calls attempted yet"
}
outcome := health.Outcome
model := ""
if outcome.Model != "" {
model = fmt.Sprintf(" (%s)", outcome.Model)
}
age := relativeAge(outcome.At, now)
switch {
case outcome.Err == "":
return fmt.Sprintf("ok %s%s", age, model)
case age == "":
return fmt.Sprintf("failed%s: %s", model, boundedDetail(outcome.Err))
default:
return fmt.Sprintf("failed %s%s: %s", age, model, boundedDetail(outcome.Err))
}
}
// lastPollAge renders when polling last began. The zero time is a real
// answer, not a missing one: a daemon that has not completed a first pass
// yet has not polled, and saying "not yet" keeps that distinct from a stale
// timestamp, which means the poller has stopped or is wedged.
func lastPollAge(at, now time.Time) string {
if at.IsZero() {
return "not yet"
}
return relativeAge(at, now)
}