diff --git a/docs/prds/status-health-surface.md b/docs/prds/status-health-surface.md index 8c411fa7c..08899fe84 100644 --- a/docs/prds/status-health-surface.md +++ b/docs/prds/status-health-surface.md @@ -1,7 +1,10 @@ # /status health surface -- what's shipped, what's proposed -**Status:** Phase 1 shipped 2026-09-03 (archie-core-wp9s). Phase 2 below is -a **proposed resolution, awaiting Sam's sign-off** -- not yet implemented. +**Status:** Phase 1 shipped 2026-09-03 (archie-core-wp9s). Phase 2 shipped +2026-09-19: broker connectivity, container pool, last poll, provider +reachability and channel state are implemented on the decisions recorded below, +with the fields that had no truthful source left out rather than faked (see +"Phase 2 -- shipped"). ## Background @@ -20,8 +23,8 @@ elsewhere (`/model` already lists active provider/model). - `/status`: itemized per-status task counts replaced with one aggregate `Queue: N in flight (...)` line (running/waiting/parked only -- queued and terminal states excluded as not a health concern). Runtime - (provider/model) is temporarily left in place; see the open question - below. + (provider/model) is temporarily left in place; see "Runtime (provider/model)" + under Phase 2's "Truthful or absent" for where that stands now. Not touched: `/agents` (`gateway.AgentReader`/`Router.Agents`) is a separate, pre-existing command that already has the right shape for "list active @@ -35,9 +38,10 @@ something to resolve by implication here. ## Phase 2 -- health checks, verified against the actual tree Each bullet below was checked against real code before being called cheap -or not; nothing here is guessed. +or not; nothing here is guessed. Both open questions were settled (see "The two +decisions") and all five fields shipped. -### Cheap -- no design decision needed, straightforward to add +### Verified as cheap against the tree - **Broker connectivity.** `internal/infrastructure/eventbus/nats.Client` already exposes `CoreConn() (*nats.Conn, error)` @@ -53,56 +57,95 @@ or not; nothing here is guessed. poll tick, with a getter. - **Queue depth.** Already shipped in Phase 1. -### Not cheap -- needs a decision before coding +### The two decisions (settled 2026-09-19, decided by position) -- **Provider reachability.** No existing check anywhere. The only way to - answer "is the provider reachable" is a real network call, which means: - a `/status` invocation either pays that latency and cost every time - (bad -- `/status` should be instant, matching `/model`'s and `/tasks`' - own current behaviour), or `/status` reports the **last** call's - outcome (success/failure/never-called) rather than probing live. The - latter needs a small success/failure counter or timestamp recorded at - the one place LLM calls already go through (`ai-sdk` responder path), - not a new probe. Recommend: last-known-outcome, not live probing -- - but this is a real product tradeoff (freshness vs. cost) worth Sam's - explicit sign-off before it's built into a command he uses daily. -- **Channel state.** `channels.Channel` (embeds `gateway.Gateway`: - `Name()/Start()/Stop()`) has no connectivity/health method today, and - telegram/email/webhook each manage their own connection lifecycle - independently. Answering "is Telegram actually connected" needs either - (a) a new method on the `Channel`/`Gateway` contract every implementation - must satisfy, or (b) each channel self-reporting into a shared registry - the daemon already holds (`ChannelManager` -- `bootstrap.go` already - calls `b.channelManager.MarkStarting/MarkRunning/MarkFailed` per - channel). (b) is very likely the right shape since `ChannelManager` - already exists and already tracks exactly this kind of state for the - dashboard; needs a read method exposed to `/status`. Flagging as - "needs a decision" only because it touches a contract multiple - implementations satisfy, not because the shape is unclear -- ChannelManager - is the obvious answer. -- **Version.** Deliberately **not** proposed for `/status`: `/version` - already owns this (`Router.Version`, `handleVersion`), and duplicating it - back into `/status` would reintroduce the exact "answers something - available elsewhere in chat" complaint this ticket exists to fix. +- **Provider reachability is last-known outcome, never a live probe.** A chat + command must not perform a blocking network call in its handler, and a probe + reports the probe's reachability rather than the daemon's. `/status` therefore + prints what actually happened -- the last chat-model call's model, age and + error, or "no calls attempted yet" when there has been none. Implemented as + `providerOutcomeRecorder`, written once in `sendChatTurn` (the single point + every chat-model call in the process passes through) and read back by + `newStatusHealth`. Nothing on this path dials a provider. The existing live + model reachability check (`readiness.go`'s `modelReachProbe`) is unchanged and + stays where it belongs: on `/health/detailed`, off the chat path. +- **Channel state comes from the existing ChannelManager.** `/status` reads + `status.Manager.Snapshot()` -- the same ledger the dashboard and the readiness + probe already use -- rather than re-deriving channel health from each adapter. + No method was added to the `Channel`/`Gateway` contract. -## Recommendation +### Phase 2 -- shipped -Ship the three cheap items (broker connectivity, container pool, last -poll) as a mechanical follow-up alongside Phase 1's `Queue:` line -- -same shape, same non-duplication reasoning, no new design surface. Land -provider reachability and channel state only after Sam confirms the -last-known-outcome approach (provider) and the ChannelManager-read -approach (channel), since both are choices about a command's behaviour -he already has strong, specific opinions about. +- **Broker connectivity.** `internal/infrastructure/eventbus/nats.Client` gained + `Connected() bool`, which reads the connection the process already holds. The + standalone Gateway (which dials NATS itself for task actions) reports the same + fact from that connection. +- **Container pool.** `container.Pool.Active()` and `Pool.Cap()` read the pool's + own counter and configured `max_concurrency` -- not a Docker listing, which + would include containers this pool does not own. A cap of zero means + unlimited, so `/status` renders the count alone rather than "1/0 active". +- **Last poll.** `daemon.Daemon.LastPollAt()` reports when the most recent poll + pass *began*; both poll paths (`poll` for single-identity `Run`, + `pollForIdentity` for `runIdentities`) stamp it. A pass that hangs leaves the + stamp stale, which is the signal -- stamping completion instead would leave a + wedged poller looking healthy. +- **Queue depth.** Already shipped in Phase 1. +- **Rendering.** `gateway.formatHealth` renders one line per reported fact + (`Broker:`, `Containers:`, `Channels:`, `Chat model:`, `Last poll:`) between + the queue line and the runtime block. The sources it reads are functions, not + captured values: occupancy and channel state change over the process's life, + and the daemon is built after the gateways, so a source captured at + construction would freeze `/status` on boot-time state. -## Packages this touches (Phase 2) +### Truthful or absent -- what is deliberately not reported -- `internal/infrastructure/eventbus/nats`: `Client.Connected() bool`. -- `internal/container`: `Pool.Active() int` (and `Pool.Cap() int` for - "N/M" display, from the already-configured `MaxConcurrency`). -- `internal/daemon`: last-poll timestamp, read accessor. -- `internal/gateway`: `formatStatus` gains the new lines; `Router` gains - whatever read surfaces back them (mirrors `StatusReader`'s existing - pattern -- a narrow interface, not a daemon-internals handle). -- Provider reachability and channel state: packages TBD pending the two - decisions above. +- **Version.** Still excluded: `/version` owns it, and duplicating it back into + `/status` is the exact "answers something available elsewhere in chat" + complaint this ticket exists to fix. +- **Sections the standalone Gateway process cannot see.** The daemon owns the + poll loop, the container pool and the channel manager; the Gateway process + owns none of them. Its `/status` reports broker and chat-model health and + omits pool, polling and channels rather than printing zeroes. Aggregating them + across processes would need a new daemon RPC -- a new subsystem, out of scope + here. For the same reason the chat-model line is per process: the daemon + records the turns it runs (Telegram, email, webhook), the Gateway records the + web chat's, and the two do not pool outcomes. Each line is true of the process + answering the command. **Decided 2026-09-19: they stay per-process.** Pooling + them would mean a new daemon↔gateway health RPC for one status line, and a + per-process line is true where an aggregate would be a claim about a process + the answering one cannot see. +- **A channel's failure reason can outlive the failure.** `status.Manager` keeps + the last non-empty `Detail` across state changes, and that one field carries + both a descriptor's standing caveat and a runtime failure reason, so a channel + that failed and later recovered can still render the old reason. Splitting the + two belongs at the producer (the manager and its descriptor), not in this + reader, which reads the field exactly as the dashboard does. +- **Container pool when Docker is unavailable.** Composition leaves + `containerPool` nil, so the section is absent. Naming a reason would mean + carrying Docker's startup error into the health surface. +- **Runtime (provider/model).** Phase 1 left this block in place; it is still + here. It is static config that `/model` and `/whoami` already answer, so it is + the only line in `/status` that this ticket's own rule says should not be + there -- but the maintainer reads `/status` daily to see the active model at a + glance, and one command answering both "is archie well" and "what is it + running on" is worth that duplication. **Decided 2026-09-19: keep it.** It is + a settled product decision, not an oversight; do not remove it as a + mechanical follow-up. +- **`/agents` copy.** The menu still advertises "List tasks currently being + worked" while `Router.Agents` is never wired (`archie-core-mxls`), so that one + published description does not match what its command does. Deleting, + rewiring or redefining `/agents` is that bead's call and is not folded in here. + +## Packages this touches (Phase 2, as shipped) + +- `internal/infrastructure/eventbus/nats`: `Client.Connected()`. +- `internal/container`: `Pool.Active()`, `Pool.Cap()`. +- `internal/daemon`: `lastPollAt`/`markPoll`/`LastPollAt()`. +- `internal/gateway`: `health.go` (health contract + renderer), `Router.Health`, + `formatStatus`. +- `internal/app/archied`: `status_health.go` (the composition-root bridge), + `ProviderOutcomes` threaded through `telegramSetup` into every turn runner, + and `sendChatTurn` recording each call's outcome. +- `internal/domain/messaging` + `internal/channels/telegram`: `/status` and + `/tasks` descriptions, which now describe both commands' actual jobs on both + published surfaces. diff --git a/internal/app/archied/bootstrap.go b/internal/app/archied/bootstrap.go index d0d8ed6b0..8972f54b8 100644 --- a/internal/app/archied/bootstrap.go +++ b/internal/app/archied/bootstrap.go @@ -21,6 +21,7 @@ import ( "syscall" "time" + natsio "github.com/nats-io/nats.go" "github.com/samcharles93/ai-sdk/runtime" "github.com/samcharles93/archie-core/internal/config" @@ -121,6 +122,19 @@ type boot struct { catalogModels []string bus *events.Bus + // taskActionsConn is the standalone Gateway process's own NATS connection: + // it dials the broker directly (the Gateway does not build the + // consumer/stream client the daemon does) and uses it for task actions. + // Held so /status can report broker connectivity from the connection this + // process actually uses instead of dialling a fresh one. Nil in the daemon. + taskActionsConn *natsio.Conn + // providerOutcomes records the last-known outcome of every chat-model call + // this process makes, read back by /status (newStatusHealth). Built before + // the chat runtime, which carries it into each turn runner. + providerOutcomes *providerOutcomeRecorder + // statusHealth is the /status health source for this process, built from + // the subsystems that exist here. + statusHealth gateway.HealthSource // rateLimiter is the shared per-(channel, sender) inbound budget every // chat Router is given. Nil when [chat.rate_limit] is not configured, // which leaves rate limiting off. diff --git a/internal/app/archied/chat_setup.go b/internal/app/archied/chat_setup.go index 119e94912..59982ab6c 100644 --- a/internal/app/archied/chat_setup.go +++ b/internal/app/archied/chat_setup.go @@ -56,6 +56,10 @@ type chatSetup struct { // the per-turn memory_create/update/delete/list tools (docs/prds/ // memory-engine-unification.md §5). Nil omits those tools. MemoryWriter gateway.MemoryWriteStore + // ProviderOutcomes records the outcome of every chat-model call this + // process makes, which the /status health source reads back. Nil disables + // recording (the recorder's methods tolerate that rather than panicking). + ProviderOutcomes *providerOutcomeRecorder } // daemonRunningVersions reports the component versions this process can @@ -177,7 +181,7 @@ func newChatTurnRunner( Sessions: sessionStore, Models: s.ChatModels, Personas: s.Personas, - Model: newChatTurnModel(s.LLM, s.ToolReg, cfg.Chat.MaxSteps, toolLimits(cfg)), + Model: newChatTurnModel(s.LLM, s.ToolReg, cfg.Chat.MaxSteps, toolLimits(cfg), s.ProviderOutcomes), TaskLister: s.ChatTaskLister, Tasks: s.ChatTasks, TaskLogs: s.ChatTaskLogs, @@ -233,7 +237,20 @@ func chatRepoEnv(cfg config.Config, identity string) []gateway.RepoEnv { return env } -func sendChatTurn(ctx context.Context, llm *runtime.Runtime, chatModel string, options core.GenerateOptions, turn gateway.TurnStream) (string, error) { +// sendChatTurn runs one chat-model call and records its outcome for /status' +// chat-model line. Recording happens in this one wrapper because every +// chat-model call in this process passes through it: recording at the call +// sites instead would let a path make a call and leave the reported +// last-known outcome stale, which is a /status that lies about the provider. +func sendChatTurn(ctx context.Context, llm *runtime.Runtime, chatModel string, options core.GenerateOptions, turn gateway.TurnStream, outcomes *providerOutcomeRecorder) (string, error) { + text, err := runChatTurn(ctx, llm, chatModel, options, turn) + // record tolerates a nil recorder: a setup built without one (tests, a + // deployment whose health surface is unwired) still makes its calls. + outcomes.record(chatModel, err) + return text, err +} + +func runChatTurn(ctx context.Context, llm *runtime.Runtime, chatModel string, options core.GenerateOptions, turn gateway.TurnStream) (string, error) { if turn == nil { result, err := llm.Chat(ctx, chatModel, options) if err != nil { @@ -357,6 +374,7 @@ type chatTitleGenerator struct { log *slog.Logger llm *runtime.Runtime chatModels gateway.ModelManager + outcomes *providerOutcomeRecorder } // newChatTitleGenerator wires an LLM-backed title generator for a chat @@ -366,7 +384,7 @@ func newChatTitleGenerator(s chatSetup) gateway.TitleGenerator { if s.LLM == nil || s.ChatModels == nil { return nil } - return &chatTitleGenerator{log: s.Log, llm: s.LLM, chatModels: s.ChatModels} + return &chatTitleGenerator{log: s.Log, llm: s.LLM, chatModels: s.ChatModels, outcomes: s.ProviderOutcomes} } func (g *chatTitleGenerator) GenerateTitle(ctx context.Context, sessionID, firstMessage string) (string, error) { @@ -377,7 +395,7 @@ func (g *chatTitleGenerator) GenerateTitle(ctx context.Context, sessionID, first // No tools: a title is a single completion. The active model is read // at call time so a /model switch applies to titles too. text, err := sendChatTurn(ctx, g.llm, g.chatModels.ActiveModel(), - core.GenerateOptions{Messages: messages, MaxSteps: 1}, nil) + core.GenerateOptions{Messages: messages, MaxSteps: 1}, nil, g.outcomes) if err != nil { if g.log != nil { g.log.Error("session title generation failed", "session", sessionID, "err", err) diff --git a/internal/app/archied/chat_turn_model.go b/internal/app/archied/chat_turn_model.go index d0536ef12..c8584207d 100644 --- a/internal/app/archied/chat_turn_model.go +++ b/internal/app/archied/chat_turn_model.go @@ -21,6 +21,8 @@ type chatTurnModel struct { registry *tools.Registry maxSteps int limits agentexec.ToolLimits + // outcomes records each call's result for /status (see sendChatTurn). + outcomes *providerOutcomeRecorder } func newChatTurnModel( @@ -28,12 +30,14 @@ func newChatTurnModel( registry *tools.Registry, maxSteps int, limits agentexec.ToolLimits, + outcomes *providerOutcomeRecorder, ) gateway.TurnModel { return &chatTurnModel{ llm: llm, registry: registry, maxSteps: maxSteps, limits: limits, + outcomes: outcomes, } } @@ -55,6 +59,7 @@ func (m *chatTurnModel) Prepare( options: options, toolInfo: toolSummaries(options.Tools), toolTokens: gateway.EstimateTokens(string(toolSchema)), + outcomes: m.outcomes, }, nil } @@ -64,6 +69,7 @@ type preparedChatTurnModel struct { options core.GenerateOptions toolInfo []gateway.ToolSummary toolTokens int + outcomes *providerOutcomeRecorder } func (m *preparedChatTurnModel) ToolSummaries() []gateway.ToolSummary { @@ -94,5 +100,5 @@ func (m *preparedChatTurnModel) Generate( // This is one provider response's output allowance, not a turn- // continuation budget. Tool loops remain free to continue. options.MaxTokens = request.MaxOutputTokens - return sendChatTurn(ctx, m.llm, m.model, options, stream) + return sendChatTurn(ctx, m.llm, m.model, options, stream, m.outcomes) } diff --git a/internal/app/archied/gateway.go b/internal/app/archied/gateway.go index 42129719d..8b3fc5437 100644 --- a/internal/app/archied/gateway.go +++ b/internal/app/archied/gateway.go @@ -99,6 +99,10 @@ func RunGateway(ctx context.Context, options GatewayOptions) error { return fmt.Errorf("connect gateway task actions: %w", err) } b.addCleanup(nc.Close) + // Held before the chat runtime is built so /status' health source can + // report this process's own broker connection (newStatusHealth reads it + // lazily, but the field must be set before requests can arrive). + b.taskActionsConn = nc contract, err := b.startGatewayRuntime(ctx, taskactions.Client{Conn: nc, Timeout: 30 * time.Second}) if err != nil { return err diff --git a/internal/app/archied/gateway_runtime.go b/internal/app/archied/gateway_runtime.go index 1032b18c9..12f0860a6 100644 --- a/internal/app/archied/gateway_runtime.go +++ b/internal/app/archied/gateway_runtime.go @@ -28,6 +28,14 @@ func (b *boot) setupChatRuntime(cfg config.Config) { // ── Persona registry ───────────────────────────────────────────── b.personas = gateway.NewPersonaRegistry(gateway.DefaultPersonas()) + // ── Operator health surface ────────────────────────────────────── + // Built before the gateways because every turn runner and router built + // later carries both: the recorder is written by sendChatTurn, the source + // is read by /status. See newStatusHealth for why the source reads boot's + // fields lazily. + b.providerOutcomes = newProviderOutcomeRecorder() + b.statusHealth = newStatusHealth(b) + profiles, defaultChatIdentity := chatTaskProfiles(cfg) var chatTasks gateway.TaskCreator if len(profiles) > 0 { @@ -59,6 +67,7 @@ func (b *boot) setupGatewayChat(ctx context.Context, actor gateway.ChatTaskActor router.Updates = b.updateService router.InitSessions(b.chatSessionStore) configureTaskCommands(router, b.chatTasks, b.chatController, chatTaskListerAdapter{tasks: b.stateStore.Tasks}, b.defaultChatIdentity) + router.Health = b.statusHealth setup := chatSetup{ Cfg: config.NewHolder(cfg), LLM: b.llm, ChatModels: b.chatModels, ToolReg: b.toolReg, Personas: b.personas, ChatTasks: b.chatTasks, @@ -69,6 +78,9 @@ func (b *boot) setupGatewayChat(ctx context.Context, actor gateway.ChatTaskActor Bus: b.bus, Log: b.log, MemoryEngine: b.memoryStore(), MemoryWriter: b.memoryWriter(), + // The Gateway executes the web chat's turns, so it is the process + // that must record their outcomes for /status. + ProviderOutcomes: b.providerOutcomes, } router.LLM, router.LLMStream = makeChatLLMResponder(ctx, "web", setup, b.chatSessionStore, router) router.Titles = newChatTitleGenerator(setup) diff --git a/internal/app/archied/status_health.go b/internal/app/archied/status_health.go new file mode 100644 index 000000000..9d03bb616 --- /dev/null +++ b/internal/app/archied/status_health.go @@ -0,0 +1,186 @@ +// status_health.go is the composition-root bridge behind /status' health +// section: it reads whichever subsystems THIS process owns and hands /status a +// narrow report, so the gateway package never takes a daemon, pool or manager +// handle. +// +// The split matters because archie's operator surface spans two processes. +// The daemon owns the forge poll loop, the container pool and the channel +// manager; the standalone Gateway owns none of those. Each process therefore +// reports what it actually has and leaves the rest absent -- /status shows a +// shorter list in the Gateway, not a list of zeroes. +package archied + +import ( + "cmp" + "sync" + "time" + + "github.com/samcharles93/archie-core/internal/channels/status" + "github.com/samcharles93/archie-core/internal/gateway" +) + +// providerOutcomeRecorder is the producer behind /status' chat-model line. It +// holds the outcome of the last chat-model call this process made, recorded at +// the one place every chat-model call passes through (sendChatTurn), so there +// is no second path that could report a different answer. +// +// It never dials a provider: the line reports what happened, and a fresh probe +// inside a chat command would report the probe's own reachability while +// costing the command a network round trip. +type providerOutcomeRecorder struct { + mu sync.Mutex + attempted bool + outcome gateway.ChatModelOutcome +} + +func newProviderOutcomeRecorder() *providerOutcomeRecorder { return &providerOutcomeRecorder{} } + +// record stores one finished call's outcome. err is the call's own error, nil +// when it succeeded. A nil recorder records nothing -- call paths built +// without one (tests, minimal setups) must not panic. +func (r *providerOutcomeRecorder) record(model string, err error) { + if r == nil { + return + } + outcome := gateway.ChatModelOutcome{Model: model, At: time.Now().UTC()} + if err != nil { + outcome.Err = err.Error() + } + r.mu.Lock() + defer r.mu.Unlock() + r.outcome = outcome + r.attempted = true +} + +// LastChatModelOutcome reports the most recent recorded call, and whether any +// call has been recorded at all. +func (r *providerOutcomeRecorder) LastChatModelOutcome() (gateway.ChatModelOutcome, bool) { + if r == nil { + return gateway.ChatModelOutcome{}, false + } + r.mu.Lock() + defer r.mu.Unlock() + return r.outcome, r.attempted +} + +// statusHealth assembles gateway.HealthReport from this process's own +// subsystems. Every source is optional; a nil source means this process holds +// no truthful input for that fact and the section is left out of the report. +// +// The sources are functions rather than captured values because they change +// over the life of the process: the container pool's occupancy, the channel +// states, and the poll timestamp all move, and the two NATS connections can +// drop. Capturing them at construction would freeze /status on the daemon's +// boot-time state -- occupancy would read 0/4 forever, and every channel would +// read as configured-but-not-started. +type statusHealth struct { + broker func() (connected, ok bool) + containers func() (active, capacity int, ok bool) + channels func() (channels []gateway.ChannelHealth, ok bool) + chatModel *providerOutcomeRecorder + lastPoll func() (at time.Time, ok bool) +} + +func (s statusHealth) Health() gateway.HealthReport { + var report gateway.HealthReport + if s.broker != nil { + if connected, ok := s.broker(); ok { + report.Broker = &gateway.BrokerHealth{Connected: connected} + } + } + if s.containers != nil { + if active, capacity, ok := s.containers(); ok { + report.Containers = &gateway.ContainerHealth{Active: active, Cap: capacity} + } + } + if s.channels != nil { + if channels, ok := s.channels(); ok { + report.Channels = channels + } + } + if s.chatModel != nil { + outcome, attempted := s.chatModel.LastChatModelOutcome() + report.ChatModel = &gateway.ChatModelHealth{Attempted: attempted, Outcome: outcome} + } + if s.lastPoll != nil { + if at, ok := s.lastPoll(); ok { + report.LastPoll = &at + } + } + return report +} + +// newStatusHealth builds the health source for this process. Sources whose +// subsystem is absent here (a deployment with no container pool, the Gateway +// process with no daemon) stay nil, and /status omits them. +// +// The closures read boot's fields at call time on purpose. Gateways are wired +// before the daemon is built, so boot.d is nil when this runs -- capturing it +// here would leave the poll section permanently absent in the one process that +// actually polls, with nothing at runtime to show the wire was dead. +func newStatusHealth(b *boot) gateway.HealthSource { + s := statusHealth{ + chatModel: b.providerOutcomes, + containers: func() (int, int, bool) { + if b.containerPool == nil { + return 0, 0, false + } + return b.containerPool.Active(), b.containerPool.Cap(), true + }, + // No channel source: the chat front-ends run in the extracted + // Messaging Service, so this process cannot observe a lifecycle it + // does not drive. Reporting anything here would be a guess, and + // "truthful or absent" says omit the section instead. Restoring it + // needs the Messaging Service to publish its channel lifecycle + // (archie-core-8cda.6.8). + channels: nil, + lastPoll: func() (time.Time, bool) { + if b.d == nil { + return time.Time{}, false + } + return b.d.LastPollAt(), true + }, + broker: func() (bool, bool) { + switch { + case b.natsClient != nil: + return b.natsClient.Connected(), true + case b.taskActionsConn != nil: + return b.taskActionsConn.IsConnected(), true + default: + return false, false + } + }, + } + return s +} + +// channelHealth projects the channel manager's snapshot into /status' view, so +// the gateway package does not depend on the channels package. It mirrors +// readiness.go's channelStates projection for the same reason. +// +// An unset display name falls back to the channel id: a line reading +// " running" would leave an operator unable to tell which channel is broken, +// which is the one thing this line exists to say. +// +// Detail is carried through whenever it is set, exactly as the dashboard reads +// it. It is one field serving two purposes -- a descriptor's standing +// configuration caveat ("token set, but the allowlist is empty") and a runtime +// failure reason -- and the manager keeps the last non-empty one, so a channel +// that failed and later recovered still shows the old reason. Filtering by +// state here would drop the caveat that matters most on a *running* channel +// (the empty allowlist one), so the split belongs at the producer, not here. +func channelHealth(m *status.Manager) []gateway.ChannelHealth { + if m == nil { + return nil + } + snapshot := m.Snapshot() + out := make([]gateway.ChannelHealth, 0, len(snapshot)) + for _, s := range snapshot { + out = append(out, gateway.ChannelHealth{ + Name: cmp.Or(s.Name, s.ID), + State: string(s.State), + Detail: s.Detail, + }) + } + return out +} diff --git a/internal/app/archied/status_health_test.go b/internal/app/archied/status_health_test.go new file mode 100644 index 000000000..8ffb518a8 --- /dev/null +++ b/internal/app/archied/status_health_test.go @@ -0,0 +1,465 @@ +package archied + +import ( + "context" + "errors" + "go/ast" + "go/parser" + "go/token" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/samcharles93/ai-sdk/chat" + "github.com/samcharles93/ai-sdk/core" + + "github.com/samcharles93/archie-core/internal/agentexec" + "github.com/samcharles93/archie-core/internal/channels/status" + "github.com/samcharles93/archie-core/internal/daemon" + "github.com/samcharles93/archie-core/internal/domain/messaging" + "github.com/samcharles93/archie-core/internal/gateway" + "github.com/samcharles93/archie-core/internal/store" +) + +// TestProviderOutcomeRecorderRecordsTheLastCallOnly pins the producer behind +// /status' chat-model line: it reports what actually happened, and the moment +// something else happens it reports that instead. There is no probe anywhere +// in this path -- a chat command must not pay a network round trip, and a +// probe would answer for the probe's connection, not the daemon's. +func TestProviderOutcomeRecorderRecordsTheLastCallOnly(t *testing.T) { + failure := errors.New("401 Unauthorized") + + tests := []struct { + name string + record func(*providerOutcomeRecorder) + wantAttempted bool + wantErr string + wantModel string + }{ + { + name: "nothing recorded yet", + record: func(*providerOutcomeRecorder) {}, + wantAttempted: false, + }, + { + name: "successful call", + record: func(r *providerOutcomeRecorder) { r.record("openai/gpt-5.6", nil) }, + wantAttempted: true, + wantModel: "openai/gpt-5.6", + }, + { + name: "failed call", + record: func(r *providerOutcomeRecorder) { r.record("openai/gpt-5.6", failure) }, + wantAttempted: true, + wantErr: "401 Unauthorized", + wantModel: "openai/gpt-5.6", + }, + { + name: "a later success replaces an earlier failure", + record: func(r *providerOutcomeRecorder) { + r.record("openai/gpt-5.6", failure) + r.record("deepseek/deepseek-v4-pro", nil) + }, + wantAttempted: true, + wantModel: "deepseek/deepseek-v4-pro", + }, + { + name: "a later failure replaces an earlier success", + record: func(r *providerOutcomeRecorder) { + r.record("openai/gpt-5.6", nil) + r.record("openai/gpt-5.6", failure) + }, + wantAttempted: true, + wantErr: "401 Unauthorized", + wantModel: "openai/gpt-5.6", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + recorder := newProviderOutcomeRecorder() + before := time.Now().Add(-time.Second) + tc.record(recorder) + + outcome, attempted := recorder.LastChatModelOutcome() + if attempted != tc.wantAttempted { + t.Fatalf("LastChatModelOutcome() attempted = %v, want %v", attempted, tc.wantAttempted) + } + if !tc.wantAttempted { + return + } + if outcome.Err != tc.wantErr { + t.Errorf("outcome.Err = %q, want %q", outcome.Err, tc.wantErr) + } + if outcome.Model != tc.wantModel { + t.Errorf("outcome.Model = %q, want %q", outcome.Model, tc.wantModel) + } + if outcome.At.Before(before) || outcome.At.After(time.Now()) { + t.Errorf("outcome.At = %v, want a time within the call's window", outcome.At) + } + }) + } + + // sendChatTurn is written to tolerate a nil recorder (a call path built + // without one must not panic), and a nil recorder must claim nothing. + var absent *providerOutcomeRecorder + absent.record("openai/gpt-5.6", failure) + if _, attempted := absent.LastChatModelOutcome(); attempted { + t.Error("a nil recorder reported an attempted call; it has nothing to report") + } +} + +// TestSendChatTurnRecordsEachCallOutcome pins the wire between the chat path +// and /status: sendChatTurn is the single point every chat-model call in this +// process passes through, and it must record what happened there rather than +// trusting each caller to remember. A recorder that is never written is the +// failure this pins -- /status would say "no calls attempted yet" forever +// while chat worked, which reads as a broken provider rather than a broken +// wire. +func TestSendChatTurnRecordsEachCallOutcome(t *testing.T) { + tests := []struct { + name string + reply string + status int + wantErr string + }{ + { + name: "successful call", + reply: `{"id":"chatcmpl-1","object":"chat.completion","created":1,"model":"gpt-5.6","choices":[{"index":0,"message":{"role":"assistant","content":"hello"},"finish_reason":"stop"}],"usage":{"prompt_tokens":1,"completion_tokens":1,"total_tokens":2}}`, + status: http.StatusOK, + }, + { + name: "failed call", + reply: `{"error":{"message":"invalid api key","type":"invalid_request_error"}}`, + status: http.StatusUnauthorized, + wantErr: "invalid api key", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + var requestedPath string + api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requestedPath = r.URL.Path + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(tc.status) + _, _ = io.WriteString(w, tc.reply) + })) + t.Cleanup(api.Close) + + llm := agentexec.NewRuntime(map[string]agentexec.Provider{ + "openai": {Class: "openai", BaseURL: api.URL}, + }) + if llm == nil { + t.Fatal("NewRuntime = nil, want a runtime for the configured provider") + } + + recorder := newProviderOutcomeRecorder() + _, err := sendChatTurn(t.Context(), llm, "openai/gpt-5.6", core.GenerateOptions{ + Messages: []chat.Message{{Role: chat.RoleUser, Content: "hi"}}, + MaxSteps: 1, + }, nil, recorder) + + if tc.wantErr == "" && err != nil { + t.Fatalf("sendChatTurn = %v, want success (request path %q)", err, requestedPath) + } + if tc.wantErr != "" && (err == nil || !strings.Contains(err.Error(), tc.wantErr)) { + t.Fatalf("sendChatTurn = %v, want an error containing %q", err, tc.wantErr) + } + + outcome, attempted := recorder.LastChatModelOutcome() + if !attempted { + t.Fatal("recorded outcome = attempted false, want the call recorded at sendChatTurn") + } + if outcome.Model != "openai/gpt-5.6" { + t.Errorf("outcome.Model = %q, want the model the call used", outcome.Model) + } + if got := outcome.Err != ""; got != (tc.wantErr != "") { + t.Errorf("outcome.Err = %q, want an error report = %v", outcome.Err, tc.wantErr != "") + } + }) + } +} + +// TestStatusHealthOmitsSourcesThisProcessDoesNotOwn pins "truthful or +// absent": with no subsystem wired, the report is empty rather than full of +// zeroes that read as measurements. This is the shape the standalone Gateway +// process produces -- it owns no container pool, no poll loop and no channel +// manager, and must not pretend otherwise. +func TestStatusHealthOmitsSourcesThisProcessDoesNotOwn(t *testing.T) { + report := statusHealth{}.Health() + + if report.Broker != nil { + t.Errorf("Broker = %+v, want nil when this process holds no broker connection", report.Broker) + } + if report.Containers != nil { + t.Errorf("Containers = %+v, want nil when this process owns no container pool", report.Containers) + } + if report.ChatModel != nil { + t.Errorf("ChatModel = %+v, want nil when this process records no model calls", report.ChatModel) + } + if report.Channels != nil { + t.Errorf("Channels = %+v, want nil when this process owns no channel manager", report.Channels) + } + if report.LastPoll != nil { + t.Errorf("LastPoll = %+v, want nil when this process does not poll", report.LastPoll) + } +} + +// TestStatusHealthReportsEveryWiredSource pins that each wired source reaches +// the report, and that a source reporting "nothing has happened yet" stays +// distinguishable from an absent source. +func TestStatusHealthReportsEveryWiredSource(t *testing.T) { + polled := time.Date(2026, 9, 19, 12, 0, 0, 0, time.UTC) + recorder := newProviderOutcomeRecorder() + recorder.record("openai/gpt-5.6", errors.New("401 Unauthorized")) + + channels := status.NewManager([]status.Descriptor{ + {ID: "telegram", Name: "Telegram", Configured: true}, + {ID: "email", Configured: true}, + }) + channels.MarkRunning("telegram") + channels.MarkFailed("email", "dial tcp: connection refused") + + tests := []struct { + name string + check func(*testing.T, gateway.HealthReport) + src statusHealth + }{ + { + name: "broker", + src: statusHealth{broker: func() (bool, bool) { return true, true }}, + check: func(t *testing.T, r gateway.HealthReport) { + if r.Broker == nil || !r.Broker.Connected { + t.Errorf("Broker = %+v, want connected", r.Broker) + } + }, + }, + { + name: "container pool", + src: statusHealth{containers: func() (int, int, bool) { return 2, 4, true }}, + check: func(t *testing.T, r gateway.HealthReport) { + if r.Containers == nil || r.Containers.Active != 2 || r.Containers.Cap != 4 { + t.Errorf("Containers = %+v, want 2/4", r.Containers) + } + }, + }, + { + name: "channel manager", + src: statusHealth{channels: func() ([]gateway.ChannelHealth, bool) { return channelHealth(channels), true }}, + check: func(t *testing.T, r gateway.HealthReport) { + want := []gateway.ChannelHealth{ + {Name: "Telegram", State: "running"}, + {Name: "email", State: "failed", Detail: "dial tcp: connection refused"}, + } + if len(r.Channels) != len(want) { + t.Fatalf("Channels = %+v, want %+v", r.Channels, want) + } + for i := range want { + if r.Channels[i] != want[i] { + t.Errorf("Channels[%d] = %+v, want %+v", i, r.Channels[i], want[i]) + } + } + }, + }, + { + name: "chat model recorder", + src: statusHealth{chatModel: recorder}, + check: func(t *testing.T, r gateway.HealthReport) { + if r.ChatModel == nil || !r.ChatModel.Attempted { + t.Fatalf("ChatModel = %+v, want the recorded call", r.ChatModel) + } + if r.ChatModel.Outcome.Err != "401 Unauthorized" { + t.Errorf("ChatModel.Outcome.Err = %q, want the recorded error", r.ChatModel.Outcome.Err) + } + }, + }, + { + name: "chat model recorder with no calls yet", + src: statusHealth{chatModel: newProviderOutcomeRecorder()}, + check: func(t *testing.T, r gateway.HealthReport) { + if r.ChatModel == nil { + t.Fatal("ChatModel = nil, want a present section saying nothing has been attempted") + } + if r.ChatModel.Attempted { + t.Error("ChatModel.Attempted = true, want false before any call") + } + }, + }, + { + name: "poll loop that has run", + src: statusHealth{lastPoll: func() (time.Time, bool) { return polled, true }}, + check: func(t *testing.T, r gateway.HealthReport) { + if r.LastPoll == nil || !r.LastPoll.Equal(polled) { + t.Errorf("LastPoll = %v, want %v", r.LastPoll, polled) + } + }, + }, + { + name: "poll loop that has not run a pass yet", + src: statusHealth{lastPoll: func() (time.Time, bool) { return time.Time{}, true }}, + check: func(t *testing.T, r gateway.HealthReport) { + if r.LastPoll == nil { + t.Fatal("LastPoll = nil, want a present \"not yet\" value") + } + if !r.LastPoll.IsZero() { + t.Errorf("LastPoll = %v, want the zero time", r.LastPoll) + } + }, + }, + { + name: "poll loop that is not this process's to report", + src: statusHealth{lastPoll: func() (time.Time, bool) { return polled, false }}, + check: func(t *testing.T, r gateway.HealthReport) { + if r.LastPoll != nil { + t.Errorf("LastPoll = %v, want nil", r.LastPoll) + } + }, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + tc.check(t, tc.src.Health()) + }) + } +} + +// TestStatusHealthReadsDeferredSourcesAtCallTime pins the composition-order +// trap this source sits behind: the daemon is built AFTER the gateways +// (main.go runs setupGateways before buildDaemon), so the health source is +// constructed while boot.d is still nil. A source that captured boot.d at +// construction would report the poll section as permanently absent -- /status +// would look wired and never say anything about polling. +func TestStatusHealthReadsDeferredSourcesAtCallTime(t *testing.T) { + b := &boot{} + src := newStatusHealth(b) + + if got := src.Health().LastPoll; got != nil { + t.Fatalf("LastPoll = %v with no daemon yet, want nil", got) + } + + b.d = &daemon.Daemon{} + + got := src.Health().LastPoll + if got == nil { + t.Fatal("LastPoll = nil after the daemon was built, want the source to read boot.d at call time") + } + if !got.IsZero() { + t.Errorf("LastPoll = %v, want the zero time: no poll pass has run yet", got) + } +} + +// TestChannelHealthProjectsManagerSnapshot pins the mapping from the channel +// manager's own state to what /status renders -- including the two edges an +// operator would otherwise have to guess at: a channel with no display name +// shows its id, and an unconfigured channel is reported as such rather than +// omitted. +func TestChannelHealthProjectsManagerSnapshot(t *testing.T) { + manager := status.NewManager([]status.Descriptor{ + {ID: "telegram", Name: "Telegram", Configured: true}, + {ID: "email", Configured: true}, + {ID: "webhook"}, + }) + manager.MarkStarting("telegram") + manager.MarkRunning("telegram") + manager.MarkFailed("email", "listen tcp: address already in use") + + tests := []struct { + name string + manager *status.Manager + want []gateway.ChannelHealth + }{ + { + name: "no channel manager", + manager: nil, + want: nil, + }, + { + name: "manager snapshot", + manager: manager, + want: []gateway.ChannelHealth{ + {Name: "Telegram", State: "running"}, + {Name: "email", State: "failed", Detail: "listen tcp: address already in use"}, + {Name: "webhook", State: "stopped"}, + }, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := channelHealth(tc.manager) + if len(got) != len(tc.want) { + t.Fatalf("channelHealth() = %+v, want %+v", got, tc.want) + } + for i := range tc.want { + if got[i] != tc.want[i] { + t.Errorf("channelHealth()[%d] = %+v, want %+v", i, got[i], tc.want[i]) + } + } + }) + } +} + +// TestSetupGatewayChatCarriesStatusHealth pins the last hop: the router the +// composition root builds must expose the health source, or /status answers +// from the queue and runtime sections alone while the composition root +// believes health is wired. +// +// Asserted on the source rather than by calling setupGatewayChat, which needs +// a fully built boot (stores, runtime, memory engines) to run. That is the +// same technique composition_order_test.go uses for wiring nothing at runtime +// observes. The predecessor of this test drove buildTelegramRouter, which the +// Messaging Service extraction deleted along with the daemon's own routers; +// setupGatewayChat is now the sole production constructor of a Router. +func TestSetupGatewayChatCarriesStatusHealth(t *testing.T) { + fileset := token.NewFileSet() + file, err := parser.ParseFile(fileset, "gateway_runtime.go", nil, 0) + if err != nil { + t.Fatal(err) + } + body := methodBody(t, file, "setupGatewayChat") + + var wired bool + ast.Inspect(body, func(n ast.Node) bool { + assign, ok := n.(*ast.AssignStmt) + if !ok || len(assign.Lhs) != 1 { + return true + } + sel, ok := assign.Lhs[0].(*ast.SelectorExpr) + if !ok || sel.Sel.Name != "Health" { + return true + } + if ident, ok := sel.X.(*ast.Ident); ok && ident.Name == "router" { + wired = true + } + return true + }) + if !wired { + t.Fatal("setupGatewayChat never assigns router.Health; /status would lose its health section") + } +} + +// TestRouterWithHealthAnswersStatus is the behavioural half: a router holding a +// health source renders it into /status. +func TestRouterWithHealthAnswersStatus(t *testing.T) { + sessions := gateway.NewSessionStoreMemory() + t.Cleanup(func() { _ = sessions.Close() }) + + st := store.OpenTest(t) + router := gateway.NewRouter(st, nil, "web") + router.InitSessions(sessions) + router.Health = statusHealth{broker: func() (bool, bool) { return true, true }} + + reply, err := router.Route(context.Background(), gateway.Inbound{ + Message: messaging.Message{Role: messaging.RoleUser, Text: "/status"}, + }) + if err != nil { + t.Fatalf("Route(/status): %v", err) + } + if !strings.Contains(reply, "Broker: connected") { + t.Errorf("reply = %q, want the wired broker line", reply) + } +} diff --git a/internal/channels/telegram/commands.go b/internal/channels/telegram/commands.go index e3f754414..64b5b353c 100644 --- a/internal/channels/telegram/commands.go +++ b/internal/channels/telegram/commands.go @@ -19,8 +19,8 @@ type commandSpec struct { // published command menu and /help. Keeping the discoverability surfaces // together prevents a newly registered command from being omitted from help. var gatewayCommandSpecs = []commandSpec{ - {Command: "status", Description: "Show daemon health", Usage: "/status"}, - {Command: "tasks", Description: "Show running and parked work", Usage: "/tasks"}, + {Command: "status", Description: "Show daemon health: broker, workers, channels, provider", Usage: "/status"}, + {Command: "tasks", Description: "Show your tasks: state, stage, age, parked reason", Usage: "/tasks"}, {Command: "version", Description: "Show installed Archie versions", Usage: "/version"}, {Command: "update", Description: "Check for Archie updates", Usage: "/update"}, {Command: "model", Description: "Choose a provider and model or switch directly", Usage: "/model [provider/model]"}, diff --git a/internal/channels/telegram/commands_test.go b/internal/channels/telegram/commands_test.go new file mode 100644 index 000000000..60f0d883f --- /dev/null +++ b/internal/channels/telegram/commands_test.go @@ -0,0 +1,67 @@ +package telegram + +import ( + "strings" + "testing" + + "github.com/samcharles93/archie-core/internal/domain/messaging" +) + +// TestStatusAndTasksCopyMatchesWhatEachCommandDoes pins the operator-facing +// division of labour archie-core-wp9s restored: /status reports daemon health +// and says nothing about the task list, /tasks reports the work view and says +// nothing about daemon health. The menu and /help are the only place an +// operator learns which of the two to reach for, so copy that promises the +// other command's job sends them to the wrong one -- which is exactly what +// "Show task counts by state" did before Phase 1. +// +// Both published surfaces are covered, because both are "registered with the +// platform": Telegram's own menu/help specs, and the shared messaging specs +// every other adapter and the dashboard's command palette render from. The two +// are also required to agree for these two commands: a menu that describes +// /status differently from the palette is the same lie told twice. +func TestStatusAndTasksCopyMatchesWhatEachCommandDoes(t *testing.T) { + telegramCopy := make(map[string]string, len(gatewayCommandSpecs)) + for _, spec := range gatewayCommandSpecs { + telegramCopy[spec.Command] = spec.Description + } + sharedCopy := make(map[string]string, len(messaging.LocalCommandSpecs())) + for _, spec := range messaging.LocalCommandSpecs() { + sharedCopy[strings.TrimPrefix(spec.Command, "/")] = spec.Description + } + + surfaces := []struct { + name string + copy map[string]string + }{ + {name: "telegram menu and help", copy: telegramCopy}, + {name: "shared command specs", copy: sharedCopy}, + } + + for _, surface := range surfaces { + t.Run(surface.name, func(t *testing.T) { + status := strings.ToLower(surface.copy["status"]) + tasks := strings.ToLower(surface.copy["tasks"]) + + if !strings.Contains(status, "health") { + t.Errorf("/status is published as %q, want it described as the health surface", status) + } + if strings.Contains(status, "task") { + t.Errorf("/status is published as %q, want no promise of a task view -- that is /tasks' job", status) + } + if !strings.Contains(tasks, "task") { + t.Errorf("/tasks is published as %q, want it described as the task view", tasks) + } + if strings.Contains(tasks, "health") { + t.Errorf("/tasks is published as %q, want no promise of daemon health -- that is /status' job", tasks) + } + }) + } + + for _, cmd := range []string{"status", "tasks"} { + if telegramCopy[cmd] != sharedCopy[cmd] { + t.Errorf("/%s description differs between the Telegram menu (%q) and the shared specs (%q)", + cmd, telegramCopy[cmd], sharedCopy[cmd]) + } + } +} diff --git a/internal/container/pool.go b/internal/container/pool.go index 323829f4a..134672191 100644 --- a/internal/container/pool.go +++ b/internal/container/pool.go @@ -501,3 +501,19 @@ func (p *Pool) recoverOrphans(ctx context.Context) { p.log.Info("orphan recovery complete", "count", len(list.Items)) } } + +// Active is the number of containers this pool currently holds: acquired but +// not yet released. It reads the pool's own counter rather than listing +// Docker containers, which would include containers this pool does not own +// (orphans from a crashed daemon, another instance's workers). +func (p *Pool) Active() int { + p.mu.Lock() + defer p.mu.Unlock() + return p.active +} + +// Cap is the concurrency cap this pool enforces, from the configured +// containers.max_concurrency. Zero means unlimited (Acquire only enforces a +// cap when MaxConcurrency > 0), so a caller rendering "active/cap" must not +// treat zero as a cap of zero. +func (p *Pool) Cap() int { return p.cfg.MaxConcurrency } diff --git a/internal/container/pool_test.go b/internal/container/pool_test.go index 9f4561b12..7269bc286 100644 --- a/internal/container/pool_test.go +++ b/internal/container/pool_test.go @@ -3,6 +3,7 @@ package container import ( "context" "encoding/json" + "fmt" "net/http" "net/http/httptest" "os" @@ -558,3 +559,93 @@ func TestReleaseAfterTheReaperFiredDoesNotTearDownTwice(t *testing.T) { t.Errorf("remove called %d times, want 1: the reaper and Release both tore the container down", got) } } + +// TestPoolActiveReportsInFlightContainers pins the worker-pool read surface +// /status reports as its container line: how many containers this pool +// currently holds, and the concurrency cap it enforces. +// +// Both come from the pool's own counter. A Docker listing would report +// containers this pool does not own (orphans from a crashed daemon, containers +// another archie instance started) and would make reading /status cost a +// Docker round trip on every invocation. +func TestPoolActiveReportsInFlightContainers(t *testing.T) { + const maxConcurrency = 2 + + var dockerCalls atomic.Int32 + created := 0 + dockerAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + dockerCalls.Add(1) + if strings.HasSuffix(r.URL.Path, "/containers/create") { + created++ + writeDockerJSON(t, w, map[string]any{"Id": fmt.Sprintf("%064d", created), "Warnings": []string{}}) + return + } + w.WriteHeader(http.StatusNoContent) + })) + t.Cleanup(dockerAPI.Close) + + dockerClient, err := client.New(client.WithHost(dockerAPI.URL), client.WithAPIVersion("1.55")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = dockerClient.Close() }) + + pool := &Pool{ + cli: dockerClient, + cfg: Config{Image: "test/image", MaxConcurrency: maxConcurrency}, + log: discardLogger(), + } + + var held []*Container + acquire := func(t *testing.T) { + t.Helper() + c, err := pool.Acquire(context.Background(), nil, nil) + if err != nil { + t.Fatalf("Acquire: %v", err) + } + held = append(held, c) + } + release := func(t *testing.T) { + t.Helper() + if len(held) == 0 { + t.Fatal("release with nothing held") + } + pool.Release(context.Background(), held[len(held)-1]) + held = held[:len(held)-1] + } + + tests := []struct { + name string + act func(*testing.T) + want int + }{ + {name: "idle pool holds nothing", act: func(*testing.T) {}, want: 0}, + {name: "one acquired container", act: acquire, want: 1}, + {name: "two acquired containers", act: acquire, want: 2}, + {name: "one released", act: release, want: 1}, + {name: "both released", act: release, want: 0}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + tc.act(t) + before := dockerCalls.Load() + got := pool.Active() + if after := dockerCalls.Load(); after != before { + t.Errorf("Active() made %d Docker API call(s), want 0: it must read the pool's own counter", after-before) + } + if got != tc.want { + t.Errorf("Active() = %d, want %d", got, tc.want) + } + }) + } + + if got := pool.Cap(); got != maxConcurrency { + t.Errorf("Cap() = %d, want the configured MaxConcurrency %d", got, maxConcurrency) + } + + // An unconfigured cap means "no limit" (Acquire treats MaxConcurrency <= 0 + // that way), so it must not be reported as a cap of zero. + if got := (&Pool{}).Cap(); got != 0 { + t.Errorf("Cap() with no MaxConcurrency = %d, want 0 (unlimited)", got) + } +} diff --git a/internal/daemon/daemon.go b/internal/daemon/daemon.go index a69d19617..633261f23 100644 --- a/internal/daemon/daemon.go +++ b/internal/daemon/daemon.go @@ -12,6 +12,7 @@ import ( "os" "strings" "sync" + "sync/atomic" "time" "github.com/samcharles93/archie-core/internal/agentexec" @@ -208,6 +209,12 @@ type Daemon struct { // so /stop can reach work already in flight. Its zero value is ready // to use. running runningTasks + + // lastPollAt is when the most recent poll pass began, in Unix + // nanoseconds (zero = no pass has started yet), read by + // LastPollAt. Atomic because runIdentities runs one poll goroutine per + // identity and they all stamp this one "is the poller alive" reading. + lastPollAt atomic.Int64 } // IdentityRunner bundles identity-specific state for a single agent @@ -433,6 +440,7 @@ func (d *Daemon) runIdentities(ctx context.Context) error { // drains or reconciles -- those are store-wide and run in the shared // maintainAndDrain loop. func (d *Daemon) pollForIdentity(ctx context.Context, id *IdentityRunner) { + d.markPoll() cfg := configForIdentity(d.Cfg.Get(), id.Cfg) for _, repo := range id.Repos { issues := d.pollIssuesWithConfig(ctx, id.Forge, cfg, repo) @@ -865,6 +873,7 @@ func (d *taskDispatcher) Wait() { } func (d *Daemon) poll(ctx context.Context) { + d.markPoll() for _, repo := range d.Cfg.Get().Repos { issues := d.pollIssues(ctx, repo) for _, is := range issues { @@ -1673,3 +1682,25 @@ func (d *Daemon) allowConcurrentForTask(task *workflow.Task) bool { repo, ok := d.repoFor(task) return ok && repo.AllowConcurrent } + +// LastPollAt reports when the daemon last began a poll pass, or the zero +// time when no pass has started yet. +func (d *Daemon) LastPollAt() time.Time { + ns := d.lastPollAt.Load() + if ns == 0 { + return time.Time{} + } + return time.Unix(0, ns).UTC() +} + +// markPoll stamps the start of a poll pass. It is called from both poll +// paths: Run's single-identity loop (via poll) and runIdentities' per-identity +// goroutines (via pollForIdentity). +// +// The stamp is the pass's START, not its completion: a pass that hangs on an +// unreachable forge leaves the stamp stale, which is exactly the signal an +// operator needs. Stamping the completion instead would leave the last +// successful pass's time in place, so a wedged poller would look healthy. +func (d *Daemon) markPoll() { + d.lastPollAt.Store(time.Now().UnixNano()) +} diff --git a/internal/daemon/poll_status_test.go b/internal/daemon/poll_status_test.go new file mode 100644 index 000000000..3261ea004 --- /dev/null +++ b/internal/daemon/poll_status_test.go @@ -0,0 +1,76 @@ +package daemon + +import ( + "context" + "log/slog" + "testing" + "time" + + "github.com/samcharles93/archie-core/internal/config" +) + +// TestPollPassStampsLastPollAt pins the "is the poller still running" fact +// /status reports: the daemon stamps the moment a poll pass begins, so a +// missing or stale timestamp is what tells an operator the poll loop has +// stopped or is stuck inside a pass. +// +// Both poll paths are covered because they are separate goroutines with +// separate entry points: Run's single-identity loop drives poll, while +// runIdentities drives one pollForIdentity per identity. A stamp on only one +// of them leaves /status permanently reporting "never" for the other +// deployment shape. +func TestPollPassStampsLastPollAt(t *testing.T) { + ctx := context.Background() + repo := config.Repo{Owner: "acme", Name: "widget"} + + pollPaths := []struct { + name string + poll func(context.Context, *Daemon) + }{ + { + name: "single-identity poll", + poll: func(ctx context.Context, d *Daemon) { d.poll(ctx) }, + }, + { + name: "per-identity poll", + poll: func(ctx context.Context, d *Daemon) { + d.pollForIdentity(ctx, &IdentityRunner{ + Name: "main", + Forge: d.Forge, + Repos: d.Cfg.Get().Repos, + Log: d.Log, + }) + }, + }, + } + + for _, path := range pollPaths { + t.Run(path.name, func(t *testing.T) { + d := &Daemon{ + Cfg: config.NewHolder(config.Config{ + BotUser: "archie", + Repos: []config.Repo{repo}, + Dispatch: config.Dispatch{Trigger: "assignee"}, + }), + Forge: &pollingForge{}, + Log: slog.New(slog.DiscardHandler), + } + + if got := d.LastPollAt(); !got.IsZero() { + t.Fatalf("LastPollAt() before any poll pass = %v, want the zero time", got) + } + + before := time.Now() + path.poll(ctx, d) + after := time.Now() + + got := d.LastPollAt() + if got.IsZero() { + t.Fatal("LastPollAt() after a poll pass = zero, want the pass's start time") + } + if got.Before(before) || got.After(after) { + t.Errorf("LastPollAt() = %v, want a time within [%v, %v]", got, before, after) + } + }) + } +} diff --git a/internal/domain/messaging/commands.go b/internal/domain/messaging/commands.go index f75c5d499..80aa7fb29 100644 --- a/internal/domain/messaging/commands.go +++ b/internal/domain/messaging/commands.go @@ -20,8 +20,8 @@ type CommandSpec struct { } var localCommandSpecs = []CommandSpec{ - {Command: "/status", Description: "Show daemon health", Usage: "/status"}, - {Command: "/tasks", Description: "Show running and parked work", Usage: "/tasks"}, + {Command: "/status", Description: "Show daemon health: broker, workers, channels, provider", Usage: "/status"}, + {Command: "/tasks", Description: "Show your tasks: state, stage, age, parked reason", Usage: "/tasks"}, {Command: "/model", Description: "Choose a provider and model", Usage: "/model [provider/model]"}, {Command: "/spawn", Description: "Create a tracked task", Usage: "/spawn