The agent-side half of the same gap. A stage failure already logged "stage failed -- parking" before parking, but a run whose OUTCOME was parked (a review that could not run, a workflow that ended with no outcome) reached the timeline as a KindParked event and the store as park_reason and never the log an operator downloads. park is the one place a park's reason is final, so it is now the one place that logs it; the stage-failure branch no longer logs its own near-copy, and the stage names itself inside the reason. The entry carries the stage as a field so nothing is lost by dropping the duplicate line. observability.md's "Implemented design" section documented the container's system-log path as though it were the only source of task-log content; it now records the daemon's and the workflow's own park writers, including why the entry follows the transition rather than preceding it.
162 lines
7.8 KiB
Markdown
162 lines
7.8 KiB
Markdown
# Task Observability
|
|
|
|
**Status:** Implemented
|
|
**Date:** 2026-08-09
|
|
|
|
## Historical problem
|
|
|
|
archied's own daemon logging (`internal/logging/`) was already solid: a
|
|
rotating file sink, a live `Feed` the dashboard subscribes to, and a reader
|
|
the `/api/logs` endpoint serves from. Before this implementation, none of
|
|
that reached inside a task.
|
|
|
|
A task's actual work happens in a task-scoped `archie-agent` container,
|
|
dispatched as one complete workflow over NATS.
|
|
Before this design was implemented, five things destroyed the evidence of what
|
|
that container did:
|
|
|
|
1. **Worker startup** constructed an stderr-only logger and never replaced its
|
|
handler after discovering a dedicated boot task, so runtime records could
|
|
not leave the container.
|
|
2. **`internal/container/pool.go`** creates every task container with
|
|
`AutoRemove: true`. The container — and its stderr — is gone at exit,
|
|
success or failure alike.
|
|
3. **`internal/domain/workflow/implement.go`, `StageBaselineGate`** captures
|
|
`go build`/`go vet`/`go test` output via `cmd.CombinedOutput()`, then
|
|
clips it to 200 characters into a `Warn` log line and discards the rest.
|
|
The eventual park reason is built from `res.Status` alone — the actual
|
|
compiler error is never persisted anywhere a human or Archie's own chat
|
|
tools can read it back.
|
|
4. **`agentexec.Result`** (`internal/agentexec/protocol.go`) carries only
|
|
`Summary`/`Detail` — no step-by-step transcript (tool calls, durations,
|
|
errors). There is no transcript to recover even in principle.
|
|
5. **`agentexec.SubjectForSystem`** is documented as the return channel for
|
|
exactly this — *"log dumps, health, and PII warnings... the daemon reads
|
|
these for observability and never forwards them"* — but has no publisher
|
|
and no consumer anywhere in the codebase except a subject-format
|
|
assertion in `nats_test.go`. The designed return path was never wired.
|
|
|
|
The practical consequence: a task parks with
|
|
`stage baseline: baseline red -- go build ./... fails and builder could not
|
|
auto-fix (status: parked)` and there is no way — not from the dashboard, not
|
|
from Telegram chat, not by SSH'ing into the deployment host — to see *why*
|
|
`go build` failed. The container that knew is already gone.
|
|
|
|
This matters beyond debugging convenience: supported deployments can combine
|
|
a resident daemon with ephemeral Docker containers that run unattended for
|
|
weeks between operator logins, and the system is
|
|
intended to be usable by people other than the original operator. A system
|
|
that is only debuggable by an operator who happens to be watching `docker ps`
|
|
in real time does not meet that bar.
|
|
|
|
## Decision: NATS is the sole autonomous handoff
|
|
|
|
The agent image must reach NATS in every deployment, because the broker is the
|
|
only autonomous task handoff and the only channel carrying task results back
|
|
to the daemon. `archied` has no workflow runner or host fallback. A container
|
|
that dies before establishing a NATS connection is a distinct, narrower
|
|
failure (connectivity/config) from the one this document addresses (a connected
|
|
container's own output being unrecoverable), and is out of scope here.
|
|
|
|
## Implemented design
|
|
|
|
The key property: **logs leave the container while it is alive**, so its
|
|
death at `AutoRemove` no longer matters. Persistence happens in the daemon
|
|
process, which already owns durable state.
|
|
|
|
### Sink: extend `internal/logging`, not a new package
|
|
|
|
`internal/logging` already owns its on-disk format end to end per
|
|
`organisation.md` ("a package owns its own format end to end"). Add a
|
|
task-scoped writer producing the same `logging.Entry` JSONL shape at
|
|
`<state_dir>/logs/tasks/<task_id>/attempt-<n>.jsonl`. `logging.Tail`,
|
|
`logging.Query`, and `logging.Components` operate on this unchanged;
|
|
`internal/webui/api_logs.go` stays transport-only, as it does today.
|
|
|
|
### Agent side: make the designed subject real
|
|
|
|
`internal/app/agentworker` replaces the stderr-only `slog.Handler` with one
|
|
that tees to stderr (kept for container-level operator debugging when a
|
|
shell is attached) and publishes each record to
|
|
`agentexec.SubjectForSystem(taskID)`, which already exists for this purpose.
|
|
The application layer obtains only a narrow publisher from
|
|
`internal/infrastructure/agenttransport/nats`; raw core-NATS types remain in
|
|
infrastructure. `internal/infrastructure/agentboot.TaskID` reads the task ID
|
|
from `<worktree>/.git/task.json` per the container/pool.go trap documented in
|
|
`CLAUDE.md`. Publishing is fire-and-forget: log shipping must never block or
|
|
fail the run it is reporting on.
|
|
|
|
### Daemon side
|
|
|
|
`cmd/archied/main.go` registers `subscribeSystemLogs` once at startup on
|
|
`agentexec.SubjectSystemWildcard`. It demultiplexes by task ID, appends to that task's
|
|
sink, and publishes into the existing `logging.Feed` so live task output
|
|
appears on the dashboard exactly like daemon output. The approved target is
|
|
for this composition to move to `internal/app/archied`; that package does not
|
|
yet exist, so this document does not claim the migration is complete.
|
|
|
|
The daemon is also a writer in its own right for the decisions only it makes.
|
|
Every park records its reason into that attempt's sink (`daemon.recordPark`),
|
|
on the daemon side after the guarded transition lands and in
|
|
`workflow.park` on the agent side, because "why did this park?" is the question
|
|
the log exists to answer and the container path cannot answer a run that never
|
|
reached a container. The entry is written only once the park actually
|
|
happened, so a transition another actor won does not leave a log claiming an
|
|
outcome the task board contradicts.
|
|
|
|
### Gate output: stop discarding it
|
|
|
|
`StageBaselineGate` and the equivalent TDD-stage gate runs write full
|
|
`cmd.CombinedOutput()` to the task's sink under a `gate` component, and put a
|
|
bounded tail (~2KB, not 200 bytes) into the park reason. This is the cheapest
|
|
fix and independently valuable — it alone would have made the `go build`
|
|
failure diagnosable without any NATS plumbing.
|
|
|
|
### Transcript
|
|
|
|
Step-level events (tool name, argument digest, duration, error) are emitted
|
|
as log entries on the same NATS system subject rather than added as new
|
|
fields on `agentexec.Result`. This keeps the wire protocol
|
|
(`internal/agentexec/protocol.go`) stable and the transcript lands in the
|
|
sink automatically, with no separate delivery path to keep consistent.
|
|
|
|
### Retention
|
|
|
|
Reuse the existing rotation (`MaxSizeMB`/`Keep`) semantics per task
|
|
directory. Prune a task's log directory when `store.ArchiveTask` archives
|
|
it, so task logs share the task's own
|
|
lifecycle rather than accumulating independently.
|
|
|
|
### Surfaces
|
|
|
|
- `/api/tasks/{id}/logs`, built on `logging.Reader` the same way
|
|
`/api/logs` is today.
|
|
- A chat tool so Archie can answer "why did task 4 park?" directly from
|
|
Telegram or the webui chat — this is the surface that actually matters
|
|
for the deployment model in `CLAUDE.md`, where an operator may not be
|
|
looking at a dashboard at all.
|
|
|
|
## Dependency direction
|
|
|
|
`internal/logging` remains cross-cutting per `organisation.md` — it imports
|
|
no domain, infrastructure, or app package, so extending it to be task-scoped
|
|
does not change its classification. The workflow domain writes through the
|
|
cross-cutting logging contract. On the agent side,
|
|
`internal/app/agentworker` composes the narrow publisher supplied by
|
|
`internal/infrastructure/agenttransport/nats`. On the daemon side,
|
|
`cmd/archied/main.go` currently composes the NATS subscription with the task
|
|
sink until the approved `internal/app/archied` migration is implemented.
|
|
|
|
## Implementation sequence (completed)
|
|
|
|
1. Gate-output capture (~30 lines, no NATS changes) — fixes the immediate
|
|
symptom that prompted this document.
|
|
2. Task log sink in `internal/logging`.
|
|
3. Agent-side publisher on `SubjectForSystem`.
|
|
4. Daemon-side subscriber + `logging.Feed` integration + retention on
|
|
archive.
|
|
5. `/api/tasks/{id}/logs` + chat tool.
|
|
|
|
Tracked as GitHub epic/subtasks per the issue list linked from this
|
|
document's PR.
|