A daemon shutdown or /restart cancels turnCtx without any guarantee
that the goroutine owning a liveReply ever runs again to notice and
clean up after itself -- a full process exit gives it no chance at
all. Gateway now tracks every liveReply mid-turn in a registry and
Stop drains it, marking each as interrupted so nothing is left ending
in the "typing" cursor forever.
finalize and abandon now run their bodies inside a per-reply
sync.Once (the terminal field): a turn finishing naturally and a
concurrent Stop draining the same reply were reaching it from two
goroutines at once, both sending their own edit for the same message
with the outcome decided nondeterministically by which HTTP response
Telegram processed last -- including a complete real answer getting
silently overwritten by the restart marker. Only one side's body now
ever runs; the other blocks on the winner and then does nothing.
The drain itself is bounded (liveDrainTimeout, default 5s): a single
stalled Telegram API call must not hang gateway, and therefore
daemon, shutdown indefinitely. Entries still in flight past the
deadline keep running in the background rather than blocking Stop.
A liveReply created in the narrow window between Stop claiming the
registry and its own registration call is abandoned immediately
instead of being registered into (and lost from) a registry nothing
will drain again.
Regression tests cover the mutual-exclusion race directly (20
iterations of finalize and abandonRestarted racing on the same
reply, asserting exactly one outcome ever wins), the bounded drain
against a deliberately hung API call, and the post-stop registration
gap.
show_tool_calls gated Telegram's tool-activity narration only; the
web dashboard's SSE chat stream narrated every completed tool call
unconditionally with no way to turn it off, leaking tool names and
parameter shapes to anyone with dashboard access even when an
operator had already opted out for Telegram.
Moved ShowToolCalls from TelegramConfig to ChatConfig so it is one
setting for every channel, and wired the webui stream to it:
chatStreamSink now carries a showToolCalls flag (snapshotted once per
turn, mirroring how Telegram's liveReply snapshots it) and ToolCall
no-ops when it's off.
BREAKING: an existing [chat.telegram] show_tool_calls = true in
config.toml now parses as an unrecognized key under [chat.telegram]
and is silently ignored (TOML decoding is non-strict) rather than
erroring -- move it to the top-level [chat] table.
The comment claimed a mount was checked before touching the
operator's global gitconfig; the code only ever checked os.Stat
(existence + directory-ness). A directory that merely exists at the
mount path -- a stale leftover, never actually bind-mounted this run
-- already satisfied the guard and got added to safe.directory,
exactly what the comment said it prevented. A real mount check would
need platform-specific mountinfo parsing for an inert failure mode
(the entry does nothing unless a git repo later exists at that exact
path), so this corrects the contract to match the code and pins it
with a test instead of changing behavior.
The finished reply is delivered by editing the message created at the
first streamed frame, and editMessageText does not change a
message's date, so a long tool-using turn's answer keeps its
turn-start timestamp and can sort before conversation that produced
it. Delete-and-resend would fix the ordering but reintroduce the
flicker/reorder streaming exists to avoid, so this documents the
accepted trade-off rather than changing behavior.
chatStreamEvent gained a Failed bool, set from ToolCallEvent.Err !=
"" -- the field the rest of the codebase already treats as
authoritative -- rather than the browser sniffing summary.startsWith
"failed:" on rendered text. A successful tool whose own output starts
with that literal string (a grep hit, a log line read back) was being
styled as a failed step.
cmd/archied carried a copy of the none/off/disabled predicate because the
configuration package's version was unexported. Two copies of a validation
predicate drift: a fourth disabled alias added in one place would silently
leave the other accepting it.
Export ForgeDisabled in internal/infrastructure/configuration as the single
definition, referenced by both validateIdentities and resolveForge, and
delete the duplicated copy. TestForgeDisabled pins the exact alias set
(including case-variant and empty-string guards); the resolveForge
degradation test now asserts on the warn log so the silent disabled path is
distinguishable from the token-unavailable fallback (verified to fail under
a re-inlined 'none'-only copy).
Three more P2s from the same code-review pass: the live frame's
single clamped buffer mixed tool lines and answer text chronologically,
so early tool lines silently scrolled out of the frame once the answer
grew past liveBodyMaxRunes and only reappeared at finalize (which
already led with tools separately). Tool lines are now tracked
exclusively in toolLines; a new framedText helper keeps the tool block
whole and clamps only the answer tail, used consistently by the live
frame, an abandoned frame, and the finished message.
A tool-only turn with no reply text used to finalize as a bare,
immutable tool line indistinguishable from a real (if terse) answer;
finalText now appends an explicit "(no response)" marker in that case.
A provider error used to funnel into the same clean-partial path as a
deliberate /stop, so a failed turn was indistinguishable from a
successful short one. abandonFailed appends a "turn failed" marker
before dropping the cursor; /stop still goes through the unmarked
abandon path.
Regression tests cover all three paths plus the degenerate case where
a tool block alone exceeds the frame budget, verified red-then-green
against the pre-fix code.
Three P1s from code review of PR #476: finalize sent the authoritative
reply through the same cancellable turnCtx a /stop kills, so a stop
landing mid-finalize stranded the live message with its cursor
forever; abandon had the identical hazard, saved only by its one
caller wrapping the context correctly. Both now strip cancellation
internally (context.WithoutCancel) rather than trusting callers to
remember.
Tool activity lines were composed unescaped into the same Markdown
body as the model's own reply, so a stray '*', '_', or backtick in
tool output/parameters could unbalance the whole message and take the
reply's formatting down with it. ToolCall now runs name, parameters,
and summary through bot.EscapeMarkdown before joining the line.
splitLongMessage sliced oversized lines on byte offsets, cutting
mid-rune for any multi-byte UTF-8 line crossing the 4000-byte bound
(CJK, emoji) and producing invalid UTF-8 parts. It now counts and
slices on runes throughout.
Regression tests cover all four paths, verified red-then-green
against the pre-fix code.
cmd/archie-agent used to own worker construction, NATS lifecycle, task execution, MCP setup, and workflow wiring. It now owns process input only and invokes internal/app/agentworker, while transport-specific NATS, boot-brief, and git-safety mechanics live under infrastructure.
ai-sdk v0.1.21 wraps the cancellation cause in aborted-stream errors
("%w: %w") instead of formatting it, so errors.Is(err, context.Canceled)
is reliable again. The Telegram channel was compensating with a comment
explaining why errors.Is could not be trusted.
The code was already correct; only the comment was obsolete. It now
states the real reason for context.WithoutCancel(turnCtx): turnCtx is
dead on both stop and fault paths, so the cursor-drop edit needs a live
context. The runtime effect of the bump is that a /stop landing inside
the step loop is now logged as "chat turn stopped" rather than
"route failed".
Refs ai-sdk-0pn
Telegram streaming callbacks now enqueue cancellable rendering rather than performing network I/O while draining the authoritative model stream. Final edit failures fall back to normal split delivery.
Completed tool events now correlate inputs by call ID and expose only bounded, conservative parameter shapes. Tool-call visibility is snapshotted per reply so reloads cannot change an in-flight turn.
Chat returned only the final answer, so a wrong tool choice, a
hallucinated action, or a footgun like `systemctl stop` against
`restart` was invisible from the conversation. Only text deltas were
threaded through the turn, and tool calls executed inside Generate with
nowhere to report.
Replace the delta callback with gateway.TurnStream, which carries text
fragments and completed tool calls on one ordered path from the model
loop to the channel. The sequence is the point: a tool line that
overtakes the sentence it interrupted describes a turn that never
happened. Text-only callers use gateway.DeltaFunc and are unaffected.
A tool is reported on its result, never on its call -- the call part
carries no outcome yet, so reporting both would list every tool twice,
once with nothing to say.
Telegram renders it behind [chat.telegram] show_tool_calls, off by
default: this is debugging visibility, and it narrates every internal
step to provide it. The dashboard always shows it, being the operator's
own observability surface; its tool lines are live-only and are not
persisted with the conversation.
The Telegram live reply changes mechanism, and with it what a frame
MEANS. It used to stream through sendMessageDraft: drafts are ephemeral
30-second previews, so a tool call longer than that outlived the
preview, and successive frames were observed concatenated with their
cursors intact. It now sends one real message and edits it. An update
used to be "the next preview of the answer"; it is now "the whole
answer as it currently stands, replacing what is there". The cursor is
appended at render time and never stored, so exactly one exists and
always at the end, and a frame too long for one message is cut to its
tail rather than rejected outright.
A stopped or failed turn now edits the cursor away instead of leaving
it blinking on a message nothing will finish. That cleanup runs on a
live context because turnCtx is already dead, and it runs on both
failure branches: ai-sdk raises an aborted stream as ErrAborted with
ctx.Err() formatted in rather than wrapped, so errors.Is cannot be
trusted to tell a /stop from a fault.
Closes#467
Repair future-dated conversation history on SQLite open and cap newly supplied timestamps so one bad platform clock cannot pin a session indefinitely.
Make message and turn identifiers injective while preserving legacy redelivery and crash-recovery compatibility, retire idle turn lanes safely, and pin the existing compression and partial-undo behavior with focused regressions.
Clone normalized nil slices to empty non-nil (make([]T, 0)), so after a
dashboard PATCH the cloned published config carried [] where the file
decodes to nil -- a subsequent SIGHUP reload's diff-and-warn compared
old [] against fresh nil and reported 'Repos changed; requires a
restart' for a field that had not changed at all. The exact warning
fatigue the allowlist exists to prevent. cloneRepos/cloneIdentities/
cloneMCPServers/cloneStringSlices now return nil for nil input (the
append-based copies already preserved nilness).
Second adversarial pass findings, fixed:
BLOCKER -- PATCH silently dropped every dotted key. The UI sends
dotted keys ('budgets.max_steps') but ApplyOverlayValues fed them
straight to yaml.Unmarshal, which cannot decode a dotted key as a
struct field and silently ignored it: 200 OK, zero effect, and the row
still persisted (poisoning later reloads with an unvalidated value).
The reload path worked only because overlay.Snapshot nests first. Fix:
applyDottedOverlay nests dotted keys via overlay.Nest before decode --
the same nesting Snapshot performs -- and is pinned by
TestApplyDottedOverlayNestsBeforeDecode, which fails against the old
behaviour.
MAJOR -- shallow copies let overlay decode mutate the published
snapshot. Holder.Get and *doc are shallow; yaml decodes maps in place,
so a crafted nested update (or a failed boot overlay apply) mutated
shared maps of the published config before validation. Fix: config.Clone
deep-copies every reference field (Models, Providers, Repos, Identities,
Dispatch.Labels, Agent.Env, Chat/Telegram slices, Tools.MCPServers,
Memory.ProviderConfig, WebFetch.Enabled pointer, Extra), used by the
PATCH path (next := d.Cfg.Get().Clone()) and by Loader.ApplyOverlay,
which now never mutates its input (clones Config and the provenance
chain internally), so boot and reload pass the base document directly
and a failed apply leaves it exactly as it was. Pinned by
TestConfigCloneDeepCopiesReferenceFields.
MAJOR -- data races on captured main.go variables. bootOverlayErr and
currentProvenance were plain values written by the SIGHUP goroutine and
the PATCH/Reset handlers and read by every GET /api/config; -race passed
only because no test exercised the composition root concurrently. Both
are now atomic.Pointer; the LastReload closure and the provenance
dedup read them atomically.
MINOR -- MaxConcurrency removed from the reloadable allowlist. The
dispatchers re-read it but the startup-built container pool captures it
(container/pool.go:94,163), so 'every consumer re-reads it' is false and
a change only partially applies. It now warns requires-restart; the
criterion comment on reloadableFields documents the pool capture.
MINOR -- ResetConfig now resolves and validates the target state BEFORE
deleting the row, so a broken file cannot leave the store and the
running config disagreeing; the provenance chain no longer grows
unboundedly (the runtime-overlay origin replaces, not appends).
Also: holder.go no longer references a file watcher that does not exist.
Verified: task check clean; -race clean on cmd/archied, webui, daemon,
config; new findings from the second adversarial pass are closed.
Three gaps closed, all in the 'your change didn't take effect and
nothing told you' family:
1. MaxRetries joins the reloadable allowlist, and the criterion is
fixed on reloadableFields: a field is reloadable when EVERY
consumer re-reads it. MaxRetries is the worked example -- its only
consumer is the webui maxRetriesFor handler, which re-reads per
request (api_tasks.go:380,383); nothing in the daemon reads it, so
warning 'requires restart' was spurious. Chat.Telegram.TokenEnv is
documented as the counter-example (webui reads it per request but a
startup-built telegram bot consumes it, so it is not reloadable).
Pinned by TestChangedNonReloadableFields alongside the comment.
2. Boot-time overlay degradation is now surfaced. config.ReloadStatus
gains OverlayUnavailable; main.go records the reason when the
overlay store cannot be opened or its values fail validation, and
/api/config carries it (merged into LastReload). Cleared by a
successful dashboard write or reset. The settings page renders a
warning banner, so a daemon running on file config alone says so
where the operator is looking, not only in logs.
3. Overlay-shadowed file edits are visible and recoverable.
ConfigView.Overridden lists the dotted keys the runtime overlay
currently sets, so the UI marks those rows (their file value is
shadowed until reset). POST /api/config/reset exposes the
store's Delete as a per-row reset: it deletes the row and
republishes file + remaining overlay, which is also the answer to
'my file edit isn't taking effect'. The dashboard can now remove
every override it can create.
Verified: task check clean; -race clean on cmd/archied, webui,
infrastructure/configuration; new handler tests cover the reset
200/400/503 paths and the overridden list.
The second half of #348's storage layer: dashboard-edited overrides
live in their own SQLite file, separate from the task and conversation
stores so each file owns its user_version and migrator with no
contention (configDBPath -> cfg.DBPath + "-config.sqlite", matching
the taskDBPath / conversationDBPath convention).
Store (internal/infrastructure/configuration/overlay):
- config_overlay(key TEXT PRIMARY KEY, value TEXT NOT NULL,
updated_at, updated_by), own migration, WAL.
- Set refuses DeniedKeys (db_path, work_dir) at write time as well as
at the API layer; the map is exported so the dashboard can show the
reason instead of silently omitting the edit affordance.
- Snapshot JSON-decodes each value and nests dotted keys
("budgets.max_steps" -> map[b][m]) ready to decode into Config.
Loader (ApplyOverlay / ApplyOverlayValues):
- Decodes a nested overlay map with the same field-level precedence as
a file overlay, records the runtime-overlay origin, then applies
defaults and validates. Used at boot, on SIGHUP reload (ReloadController
gained an optional WithOverlay source) and by the PATCH path.
Boot and reload:
- A broken overlay (unopenable store, unreadable snapshot, values that
fail validation) degrades to file config alone -- it must never brick
the daemon. --no-config-overlay / ARCHIE_SKIP_CONFIG_OVERLAY=1 skips
the store entirely; the flag is now a real gate, not a log-only stub.
PATCH /api/config (webui):
- handleConfigUpdate decodes {"updates": {dotted.key: value}}, calls
the composition-wired UpdateConfig seam, and classifies errors via
webui sentinels: ErrConfigUpdateInvalid -> 400 (denylisted key,
failed validation, unparseable value), ErrConfigUpdateUnavailable ->
503 (overlay skipped by recovery flag), anything else -> 500.
- The seam (main.go) applies updates to a COPY of the published
config, validates the materialised result, persists to the overlay,
then publishes through the same path as reload -- it never calls
Set directly from the handler, since web.Cfg is the daemon's own
Holder.
- ConfigView gains Locked (dotted key -> reason) so the UI disables
denylisted rows explicitly.
Verified: task check clean; new tests cover the store round-trip,
denied keys, loader overlay layering/validation/provenance, reload
with and without overlay, and the PATCH handler's 200/400/503 paths.
The allowlist drifted before the chunk shipped. ForTask (config.go:457-465)
is the authoritative per-dispatch snapshot and carries Models, Budgets,
Dispatch, DiffCapLines, Notify and Forge.Host -- I allowlisted only three.
And the poll path reads cfg.Label and cfg.BotUser live per cycle
(daemon.go:359,390 and :368,381) off the same per-poll config as
Dispatch.Trigger. Five fields would have reported 'requires a restart'
while actually taking effect immediately: the safe direction, but exactly
the warning-fatigue failure that kills the feature (an operator who is
told to restart for a label change that already applied learns the
warning is noise).
The fix is enforcement, not a bigger list: TestReloadableFieldsCoverForTaskSnapshot
reflects over TaskConfig and fails if any field ForTask carries is not in
the allowlist (Forge maps to the Forge.Host sub-field entry). A field
added to ForTask later now fails the test instead of silently producing a
wrong warning. Label and BotUser are not carried by ForTask -- they come
off the poll path -- so they are allowlisted by hand with a comment, and
the same test pins them explicitly.
Also documents on webui.Server.Cfg that in production it is the daemon's
OWN Holder (web.Cfg = d.Cfg), so Set publishes to the running daemon: a
config-mutating handler must apply to a copy, validate the materialised
config, persist, then Set -- never Set directly.
task check clean.
The daemon can now re-read its configuration on SIGHUP (or on a
future in-process trigger) and republish it through the Holder,
closing the first half of issue #348: 'config can change at runtime,
safely, and can't brick the daemon.'
Reload path:
- cmd/archied/reload.go: ReloadController re-runs loader.Resolve on
the same paths main.go passed at startup (Resolve is pure -- no
reload state added to the loader). apply is called only with a
Document that passed validation; a failed reload records the error
and its timestamp and leaves the running config untouched. That
Set-not-called-on-error property is the safety core and is pinned
by TestReloadValidationFailureDoesNotApply.
- SIGHUP listener in main.go runs reloadLoop(ctx, ch, controller, log);
the loop is extracted as a testable seam (syscall.SIGHUP is Unix-only,
fine given the Linux target).
PollInterval is genuinely reloadable now, not just re-read:
the three ticker sites (Run, per-identity goroutine, shared
maintenance loop) re-read the interval each tick and Reset the ticker
on change. A reloaded poll_interval takes effect on the next cycle.
Diff-and-warn so an operator never has to read the source to know
whether an edit took effect: changedNonReloadableFields compares the
old and new snapshots against an allowlist of re-read fields
(PollInterval, Repos, DiffCapLines, Budgets, Dispatch, and the
Containers sub-fields VolumeTTL + MaxConcurrency). Anything else that
changed logs Warn 'requires a restart'; a fully-applied reload logs
Info. The allowlist is deliberate: a field added to Config later
defaults to requires-restart, the safe direction.
NATS split-brain closed: containerEnv now hands containers the
endpoint the daemon's own client connected with at startup
(Daemon.ConnectedNATS, captured at construction) instead of the live
config, so a reloaded nats.url cannot point new containers at a
server the daemon is not publishing on. A nats change is therefore
classified requires-restart, and CLAUDE.md's 'hands its own nats.url
verbatim' note is updated to the meaning it now has (same today,
deliberately different after a reload).
Validation guard: poll_interval <= 0 is rejected in validate.go
beside the VolumeTTL guard. applyDefaults fills zero with 60s, so
this fires on an explicit negative -- which would panic
time.NewTicker/Reset at boot and, after the per-iteration reset,
on a SIGHUP mid-run.
Observability: config.ReloadStatus (last_error, last_error_at,
last_reload_at) is recorded by the controller and surfaced in
/api/config via Server.LastReload, so a stale running config is
visible in the dashboard, not just the logs. Rendering waits for the
UI step.
Recovery hatch: --no-config-overlay flag plus ARCHIE_SKIP_CONFIG_OVERLAY=1
(configuration.SkipOverlay), logged at startup. No overlay source
exists yet; the flag is the documented escape hatch the DB overlay
(step 3 of #348) will consult.
webui.Server.ConfigProvenance becomes atomic.Pointer so a reload can
swap it via SetProvenance independently of Cfg; the accepted skew
(new config alongside old provenance) is documented on the field.
Verified: task check clean; -race on daemon, webui, cmd/archied,
config, infrastructure/configuration.
Daemon.Cfg, webui.Server.Cfg, and telegramSetup.Cfg were all held by
value (or as a raw shared pointer in webui's case), so a future reload
path could not swap the running configuration -- the consumers held
frozen snapshots made at construction. This is the prerequisite for
issue #348.
Add config.Holder, a small RWMutex-backed snapshot holder:
- Get returns a value copy under RLock; the copy shares its
reference-type fields with the published snapshot, so a doc
comment pins the immutability invariant (a published Config is
immutable; reload constructs a fresh one).
- Set replaces the snapshot under WLock.
- Get is nil-safe; Set is not.
Convert every consumer to read through the accessor. The convention
'cfg := h.Get() once per logical operation' is documented on the
type so the next reader knows that inline d.Cfg.Get().Foo is a code
smell (a Set could land between two consecutive Gets).
Special cases:
- webui.Server.Cfg: was *config.Config, a raw shared pointer read
from handler goroutines. Convert to *config.Holder. The
pre-existing 'cfg := s.Cfg' aliasing bug at api_config.go:295
(which was *trying* to copy but only copied the pointer) now
becomes 'cfg := s.Cfg.Get()', which is the value copy the
read-only view always wanted.
- daemon_test.go: one test mutated d.Cfg.DiffCapLines in place,
which the Holder forbids. Replace with the documented
read-modify-write pattern (prev := d.Cfg.Get(); prev.X = Y;
d.Cfg.Set(prev)).
- telegramSetup.Cfg: matches the daemon's API for consistency
even though setup runs before the daemon is alive.
TaskContext.Cfg in internal/domain/workflow/workflow.go is
deliberately NOT converted -- a task that starts under one config
should finish under it, and a config changing between TDD and gate
stages is a bug, not a feature. Snapshotted at dispatch, as before.
Verified: go test ./... (61 packages), go vet, -race on
internal/daemon and internal/config.
Task and event persistence now uses a dedicated SQLite sidecar derived from the configured state prefix, matching the independently persisted conversation store. The configured path now means a state-path prefix rather than a database file so fresh SQLite state never attempts to interpret the previous storage format.
Remove the retired implementation and dependency completely, strengthen SQLite migrations and concurrency behavior, restore backend-neutral coverage, and align configuration, operational guidance, UI labels, and deployment documentation with the new storage and host-daemon topology.
Adds two read surfaces for per-task persisted log history so an operator
can answer "why did task N park?" without needing to watch docker ps in
real time — the deployment model documented in CLAUDE.md.
/api/tasks/{id}/logs — built on logging.Tail the same way /api/logs
already is, defaults attempt to the tasks current Attempt from the store.
task_logs chat tool — lets Archie read task logs from Telegram or webui
chat. Identity enforcement matches chatTaskListerAdapter: a model bound
to one identity cannot read another identitys logs, and forge tasks
(empty identity) are denied.
internal/logging: add TaskRegistry.Path() for read-side access without
threading baseDir separately through every caller.
Fixes from adversarial review: strict identity check (no empty-identity
bypass), non-null Entries when disabled, component doc accuracy, and
adapter tests covering the security boundary.
Ref: #455
Closes the loop #451-#453 opened. archie-agent (#453) now publishes its
own logs to agentexec.SubjectForSystem(taskID) over NATS, and a task
log sink (#452) exists to persist them, but nothing on the daemon side
subscribed, demuxed, or wrote anything -- the messages had nowhere to
land.
- agentexec: SubjectSystemWildcard + TaskIDFromSystemSubject, the
inverse of SubjectForSystem, for a daemon subscribing to every
task's system subject with one wildcard subscription.
- logging: TaskRegistry manages one open TaskSink per running task
attempt, mirroring every write onto the live dashboard Feed the same
way a locally-written log would appear. Nil-receiver-safe, matching
every other optional composition-wired Daemon field.
- daemon: Daemon.process opens a task's log sink for its own duration
(covering both the sandboxed-container and in-process dispatch
paths uniformly) and closes it on every return path.
- cmd/archied: constructs the registry, subscribes core NATS to
SubjectSystemWildcard, and wires TaskLogs into both Daemon and
webui.Server.
- webui: archiving a task now removes its log directory too, so
archived tasks' logs don't accumulate forever. Documented as a
known, accepted race: a task can reach terminal status (satisfying
ArchiveTask's precondition) before process() itself returns and
closes the sink, so a very late system-log write can race an
archive's Remove. Not corrupting -- the write lands on a
soon-to-be-unlinked inode and is discarded -- and not worth fixing
here: it would mean tying Close to the store transition instead of
to process()'s return, a larger change than this issue's scope.
Adversarial review (fresh subagent) found and fixed two real issues in
the first draft: a doc comment accidentally spliced between two
functions during an extraction meant to fix a lint regression, and a
genuine test gap on openTaskLog (added, using TaskRegistry.Write's own
true/false contract as the observable -- no Store/Forge/Trees fakes
needed, since openTaskLog only touches TaskLogs and Log).
Part of #453 (task-observability epic #450). archie-agent logged to
os.Stderr only, and its container is created with AutoRemove: true
(internal/container/pool.go), so all of it disappeared the moment a
task's container exited -- there was no way to see why, say, a gate
command failed inside one.
Adds agentexec.SystemLogHandler: a slog.Handler that tees to whatever
it wraps (kept for an operator attached to a live container) and
additionally publishes each record, JSON-encoded as a logging.Entry,
to SubjectForSystem(taskID) -- a subject that already existed,
documented for exactly this ("log dumps, health... observability"),
with zero publishers anywhere in the codebase.
cmd/archie-agent/main.go wires it in only when this container was
spawned for one specific task (bootTaskID / .git/task.json present);
shared queue-group mode serves many unrelated tasks per process, so
there's no single task subject to ship to and keeps the stderr-only
logger.
Building this surfaced a real pre-existing bug: FeedHandler's attr
flattening applied the group chain active at Handle-time to *all*
accumulated attrs, including ones added via WithAttrs before a later
WithGroup call -- nesting them under a group they were never
actually inside, contrary to slog's own documented Handler contract.
It went unnoticed because nothing in the codebase called WithGroup
until this change needed to. Extracting the shared flattening logic
into exported FlattenAttrs/PrefixAttrs (so SystemLogHandler doesn't
duplicate it) is what surfaced it; fixed in the same change since I
was the one making it public API.
Docs on the wire format's own trap for the daemon-side consumer
(#454, not yet written): the published bytes are a marshaled
logging.Entry, meant for json.Unmarshal, not logging.decode() --
decode() parses raw slog JSON *log lines* off disk, a different and
incompatible shape. See the doc comment on SystemLogHandler.
Part of #452 (task-observability epic #450). A task's real work
happens in an ephemeral archie-agent container; currently nothing
persists what it logs once the container exits (AutoRemove: true).
Adds TaskSink: a per-task, per-attempt JSONL log destination sharing
the exact on-disk format the daemon-wide logger already writes (see
Entry/Tail/Query/Components in reader.go), so a task's history is
readable with the same reader rather than a second format to keep in
sync. Backed by the existing rotatingFile, so it gets the same
size-based rotation and retention as the daemon-wide log for free.
Deliberately does not fall back to stderr on open failure the way the
daemon-wide New does: that fallback exists because losing the daemon
over a broken log path is worse than losing the durable copy, and
there is no equivalent daemon-level fallback for one task's log.
This lands only the sink primitive and its own tests. Wiring an
agent-side publisher and a daemon-side NATS consumer into it is
tracked separately in #453 and #454.
StageBaselineGate captured a failing gate command's full output but
discarded it: a 200-char clip went into a Warn log, and the terminal
error carried only res.Status. A park reason like "go build ./...
fails ... (status: failed)" gave no way to see the actual compiler
error, and the container that had it is already gone (AutoRemove).
Now the park error carries a bounded tail of the real output (tail,
not head, since a failing test's explanation is at the end after
pages of "ok" lines), rune-safe so it can't emit invalid UTF-8 into
a persisted park reason. The three existing byte bounds in this
stage (log line, agent mission, park error) are named and documented
as independently sized rather than left as bare numbers.
Also adds docs/architecture/observability.md, the design doc for the
larger task-observability epic (#450) this fix is the first step of.
Composition root work tying the config resolver, channel manager, log
feed, and executable workflow registry into the running daemon and
dashboard server: web.Channels/LogFeed/Workflows/ConfigProvenance,
channel-reload wiring, TaskStopper for the Stop action, and
SpawnRequest.Body/WorkRequests threading dashboard "Start work" requests
through the same admission path chat uses. Touches all three features at
once because this is where they're composed, not owned by any single one.
Refs #53, #54, #55
Follow-up pieces of the task-controls work: store.InsertEvent factors out
an execer-generic insertEvent so SQLite gets the same zero-time defaulting
ArchiveTask's audit path relies on; gateway tests assert against
taskstate constants instead of magic status strings; the navigation
command bar gets an accessible mobile search toggle.
Refs #53
Add internal/logging.Feed, a logging-owned bounded replay buffer separate
from the persisted task-lifecycle event stream, with a live SSE endpoint.
Expose the executable workflow registry (skillbuild.Catalog) including
zero-run definitions and their origin (builtin vs. installed skill), not
just historical run statistics. Route() now returns an explicit failure
workflow for an unknown pre-assigned workflow name instead of silently
falling through to label-based routing.
Fixes#54
Loader.Resolve() selects TOML/YAML files or a config directory as the
production entry point, replacing the TOML-only Overlay() as the default
path. Add internal/channels.Manager, reporting each adapter's actual
runtime lifecycle (configured/starting/running/degraded/failed) instead of
inferring health from configuration presence alone. Wire channel reload
through a per-adapter seam.
Fixes#55
Add cancel/stop/approve/reject/retry/abandon/archive to the dashboard task
action matrix, each guarded by an optimistic status check against the
store. Retry accounting (RetryCount, max_retries) is now atomic with the
status transition instead of a separate write. Archive removes a terminal
task's local record with a fail-safe audit event first. Add always-labelled
responsive navigation, mobile Jump-to search, and versioned Setup-complete
dismissal.
Fixes#53
clean.RequestURI() echoed the incoming request's raw path/query straight
into http.Redirect's Location header. A path starting with "//" is treated
by browsers as protocol-relative, so a crafted token-exchange link could
redirect off-host after a legitimate token exchange. sameOriginPath()
collapses any leading "//" before the value is ever used.
Fixes#440
Adds a human-consent gate for tools marked RequiresApproval, exposes
session management as LLM-invokable chat tools, and gives the operator
/delete to retire conversations permanently.
**Tool-approval gate (internal/tools, internal/agentexec)**
- ApprovalRequester contract lives in internal/tools (not gateway) so
agentexec can import it without pulling in the chat-adapter stack.
- ToolSetOptions.Approval wires the approver through to toolExecute.
- BuildToolSetFrom omits RequiresApproval tools when no approver is
configured; dispatch paths (invokeTool, tool_call bridge) refuse
them too, so the invariant holds at every indirection.
- The gate fails closed: only Approved/PermanentlyApproved pass;
an unknown decision value from a faulty approver does not execute.
- Tool-level timeout (2 min) returns a plain error, not a context
cancellation, so the model can report "approval timed out" instead
of the turn aborting.
- BuildApprovalDescription callback lets a tool surface the specific
resource being acted on ("Delete session \"abc123\"") rather than
showing a static tool description.
**Session tools (internal/gateware/session_tools.go)**
- session_list (idempotent), session_resume (mutating), session_title
(mutating), session_delete (mutating + RequiresApproval).
- Scoped to the current platform + channel: GetByChannel replaces
List() so a model in chat A cannot list or manipulate chat B.
- session_title validates the session belongs to this channel before
writing; session_resume/session_delete resolve refs against the
channel-scoped list.
- Shared limit helper extracted to tools.ListLimit.
**Telegram approval adapter (internal/channels/telegram/approval.go)**
- telegramApprover implements ApprovalRequester via the three-button
inline keyboard (Approve/Permanent/Deny). 2-min token expiry.
- Own map + mutex + "approval:" callback prefix — separate from the
dangerous-command system.
- Permanent approval uses exact-match (not prefix) lookup keyed by
action + resource description, so permanently approving "delete
session abc123" only covers that one session, not every call to
session_delete for 24h.
**Session lifecycle**
- /delete <session-id>: removes a session and its history. Refuses
prefixes shorter than 4 chars even when unique. Exact ID wins over
an ambiguous prefix. sessionTracker.forget sweeps every channel+
thread pointing at the deleted session.
- Extracted resolveSessionRef and shortSessionID for /resume and
/delete to share.
**Wiring**
- submitTurn builds the approver onto ctx before the LLM turn.
- chatGenerateOptions reads approver from ctx, passes it through
ToolSetOptions.
- SessionTools built per turn on the current message, combined with
taskTools into a single toolset.
Multi-identity daemon isolation:
- Split runIdentities into per-identity poll loops plus one shared
maintenance-and-drain loop, so reconcilePRs and storage cleanup run in
multi-identity mode and the global max_concurrency / per-repo
serialization holds across identities.
- Make the poll path identity-aware: each identity polls with its own
forge client, bot user, label, trigger, and ack reaction.
- Make allowConcurrentFor task-based and identity-aware via repoFor.
- Roll back storage mounts when container acquisition fails; log
Store.Update failures after branch set.
Container-mode RPC identity routing:
- forgerpc/worktreerpc gain identity-scoped subjects (SubjectFor,
RegisterFor) and identity-carrying clients.
- registerTaskRPCServers registers one server pair per identity (plus
root), so a container-mode task owned by a non-root identity is served
by its own forge client and worktree manager.
Gateway/Telegram:
- LLM-proposed session titles generated in the background after turns
(TitleGenerator + chatTitleGenerator), persisted by the gateway.
- /delete session command with prefix guard and tracker forget.
- Shared session display helpers (shortSessionID, sessionDisplayTitle,
resolveSessionRef).
- Reword docs and comments to drop vague "legacy" qualifiers where they no longer describe deprecation status
- Remove unused code: RecentEvents, MessageEvent.ToLegacy, index setState, stderr redaction tool, telegramOnlyCommands
- Delete or comment out tests covering removed or superseded behavior
NellDB v0.3.0 skips meta:* ids when rebuilding its rev cache on open, so
after any restart the counter Get returned a _rev with no cache entry and
every RMW Put failed with sdk: revision conflict -- task enqueues and
event inserts were permanently broken.
- Counters move to counter:task_id / counter:event_id.
- migrateCounters seeds the new keys from the legacy docs at open; any
read/write/validation failure aborts OpenStore so allocation can never
silently restart at 1. Legacy docs stay in place, skipped by scans.
- isReservedKey matches the counter ids exactly (a counter: prefix rule
would hide every task in a repo owned by "counter").
- Existing counters are validated (next_id >= 1) at open and at allocation.
- EnqueueIssue persists identity, which it previously dropped.
- Tests: reopen regression with exact post-reopen IDs and event
preservation, migration sequence carry-over for both counters,
fail-closed on read/write/invalid values, counter-owned repos,
identity round-trip.
Adversarial review: zero surviving findings.
A completed chat turn is published as turn_completed from the single
responder choke point shared by the telegram and web gateways
(redeliveries return earlier and never emit). The runtime wake path
forwards ONLY primary-input kinds to input-driven curators; curator-
produced activity (curator_run/action/error and any future derived
kinds) never appears on that path, so a curator that writes memories
cannot schedule itself again — the feedback-loop guard, with the
rationale recorded in the regression test so it is not removed as
redundant.
The bus subscription is bounded and dropping, so the wake path can
never backpressure the chat publisher; a dropped event only delays a
run to the next check-in.
internal/curator is a conforming engine family per the strict plugin
engine rule: typed CuratorEngine contract (Lifecycle + domain ops),
owning Registry (registration, discovery, start/health/stop with
failure isolation, reverse-order shutdown, ctx-bounded stop), and a
narrow typed Registrar — each curator receives a capability-filtered
view of the host at Bind, never the daemon.
Runtime: one loop per curator on an injectable clock (fake-clock
testable), best-effort wake nudges with the trigger decision kept in
Check (a dropped nudge only delays to the next check-in, never
corrupts accounting), per-pass timeout and a max-concurrent-passes
slot so curator bursts cannot starve chat turns, per-pass panic
recovery, and bounded shutdown even when a pass ignores cancellation.
curator_run / curator_action / curator_error ride the in-process
event bus so activity stays attributable.
Wired in cmd/archied/main.go (constructed beside memory, Start after
daemon startup, shutdown ordered before memory/LLM teardown) and
exposed as daemon.Daemon.Curators.
internal/infrastructure/configuration/setup.Run drives operator
identity, LLM provider, models, forge, and Telegram chat channel
prompts and returns TOML edits -- not a config.Config. Building a
parallel in-memory representation to validate against was rejected
during design review: it's a second source of truth about the exact
thing (text vs. struct disagreeing) this whole feature exists to stop.
Tests instead render the edits against the real embedded template and
load the result through a real configuration.Loader, proving the
actual defaults/decode/validate path accepts it.
Two boot-time landmines in config.example.toml surfaced by that
approach, both fixed here:
- [providers.openai] ships active with api_key pointed at the bws
engine. cmd/archied/provider_secrets.go resolves every entry in
cfg.Providers at startup, not just the ones [models] references, and
refuses to start the daemon if any fails -- so picking any other
provider (or picking openai but skipping the key) left a config that
wouldn't boot on a machine without the bws CLI installed (the common
case). stepProvider now neutralises providers.openai.api_key to the
zero-value secret ref whenever openai wasn't the actual choice.
- [[repos]] shipped active with a placeholder owner/name that
validates cleanly (owner/name are only checked non-empty) and would
have archied silently polling a repo that doesn't exist. Commented
out in the template with an explanation; setup does not write this
section since tomlwrite cannot target array-of-tables.
TestExampleConfigLoads updated to expect zero repos.
Also: forge and provider secret env-var names are each defined in one
table (forgeToken, cloudProviders) read by both the config edit and
the secret sink, so the two can't name a secret differently -- the
exact bug class this feature exists to close for the installer, tested
directly (TestRun_ForgeTokenKeyMatchesStoredSecretKey compares the
decoded key against the key the secret was actually stored under, not
a hardcoded literal on both sides). Telegram writes chat.telegram.
token_env, not a secret.SecretRef -- TelegramConfig has no such field.
Telegram also requires at least one allowed_user_id: AllowedUserIDs is
deny-by-default, so a bot token with none configured answers nobody,
which reads as broken rather than as the safe default it is.
SecretSink is a caller-supplied sink (Put buffers, Commit persists),
not a return value: Run never hands back plaintext secrets, and the
caller is expected to call Commit only after proving the generated
config loads -- so a validation failure never leaves a secret written
without the config that references it.
Refs archie-core-cbk
archie-core-via. internal/infrastructure/configuration/setup defines
Prompter -- Select, ReadLine, ReadSecret, Confirm, all context-aware --
which is all the setup step logic (to follow) needs from a prompt
surface. No terminal dependency, so steps are testable without a TTY.
internal/infrastructure/terminalprompt.Terminal is the real
implementation, using golang.org/x/term. New refuses construction when
its input isn't a real terminal, with a message naming the problem --
enforcing "a non-TTY invocation fails with a clear message rather than
writing a partial config" at the earliest possible point, rather than
falling back to a plain read that would silently accept piped/scripted
input (ssh, `archied setup < answers.txt`, CI). Terminal satisfies
Prompter structurally; this package does not import that one, so
there's no infrastructure -> infrastructure edge for what Go's type
system already gives for free.
Each blocking read runs in a goroutine raced against ctx.Done(), so
Ctrl+C returns promptly instead of leaving the operator stuck; the
losing goroutine is abandoned on cancellation; term.ReadPassword's own
raw-mode restoration is not interruptible, so an abandoned secret read
can leave the terminal in raw mode until it unblocks on its own --
documented on Terminal.readSecret rather than worked around, since a
cancelled setup command is expected to exit shortly after via its own
signal handling anyway.
Tested by constructing a Terminal directly (bypassing New, same
package) over a plain pipe with an injected readSecret seam -- New's
TTY gate itself is exercised separately against a real os.Pipe(). No
weakening of the production constructor for testability.
Refs archie-core-via