feat(logging): cursor pagination and filtering for task_logs

task_logs could only read the newest N entries with no way to walk the
rest, so a task's log beyond the page cap was unreachable and the agent
had no honest way to say what was missing. Add logging.Page -- a
byte-offset cursor over the tail-most 8 MiB scan window -- plus
Query.Since/Until time bounds, and wire the gateway's task_logs tool to
expose level/since/until/after_id with cursor and more_available in the
result.

The reader now reads the scan window into a single in-memory buffer and
walks it by exact slice offsets. Offsets into a contiguous buffer are
trivially correct, which an incremental reader over the file is not:
bufio.Reader buffers ahead, so its underlying file position runs one
buffer past its logical position and cannot be used as a cursor. The
same rewrite removes f.Read in favour of f.ReadAt (full-or-error), since
a silent short read would drop the newest entries and still report
success.

The scan window is a documented hard ceiling: for a log over 8 MiB the
oldest bytes are unreachable by any cursor -- that is what Truncated
reports, and send_file remains the path for a complete archive. The
tool description teaches the model the difference between more_available
(more this call did not return) and truncated (older entries the scan
cap hides).

Closes archie-core-vq92 (#625).
This commit is contained in:
sam committed 2026-08-26 02:02:36 +10:00
1 parent 36501daf8a
commit 5e240721e2
7 files changed
+1247 -85

No files matched your search

+2 -2
View File
@@ -13,7 +13,7 @@
{"_type":"issue","id":"archie-core-jy5m","title":"Make task timeline events operationally informative","description":"Task timelines currently render generic stage_start/stage_finish entries with no stage name or outcome, and task_retried entries add little beyond the event kind. Surface the existing event stage, workflow, detail, and relevant structured data so operators can understand what happened without opening raw logs.","status":"closed","priority":1,"issue_type":"bug","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-24T17:39:51Z","created_by":"Sam Catlow","updated_at":"2026-08-24T19:49:36Z","started_at":"2026-08-24T17:39:57Z","closed_at":"2026-08-24T17:55:34Z","close_reason":"Task timelines now render stage/retry context; baseline and gate-cycle logs expose useful bounded failure output. task check and independent review pass.","external_ref":"https://github.com/samcharles93/archie-core/issues/643","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"archie-core-y1hv","title":"Relay dashboard update completion through Telegram","description":"Integrate the existing Carina-side behavior so dashboard-initiated updates can deliver their post-restart watchdog result to an authorized Telegram recipient, while retaining the web UI report path fallback.","status":"closed","priority":1,"issue_type":"bug","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-24T17:18:30Z","created_by":"Sam Catlow","updated_at":"2026-08-24T19:49:36Z","started_at":"2026-08-24T17:18:35Z","closed_at":"2026-08-24T17:19:42Z","close_reason":"Implemented, regression-tested, full task check passed, and adversarial review finding resolved.","external_ref":"https://github.com/samcharles93/archie-core/issues/644","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"archie-core-b7f1","title":"Fix Web UI session UUID compatibility and update-status synchronization","description":"The production Web UI crashes when crypto.randomUUID is unavailable, preventing chat session creation. Review and fix UUID generation compatibility plus the recovered Web UI/Telegram update-status synchronization issue, with regression tests and production-path verification.","status":"closed","priority":1,"issue_type":"bug","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-24T16:50:12Z","created_by":"Sam Catlow","updated_at":"2026-08-24T19:49:37Z","started_at":"2026-08-24T16:50:18Z","closed_at":"2026-08-24T17:00:21Z","close_reason":"Implemented UUIDv7 identities, replayable archived chat tool activity, and post-start update report delivery; full quality gate passes.","external_ref":"https://github.com/samcharles93/archie-core/issues/645","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"archie-core-vq92","title":"task_logs truncates at 50KB with no pagination, so full logs are unreachable","description":"The task_logs chat tool caps its response at 50 KB and offers no way to fetch the rest. An operator asking the agent to retrieve or archive a task's logs gets a silently clipped window, and the agent has no mechanism to page past it. Observed 2026-08-24: asked to save a full transcript and agent logs, the agent could only produce a partial archive and had to document the truncation as a limitation rather than satisfy the request.\n\nThe failure compounds with archie-core-p1bl (errors logging as `{}`): a truncated log of uninformative errors leaves an operator with nothing at all, precisely when a task is stuck.\n\n## What is needed\n\nTruncation itself is legitimate -- a tool result must be bounded, and unbounded log text would blow the model's context. The defect is that truncation is TERMINAL: there is no continuation, no offset/cursor, no way to narrow the window, and the caller cannot tell how much was lost.\n\nEstablish first where the cap is applied (the task_logs tool in internal/gateway/task_tools.go and whatever log reader it calls in internal/logging -- note internal/logging owns the on-disk format end to end and internal/webui parses nothing, per CLAUDE.md, so the reader is the right place for range support).\n\nDesign points to decide:\n- A cursor or offset so successive calls walk the whole log, rather than always returning the same head or tail.\n- Filters that make bounded retrieval useful rather than merely possible: by level (errors only), by time range, by stage.\n- The response must state explicitly that it was truncated and how much remains, so the agent can report honestly instead of implying completeness.\n- Consider whether bulk retrieval belongs in a tool result at all, or whether the right answer for \"save the full logs\" is to write the file and deliver it via the new send_file path (archie-core-absb), keeping the tool result small.\n\n## Acceptance Criteria\n\nAn operator can obtain a task's complete logs through the agent, by paging or by file delivery; a truncated response says so and reports what remains rather than ending silently; level and time-range filtering exist so a bounded call can still answer a specific question; the cap remains enforced so no single response can exhaust the model's context; tests cover a log larger than the cap and assert that repeated calls retrieve all of it without gaps or duplication.\n","status":"in_progress","priority":1,"issue_type":"bug","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-24T15:48:15Z","created_by":"Sam Catlow","updated_at":"2026-08-25T15:03:46Z","started_at":"2026-08-25T15:03:46Z","lease_expires_at":"2026-08-25T15:08:46Z","heartbeat_at":"2026-08-25T15:03:46Z","external_ref":"https://github.com/samcharles93/archie-core/issues/625","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"archie-core-vq92","title":"task_logs truncates at 50KB with no pagination, so full logs are unreachable","description":"The task_logs chat tool caps its response at 50 KB and offers no way to fetch the rest. An operator asking the agent to retrieve or archive a task's logs gets a silently clipped window, and the agent has no mechanism to page past it. Observed 2026-08-24: asked to save a full transcript and agent logs, the agent could only produce a partial archive and had to document the truncation as a limitation rather than satisfy the request.\n\nThe failure compounds with archie-core-p1bl (errors logging as `{}`): a truncated log of uninformative errors leaves an operator with nothing at all, precisely when a task is stuck.\n\n## What is needed\n\nTruncation itself is legitimate -- a tool result must be bounded, and unbounded log text would blow the model's context. The defect is that truncation is TERMINAL: there is no continuation, no offset/cursor, no way to narrow the window, and the caller cannot tell how much was lost.\n\nEstablish first where the cap is applied (the task_logs tool in internal/gateway/task_tools.go and whatever log reader it calls in internal/logging -- note internal/logging owns the on-disk format end to end and internal/webui parses nothing, per CLAUDE.md, so the reader is the right place for range support).\n\nDesign points to decide:\n- A cursor or offset so successive calls walk the whole log, rather than always returning the same head or tail.\n- Filters that make bounded retrieval useful rather than merely possible: by level (errors only), by time range, by stage.\n- The response must state explicitly that it was truncated and how much remains, so the agent can report honestly instead of implying completeness.\n- Consider whether bulk retrieval belongs in a tool result at all, or whether the right answer for \"save the full logs\" is to write the file and deliver it via the new send_file path (archie-core-absb), keeping the tool result small.\n\n## Acceptance Criteria\n\nAn operator can obtain a task's complete logs through the agent, by paging or by file delivery; a truncated response says so and reports what remains rather than ending silently; level and time-range filtering exist so a bounded call can still answer a specific question; the cap remains enforced so no single response can exhaust the model's context; tests cover a log larger than the cap and assert that repeated calls retrieve all of it without gaps or duplication.\n","status":"closed","priority":1,"issue_type":"bug","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-24T15:48:15Z","created_by":"Sam Catlow","updated_at":"2026-08-25T16:02:24Z","started_at":"2026-08-25T15:03:46Z","closed_at":"2026-08-25T16:02:24Z","close_reason":"task_logs cursor pagination, level/time filters, and honest truncation reporting","external_ref":"https://github.com/samcharles93/archie-core/issues/625","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"archie-core-p1bl","title":"Task logs serialise errors to an empty object, destroying diagnostics","description":"Error values in task logs sometimes render as an empty JSON object, so the one field that says WHY something failed carries no information. Observed in production task logs 2026-08-24, in the same stream:\n\n {\"time\":\"...\",\"level\":\"ERROR\",\"message\":\"handle failed\",\"fields\":{\"err\":{}}}\n {\"time\":\"...\",\"level\":\"ERROR\",\"message\":\"handle failed\",\"fields\":{\"err\":\"stage re…\"}}\n\nSame message, same field, two different encodings -- one useful, one useless. The empty-object form is what an operator hits when diagnosing a stuck task, which is exactly when the log is the only evidence available.\n\n## Likely mechanism (to be confirmed, not assumed)\n\n`{}` is what encoding/json produces for a struct with no exported fields. A custom error type (a struct implementing `error` via a value or pointer receiver, with all fields unexported) marshals to `{}` rather than to its Error() string, because encoding/json does not consult the error interface. Errors that happen to be `*errors.errorString` or `fmt.Errorf` results go through a different path and survive as text.\n\nStart at internal/logging (attrs.go and its tests already reference a \"handle failed\" record, so the attribute handling is there) and establish where an `error` attribute is converted for the JSONL sink. The fix is to render errors through their Error() string -- and, where the error carries structured context worth keeping, to do so ADDITIONALLY rather than instead.\n\n## Why P1\n\nSam's words: \"LOGGING! ... this is another critical requirement.\" Diagnosis of every other defect depends on it. Two separate investigations this week were slowed by log output that recorded that something failed without recording what.\n\nNote this compounds archie-core-1ysw-era debugging and the parked-task investigation: an operator seeing `{\"err\":{}}` has strictly less information than one seeing no log line at all, because the empty object implies the error was captured.\n\n## Steps to Reproduce\n\n1. Arrange for a code path that logs a custom error type (unexported fields) as an `err` attribute.\n2. Read the task's JSONL log via the dashboard, /api logs, or the task_logs tool.\n3. Observe `\"err\":{}` where the message should be.\n4. Compare with a path that logs a fmt.Errorf value and observe `\"err\":\"...\"` in the same file.\n\n## Acceptance Criteria\n\nEvery error logged as an attribute renders its Error() text in the JSONL sink regardless of the concrete error type; a custom error type with only unexported fields is covered by a test asserting the message survives; structured context on an error is preserved in addition to, never instead of, the message; a regression test fails against the current encoder; no log line can report an error as an empty object.\n","status":"closed","priority":1,"issue_type":"bug","owner":"sam@catlow.tech","created_at":"2026-08-24T15:47:44Z","created_by":"Sam Catlow","updated_at":"2026-08-25T15:05:24Z","closed_at":"2026-08-25T15:05:24Z","close_reason":"Shipped in 6a23f8d: logging resolves every non-nil error attribute through Error(), with custom unexported-field and typed-nil regression coverage; included in archied v1.18.0.","external_ref":"https://github.com/samcharles93/archie-core/issues/626","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"archie-core-440c","title":"Dashboard mutations 403 behind a TLS-terminating reverse proxy: Origin check ignores X-Forwarded-Proto","description":"Every operator action from the dashboard -- Stop, Cancel, Retry, Approve, Reject, Abandon, Archive -- fails with HTTP 403 in the production deployment. Observed 2026-08-24: `POST https://archie.catlow.cloud/api/tasks/3/action` returned 403 in 7ms with a 30-byte body. This left the operator with no way to stop a running task from the web UI at all.\n\n## Root cause\n\n`authorizeTaskMutation` (internal/webui/api_tasks.go:197-221) derives the expected Origin scheme from the daemon's OWN connection:\n\n wantScheme := \"http\"\n if r.TLS != nil {\n wantScheme = \"https\"\n }\n ...\n if ... || !strings.EqualFold(u.Scheme, wantScheme) || !strings.EqualFold(u.Host, r.Host) {\n http.Error(w, \"cross-origin mutation refused\", http.StatusForbidden)\n\narchied runs behind a reverse proxy that terminates TLS (confirmed: archie.catlow.cloud redirects to auth.catlow.cloud for forward auth, and the request was HTTP/2 to the proxy). The daemon therefore receives a PLAIN HTTP connection, so `r.TLS == nil` and `wantScheme == \"http\"`, while the browser correctly sends `Origin: https://archie.catlow.cloud`. Scheme mismatch -\u003e 403 on every mutation.\n\n`X-Forwarded-Proto` is never consulted anywhere in the codebase (`grep -rn 'X-Forwarded' internal/webui/ internal/config/` returns nothing).\n\n## Evidence the body confirms which branch fired\n\nThere are exactly two 403 paths in this handler. Their response bodies differ in length:\n- \"missing CSRF header\\n\" = 20 bytes\n- \"cross-origin mutation refused\\n\" = 30 bytes\n\nThe observed response was 30 B, so the Origin comparison is what rejected it, NOT the CSRF header. The dashboard is sending X-Archie-CSRF correctly.\n\n## Why this is P1\n\nIt is a total loss of the operator control surface in the supported deployment shape. It also removes the documented remedy for other faults: archie-core-0dic leaves a stopped task's row reading `running`, and `Parked`'s retry action is the stated recovery for a review that fails closed (docs/prds/adversarial-self-review.md section 4) -- both assume dashboard actions work.\n\nNote the deployment model is explicitly plural (CLAUDE.md \"Deployment model\": several supported shapes, systemd is not required), so \"runs behind a proxy\" is a first-class configuration, not an edge case.\n\n## Steps to Reproduce\n\n1. Serve archied behind any TLS-terminating reverse proxy (nginx, Caddy, Traefik, or a forward-auth proxy such as Authelia), with the daemon listening on plain HTTP.\n2. Load the dashboard over https and press Stop on a running task.\n3. Observe `POST /api/tasks/\u003cid\u003e/action` -\u003e 403, body \"cross-origin mutation refused\".\n4. The same build works when reached directly over http, or over https terminated by the daemon itself, which is why this does not reproduce in local development.\n\n## Acceptance Criteria\n\nDashboard mutations succeed when archied is behind a TLS-terminating reverse proxy; the effective scheme is derived from X-Forwarded-Proto (and the effective host from X-Forwarded-Host where applicable) ONLY when the request arrives from a trusted proxy, configured explicitly -- an unconditional trust of these headers would let any client forge the Origin check and defeat the CSRF protection this function exists to provide; direct (non-proxied) deployments keep their current strict behaviour; tests cover proxied-https, direct-http, direct-https, and a spoofed X-Forwarded-Proto from an untrusted source; the config surface and its default are documented in config.example.toml and the deployment profiles.\n","notes":"CONFIRMED LIVE against production 2026-08-25 via Playwright, signed in as the operator.\n\nReproduced the dashboard's own Stop request from the loaded page:\n\n fetch('/api/tasks/3/action', {method:'POST',\n headers:{'X-Archie-CSRF':'1','Content-Type':'application/json'},\n body:'{\"action\":\"stop\"}'})\n\nResult: status 403, body exactly \"cross-origin mutation refused\\n\" (30 bytes), with location.origin = https://archie.catlow.cloud and location.protocol = https:.\n\nThis settles which of the two 403 branches fires. The request carried a correct X-Archie-CSRF header and same-origin credentials, so it passed the CSRF check and was rejected by the Origin comparison in authorizeTaskMutation. Combined with the deployment shape (archie.catlow.cloud 302s to auth.catlow.cloud for forward auth, so TLS terminates at the proxy and the daemon sees plain HTTP), the scheme comparison is the failing term: browser Origin scheme 'https' vs the daemon's computed wantScheme 'http' from r.TLS == nil.\n\nThe dashboard confirms the operator impact at the same time: the header reads '1 running' and task #3 is still listed as running (that stale row is archie-core-0dic, a separate defect). So in production the Line truncated
{"_type":"issue","id":"archie-core-0dic","title":"Stopped tasks stay 'running' forever: terminal writes ride the cancelled context","description":"/stop reports success and genuinely does stop the container, but the task row is never updated, so the dashboard, /status and task_list all show the task as running indefinitely. Observed 2026-08-24: task #3 was stopped and was still displayed as running the following day.\n\n## Root cause\n\n`Daemon.runViaAgent` (internal/daemon/daemon.go:1010-1100) writes its terminal state with the SAME context that was just cancelled, and discards the result:\n\n reply, err := d.requestTaskRun(ctx, task.ID, data) // returns ctx.Err() on /stop\n if err != nil {\n _ = d.Store.Transition(ctx, task.ID, store.StatusRunning, store.StatusParked, \"taskrun request failed: \"+err.Error())\n\n`store.Transition` writes through `tx.ExecContext(ctx, ...)` (internal/store/store.go:358-365), so on a cancelled context the write fails immediately and the `_ =` throws the error away. The row stays `running` with nothing left to correct it.\n\nThere are SIX such writes in runViaAgent (grant unavailable, grant failed, encode failed, request failed, decode failed, run failed). Every one is unreachable in exactly the situation it exists for -- cancellation -- because cancellation is what invalidates the context it writes with.\n\n`cleanupTerminalTaskWorktree` (daemon.go:897) has the same defect: it re-reads via `d.Store.TaskByID(ctx, ...)` and returns early on error, so a cancelled task also skips worktree cleanup.\n\n## Why the container is NOT the problem\n\n`Pool.Release` already gets this right and is the precedent for the fix: it detaches deliberately with `context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)` and documents why -- \"the most important reason a task is on its way out is that it was cancelled... Teardown is cleanup, so it gets its own deadline rather than inheriting the caller's cancellation.\" So teardown works; only state recording is broken.\n\n## Contributing factor\n\n`StoreTaskController.StopRunning` (internal/gateway/tasks.go:227) deliberately writes no terminal state, documented as \"A stopped task unwinds through its own error paths and lands wherever that leaves it, usually parked\". That assumption is now false: the unwind path cannot record anything, so nothing lands. Note `CancelChatTask` (the /cancel path) DOES write state explicitly and is unaffected.\n\n## Steps to Reproduce\n\n1. Start an autonomous task so it reaches `running` with a live archie-agent container.\n2. Issue `/stop` from chat (or Stop from the dashboard).\n3. Observe the container is stopped and removed by Pool.Release.\n4. Observe the task row is still `status = running` in the store, in `task_list`, in `/status` and on the dashboard.\n5. Wait: nothing ever corrects it. Only a daemon restart re-queues it, via the crash-recovery path that re-queues tasks left `running`.\n\n## Acceptance Criteria\n\nTerminal state writes on the cancellation path use a context that survives cancellation (`context.WithoutCancel`, with its own timeout, mirroring Pool.Release); their errors are logged rather than discarded; a stopped task reaches a terminal state observable in the store, task_list and the dashboard without a daemon restart; cleanupTerminalTaskWorktree also runs for a cancelled task; a regression test cancels a task mid-run and asserts the row is no longer `running`, and fails against the current code.\n","notes":"Timeline correction: the task was observed still 'running' roughly ONE HOUR after /stop, not a day (an earlier note said a day; that was inferred wrongly from a date rollover). An hour is if anything stronger evidence, since no plausible refresh interval covers it.\n\nChronology correction: the agent's pre-/stop task_list reading 'still marked running' is NOT evidence of this bug -- nothing had stopped the task at that point, so it was reporting correctly. The genuine evidence is (a) the row still 'running' an hour after /stop, and (b) the code path itself, which predicts exactly that and was found by reading runViaAgent rather than from the transcript.\n\nScope note: this bug explains a stale ROW only. It does not explain a still-live container, because container.Pool.Release deliberately detaches from the cancelled context and does tear down. If a container is ever found alive after /stop, that is a SEPARATE defect, not this one.","status":"closed","priority":1,"issue_type":"bug","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-24T15:22:19Z","created_by":"Sam Catlow","updated_at":"2026-08-24T15:50:13Z","started_at":"2026-08-24T15:31:34Z","closed_at":"2026-08-24T15:42:47Z","close_reason":"Closed","external_ref":"https://github.com/samcharles93/archie-core/issues/628","dependency_count":0,"dependent_count":0,"comment_count":0}
@@ -51,7 +51,7 @@
{"_type":"issue","id":"archie-core-7d5u.2","title":"Reaction delivery via embedded NATS, no external server required","description":"Reactions need at-least-once delivery distinct from the lossy events.Bus (see epic description). The delivery mechanism is NATS -- that part is settled, not a design question. What's open is deployment shape.\n\nVERIFIED 2026-08-22: config.NATSConfig.URL empty currently means 'no NATS, use SQLite ClaimNext' (internal/config/config.go:573-582). That fallback was designed for task distribution working without a broker at all. It was never meant to generalise to 'archie has no NATS' as a permanent architectural stance -- the intended second deployment shape alongside an external NATS server was always NATS EMBEDDED (in-process), not a different eventbus.\n\ngithub.com/nats-io/nats-server/v2 is already a direct dependency and is proven to work in this codebase: internal/infrastructure/agenttransport/nats/transport_test.go and others already start a real embedded server via nats-server/v2/test for tests. It has just never been wired into production startup -- cmd/archied only knows 'external URL configured' vs 'fall back to SQLite'. No new library, no new bus, no architectural gap to design around: this is 'wire the dependency we already have and already test against into the daemon's own startup path.'\n\nDesign questions that remain, now correctly scoped: does an embedded NATS instance replace the SQLite fallback outright for single-process deployments, or run alongside it; what config controls embedded vs external vs disabled; does the daemon own the embedded server's lifecycle the same way it owns everything else (start/stop paired with daemon start/stop, per the shutdown-ordering rules in ARCHITECTURE.md).","acceptance_criteria":"A single-process deployment with no external NATS URL configured still gets reaction delivery via an embedded NATS instance; the daemon starts and stops the embedded server as part of its own lifecycle; no second eventbus type is introduced; internal/infrastructure/eventbus/nats is reused as the transport rather than duplicated.","notes":"2026-08-22, from docs/prds/event-sources-and-reactions.md: do not reuse the existing nats.Client's stream creation as-is. internal/infrastructure/eventbus/nats/client.go:65 hardcodes Retention: jetstream.WorkQueuePolicy -- correct for ARCHIE_TASKS (task distribution, one consumer claims each message) and wrong for reactions, which need fan-out (every registered reaction sees every matching event). Two reactions on overlapping subjects under WorkQueuePolicy hit the exact trap CLAUDE.md documents: the second consumer silently gets nothing. Config.StreamName/Config.Subjects are already composition-supplied (config.go:30-38); Retention is not parameterized and needs to become a Config field, defaulting to today's WorkQueuePolicy so ARCHIE_TASKS is unaffected, with reactions getting a second stream under a fan-out-capable policy and their own subject namespace.","status":"closed","priority":1,"issue_type":"feature","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-21T13:26:26Z","created_by":"Sam Catlow","updated_at":"2026-08-22T09:25:10Z","started_at":"2026-08-21T18:57:41Z","closed_at":"2026-08-21T19:18:55Z","close_reason":"Embedded NATS wired into production startup: mode=embedded/external/off resolves from url, daemon owns the in-process server lifecycle, Retention parameterized for a future fan-out reaction stream. task check + lint clean; adversarial review zero surviving findings. Committed 2c7db34.","external_ref":"https://github.com/samcharles93/archie-core/issues/545","dependencies":[{"issue_id":"archie-core-7d5u.2","depends_on_id":"archie-core-7d5u","type":"parent-child","created_at":"2026-08-21T23:26:26Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-7d5u.2","depends_on_id":"archie-core-7d5u.1","type":"blocks","created_at":"2026-08-21T23:26:57Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"archie-core-7d5u.3","title":"Shared webhook mechanics: HMAC verification, per-source rate limiting, payload redaction","description":"NARROWED 2026-08-22 from this bead's prior scope, which bundled capture-storage and an approval state machine into a single 'shared mechanics' package. On reflection those are not genuinely shared yet:\n\n- Capture STORAGE is t2db.1's job -- it depends on the NATS-vs-SQLite decision 7d5u.2 owns, and building a storage engine here would either duplicate or contradict that. This package may be USED by t2db.1's capture path, but does not own persistence.\n- The draft/pending_approval/armed state machine belongs to the entity that has that state -- t2db.4's binding -- not a generic package. There is exactly one consumer (bindings) today; per 'extract into a shared primitive on the second real consumer, never in anticipation of one,' building it here now would be speculative.\n\nWhat remains, and is genuinely storage-agnostic and reusable by any inbound HTTP source regardless of what consumes it:\n\n1. HMAC-SHA256 verification -- generalize internal/channels/webhook/webhook.go's existing unexported validateHMAC into an exported function here, and have that package call into it instead of keeping its own copy. One implementation of signature checking, not two that can drift.\n2. Per-source rate limiting -- a token bucket keyed by source ID (not global), so one noisy sender cannot exhaust another's budget. Limit-exceeded is an observable rejection, not a silent drop, per t2db.5's decision.\n3. Payload redaction -- generalize internal/gateway/stream.go's unexported sensitiveParameterKey/redactToolParameters heuristic (matches token/secret/password/passwd/api_key/apikey/authorization/credential/cookie/private_key against normalized JSON keys) into a reusable, storage-agnostic transform any capture path can apply before persisting.","acceptance_criteria":"webhookguard.VerifyHMAC replaces internal/channels/webhook's private copy with no behaviour change (its existing tests still pass); a per-source rate limiter rejects over-budget requests observably; a redaction function matches the same key heuristic already proven in gateway/stream.go; the package has zero imports of any domain, infrastructure or app package.","notes":"--notes=2026-08-22: merged to main as f3d3d32 (PR #534, squash of aa23ac9+46401ab+ce4f945).","status":"closed","priority":1,"issue_type":"feature","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-21T13:26:26Z","created_by":"Sam Catlow","updated_at":"2026-08-22T09:25:11Z","started_at":"2026-08-21T15:19:09Z","closed_at":"2026-08-21T15:26:01Z","close_reason":"Built internal/webhookguard: VerifyHMAC (behavior-preserving extraction from internal/channels/webhook, which now calls it instead of keeping a private copy), RateLimiter (per-source token bucket, independent budgets, observable rejection), RedactPayload (key-name heuristic matching gateway/stream.go's marker list, but -- deliberately unlike that function -- redacts only values under sensitive keys and leaves ordinary strings/numbers/bools untouched, since a captured payload exists to be inspected for field mapping in t2db.3).\n\nAll four acceptance criteria met and independently confirmed by a fresh adversarial review with zero surviving findings: HMAC extraction is byte-for-byte behavior-preserving (diffed against the deleted function); rate limiter hand-traced correct on two scenarios plus race-clean under -race; redaction's divergence from gateway/stream.go verified by reading both functions side by side; zero non-stdlib imports confirmed by grep. The reviewer also mutation-tested both the redaction and rate-limit guards (disabled each in turn) and confirmed the test suite catches both, then reverted and re-verified go test -race and git status were clean.\n\ntask check exit 0, golangci-lint 0 issues (one forcetypeassert finding was caught and fixed properly during build, not suppressed), gofumpt clean. All 10 pre-existing internal/channels/webhook tests pass unmodified; the one now-redundant unit test was removed with a comment pointing to where its coverage relocated.","external_ref":"https://github.com/samcharles93/archie-core/issues/546","dependencies":[{"issue_id":"archie-core-7d5u.3","depends_on_id":"archie-core-1786637497036-275-8838629e","type":"relates-to","created_at":"2026-08-21T23:27:06Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-7d5u.3","depends_on_id":"archie-core-7d5u","type":"parent-child","created_at":"2026-08-21T23:26:26Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-7d5u.3","depends_on_id":"archie-core-7d5u.1","type":"blocks","created_at":"2026-08-21T23:26:58Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0}
{"_type":"issue","id":"archie-core-7d5u","title":"[EPIC] Event sources and typed reactions: archie as an extensible platform","description":"Goal: let archie react to things happening -- forge activity, inbound callers, schedules -- and let extensions hook those reactions, without a generic untyped event bus.\n\nTwo findings shape this, both verified in the tree on 2026-08-21.\n\n1. ARCHIE HAS TWO EVENT NEEDS AND ONE BUS. internal/events is documented as 'archied's in-process observability stream' with 'bounded per-subscriber buffers that DROP on overflow, so a stalled dashboard connection can never apply backpressure to the task engine'. Publish is fire-and-forget; subscribers get a read-only channel and cannot influence anything. Correct for the dashboard, wrong for reactions: a hook that silently misses events under load and cannot affect an outcome is worse than no hook, because it looks like it ran. Reactions need at-least-once delivery and the ability to change what happens. Do not bolt reactions onto events.Bus.\n\n2. THE OBVIOUS DESIGN IS PROHIBITED, DELIBERATELY. The tau model this is drawn from dispatches generically: DispatchEvent(ctx, eventName string, sessionID, ...) to out-of-process gRPC plugins. archie's approved foundation (docs/architecture/plugins-and-extensions.md) forbids that shape -- the generic plugin contract stays metadata-only and MUST NOT expose untyped hooks. plugin.Plugin is Name()/Version() only, enforced by internal/plugin/architecture_test.go and by the plugin engine rule in ARCHITECTURE.md. tau already carries the scar the rule avoids: TestUnload_DoesNotWaitForInFlightDispatch exists because a stuck plugin blocked plugin unload.\n\nThe reconciliation, and why this is an epic rather than a feature: separate INBOUND (callers push work in -- forge webhook, chat webhook, cron, channel) from REACTION (extensions respond to something that happened). Inbound becomes a typed source/intake capability family with an owning registry. Reactions become typed capability families owned by the domain whose semantics they affect. 'Webhooks aligned to types of callers' is exactly the typed-family model applied to intake: the instinct is right, the generic bus is what has to go.\n\nNear-term payoff: archie reacts to an issue being labelled the moment it happens, instead of up to poll_interval later.","acceptance_criteria":"A forge webhook triggers work behind a feature flag with polling still available as a backstop; inbound callers register through one typed family rather than bespoke wiring per caller; reactions have delivery guarantees distinct from the lossy observability bus; the generic plugin contract remains metadata-only and internal/plugin/architecture_test.go still passes.","notes":"CORRECTION 2026-08-22 (second): the earlier summary I gave characterized 'NATS is currently optional' as an open architectural gap for this epic to navigate around. That's wrong, and Sam corrected it directly: NATS was never meant to be optional as a technology. What's optional is running a standalone NATS SERVER -- config.NATSConfig.URL empty falls back to the existing SQLite ClaimNext flow (internal/config/config.go:573-582, internal/infrastructure/eventbus/nats/doc.go:24). The intended second deployment shape is NATS EMBEDDED (in-process, no external server to stand up), not 'no NATS, use some other eventbus'. There is no other eventbus in scope, was never meant to be one, and this epic must not introduce one.\n\nCurrent code state, verified: github.com/nats-io/nats-server/v2 (the embeddable NATS server package) is already a direct, non-indirect dependency, and is genuinely used to spin up an in-process server -- but only inside _test.go files (internal/infrastructure/agenttransport/nats/transport_test.go and others use github.com/nats-io/nats-server/v2/test to run tests against a real embedded server). It has never been wired into production as an actual deployment mode. So embedded NATS is proven to work in this codebase's own test harness, and the gap is purely that production startup only knows 'external URL configured' vs 'no NATS, fall back to SQLite' -- there is no production code path that starts nats-server in-process.\n\nThis changes 7d5u.2 (reaction delivery) materially: 'ride the existing NATS eventbus, or a second in-process bus' was the wrong framing. The actual question is whether reaction delivery requires an external NATS deployment or can run on an embedded NATS instance so single-process deployments still get it -- not whether to use NATS at all versus some other bus. If embedded NATS is wired for reaction delivery, capture (t2db.1) gets it too essentially for free, since they were flagged as needing the same underlying decision.","status":"open","priority":1,"issue_type":"epic","owner":"sam@catlow.tech","created_at":"2026-08-21T13:25:57Z","created_by":"Sam Catlow","updated_at":"2026-08-22T09:25:11Z","external_ref":"https://github.com/samcharles93/archie-core/issues/547","dependenLine truncated
{"_type":"issue","id":"archie-core-h019.3","title":"Reviewer runs in a context that provably never saw the implementation","description":"A reviewer that shares the implementer's context is a second opinion from the same mind, and it will rationalise the same mistakes -- the failure mode CLAUDE.md's \"never self-certified by the implementer\" rule exists to prevent.\n\nUNBLOCKED 2026-08-25. The mechanism is decided in docs/prds/adversarial-self-review.md section 1 (rewritten, superseding the version q15y invalidated): the reviewer is an `agent.Subagent` run from inside the task-scoped archie-agent worker. ai-sdk v0.1.22 is already the pinned version, so no SDK bump. This adds no host executor, no second Pool.Acquire and no nested container lifecycle -- the open question the supersession left.\n\nTwo isolations are required and only one is free:\n\n- CONVERSATION isolation is structural. `Subagent.Run` (agent/subagent.go:76-96) calls GenerateText with Prompt set and Messages never populated, so the implementer's history has no code path into the nested run. This is the property to assert in the required test.\n- WORKSPACE isolation is ours to build. A sub-agent shares the parent's process, working directory and credentials, so the `.git`-free snapshot survives from the original design and is load-bearing: `git archive \u003ccommit\u003e | tar -x` into a fresh scratch directory BESIDE the task worktree, plus `diff.patch` and `issue.md`. Stripping `.git` is what makes commit messages, branch name and reflog structurally unreachable rather than merely undisclosed.\n\nThree construction rules:\n1. `Subagent.Tools` gets a distinct read-only toolset rooted at the snapshot. NEVER the worker's own toolset -- it carries the worktreerpc publication grant, so a reviewer holding it could push the branch it exists to block.\n2. Call `Run()`, not `Tool()`. `Tool()` would let the implementer's own model elect to review itself, which is the self-certification this feature prevents. It also avoids Tool()'s two-step-parent and recursion footguns.\n3. Findings return via a closure-captured capture tool, not the return value -- `Run` yields only text. Because the sub-agent is in-process, the capture tool's Go func appends to a []workflow.ReviewFinding the stage owns. Read that slice EVEN WHEN `Run` returns an error: findings already captured survive, so a reviewer that logged three defects then hit its step limit still yields those three.\n\n## Acceptance Criteria\n\nThe reviewer is launched with only the snapshot, diff.patch and issue.md; a test fails if implementer context reaches it, asserting isolation as a construction fact rather than prompt text; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses agent.Subagent inside the existing worker rather than inventing a second execution path; the reviewer's toolset is read-only, rooted at the snapshot, and provably excludes the worktreerpc publication grant; captured findings are recovered even when Run returns an error.\n","acceptance_criteria":"The reviewer is launched with only diff + issue; a test fails if implementer context reaches it; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses the existing container/worktree isolation rather than inventing a second one.","notes":"BLOCKED 2026-08-24: not implementable as specified. docs/prds/adversarial-self-review.md is marked 'Superseded; requires redesign before implementation' -- superseded by the full-task worker boundary landed in archie-core-q15y. Its mechanism for this bead was a daemon-side second container.Pool.Acquire plus an in-process agentexec path; q15y.2 and q15y.3 removed BOTH call sites, so the isolation mechanism this bead reuses no longer exists. Needs a design decision first: how a fresh reviewer receives a .git-stripped snapshot from INSIDE the existing task-scoped archie-agent worker, without introducing a host executor or a nested container lifecycle. Do not implement against the superseded PRD. The sibling bead h019.2 (findings contract) is unaffected and is being implemented, since it is a pure domain type with no execution dependency.","status":"open","priority":1,"issue_type":"feature","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","updated_at":"2026-08-24T14:55:25Z","external_ref":"https://github.com/samcharles93/archie-core/issues/548","dependencies":[{"issue_id":"archie-core-h019.3","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.3","depends_on_id":"archie-core-h019.1","type":"blocks","created_at":"2026-08-21T13:03:25Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0}
{"_type":"issue","id":"archie-core-h019.3","title":"Reviewer runs in a context that provably never saw the implementation","description":"A reviewer that shares the implementer's context is a second opinion from the same mind, and it will rationalise the same mistakes -- the failure mode CLAUDE.md's \"never self-certified by the implementer\" rule exists to prevent.\n\nUNBLOCKED 2026-08-25. The mechanism is decided in docs/prds/adversarial-self-review.md section 1 (rewritten, superseding the version q15y invalidated): the reviewer is an `agent.Subagent` run from inside the task-scoped archie-agent worker. ai-sdk v0.1.22 is already the pinned version, so no SDK bump. This adds no host executor, no second Pool.Acquire and no nested container lifecycle -- the open question the supersession left.\n\nTwo isolations are required and only one is free:\n\n- CONVERSATION isolation is structural. `Subagent.Run` (agent/subagent.go:76-96) calls GenerateText with Prompt set and Messages never populated, so the implementer's history has no code path into the nested run. This is the property to assert in the required test.\n- WORKSPACE isolation is ours to build. A sub-agent shares the parent's process, working directory and credentials, so the `.git`-free snapshot survives from the original design and is load-bearing: `git archive \u003ccommit\u003e | tar -x` into a fresh scratch directory BESIDE the task worktree, plus `diff.patch` and `issue.md`. Stripping `.git` is what makes commit messages, branch name and reflog structurally unreachable rather than merely undisclosed.\n\nThree construction rules:\n1. `Subagent.Tools` gets a distinct read-only toolset rooted at the snapshot. NEVER the worker's own toolset -- it carries the worktreerpc publication grant, so a reviewer holding it could push the branch it exists to block.\n2. Call `Run()`, not `Tool()`. `Tool()` would let the implementer's own model elect to review itself, which is the self-certification this feature prevents. It also avoids Tool()'s two-step-parent and recursion footguns.\n3. Findings return via a closure-captured capture tool, not the return value -- `Run` yields only text. Because the sub-agent is in-process, the capture tool's Go func appends to a []workflow.ReviewFinding the stage owns. Read that slice EVEN WHEN `Run` returns an error: findings already captured survive, so a reviewer that logged three defects then hit its step limit still yields those three.\n\n## Acceptance Criteria\n\nThe reviewer is launched with only the snapshot, diff.patch and issue.md; a test fails if implementer context reaches it, asserting isolation as a construction fact rather than prompt text; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses agent.Subagent inside the existing worker rather than inventing a second execution path; the reviewer's toolset is read-only, rooted at the snapshot, and provably excludes the worktreerpc publication grant; captured findings are recovered even when Run returns an error.\n","acceptance_criteria":"The reviewer is launched with only diff + issue; a test fails if implementer context reaches it; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses the existing container/worktree isolation rather than inventing a second one.","notes":"BLOCKED 2026-08-24: not implementable as specified. docs/prds/adversarial-self-review.md is marked 'Superseded; requires redesign before implementation' -- superseded by the full-task worker boundary landed in archie-core-q15y. Its mechanism for this bead was a daemon-side second container.Pool.Acquire plus an in-process agentexec path; q15y.2 and q15y.3 removed BOTH call sites, so the isolation mechanism this bead reuses no longer exists. Needs a design decision first: how a fresh reviewer receives a .git-stripped snapshot from INSIDE the existing task-scoped archie-agent worker, without introducing a host executor or a nested container lifecycle. Do not implement against the superseded PRD. The sibling bead h019.2 (findings contract) is unaffected and is being implemented, since it is a pure domain type with no execution dependency.","status":"in_progress","priority":1,"issue_type":"feature","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","updated_at":"2026-08-25T15:06:11Z","started_at":"2026-08-25T15:06:11Z","lease_expires_at":"2026-08-25T15:11:11Z","heartbeat_at":"2026-08-25T15:06:11Z","external_ref":"https://github.com/samcharles93/archie-core/issues/548","dependencies":[{"issue_id":"archie-core-h019.3","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.3","depends_on_id":"archie-core-h019.1","type":"blocks","created_at":"2026-08-21T13:03:25Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"depeLine truncated
{"_type":"issue","id":"archie-core-h019.4","title":"Wire review as an implement-workflow stage: after gates, before PR open","description":"Sequence matters. The review runs after gates pass -- there is no point reviewing code that does not build or test -- and before the PR is opened, because the point is to not spend the operator's attention on a PR that is not ready. The slot is between StageDiffCap() and StageOpenPR() in internal/domain/workflow/implement.go.\n\nA surviving CONFIRMED finding must not open a PR. Per CLAUDE.md, a finding that survives blocks closing the work; it does not get waved through.\n\nLIFECYCLE CORRECTION (2026-08-25): this bead previously said \"parks the task as waiting_human\", which names two different states in the actual machine (internal/taskstate/taskstate.go:44-59). docs/prds/adversarial-self-review.md section 4 decided this: use store.StatusParked, following StageDiffCap's existing precedent (steps.go:91-94) -- a stage-computed policy stop with a Detail explaining why, before PR open. Parked's available actions, retry and abandon, are the right shape: fix the flagged code and retry, or abandon/override a false positive. WaitingHuman's approve/reject actions have no referent here because there is no PR yet to approve or reject.\n\nBlocking is workflow.ReviewBlocking (shipped in h019.2): Verdict == confirmed AND Level == error. A PLAUSIBLE finding never blocks, however severe its level claims to be.\n\nFAIL-CLOSED (decided 2026-08-25): a review that fails to RUN is distinct from a review that ran and found nothing, and it must not open a PR. A sub-agent that errors, is truncated, or produces no conclusion yields ReviewStatusNotRun, for which Passed() is false and Blocking() is also false -- the contract deliberately refuses to guess, and this stage resolves it as park. Rationale: Repo.ReviewEnabled is opt-in, so an operator who turned the gate on wants the gate, and a provider outage silently degrading into \"PR opened unreviewed\" is what they were buying protection against. Accepted cost: a transient model error parks a task, and Parked's retry action is the remedy. The park Detail must say the review did not run, never that it passed.\n\nScope to the implement workflow first. Routing to tdd/bootstrap/feasibility is a later decision -- feasibility in particular produces a document rather than a diff, so \"review the diff\" may not even apply.\n\nAlso adds to internal/config: Models[\"reviewer\"] (falling back to Models[\"builder\"], no schema change) and Repo.ReviewEnabled bool (default false).\n\n## Acceptance Criteria\n\nAn implement task with a clean gate and no confirmed findings opens a PR as it does today; one with a surviving confirmed finding parks as store.StatusParked with findings recorded in Detail and opens no PR; a review that fails to run also parks, with a Detail that says so rather than implying a pass; a warn-level or plausible finding does not block; the stage is observable in the task's running(workflow:stage) status and emits task events despite the nested run being non-streaming; retry after operator action re-runs the stage; the stage is inert unless Repo.ReviewEnabled is true.\n","acceptance_criteria":"An implement task with a clean gate and no confirmed findings opens a PR as it does today; one with a surviving confirmed finding parks as waiting_human with findings recorded and opens no PR; the stage is observable in the task's running(workflow:stage) status; retry after operator action re-runs the review.","status":"open","priority":1,"issue_type":"feature","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","updated_at":"2026-08-24T14:55:47Z","external_ref":"https://github.com/samcharles93/archie-core/issues/549","dependencies":[{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019.1","type":"blocks","created_at":"2026-08-21T13:03:25Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019.2","type":"blocks","created_at":"2026-08-21T13:03:25Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019.3","type":"blocks","created_at":"2026-08-21T13:03:26Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":3,"dependent_count":1,"comment_count":0}
{"_type":"issue","id":"archie-core-h019.1","title":"Design: adversarial self-review stage (one-page decision in docs/prds/)","description":"No settled design exists for this in docs/architecture/, so under the scope-discipline rule merged as PR #533 the design comes first -- briefly and decisively -- and is not finished until a diff follows it.\n\nDecide and record, in one page under docs/prds/: where the reviewer runs (reuse the per-task container pool, or a separate short-lived container); what it is given (diff plus originating issue, and explicitly what it must NOT be given); the findings contract and who owns the type; what a surviving finding does to the task lifecycle; and how the reviewer's model/effort is configured.\n\nConstraints from the repo that the design must respect: dependency direction is cmd -\u003e app -\u003e {domain, infrastructure}, so the reviewer contract belongs to the workflow domain and the execution to infrastructure/app. Agent execution is a data boundary. The model never runs git. Prefer environmental enforcement over prompt rules -- 'the reviewer cannot see the implementation' must be a property of how it is launched, not an instruction in its prompt.\n\nDo not write an architecture document, migration plan or review campaign. One page that decides.","acceptance_criteria":"A one-page decision exists under docs/prds/ answering: reviewer isolation mechanism, inputs and forbidden inputs, findings contract owner, lifecycle effect of a surviving finding, and model/effort configuration. It decides rather than surveys, and cites the packages it will touch.","status":"closed","priority":1,"issue_type":"decision","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:39Z","created_by":"Sam Catlow","updated_at":"2026-08-24T07:12:05Z","started_at":"2026-08-22T09:30:19Z","closed_at":"2026-08-24T07:12:05Z","close_reason":"One-page decision written at docs/prds/adversarial-self-review.md: reviewer runs in a second workspace (container or in-process) pointed at a .git-stripped snapshot of the reviewed commit, never the implementer's worktree; findings contract is workflow.ReviewFinding, owned by internal/domain/workflow, reported via a structured capture tool; a surviving error-level finding sets StatusParked (not WaitingHuman -- epic wording named two different states, this resolves which); model/effort reuse the existing Models[role] map and per-stage budget override, gated by a new opt-in Repo.ReviewEnabled so h019.4 doesn't gate every repo on an uncalibrated reviewer before h019.5 lands. [Closure restored 2026-08-24: originally closed 2026-08-22, lost when the beads DB was rebuilt (.beads.backup-pre-recovery).]","external_ref":"https://github.com/samcharles93/archie-core/issues/550","metadata":null,"dependencies":[{"issue_id":"archie-core-h019.1","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:39Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":0,"dependent_count":5,"comment_count":0}
{"_type":"issue","id":"archie-core-h019.2","title":"Findings contract: a structured review result the workflow can act on","description":"The reviewer's output must be data the workflow can branch on, not prose a human has to read to find out whether the PR is safe to open. Prose is what makes a review advisory; structure is what lets a finding block.\n\nDefine the finding type in the workflow domain (the domain declares the contract it needs; it never names who implements it). At minimum a finding carries: file and line, a one-sentence defect, a concrete failure scenario (inputs/state -\u003e wrong output), a verdict distinguishing CONFIRMED from PLAUSIBLE, and a disposition once acted on (fixed / skipped / no_change_needed).\n\nThe CONFIRMED/PLAUSIBLE split is load-bearing and must not be collapsed into a single boolean: only confirmed findings may block, or the feature degrades into crying wolf and the operator learns to skim past it.","acceptance_criteria":"A finding type exists in internal/domain/workflow with file, line, summary, failure scenario, verdict and disposition; it is serialisable across the agent-execution data boundary; a review result with zero findings is representable and distinct from a review that did not run.","status":"closed","priority":1,"issue_type":"feature","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:39Z","created_by":"Sam Catlow","updated_at":"2026-08-24T09:15:56Z","started_at":"2026-08-24T08:57:20Z","closed_at":"2026-08-24T09:15:56Z","close_reason":"Implemented: workflow.ReviewFinding + ReviewReport in internal/domain/workflow/review_findings.go, merged 53bb841. The domain owns the type; gate.Finding was used as a shape precedent only, not reused, so the review contract is not coupled to the Yaegi gate's evolution. Blocking requires Verdict==confirmed AND Level==error, so a PLAUSIBLE finding never blocks however severe it claims to be. ReviewStatus makes 'ran and found nothing' structurally distinct from 'never ran' -- Passed() is false for a zero-value report, and the distinction does not rest on nil-vs-empty-slice. JSON round-trip covered; Line 0 is preserved (no omitempty) for whole-file findings. Review found and fixed a duplicate package doc comment and a redundant ReviewBlocking/BlockingReviewFindings alias pair. task check green. Fresh adversarial review clean across 8 dimensions; I independently mutation-tested the blocking rule: dropping the verdict check fails 5 tests (incl. plausible_error_MUST_NOT_block) and dropping the level check fails 5 more, so the tests genuinely pin both halves.","external_ref":"https://github.com/samcharles93/archie-core/issues/551","dependencies":[{"issue_id":"archie-core-h019.2","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:39Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.2","depends_on_id":"archie-core-h019.1","type":"blocks","created_at":"2026-08-21T13:03:24Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":3,"comment_count":0}
+12 -7
View File
@@ -576,17 +576,20 @@ func (a chatTaskLogReaderAdapter) ReadChatTaskLogs(
return gateway.ChatTaskLogResult{Attempt: attempt, Entries: []gateway.ChatTaskLogEntry{}}, nil
}
result, err := logging.Tail(path, logging.Query{
page, err := logging.Page(path, logging.Query{
Component: q.Component,
Contains: q.Contains,
Levels: q.Levels,
Since: q.Since,
Until: q.Until,
Limit: q.Limit,
})
}, q.AfterID)
if err != nil {
return gateway.ChatTaskLogResult{}, err
}
entries := make([]gateway.ChatTaskLogEntry, len(result.Entries))
for i, e := range result.Entries {
entries := make([]gateway.ChatTaskLogEntry, len(page.Entries))
for i, e := range page.Entries {
entries[i] = gateway.ChatTaskLogEntry{
Time: e.Time,
Level: e.Level,
@@ -595,9 +598,11 @@ func (a chatTaskLogReaderAdapter) ReadChatTaskLogs(
}
}
return gateway.ChatTaskLogResult{
Entries: entries,
Attempt: attempt,
Truncated: result.Truncated,
Entries: entries,
Attempt: attempt,
Truncated: page.Truncated,
Cursor: page.Cursor,
MoreAvailable: page.MoreAvailable,
}, nil
}
+196
View File
@@ -0,0 +1,196 @@
# task_logs pagination and bounded retrieval -- decision
**Status:** Approved for vq92 implementation
**Date:** 2026-08-26
**Beads issue:** `archie-core-vq92`
**Compounds with:** `archie-core-p1bl` (errors serialise as `{}`)
## Decision
`task_logs` becomes a *cursor-paginated, filterable* bounded read, not a
one-shot head-of-file window. The reader in `internal/logging` gains a
second entry point (`Page`) that walks the file forward from a byte
offset; the tool schema in `internal/gateway/task_tools.go` surfaces
level, time-range and cursor parameters; the response carries an explicit
"more available" signal so a model can either keep paging or hand the
rest to `send_file` when the operator wants a complete archive.
We do NOT add an unbounded read path. A single response is still capped at
`MaxTailLines` (2000) entries. What changes is that pagination makes the
cap reachable in practice instead of silently terminal.
## Why now
`archie-core-vq92` is the right place to fix this: the chat tool is the
user-visible failure mode, but the same defect propagates to every caller
of `internal/logging.Tail` (`internal/webui/api_logs.go` and the
`chatTaskLogReaderAdapter` in `cmd/archied/main.go`). Patching the
reader and the tool together is one change in two layers, not two
changes; CLAUDE.md's "package owns its format end-to-end" rule means the
reader is the right place for cursor support regardless.
The compound with `p1bl` matters: a truncated window of uninformative
errors leaves an operator with nothing. Cursor pagination lets the model
walk the whole file when a level filter does not narrow enough, and the
response's "more available" signal means it can honestly say "I read N of
M and stopped there" instead of presenting the head as complete.
## Shape of the API
### Reader (`internal/logging`)
Add a second entry point next to `Tail`:
```go
// Page walks the log forward from cursor (a byte offset; 0 starts at
// the beginning of the scanned window), applying the same filters as
// Tail. PageResult.Cursor is the file offset just past the last line
// the walk returned; pass it back as Cursor to continue, and stop when
// MoreAvailable is false.
func Page(path string, q Query, cursor int64) (PageResult, error)
```
`PageResult` carries `Entries`, `Truncated` (the existing meaning --
older entries exist that were not examined within `maxScanBytes`),
`Cursor` (the byte offset the next page should resume from), and
`MoreAvailable` (true iff the scan saw more matching entries beyond the
returned page within the same call).
`Cursor` is a byte offset rather than an entry ID for two reasons.
First, `TaskSink` writes JSONL through `slog.NewJSONHandler` directly --
without routing through `FeedHandler` -- so the on-disk entries carry no
`id` field today, and a per-task attempt monotonic counter would require
either a sink-side change or a reader-side reconstruction that is
brittle against rotation. Second, the byte offset uniquely identifies
"the next unread position" in a sequential scan, which is exactly what
the call pattern needs. A cursor that points at or past the file's
current size is treated as end-of-stream (empty page, `MoreAvailable`
false).
The implementation is `readWindow` (reads the tail-most `maxScanBytes`
into a single in-memory buffer) plus `readLines` (clamps the cursor into
the window, drops a partial first line on the size-cap seek, then
delegates) plus `walkWindow` (walks the buffer line-by-line by exact
slice offset). Offsets are into one contiguous buffer, so they are
trivially correct -- unlike an incremental reader over the file, whose
`bufio.Reader` buffer-ahead would make its underlying file position
unusable as a cursor.
`Tail` is NOT a wrapper over `Page`: it keeps its legacy ring-buffer
semantics (newest `Limit` matches, oldest dropped, `Truncated` set on
ring overflow). They share `readWindow`/`walkWindow` but stay separate
entry points precisely so Tail's established behaviour is unchanged.
### `Query` gains two optional fields
```go
Since time.Time // inclusive lower bound on entry.Time
Until time.Time // inclusive upper bound on entry.Time
```
`Levels`, `Component`, `Contains`, `Limit` are unchanged. The existing
filter logic and tests move unchanged.
### Tool schema (`internal/gateway/task_tools.go`)
`ChatTaskLogQuery` gains three fields:
- `Level []string` (forwarded to `Query.Levels`)
- `Since time.Time` / `Until time.Time` (forwarded as `Query.Since/Until`)
- `AfterID int64` (the cursor; 0 to start at the beginning)
`ChatTaskLogResult` gains two fields:
- `Cursor int64` -- highest ID in this page; pass back as `AfterID` to continue
- `MoreAvailable bool` -- true iff the scan saw more matching entries that
did not fit in this page
The tool description is updated to teach the model the pagination
protocol in one sentence: "pass the previous `cursor` back as `after_id`
until `more_available` is false; for an entire archive, prefer
`send_file`."
### Production adapter (`cmd/archied/main.go`)
`chatTaskLogReaderAdapter` passes the new fields through to the reader.
`TaskRegistry.Path` is already attempt-scoped, so paging stays inside
one attempt.
## What we deliberately do NOT do
- **No time-range-only read without a cursor.** Time bounds are a filter,
not a window. A model that wants only "errors in the last hour" passes
both `Levels=[ERROR]` and `Since=now-1h`. The page is still the same
size; the response's `MoreAvailable` tells the caller whether the hour
fit.
- **No "save the full log" via the tool.** That is what
`archie-core-absb`'s `send_file` is for. The PRD description for vq92
raised this as an open question; the answer is "use the existing tool"
rather than building a parallel archive endpoint.
- **No reversal or random access.** Paging is forward-only on the byte
offset. Reverse scan would require either an index or a full file
pass; neither earns its cost given `Tail` already handles "show me
the end."
- **No bumping `MaxTailLines` past 2000.** A single response must not be
able to exhaust the model's context. Paging makes the cap *reachable*
but does not relax it.
- **No raising `maxScanBytes`, and no walking rotated generations.** See
the ceiling below; lifting it is a separate decision with its own cost.
## The scan-window ceiling, and what it means for "complete"
`maxScanBytes` (8 MiB) bounds the tail of the file any single call will
examine. Paging does **not** lift that bound: every call re-derives the
window from the file's *current* size and clamps the incoming cursor up
into it, so a paging loop walks the window exhaustively and then stops.
For a log larger than 8 MiB the oldest bytes are unreachable through
`task_logs` at any cursor, and rotated generations (`attempt-N.jsonl.1`,
`.2`, ...) are separate files the reader never opens.
This is a deliberate limit, not an oversight, and it is why criterion 1
below says "paging **or** file delivery" rather than "paging". The
honest contract is:
- Within the window: paging reaches every matching entry, in order,
exactly once.
- Beyond it: `Truncated` is true, and that is the *only* correct thing
to report. An agent that pages to `more_available == false` while
`truncated` is true has read the window, not the log, and must say so.
Lifting the ceiling would mean either buffering an unbounded file or
teaching the reader to walk rotation history. Neither is needed to close
vq92 — the issue's own text accepts file delivery as the answer for
"save the full logs" — so both are deliberately out of scope here.
## Acceptance criteria
The issue's criteria, restated against the new shape:
1. An operator can obtain a task's complete logs through the agent: by
paging when the log fits the scan window, and by file delivery
(`send_file`) when it does not.
2. A bounded response says so explicitly -- `MoreAvailable` for "more
matches this call did not return", `Truncated` for "the scan cap hid
older entries" -- so the agent can report honestly instead of
implying completeness.
3. `Levels`, `Since`, `Until` filters exist so a bounded call can answer
a specific question without paging.
4. The cap stays enforced: one response cannot exceed `MaxTailLines`,
and one call cannot examine more than `maxScanBytes`.
5. Tests cover a log larger than `MaxTailLines` and assert that repeated
paged calls retrieve every entry without gaps or duplication, plus a
log larger than `maxScanBytes` asserting the window is walked
contiguously and `Truncated` is set.
## Files this change touches
- `internal/logging/reader.go` -- add `Page`, expand `Query` with `Since/Until`
- `internal/logging/reader_test.go` -- new tests for `Page`, `Since/Until`, paging without gaps
- `internal/gateway/task_tools.go` -- new `ChatTaskLogQuery` and `ChatTaskLogResult` fields, schema description
- `internal/gateway/task_tools_test.go` -- new tests for the new fields on the tool
- `cmd/archied/main.go` -- adapter passes through the new fields
The webui handler in `internal/webui/api_logs.go` is untouched: it is
already filter-aware (Levels, Component, Contains, Limit) and is not the
failing surface vq92 reports. If the dashboard later wants the same
pagination, it consumes the same reader.
+151 -31
View File
@@ -90,6 +90,20 @@ type ChatTaskLogQuery struct {
Component string
// Contains matches the message or any field value, case-insensitively.
Contains string
// Levels restricts results to these levels (case-insensitive). Empty
// means all levels.
Levels []string
// Since excludes entries whose timestamp is strictly before this. Zero
// time disables the bound.
Since time.Time
// Until excludes entries whose timestamp is strictly after this. Zero
// time disables the bound.
Until time.Time
// AfterID is the byte-offset cursor returned in the previous page's
// Cursor field. Zero means "start at the beginning of the scanned
// window". The model treats this as opaque and just passes it back
// unchanged; the byte offset is not intended to be human-readable.
AfterID int64
// Limit caps returned entries.
Limit int
}
@@ -101,8 +115,20 @@ type ChatTaskLogResult struct {
// the caller which run it looked at when none was requested explicitly.
Attempt int `json:"attempt"`
// Truncated reports that older matching entries exist beyond what this
// page examined.
// page examined (the on-disk scan hit its size cap before EOF).
Truncated bool `json:"truncated"`
// Cursor is the opaque byte offset to pass back as AfterID to read the
// next page. It is the file offset just past the last line this page
// returned, so a non-zero value after a full page means "resume here";
// when MoreAvailable is false the caller is done regardless of the
// value. Zero means the log was missing or empty, not "exhausted".
Cursor int64 `json:"cursor"`
// MoreAvailable is true when the scan saw more matching entries that
// did not fit in this page. Combine with Truncated: a result can be
// MoreAvailable=false while still Truncated=true when the scan ended
// at the size cap, or MoreAvailable=true while Truncated=false when
// more matches remain within the same scan window.
MoreAvailable bool `json:"more_available"`
}
// ChatTaskLogReader is the read surface task_logs needs. It mirrors the
@@ -294,37 +320,16 @@ func taskLogsTool(reader ChatTaskLogReader, identity string) tools.ToolEntry {
Description: "Read a task's persisted log history -- gate output, agent activity, errors -- " +
"to answer questions like \"why did task N park?\" or \"what happened on task N?\" " +
"without an operator needing to open the dashboard. " +
"Defaults to the task's latest attempt when attempt is omitted.",
"Defaults to the task's latest attempt when attempt is omitted. " +
fmt.Sprintf("One call returns at most %d entries. ", maxTaskLogLimit) +
"Two separate signals report incompleteness and they mean different things: " +
"more_available means this call left matching entries unread, so pass its " +
"cursor back as after_id to continue; truncated means the log is larger than " +
"the readable window, so its OLDEST entries cannot be reached by paging at all. " +
"If truncated is true, say so rather than implying you read the whole log, and " +
"use send_file when the operator wants the complete archive.",
Classification: tools.ClassIdempotent,
Schema: tools.JSONSchema{
"type": "object",
"properties": map[string]any{
"task_id": map[string]any{
"type": "integer",
"description": "ID of the task to read logs for.",
},
"attempt": map[string]any{
"type": "integer",
"description": "Which attempt to read. Omit for the task's current/latest attempt.",
},
"component": map[string]any{
"type": "string",
"description": "Only return entries from this component (e.g. \"gate\"). Omit for all components.",
},
"q": map[string]any{
"type": "string",
"description": "Only return entries whose message or fields contain this text (case-insensitive).",
},
"limit": map[string]any{
"type": "integer",
"description": fmt.Sprintf(
"Maximum entries to return. Defaults to %d, capped at %d.",
defaultTaskLogLimit, maxTaskLogLimit,
),
},
},
"required": []any{"task_id"},
},
Schema: taskLogsSchema(),
Handler: func(ctx context.Context, input map[string]any) (any, error) {
taskID, ok := asInt64(input["task_id"])
if !ok {
@@ -336,10 +341,28 @@ func taskLogsTool(reader ChatTaskLogReader, identity string) tools.ToolEntry {
}
limit := tools.ListLimit(input, defaultTaskLogLimit, maxTaskLogLimit)
levels := asStringSlice(input["level"])
since, err := asRFC3339(input["since"])
if err != nil {
return nil, fmt.Errorf("task_logs: since: %w", err)
}
until, err := asRFC3339(input["until"])
if err != nil {
return nil, fmt.Errorf("task_logs: until: %w", err)
}
var afterID int64
if v, ok := asInt64(input["after_id"]); ok {
afterID = v
}
// The bound identity, never input["identity"].
result, err := reader.ReadChatTaskLogs(ctx, identity, taskID, attempt, ChatTaskLogQuery{
Component: strings.TrimSpace(asString(input["component"])),
Contains: strings.TrimSpace(asString(input["q"])),
Levels: levels,
Since: since,
Until: until,
AfterID: afterID,
Limit: limit,
})
if err != nil {
@@ -350,6 +373,103 @@ func taskLogsTool(reader ChatTaskLogReader, identity string) tools.ToolEntry {
}
}
// taskLogsSchema is the task_logs input schema. It lives apart from the
// tool entry only because the literal is long; it is pure data, and the
// handler is the sole consumer.
func taskLogsSchema() tools.JSONSchema {
return tools.JSONSchema{
"type": "object",
"properties": map[string]any{
"task_id": map[string]any{
"type": "integer",
"description": "ID of the task to read logs for.",
},
"attempt": map[string]any{
"type": "integer",
"description": "Which attempt to read. Omit for the task's current/latest attempt.",
},
"component": map[string]any{
"type": "string",
"description": "Only return entries from this component (e.g. \"gate\"). Omit for all components.",
},
"q": map[string]any{
"type": "string",
"description": "Only return entries whose message or fields contain this text (case-insensitive).",
},
"level": map[string]any{
"type": "array",
"items": map[string]any{"type": "string"},
"description": "Only return entries whose level matches one of these (case-insensitive). Omit for all levels.",
},
"since": map[string]any{
"type": "string",
"format": "date-time",
"description": "RFC3339 timestamp; exclude entries strictly before this. Omit for no lower bound.",
},
"until": map[string]any{
"type": "string",
"format": "date-time",
"description": "RFC3339 timestamp; exclude entries strictly after this. Omit for no upper bound.",
},
"after_id": map[string]any{
"type": "integer",
"description": "Opaque cursor from a previous task_logs result's \"cursor\" field. Omit for the first page.",
},
"limit": map[string]any{
"type": "integer",
"description": fmt.Sprintf(
"Maximum entries to return. Defaults to %d, capped at %d.",
defaultTaskLogLimit, maxTaskLogLimit,
),
},
},
"required": []any{"task_id"},
}
}
// asStringSlice reads a JSON array of strings, tolerating absent or
// non-array values by returning nil. A model that emits a single string
// instead of an array is normalised to a one-element slice.
func asStringSlice(v any) []string {
switch t := v.(type) {
case []string:
return t
case []any:
out := make([]string, 0, len(t))
for _, item := range t {
if s, ok := item.(string); ok && strings.TrimSpace(s) != "" {
out = append(out, strings.TrimSpace(s))
}
}
return out
case string:
if s := strings.TrimSpace(t); s != "" {
return []string{s}
}
}
return nil
}
// asRFC3339 parses an RFC3339 timestamp string, treating the zero value
// as "no bound" (returns time.Time{} without error). An obviously
// malformed value is rejected so a model that emits nonsense does not
// silently get every entry. The parser accepts RFC3339Nano (with
// fractional seconds) as well as bare RFC3339 -- the schema advertises
// "RFC3339 timestamp" and per Go's time package the two layouts both
// match when no fractional component is present, but RFC3339 alone
// rejects the fractional case. RFC3339Nano is a strict superset.
func asRFC3339(v any) (time.Time, error) {
s, ok := v.(string)
if !ok || strings.TrimSpace(s) == "" {
return time.Time{}, nil
}
t, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(s))
if err != nil {
return time.Time{}, fmt.Errorf("invalid timestamp %q: %w", s, err)
}
return t, nil
}
// asInt64 reads an integer field. JSON numbers decode as float64, but a
// model that emits an integer through a different provider path can arrive
// as int or int64, so all three are accepted.
+106
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"strings"
"testing"
"time"
"github.com/samcharles93/archie-core/internal/tools"
)
@@ -348,6 +349,111 @@ func TestTaskLogsDefaultsAttemptToZero(t *testing.T) {
}
}
// TestTaskLogsForwardsPaginationQuery pins the schema fields the issue
// adds: Levels, Since, Until, AfterID, and Limit flow through to the
// adapter exactly as supplied. The reader fake records the query it
// received so a regression that drops or rewrites one of these would
// fail here.
func TestTaskLogsForwardsPaginationQuery(t *testing.T) {
reader := &fakeLogReader{}
entry := toolNamed(t, TaskTools(nil, nil, reader, nil, "archie"), "task_logs")
since := "2026-08-04T01:00:00Z"
until := "2026-08-04T01:59:59Z"
if _, err := entry.Handler(context.Background(), map[string]any{
"task_id": float64(7),
"attempt": float64(3),
"level": []any{"ERROR", "WARN"},
"since": since,
"until": until,
"after_id": float64(12345),
"limit": float64(500),
}); err != nil {
t.Fatal(err)
}
got := reader.gotQuery
if got.Component != "" {
t.Errorf("Component = %q, want empty (none supplied)", got.Component)
}
if len(got.Levels) != 2 || got.Levels[0] != "ERROR" || got.Levels[1] != "WARN" {
t.Errorf("Levels = %v, want [ERROR WARN]", got.Levels)
}
if got.Since.Format(time.RFC3339) != since {
t.Errorf("Since = %v, want %s", got.Since, since)
}
if got.Until.Format(time.RFC3339) != until {
t.Errorf("Until = %v, want %s", got.Until, until)
}
if got.AfterID != 12345 {
t.Errorf("AfterID = %d, want 12345", got.AfterID)
}
if got.Limit != 500 {
t.Errorf("Limit = %d, want 500", got.Limit)
}
}
// TestTaskLogsAcceptsSingleLevelString confirms the JSON flexibility: a
// model that emits a single string instead of an array still gets a
// single-element Levels list. The schema declares array but real models
// are inconsistent.
func TestTaskLogsAcceptsSingleLevelString(t *testing.T) {
reader := &fakeLogReader{}
entry := toolNamed(t, TaskTools(nil, nil, reader, nil, "archie"), "task_logs")
if _, err := entry.Handler(context.Background(), map[string]any{
"task_id": float64(1), "level": "ERROR",
}); err != nil {
t.Fatal(err)
}
if len(reader.gotQuery.Levels) != 1 || reader.gotQuery.Levels[0] != "ERROR" {
t.Errorf("Levels = %v, want [ERROR]", reader.gotQuery.Levels)
}
}
// TestTaskLogsRejectsInvalidTimestamp pins the boundary: a malformed
// since/until is rejected so the model cannot accidentally get the
// entire log back by emitting nonsense.
func TestTaskLogsRejectsInvalidTimestamp(t *testing.T) {
for field, key := range map[string]string{"since": "since", "until": "until"} {
t.Run(field, func(t *testing.T) {
reader := &fakeLogReader{}
entry := toolNamed(t, TaskTools(nil, nil, reader, nil, "archie"), "task_logs")
_, err := entry.Handler(context.Background(), map[string]any{
"task_id": float64(1),
key: "not a timestamp",
})
if err == nil {
t.Errorf("%s: expected error on malformed timestamp, got nil", field)
}
})
}
}
// TestTaskLogsResultSurfacesCursorAndMoreAvailable confirms the response
// shape: the cursor and more_available fields are passed through to the
// caller so the model can continue paging without re-deriving them.
func TestTaskLogsResultSurfacesCursorAndMoreAvailable(t *testing.T) {
reader := &fakeLogReader{result: ChatTaskLogResult{
Attempt: 2,
Cursor: 9999,
MoreAvailable: true,
Truncated: true,
Entries: []ChatTaskLogEntry{{Level: "INFO", Message: "x"}},
}}
entry := toolNamed(t, TaskTools(nil, nil, reader, nil, "archie"), "task_logs")
out, err := entry.Handler(context.Background(), map[string]any{"task_id": float64(7)})
if err != nil {
t.Fatal(err)
}
got, ok := out.(ChatTaskLogResult)
if !ok {
t.Fatalf("output type = %T, want ChatTaskLogResult", out)
}
if got.Cursor != 9999 || !got.MoreAvailable || !got.Truncated {
t.Errorf("page flags = (cursor=%d, more=%v, trunc=%v), want 9999/true/true",
got.Cursor, got.MoreAvailable, got.Truncated)
}
}
func TestTaskLogsHonoursExplicitAttempt(t *testing.T) {
reader := &fakeLogReader{result: ChatTaskLogResult{Attempt: 2}}
entry := toolNamed(t, TaskTools(nil, nil, reader, nil, "archie"), "task_logs")
+336 -45
View File
@@ -1,7 +1,7 @@
package logging
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"os"
@@ -40,6 +40,11 @@ type Query struct {
// Limit caps returned entries. Zero selects DefaultTailLines; values
// above MaxTailLines are clamped.
Limit int
// Since, when set, excludes entries whose Time is strictly before it.
// Until, when set, excludes entries whose Time is strictly after it.
// Both bounds are inclusive. The zero time disables its bound.
Since time.Time
Until time.Time
}
// Read bounds. A log file is unbounded input, so a request must not be able to
@@ -62,75 +67,333 @@ type Result struct {
File string `json:"file"`
}
// PageResult is what a single forward-paged call returns.
//
// Cursor is the byte offset within path at which the next page should
// resume -- pass it back as Page's cursor argument. A Cursor equal to
// or past the file's current size means there is nothing more to read.
// MoreAvailable is true iff the scan saw at least one matching entry it
// did not return because Limit was already satisfied; it does NOT imply
// the file has been read to the end (combine with Truncated).
type PageResult struct {
Entries []Entry `json:"entries"`
Truncated bool `json:"truncated"`
MoreAvailable bool `json:"more_available"`
Cursor int64 `json:"cursor"`
File string `json:"file"`
}
// logWindow is the region of a log file one read examines: the
// tail-most maxScanBytes, held in memory so offset arithmetic is
// against a contiguous buffer rather than a live file position.
type logWindow struct {
// data is the window's bytes; data[0] is at file offset start.
data []byte
// start is the file offset data begins at (0 when the whole file fits).
start int64
// fileSize is the file's size when the window was taken.
fileSize int64
// truncated reports that data is a strict suffix of the file, so
// older bytes exist that this window does not contain.
truncated bool
}
// readWindow opens path and reads its scan window into memory.
// found is false (with a nil error) for an absent file or empty path:
// file logging is optional, so neither is an error condition.
func readWindow(path string) (w logWindow, found bool, err error) {
if strings.TrimSpace(path) == "" {
return logWindow{}, false, nil
}
f, err := os.Open(path)
if err != nil {
if os.IsNotExist(err) {
return logWindow{}, false, nil
}
return logWindow{}, false, fmt.Errorf("logging: open %s: %w", path, err)
}
defer func() { _ = f.Close() }()
info, err := f.Stat()
if err != nil {
return logWindow{}, false, fmt.Errorf("logging: stat %s: %w", path, err)
}
w = logWindow{fileSize: info.Size()}
size := info.Size()
if size > maxScanBytes {
w.start = size - maxScanBytes
size = maxScanBytes
w.truncated = true
}
w.data = make([]byte, size)
// ReadAt, never Read: io.ReaderAt guarantees it either fills the
// buffer or returns a non-nil error, whereas Read may legally return
// fewer bytes with a nil error. A silent short read here would leave
// the tail of the buffer as zero bytes, which decode rejects -- so
// the NEWEST entries would vanish from every result while the call
// still reported success. That is the worst possible failure for a
// log reader, and it is the one Read invites.
//
// Guarded on len>0 because a zero-length ReadAt at end-of-file is
// permitted to return io.EOF, and an empty log is a valid empty
// result rather than an error.
if len(w.data) > 0 {
if _, err := f.ReadAt(w.data, w.start); err != nil {
return logWindow{}, false, fmt.Errorf("logging: read %s: %w", path, err)
}
}
return w, true, nil
}
// readLines walks every JSONL line in path within the scan window
// (tail-most maxScanBytes), invoking step with each decoded entry that
// matches q and the file offset just past the line's terminating '\n'.
// It returns whether the scan hit maxScanBytes before EOF (so older
// matching entries exist that were not examined) and the file offset
// the caller should use as its next-page Cursor.
//
// cursor=0 begins at the scan window's start (same as Tail); a cursor
// past the scan window's start resumes partway through. A cursor at or
// past EOF returns ok=false without invoking step -- either the caller
// has read everything, or the file shrank under rotation.
//
// SCAN WINDOW IS A HARD CEILING, and it bounds pagination too. For a
// file larger than maxScanBytes the oldest bytes are never examined by
// ANY call, no matter how the caller pages: every call re-derives the
// window from the CURRENT file size, and the cursor is clamped up into
// it. Paging therefore walks the window exhaustively and stops; it does
// not walk backwards into history. truncated=true is the signal that
// this happened, and it is the caller's job to surface that rather than
// implying the log was read whole. Rotated generations
// (<path>.1, .2, ...) are separate files this function never opens.
//
// Implementation: the window is read into memory in one Read, then
// walked by exact slice offsets. Offsets into a single contiguous
// buffer are trivially correct, which an incremental reader over the
// file is not: bufio.Reader buffers ahead, so its underlying file
// position runs up to one buffer past its logical position and cannot
// be used to derive a cursor.
func readLines(path string, q Query, cursor int64, step func(e Entry, endOff int64) bool) (truncated bool, cursorOut int64, ok bool, err error) {
w, found, err := readWindow(path)
if err != nil || !found {
return false, 0, false, err
}
truncated = w.truncated
windowStart, data := w.start, w.data
// Clamp the caller's cursor to the scan window. A cursor below
// windowStart cannot read older history than Tail would have shown;
// we clamp upward so the returned cursor stays inside the window.
if cursor < windowStart {
cursor = windowStart
}
// A cursor at or past EOF means the caller has read everything we
// can serve. Return ok=false so the caller stops looping; the
// preserved cursor lets it detect overshoot (file shrank under
// rotation) versus exact-fit.
if cursor >= w.fileSize {
return truncated, cursor, false, nil
}
// The caller's cursor is a file offset. Convert to an offset within
// `data` (which begins at windowStart). The clamp above guarantees
// cursor >= windowStart, so this cannot go negative; min() only
// guards against a caller-supplied cursor inside the window but past
// its end, which the EOF check above does not catch when the file is
// larger than maxScanBytes.
cursorInWindow := min(cursor-windowStart, int64(len(data)))
// When the caller resumes at the very start of the scan window, the
// first line may have been truncated by the size-cap seek. Drop
// bytes up to and including the first '\\n' so the caller never sees
// a partial first entry. If there is no '\\n' at all, the file (or
// the visible window) is one giant unterminated line: report it as
// truncated (already true) and stop -- there is nothing for the
// caller to page through.
if cursor == windowStart && truncated {
idx := bytes.IndexByte(data, '\n')
if idx < 0 {
return truncated, w.fileSize, false, nil
}
cursorInWindow = int64(idx + 1)
}
return truncated, walkWindow(data, windowStart, cursorInWindow, q, step), true, nil
}
// walkWindow invokes step for each entry in data at or after the
// in-buffer offset from that decodes and matches q, and returns the FILE
// offset at which the next page should resume.
//
// step receives the file offset just past the line it was given, and
// returns false to stop the walk; walkWindow then returns the cursor
// exactly there. A line that fails to decode is skipped rather than
// ending the walk -- a log may carry subprocess output that does not
// share our format.
//
// NOTE: a `false` return from step does NOT mean "no more matches". It
// means "you have enough for this page; stop here so you can return to
// the caller." The next call, started from the cursor walkWindow
// returns, will revisit the file offset step saw and continue walking.
// The caller is responsible for distinguishing the two outcomes via
// the limit it passed.
//
// The returned cursor is the FILE offset of the first byte after the
// last line walked, regardless of whether step stopped the walk early
// or the buffer was exhausted. Cursor values are in the same coordinate
// system across calls (file offset), so passing one straight back into
// the next Page call resumes the walk cleanly.
func walkWindow(data []byte, windowStart, from int64, q Query, step func(e Entry, endOff int64) bool) int64 {
pos := from
for pos < int64(len(data)) {
// lineEnd indexes the line's terminating '\n'; when the final
// line is unterminated (daemon killed mid-write) it indexes the
// end of the buffer and endOff stops there.
nl := bytes.IndexByte(data[pos:], '\n')
lineEnd := int64(len(data))
endOff := windowStart + int64(len(data))
if nl >= 0 {
lineEnd = pos + int64(nl)
endOff = windowStart + lineEnd + 1 // past the '\n'
}
raw := data[pos:lineEnd] // excludes the '\n'
if len(raw) > 0 && raw[len(raw)-1] == '\r' {
raw = raw[:len(raw)-1] // CRLF
}
if entry, decoded := decode(raw); decoded && q.matches(entry) && !step(entry, endOff) {
return endOff
}
if nl < 0 {
return windowStart + int64(len(data))
}
pos = lineEnd + 1
}
return windowStart + pos
}
// Tail returns the most recent entries in path matching q.
//
// Entries are returned oldest-first so the UI can append newer ones from the
// live stream without reordering. A missing file is not an error: file logging
// is optional, and an empty result with a clear File value lets the caller say
// so rather than showing a failure.
//
// Tail preserves the legacy ring-buffer semantics: when more than Limit
// matching entries exist within the scan window, the OLDEST matches are
// dropped and Truncated is set. Page is the forward-paged alternative --
// callers that want every entry reachable should use Page, not Tail.
func Tail(path string, q Query) (Result, error) {
res := Result{Entries: []Entry{}, File: path}
if strings.TrimSpace(path) == "" {
return res, nil
}
limit := q.Limit
if limit <= 0 {
limit = DefaultTailLines
}
limit = min(limit, MaxTailLines)
f, err := os.Open(path)
if err != nil {
if os.IsNotExist(err) {
return res, nil
}
return res, fmt.Errorf("logging: open %s: %w", path, err)
}
defer func() { _ = f.Close() }()
info, err := f.Stat()
if err != nil {
return res, fmt.Errorf("logging: stat %s: %w", path, err)
}
// Read only the tail: a long-lived daemon's log is far larger than any
// page of it, and seeking past the head keeps memory bounded.
if info.Size() > maxScanBytes {
if _, err := f.Seek(info.Size()-maxScanBytes, 0); err != nil {
return res, fmt.Errorf("logging: seek %s: %w", path, err)
}
res.Truncated = true
}
scanner := bufio.NewScanner(f)
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)
if res.Truncated {
scanner.Scan() // discard the partial first line after seeking
}
// Ring buffer: keep only the newest `limit` matches without holding the
// whole file.
matches := make([]Entry, 0, limit)
for scanner.Scan() {
entry, ok := decode(scanner.Bytes())
if !ok || !q.matches(entry) {
continue
}
truncated, _, ok, err := readLines(path, q, 0, func(e Entry, _ int64) bool {
if len(matches) == limit {
matches = append(matches[:0], matches[1:]...)
res.Truncated = true
}
matches = append(matches, entry)
matches = append(matches, e)
return true
})
if err != nil {
return Result{}, err
}
if err := scanner.Err(); err != nil {
return res, fmt.Errorf("logging: read %s: %w", path, err)
if !ok {
// A "nothing to read" early return (file > maxScanBytes with no
// terminator in the window) is still Truncated: the scan cap
// fired. Preserve that signal even though we have no entries to
// surface -- the caller must know that older matching entries
// exist beyond what we examined.
res.Truncated = truncated
return res, nil
}
res.Truncated = res.Truncated || truncated
res.Entries = matches
return res, nil
}
// Page walks path forward from cursor, returning up to Limit matching
// entries (clamped to MaxTailLines). A cursor of 0 starts at the
// beginning of the scanned window, exactly as Tail does.
//
// The returned Cursor is the byte offset at which the next call should
// resume; pass it back unchanged. When the file has been read to EOF
// within maxScanBytes, Cursor equals the file's size and MoreAvailable
// is false. Truncated is true when the scan cap fired before EOF,
// meaning older matching entries exist that were never examined -- the
// caller must decide whether to walk the rotation history or surface
// the gap.
//
// A cursor that points past the file's current size returns an empty
// page with MoreAvailable=false: rotation may have shrunk the file, and
// the next page is whatever the file now contains.
//
// Cursor=0 with a file larger than maxScanBytes begins at the first
// complete line AFTER the size-cap seek, exactly as Tail does --
// identical output to Tail under the same query when Limit is large
// enough to hold everything.
func Page(path string, q Query, cursor int64) (PageResult, error) {
res := PageResult{Entries: []Entry{}, File: path}
limit := q.Limit
if limit <= 0 {
limit = DefaultTailLines
}
limit = min(limit, MaxTailLines)
type lineAt struct {
entry Entry
endOff int64
}
matches := make([]lineAt, 0, limit)
truncated, endOff, ok, err := readLines(path, q, cursor, func(e Entry, end int64) bool {
if len(matches) < limit {
matches = append(matches, lineAt{entry: e, endOff: end})
return true
}
// Once we have limit matches, any further match means more is
// available; we keep walking only to maintain endOff accuracy.
res.MoreAvailable = true
return true
})
if err != nil {
return PageResult{}, err
}
if !ok {
// Preserved endOff carries the cursor the caller asked for
// (when ok=false because cursor > EOF, so the next call can
// detect overshoot). The scan-cap signal is also preserved.
res.Truncated = truncated
res.Cursor = endOff
return res, nil
}
res.Truncated = truncated
res.Entries = make([]Entry, len(matches))
for i, m := range matches {
res.Entries[i] = m.entry
}
// The cursor must resume AFTER the last entry this page returned, so
// the next call does not re-walk the same entries. When the walk
// stopped because the limit was reached (MoreAvailable), that resume
// point is the end of the last matching line. When the walk ran the
// whole buffer to completion (no more matches), the cursor is the
// walk's own end offset -- which can sit past trailing lines that
// failed the filter or failed to decode, and must, otherwise the next
// call would re-walk those trailing bytes forever. The two values
// differ only in the MoreAvailable=false case, which is exactly the
// case where the doc comment promises "Cursor equals the file's size".
if res.MoreAvailable {
res.Cursor = matches[len(matches)-1].endOff
} else {
res.Cursor = endOff
}
return res, nil
}
// decode parses one slog JSON line, lifting the known keys out and keeping the
// rest as structured fields. A line that is not JSON is skipped: a log file may
// contain output from a subprocess that does not share our format.
@@ -161,6 +424,31 @@ func decode(line []byte) (Entry, bool) {
return entry, true
}
// withinTimeBounds reports whether t satisfies q's Since/Until bounds.
// Both bounds are inclusive, and an unset (zero) bound is no bound.
//
// An entry whose timestamp did not parse -- missing, non-string, or
// malformed "time", all of which leave decode's Entry.Time zero rather
// than dropping the record -- cannot be placed in a range, so ANY
// time-bounded query excludes it. That check is explicit rather than
// implied, because the two comparisons are not symmetric about the zero
// time on their own: zero.Before(since) is true so Since would drop such
// entries, while zero.After(until) is false so Until would silently keep
// them, and "errors before noon" would come back carrying records with
// no known time at all.
func (q Query) withinTimeBounds(t time.Time) bool {
if q.Since.IsZero() && q.Until.IsZero() {
return true
}
if t.IsZero() {
return false
}
if !q.Since.IsZero() && t.Before(q.Since) {
return false
}
return q.Until.IsZero() || !t.After(q.Until)
}
func (q Query) matches(e Entry) bool {
if len(q.Levels) > 0 && !slices.ContainsFunc(q.Levels, func(l string) bool {
return strings.EqualFold(l, e.Level)
@@ -173,6 +461,9 @@ func (q Query) matches(e Entry) bool {
return false
}
}
if !q.withinTimeBounds(e.Time) {
return false
}
if q.Contains != "" {
needle := strings.ToLower(q.Contains)
if strings.Contains(strings.ToLower(e.Message), needle) {
+444
View File
@@ -1,11 +1,14 @@
package logging
import (
"bytes"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
)
func writeLog(t *testing.T, lines ...string) string {
@@ -175,6 +178,447 @@ func TestDecodeKeepsStructuredFields(t *testing.T) {
}
}
// TestPageMatchesTailWhenLimitFitsWholeFile pins the first-cut promise of
// Page: with cursor=0 and a Limit large enough to hold every match, the
// page is identical to what Tail would have returned.
func TestPageMatchesTailWhenLimitFitsWholeFile(t *testing.T) {
lines := []string{
line("INFO", "started", `"component":"daemon"`),
line("WARN", "rate limit", `"component":"telegram"`),
line("ERROR", "pull failed", `"component":"container"`),
line("DEBUG", "noisy", `"component":"daemon"`),
}
path := writeLog(t, lines...)
tail, err := Tail(path, Query{})
if err != nil {
t.Fatal(err)
}
page, err := Page(path, Query{}, 0)
if err != nil {
t.Fatal(err)
}
if page.MoreAvailable || page.Truncated {
t.Errorf("page flags = (more=%v, trunc=%v), want both false", page.MoreAvailable, page.Truncated)
}
if len(page.Entries) != len(tail.Entries) {
t.Fatalf("page has %d entries, tail has %d", len(page.Entries), len(tail.Entries))
}
for i := range page.Entries {
if page.Entries[i].Message != tail.Entries[i].Message {
t.Errorf("entry %d: page=%q tail=%q", i, page.Entries[i].Message, tail.Entries[i].Message)
}
}
}
// TestPageWalksWholeFileWithoutGapsOrDuplication is the issue's acceptance
// criterion #5: repeated paged calls against a log LARGER THAN THE CAP must
// retrieve every entry exactly once.
//
// total deliberately exceeds MaxTailLines (2000), because that is the cap the
// criterion is about: a single call can never return more than MaxTailLines
// entries, so a log below that size would be satisfied by one call and would
// not exercise paging at all. It stays well under maxScanBytes (8 MiB) so that
// EVERY entry is genuinely reachable -- see TestPageAcrossSizeCapSeek for the
// oversized case, where the scan window makes the oldest entries unreachable
// by design.
func TestPageWalksWholeFileWithoutGapsOrDuplication(t *testing.T) {
const total = MaxTailLines + 500
const pageSize = 700
lines := make([]string, total)
for i := range lines {
lines[i] = line("INFO", fmt.Sprintf("entry-%05d", i))
}
path := writeLog(t, lines...)
var got []string
cursor := int64(0)
for call := range 100 {
page, err := Page(path, Query{Limit: pageSize}, cursor)
if err != nil {
t.Fatalf("call %d: %v", call, err)
}
for _, e := range page.Entries {
got = append(got, e.Message)
}
if !page.MoreAvailable {
if page.Cursor < cursor {
t.Fatalf("call %d: cursor went backwards: prev=%d new=%d", call, cursor, page.Cursor)
}
break
}
if page.Cursor <= cursor {
t.Fatalf("call %d: cursor did not advance: prev=%d new=%d", call, cursor, page.Cursor)
}
cursor = page.Cursor
}
if len(got) != total {
t.Fatalf("walked %d entries, want %d (first few: %v)", len(got), total, got[:min(len(got), 5)])
}
seen := map[string]int{}
for _, m := range got {
seen[m]++
}
if len(seen) != total {
t.Errorf("got %d distinct messages, want %d (duplicates: %v)", len(seen), total, dupCount(seen))
}
}
// dupCount returns the messages seen more than once, for test diagnostics.
func dupCount(seen map[string]int) []string {
out := []string{}
for k, v := range seen {
if v > 1 {
out = append(out, fmt.Sprintf("%s×%d", k, v))
}
}
return out
}
// appendInt formats n in decimal into dst and returns the extended
// slice. Used by TestPageAcrossSizeCapSeek so every line is exactly
// the bytes the JSON encoder produced -- a fixed byte boundary would
// truncate some lines mid-string and the JSON decoder would drop them.
func appendInt(dst []byte, n int) []byte {
if n == 0 {
return append(dst, '0')
}
var digits [20]byte
i := len(digits)
for n > 0 {
i--
digits[i] = byte('0' + n%10)
n /= 10
}
return append(dst, digits[i:]...)
}
// TestPageCursorSurvivesNoOpAdvances: a page that returns no entries
// must still produce a Cursor equal to the input -- otherwise the caller
// cannot distinguish "no matches ahead" from "I lost my place".
func TestPageCursorAdvancesWithFile(t *testing.T) {
path := writeLog(t,
line("INFO", "first"),
line("INFO", "second"),
line("INFO", "third"),
)
page, err := Page(path, Query{Limit: 10}, 0)
if err != nil {
t.Fatal(err)
}
if page.MoreAvailable {
t.Errorf("MoreAvailable = true with 3 lines and Limit 10, want false")
}
if page.Cursor <= 0 {
t.Errorf("Cursor = %d, want > 0 (the file is non-empty)", page.Cursor)
}
}
// TestPageCursorPastEOFIsEmpty confirms the rotation-shrank case: a cursor
// past the current file size returns no entries and no MoreAvailable.
func TestPageCursorPastEOFIsEmpty(t *testing.T) {
path := writeLog(t, line("INFO", "only entry"))
page, err := Page(path, Query{}, 1<<20)
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 0 || page.MoreAvailable {
t.Errorf("cursor-past-EOF page = %+v, want empty + no more", page)
}
}
// TestQuerySinceUntilFilters checks the time-range filter on the new
// fields. Times are read from each entry's `time` field; the helper
// `line` writes a fixed timestamp, so the test pins the inclusive
// boundary behaviour directly.
func TestQuerySinceUntilFilters(t *testing.T) {
// Three entries: two seconds apart on the same minute.
path := writeLog(t,
`{"time":"2026-08-04T01:00:00Z","level":"INFO","msg":"early"}`,
`{"time":"2026-08-04T01:00:02Z","level":"INFO","msg":"middle"}`,
`{"time":"2026-08-04T01:00:04Z","level":"INFO","msg":"late"}`,
)
since, _ := time.Parse(time.RFC3339Nano, "2026-08-04T01:00:02Z")
until, _ := time.Parse(time.RFC3339Nano, "2026-08-04T01:00:02Z")
page, err := Page(path, Query{Since: since, Until: until, Limit: 10}, 0)
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 1 || page.Entries[0].Message != "middle" {
got := make([]string, len(page.Entries))
for i, e := range page.Entries {
got[i] = e.Message
}
t.Fatalf("Since==Until page = %v, want [middle] (inclusive on both bounds)", got)
}
// Loose bounds: only the late entry passes.
page, err = Page(path, Query{Since: since.Add(2 * time.Second), Limit: 10}, 0)
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 1 || page.Entries[0].Message != "late" {
got := make([]string, len(page.Entries))
for i, e := range page.Entries {
got[i] = e.Message
}
t.Fatalf("Since=since+2s page = %v, want [late]", got)
}
// Loose bounds: only the early entry passes (Until is strictly less
// than middle so middle is excluded).
page, err = Page(path, Query{Until: since.Add(-time.Nanosecond), Limit: 10}, 0)
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 1 || page.Entries[0].Message != "early" {
got := make([]string, len(page.Entries))
for i, e := range page.Entries {
got[i] = e.Message
}
t.Fatalf("Until=since-1ns page = %v, want [early]", got)
}
}
// TestTimeBoundsExcludeUnparseableTimestamps pins the symmetry between
// Since and Until for a record whose "time" did not parse. Such a record
// cannot be placed in a range, so BOTH bounds must exclude it -- the
// naive comparison keeps it under Until (zero.After(x) is false) while
// dropping it under Since (zero.Before(x) is true).
func TestTimeBoundsExcludeUnparseableTimestamps(t *testing.T) {
bound, err := time.Parse(time.RFC3339Nano, "2026-08-04T01:00:02Z")
if err != nil {
t.Fatal(err)
}
path := writeLog(t,
`{"level":"INFO","msg":"no-time-field"}`,
`{"time":"not a timestamp","level":"INFO","msg":"unparseable-time"}`,
`{"time":"2026-08-04T01:00:02Z","level":"INFO","msg":"has-time"}`,
)
for _, tc := range []struct {
name string
query Query
}{
{"since only", Query{Since: bound, Limit: 10}},
{"until only", Query{Until: bound, Limit: 10}},
{"both bounds", Query{Since: bound, Until: bound, Limit: 10}},
} {
t.Run(tc.name, func(t *testing.T) {
page, err := Page(path, tc.query, 0)
if err != nil {
t.Fatal(err)
}
got := make([]string, len(page.Entries))
for i, e := range page.Entries {
got[i] = e.Message
}
if len(got) != 1 || got[0] != "has-time" {
t.Fatalf("entries = %v, want only [has-time]: a record with no usable "+
"timestamp must not satisfy a time-bounded query", got)
}
})
}
// Without any bound, all three are returned -- the exclusion is a
// property of time-bounded queries, not of the records themselves.
page, err := Page(path, Query{Limit: 10}, 0)
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 3 {
t.Errorf("unbounded query returned %d entries, want all 3", len(page.Entries))
}
}
// TestPageRespectsFilter ensures paging with a level filter walks only
// matching entries and the MoreAvailable/Cursor fields account for
// every line in the scan window, not just the matches.
func TestPageRespectsFilter(t *testing.T) {
lines := []string{
line("INFO", "i1"),
line("ERROR", "e1"),
line("INFO", "i2"),
line("ERROR", "e2"),
line("INFO", "i3"),
line("ERROR", "e3"),
}
path := writeLog(t, lines...)
page, err := Page(path, Query{Levels: []string{"ERROR"}, Limit: 2}, 0)
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 2 || page.Entries[0].Message != "e1" || page.Entries[1].Message != "e2" {
t.Fatalf("first page = %+v, want [e1, e2]", page.Entries)
}
if !page.MoreAvailable {
t.Errorf("MoreAvailable = false, want true (one match remains)")
}
page2, err := Page(path, Query{Levels: []string{"ERROR"}, Limit: 2}, page.Cursor)
if err != nil {
t.Fatal(err)
}
if len(page2.Entries) != 1 || page2.Entries[0].Message != "e3" {
t.Fatalf("second page = %+v, want [e3]", page2.Entries)
}
if page2.MoreAvailable {
t.Errorf("MoreAvailable = true on final page, want false")
}
}
// TestPageCursorAtEOFWithTrailingNonMatchingLines pins the fix for the
// doc-vs-code contradiction the adversarial pass found: when the walk
// runs to EOF but the final lines fail the filter, Cursor must equal the
// file size (the walk's own end offset), NOT the end of the last
// matching line. Returning the latter would leave the trailing bytes
// unread, and a caller that follows PageResult's documented "cursor ==
// file size means done" rule would loop.
func TestPageCursorAtEOFWithTrailingNonMatchingLines(t *testing.T) {
path := writeLog(t,
line("ERROR", "e1"),
line("INFO", "i1"),
line("INFO", "i2"),
)
info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
page, err := Page(path, Query{Levels: []string{"ERROR"}, Limit: 100}, 0)
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 1 || page.Entries[0].Message != "e1" {
t.Fatalf("entries = %+v, want just [e1]", page.Entries)
}
if page.MoreAvailable {
t.Errorf("MoreAvailable = true, want false (only one ERROR line)")
}
if page.Cursor != info.Size() {
t.Errorf("Cursor = %d, want file size %d: the walk ran to EOF past the "+
"trailing INFO lines, so the cursor must sit at end-of-file, not at "+
"the end of the last match", page.Cursor, info.Size())
}
}
// TestPageAcrossSizeCapSeek is the regression test for the readLines
// refactor: a file larger than maxScanBytes must trigger the seek to
// windowStart, and paged reads across that boundary must visit every
// line without duplicates or gaps. The earlier "no-gap" test used a
// 3 KB file that never crossed maxScanBytes (8 MiB), so the seek
// branch was uncovered.
//
// The test writes a file whose size is well above maxScanBytes and
// pages through it in chunks of MaxTailLines, asserting that the union
// of returned pages contains every line exactly once. It also pins
// the Truncated flag: the first page should report Truncated=true.
func TestPageAcrossSizeCapSeek(t *testing.T) {
// 12 MiB of JSONL, fixed-width lines that decode unambiguously.
// We do NOT truncate the line at a fixed byte boundary, because a
// truncated JSON object silently fails to decode and would
// invalidate the no-gap invariant. Instead, every line is the
// same fixed size, computed from a per-line message id that
// always fits.
const totalBytes = 12 << 20
path := filepath.Join(t.TempDir(), "big.log")
f, err := os.Create(path)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = f.Close() })
var total int
buf := make([]byte, 0, 128)
for written := 0; written < totalBytes; {
buf = buf[:0]
buf = append(buf, []byte(`{"time":"2026-08-04T01:00:00Z","level":"INFO","msg":"x`)...)
buf = appendInt(buf, total)
buf = append(buf, '"', '}', '\n')
if _, err := f.Write(buf); err != nil {
t.Fatal(err)
}
written += len(buf)
total++
}
// Walk the whole scan window by paging.
var ids []int
seen := map[int]struct{}{}
cursor := int64(0)
for call := range 500 {
page, err := Page(path, Query{Limit: MaxTailLines}, cursor)
if err != nil {
t.Fatalf("call %d: %v", call, err)
}
if call == 0 && !page.Truncated {
t.Fatal("first page Truncated = false, want true: the file exceeds maxScanBytes " +
"so the oldest entries were never examined, and the caller must be told")
}
for _, e := range page.Entries {
id, err := strconv.Atoi(strings.TrimPrefix(e.Message, "x"))
if err != nil {
t.Fatalf("call %d: unparseable message %q -- a line was split mid-record", call, e.Message)
}
if _, dup := seen[id]; dup {
t.Fatalf("call %d: duplicate entry x%d", call, id)
}
seen[id] = struct{}{}
ids = append(ids, id)
}
if !page.MoreAvailable {
break
}
if page.Cursor <= cursor {
t.Fatalf("call %d: cursor did not advance: prev=%d new=%d", call, cursor, page.Cursor)
}
cursor = page.Cursor
}
// The contract is NOT "every line in the file" -- maxScanBytes caps
// the window, and for this deliberately-oversized file the oldest
// entries are out of reach by design (that is what Truncated says).
// The contract IS: everything inside the window, in order, exactly
// once, ending at the file's final entry.
if len(ids) == 0 {
t.Fatal("paged zero entries from a 12 MiB log")
}
for i := 1; i < len(ids); i++ {
if ids[i] != ids[i-1]+1 {
t.Fatalf("gap in paged walk at index %d: x%d followed by x%d", i, ids[i-1], ids[i])
}
}
if last := ids[len(ids)-1]; last != total-1 {
t.Errorf("walk ended at x%d, want the file's final entry x%d", last, total-1)
}
// Sanity-check the window arithmetic itself: the reachable count is
// determined by where maxScanBytes lands, so assert the walk covered
// a large majority of the window rather than a hardcoded count that
// would rot the moment the line format changes.
if len(ids) < 100_000 {
t.Errorf("paged only %d entries out of a %d-entry file; the 8 MiB "+
"scan window should reach far more than that", len(ids), total)
}
}
// TestPagePreservesTruncatedOnGiantLine pins the second regression the
// refactor fixes: a file that exceeds maxScanBytes with no '\\n' in the
// scan window still reports Truncated, because older matching entries
// exist beyond what was examined.
func TestPagePreservesTruncatedOnGiantLine(t *testing.T) {
path := filepath.Join(t.TempDir(), "giant.log")
// One JSON object spanning 10 MiB -- no terminator at all within
// the 8 MiB scan window.
body := []byte(`{"time":"2026-08-04T01:00:00Z","level":"INFO","msg":"x","pad":`)
body = append(body, bytes.Repeat([]byte("a"), 10<<20)...)
body = append(body, '}')
if err := os.WriteFile(path, body, 0o600); err != nil {
t.Fatal(err)
}
page, err := Page(path, Query{}, 0)
if err != nil {
t.Fatal(err)
}
if !page.Truncated {
t.Errorf("Truncated = false on giant-line file > maxScanBytes, want true")
}
}
func TestComponents(t *testing.T) {
path := writeLog(
t,