Token usage totals mislead: raw prompt sum not netted against cache hits #1558

Closed
opened 2026-10-27 13:40:52 +00:00 by sam · 0 comments
Owner

Confirmed on production task 6 (2026-09-02, carina): dashboard/PR summary showed '14 iterations · 405,004 tokens · 400,021 prompt · 4,983 completion · 360,018 cached'. Cross-checked against the per-iteration daemon log: nothing is double-counted or hardcoded -- the totals are the correct sum of 14 real, separately-billed API calls with growing context. But 360,018 of the 400,021 'prompt tokens' shown were cache hits (billed at a steep discount), not full price -- real fresh spend was ~40,003 tokens. Showing only the raw prompt-token sum makes every agent run's cost read roughly 10x worse than what was actually billed, everywhere this aggregate surfaces (PR body, dashboard timeline).

Separately (real, not just cosmetic cost): StageBaselineGate's retry-mission prompt re-injected the ENTIRE go test ./... output -- hundreds of lines covering every passing package -- into every retry iteration's prompt, not just the failing ones. Direct mechanical contributor to the growing per-iteration prompt sizes.

Design

TaskContext.RunUsage (in-memory, not persisted -- Task.TokensUsed remains authoritative) accumulates agentexec.Usage across every agent run in one workflow execution. formatTokenUsage(total, usage) renders 'N tokens (F fresh + C cached)' with a plain fallback. extractFailingGateOutput(s) strips go test 'ok' package lines, only when it recognizes --- FAIL:/FAIL\t markers, else returns input unchanged. Known gap: a task resumed after a daemon restart mid-workflow loses RunUsage and falls back to the plain total -- an honest degrade, not a bug; persisting the breakdown would need a store migration for a purely presentational value, not worth it yet.

Acceptance criteria

PR bodies and the dashboard timeline show a fresh-vs-cached breakdown (or an equivalent net figure) instead of a raw prompt-token sum, falling back gracefully when no usage breakdown is available. Gate-retry mission prompts drop passing-package output, keeping only failure-relevant content, with graceful degradation for non-Go gates or unrecognized output shapes. task check passes.

Close reason

Added TaskContext.RunUsage + accumulateUsage (agent.go, implement.go's baseline-fix branch) and formatTokenUsage (workflow.go), wired into implement.go's and tdd.go's PR-body footers plus ui/src/tasks/timeline-event.js's agent_finish rendering (raw prompt replaced with computed fresh = prompt - cached). Added extractFailingGateOutput (workflow.go) trimming go test 'ok' package lines from StageBaselineGate's retry mission, degrading to unchanged output when no --- FAIL:/FAIL\t marker survives the filter. 5 new Go tests, 1 updated JS test. task check clean, 100/100 JS tests.


Imported from beads archie-core-rwzo · GitHub #699 · type bug · priority P2 · created 2026-09-02 · closed 2026-09-02

Confirmed on production task 6 (2026-09-02, carina): dashboard/PR summary showed '14 iterations · 405,004 tokens · 400,021 prompt · 4,983 completion · 360,018 cached'. Cross-checked against the per-iteration daemon log: nothing is double-counted or hardcoded -- the totals are the correct sum of 14 real, separately-billed API calls with growing context. But 360,018 of the 400,021 'prompt tokens' shown were cache hits (billed at a steep discount), not full price -- real fresh spend was ~40,003 tokens. Showing only the raw prompt-token sum makes every agent run's cost read roughly 10x worse than what was actually billed, everywhere this aggregate surfaces (PR body, dashboard timeline). Separately (real, not just cosmetic cost): StageBaselineGate's retry-mission prompt re-injected the ENTIRE go test ./... output -- hundreds of lines covering every passing package -- into every retry iteration's prompt, not just the failing ones. Direct mechanical contributor to the growing per-iteration prompt sizes. ### Design TaskContext.RunUsage (in-memory, not persisted -- Task.TokensUsed remains authoritative) accumulates agentexec.Usage across every agent run in one workflow execution. formatTokenUsage(total, usage) renders 'N tokens (F fresh + C cached)' with a plain fallback. extractFailingGateOutput(s) strips go test 'ok' package lines, only when it recognizes --- FAIL:/FAIL\t markers, else returns input unchanged. Known gap: a task resumed after a daemon restart mid-workflow loses RunUsage and falls back to the plain total -- an honest degrade, not a bug; persisting the breakdown would need a store migration for a purely presentational value, not worth it yet. ### Acceptance criteria PR bodies and the dashboard timeline show a fresh-vs-cached breakdown (or an equivalent net figure) instead of a raw prompt-token sum, falling back gracefully when no usage breakdown is available. Gate-retry mission prompts drop passing-package output, keeping only failure-relevant content, with graceful degradation for non-Go gates or unrecognized output shapes. task check passes. ### Close reason Added TaskContext.RunUsage + accumulateUsage (agent.go, implement.go's baseline-fix branch) and formatTokenUsage (workflow.go), wired into implement.go's and tdd.go's PR-body footers plus ui/src/tasks/timeline-event.js's agent_finish rendering (raw prompt replaced with computed fresh = prompt - cached). Added extractFailingGateOutput (workflow.go) trimming go test 'ok' package lines from StageBaselineGate's retry mission, degrading to unchanged output when no --- FAIL:/FAIL\t marker survives the filter. 5 new Go tests, 1 updated JS test. task check clean, 100/100 JS tests. --- _Imported from beads `archie-core-rwzo` · GitHub #699 · type bug · priority P2 · created 2026-09-02 · closed 2026-09-02_
sam added the mediumbug labels 2026-10-27 13:40:52 +00:00
sam self-assigned this 2026-10-27 13:40:52 +00:00
sam closed this issue 2026-10-27 13:40:52 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1558