Record every model message and tool call of an agent run #1718

Closed
opened 2026-11-01 10:25:42 +00:00 by sam · 1 comment
Owner

What's wrong

An agent.run step records only its result: status, summary, usage and iteration count. There's no way to see what the agent was told, what it said, which tools it called with what arguments, or what each tool returned. That makes it guesswork to tell whether a prompt is right, why a stage stalled or parked, or where a stage could be cut or improved.

What exists

agentloop.Config has ModelHooks (every model call, including compaction and force-finish recovery, with request, latency and usage) and ToolHooks (every tool, built-ins included). The loop runner in internal/agentexec/modelloop sets neither. It only reports central-tool calls through OnToolCall.

Done when

  • Each attempt of an agent.run step keeps an ordered transcript: system prompt and mission, each model response (text and tool calls), each tool call's arguments and result (truncated past a size limit, with the full size noted), compaction points, and per-call usage and latency.
  • The transcript reaches the State Store through the existing task contracts, and survives the container ending.
  • The dashboard's run view shows it per step and attempt, readable as a conversation with tool calls collapsible.
  • Secrets that pass through tool output are redacted the same way task logs are.
## What's wrong An `agent.run` step records only its result: status, summary, usage and iteration count. There's no way to see what the agent was told, what it said, which tools it called with what arguments, or what each tool returned. That makes it guesswork to tell whether a prompt is right, why a stage stalled or parked, or where a stage could be cut or improved. ## What exists `agentloop.Config` has `ModelHooks` (every model call, including compaction and force-finish recovery, with request, latency and usage) and `ToolHooks` (every tool, built-ins included). The loop runner in `internal/agentexec/modelloop` sets neither. It only reports central-tool calls through `OnToolCall`. ## Done when - Each attempt of an `agent.run` step keeps an ordered transcript: system prompt and mission, each model response (text and tool calls), each tool call's arguments and result (truncated past a size limit, with the full size noted), compaction points, and per-call usage and latency. - The transcript reaches the State Store through the existing task contracts, and survives the container ending. - The dashboard's run view shows it per step and attempt, readable as a conversation with tool calls collapsible. - Secrets that pass through tool output are redacted the same way task logs are.
Author
Owner

Done in cf21d5835 and 32dc58a0b. The loop runner records each model call and tool call through ai-sdk's hooks as ordered, redacted, clipped entries, including reasoning. Each is persisted as an agent_transcript task event through InsertEvent, left out of the timeline read, and served per attempt at /api/tasks/{id}/transcript. The step run view draws it with Trace patterns. Verified on the e2e stack: a triage run in an agent image built from this tree recorded six entries, and the classify step's transcript rendered with prompt, mission, reasoning, steps with usage and latency, the capture call and the finish. Follow-ups: 1720 (task log redaction), 1721 (harness transcripts), 1722 (todo list for Queue and Plan).

Done in cf21d5835 and 32dc58a0b. The loop runner records each model call and tool call through ai-sdk's hooks as ordered, redacted, clipped entries, including reasoning. Each is persisted as an agent_transcript task event through InsertEvent, left out of the timeline read, and served per attempt at /api/tasks/{id}/transcript. The step run view draws it with Trace patterns. Verified on the e2e stack: a triage run in an agent image built from this tree recorded six entries, and the classify step's transcript rendered with prompt, mission, reasoning, steps with usage and latency, the capture call and the finish. Follow-ups: 1720 (task log redaction), 1721 (harness transcripts), 1722 (todo list for Queue and Plan).
sam closed this issue 2026-11-01 10:51:25 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1718