LLM calls: trace model and tool calls in gateway and archie-agent #254

Closed
opened 2026-10-27 13:35:34 +00:00 by sam · 0 comments
Owner

archie-core uses neither ai-sdk's middleware nor its telemetry package, and cannot: ai-sdk runtime builds providers internally with no transport or middleware option. LLM calls run in archie-gateway (chat turns, curators, MCP sampling) and archie-agent (modelloop LoopRunner). The real fix is in ai-sdk (ai-sdk-161: core emits OTel spans with gen_ai attributes, runtime accepts a transport).

Design

After ai-sdk-161 ships: bump ai-sdk; LLM calls already receive the request/task ctx (gateway chat_setup Chat/ChatStream, modelloop agentloop.Run), so spans parent automatically once the provider from fzb3x.3 is installed. Pass a transport via the new runtime.Config option: otelhttp.NewTransport(base) outermost, where base in archie-agent is the egress.ModelTransport that agentworker currently installs as http.DefaultTransport (replace that global swap with the option). Message content capture stays off.

Interim if ai-sdk work slips: otelhttp.NewTransport around http.DefaultTransport in gateway and agent startup gives bare HTTP client spans with no model/token data.

Acceptance criteria

A chat turn and a workflow stage each produce model-call spans with model, token usage and finish reason, and tool-call child spans, parented to the gateway request or task span.

Close reason

b1ef0be3: ai-sdk v0.5.0, otelhttp model transport in gateway and agent; gen_ai spans parent to caller span (verified with a throwaway SDK test)


Imported from beads archie-core-fzb3x.2 · GitHub #1853 · type feature · priority P2 · created 2026-10-25 · closed 2026-10-25

archie-core uses neither ai-sdk's middleware nor its telemetry package, and cannot: ai-sdk runtime builds providers internally with no transport or middleware option. LLM calls run in archie-gateway (chat turns, curators, MCP sampling) and archie-agent (modelloop LoopRunner). The real fix is in ai-sdk (ai-sdk-161: core emits OTel spans with gen_ai attributes, runtime accepts a transport). ### Design After ai-sdk-161 ships: bump ai-sdk; LLM calls already receive the request/task ctx (gateway chat_setup Chat/ChatStream, modelloop agentloop.Run), so spans parent automatically once the provider from fzb3x.3 is installed. Pass a transport via the new runtime.Config option: otelhttp.NewTransport(base) outermost, where base in archie-agent is the egress.ModelTransport that agentworker currently installs as http.DefaultTransport (replace that global swap with the option). Message content capture stays off. Interim if ai-sdk work slips: otelhttp.NewTransport around http.DefaultTransport in gateway and agent startup gives bare HTTP client spans with no model/token data. ### Acceptance criteria A chat turn and a workflow stage each produce model-call spans with model, token usage and finish reason, and tool-call child spans, parented to the gateway request or task span. ### Close reason b1ef0be3: ai-sdk v0.5.0, otelhttp model transport in gateway and agent; gen_ai spans parent to caller span (verified with a throwaway SDK test) --- _Imported from beads `archie-core-fzb3x.2` · GitHub #1853 · type feature · priority P2 · created 2026-10-25 · closed 2026-10-25_
sam added the mediumfeature labels 2026-10-27 13:35:34 +00:00
sam self-assigned this 2026-10-27 13:35:34 +00:00
sam closed this issue 2026-10-27 13:35:34 +00:00
sam added a new dependency 2026-10-27 13:41:50 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#254