NATS and container boundary: carry traceparent across messages and into task containers #256

Closed
opened 2026-10-27 13:35:35 +00:00 by sam · 0 comments
Owner

Traces stop at every NATS hop and at the container boundary. natsrpc.Request/Respond send bare payloads with no header; eventbus.Message has no Header(). Context is dropped on request paths: forgerpc (7 forge-client calls on context.Background), worktreerpc handlerContext, taskactions Apply, webui api_config, staterpc task_grants; events.Bus.Publish takes no ctx. task.json and taskrun.Request carry no trace field, containers inherit no daemon env, and Kit containers can reach only the egress relay.

Design

No NATS instrumentation exists in nats.go v1.54.0 or otel contrib; hand-roll it.

Carrier (nats.Header is case-sensitive; propagation.HeaderCarrier canonicalises to "Traceparent" and misses lowercase senders; nats.Header has no Keys()):
type natsCarrier nats.Header
func (c natsCarrier) Get(k string) string { return nats.Header(c).Get(k) }
func (c natsCarrier) Set(k, v string) { nats.Header(c).Set(k, v) }
func (c natsCarrier) Keys() []string { return slices.Collect(maps.Keys(c)) }
Set panics on a nil map: build messages with nats.NewMsg.

API switches: natsrpc Call uses nc.RequestMsgWithContext (RequestWithContext sends no headers); Respond uses msg.RespondMsg; logrpc and agenttransport bypasses likewise. JetStream: js.PublishMsg with headers, eventbus.Message gains Header() backed by jetstream.Msg.Headers(). Request/reply: CLIENT/SERVER span pair. JetStream consume: SpanKindConsumer started with a link to the producer span context, not as its child (producer span has usually ended). messaging.* attrs (semconv/v1.43.0) are Development stability.

Request-path ctx drops to fix: forgerpc (7 forge-client calls), worktreerpc handlerContext, taskactions Apply, webui api_config, staterpc task_grants; events.Bus.Publish takes ctx.

Container trace context: otel-go has no TRACEPARENT env carrier. Add Trace map[string]string json:"trace,omitempty" to the task brief (/.git/task.json) or taskrun.Request:
c := propagation.MapCarrier{}; otel.GetTextMapPropagator().Inject(ctx, c); brief.Trace = c // daemon
ctx = otel.GetTextMapPropagator().Extract(ctx, propagation.MapCarrier(brief.Trace)) // archie-agent
Sampling follows the parent flag (ParentBased), so the container keeps the daemon decision.

Container export over NATS (compile-checked against v1.47.0; the ReadOnlySpan->proto converter is internal, but not needed):

  • archie-agent: implement otlptrace.Client {Start, Stop, UploadTraces(ctx, []*tracepb.ResourceSpans)}; register sdktrace.WithBatcher(otlptrace.NewUnstarted(natsClient), sdktrace.WithMaxExportBatchSize(n)) with n sized under the NATS max payload. UploadTraces marshals coltracepb.ExportTraceServiceRequest{ResourceSpans: rs} (go.opentelemetry.io/proto/otlp v1.11.0) and publishes via JetStream, returning after the ack so the batch is durable.
  • archied: consume, unmarshal, hand ResourceSpans to otlptracegrpc.NewClient(opts...).UploadTraces. No converter on either side.
  • Shutdown the container provider with a bounded ctx before the agent exits (stderr is lost on AutoRemove).
    Containers get no exporter config, no egress rule, no collector reachability.

Acceptance criteria

One task shows as a single trace from event intake through dispatch, the container's agent loop and the result, including spans produced inside a Kit container, with no new egress rule.

Notes

Part A landed (74b11b33): NATS header propagation, JetStream consumer links, ctx-drop fixes, taskrun.Request.Trace injected/extracted. Remaining: agent otlptrace NATS exporter, archied relay to otlptracegrpc, agent provider+shutdown (after fzb3x.3). Subscribe handlers get no ctx, so only the consumer span exists there.

Close reason

Part A 74b11b33, part B 812f1bdd: container spans exported over ARCHIE_SPANS and relayed by archied to OTLP


Imported from beads archie-core-fzb3x.4 · GitHub #1852 · type feature · priority P2 · created 2026-10-25 · closed 2026-10-25

Traces stop at every NATS hop and at the container boundary. natsrpc.Request/Respond send bare payloads with no header; eventbus.Message has no Header(). Context is dropped on request paths: forgerpc (7 forge-client calls on context.Background), worktreerpc handlerContext, taskactions Apply, webui api_config, staterpc task_grants; events.Bus.Publish takes no ctx. task.json and taskrun.Request carry no trace field, containers inherit no daemon env, and Kit containers can reach only the egress relay. ### Design No NATS instrumentation exists in nats.go v1.54.0 or otel contrib; hand-roll it. Carrier (nats.Header is case-sensitive; propagation.HeaderCarrier canonicalises to "Traceparent" and misses lowercase senders; nats.Header has no Keys()): type natsCarrier nats.Header func (c natsCarrier) Get(k string) string { return nats.Header(c).Get(k) } func (c natsCarrier) Set(k, v string) { nats.Header(c).Set(k, v) } func (c natsCarrier) Keys() []string { return slices.Collect(maps.Keys(c)) } Set panics on a nil map: build messages with nats.NewMsg. API switches: natsrpc Call uses nc.RequestMsgWithContext (RequestWithContext sends no headers); Respond uses msg.RespondMsg; logrpc and agenttransport bypasses likewise. JetStream: js.PublishMsg with headers, eventbus.Message gains Header() backed by jetstream.Msg.Headers(). Request/reply: CLIENT/SERVER span pair. JetStream consume: SpanKindConsumer started with a link to the producer span context, not as its child (producer span has usually ended). messaging.* attrs (semconv/v1.43.0) are Development stability. Request-path ctx drops to fix: forgerpc (7 forge-client calls), worktreerpc handlerContext, taskactions Apply, webui api_config, staterpc task_grants; events.Bus.Publish takes ctx. Container trace context: otel-go has no TRACEPARENT env carrier. Add `Trace map[string]string json:"trace,omitempty"` to the task brief (<worktree>/.git/task.json) or taskrun.Request: c := propagation.MapCarrier{}; otel.GetTextMapPropagator().Inject(ctx, c); brief.Trace = c // daemon ctx = otel.GetTextMapPropagator().Extract(ctx, propagation.MapCarrier(brief.Trace)) // archie-agent Sampling follows the parent flag (ParentBased), so the container keeps the daemon decision. Container export over NATS (compile-checked against v1.47.0; the ReadOnlySpan->proto converter is internal, but not needed): - archie-agent: implement otlptrace.Client {Start, Stop, UploadTraces(ctx, []*tracepb.ResourceSpans)}; register sdktrace.WithBatcher(otlptrace.NewUnstarted(natsClient), sdktrace.WithMaxExportBatchSize(n)) with n sized under the NATS max payload. UploadTraces marshals coltracepb.ExportTraceServiceRequest{ResourceSpans: rs} (go.opentelemetry.io/proto/otlp v1.11.0) and publishes via JetStream, returning after the ack so the batch is durable. - archied: consume, unmarshal, hand ResourceSpans to otlptracegrpc.NewClient(opts...).UploadTraces. No converter on either side. - Shutdown the container provider with a bounded ctx before the agent exits (stderr is lost on AutoRemove). Containers get no exporter config, no egress rule, no collector reachability. ### Acceptance criteria One task shows as a single trace from event intake through dispatch, the container's agent loop and the result, including spans produced inside a Kit container, with no new egress rule. ### Notes Part A landed (74b11b33): NATS header propagation, JetStream consumer links, ctx-drop fixes, taskrun.Request.Trace injected/extracted. Remaining: agent otlptrace NATS exporter, archied relay to otlptracegrpc, agent provider+shutdown (after fzb3x.3). Subscribe handlers get no ctx, so only the consumer span exists there. ### Close reason Part A 74b11b33, part B 812f1bdd: container spans exported over ARCHIE_SPANS and relayed by archied to OTLP --- _Imported from beads `archie-core-fzb3x.4` · GitHub #1852 · type feature · priority P2 · created 2026-10-25 · closed 2026-10-25_
sam added the mediumfeature labels 2026-10-27 13:35:35 +00:00
sam self-assigned this 2026-10-27 13:35:35 +00:00
sam closed this issue 2026-10-27 13:35:35 +00:00
sam added a new dependency 2026-10-27 13:41:50 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#256