servicekit: tracer provider plus gRPC and HTTP span propagation #255
Notifications
Due Date
No due date set.
Blocks
Reference: sam/archie#255
Reference in new issue
Block a user
No process installs a tracer provider and no RPC edge propagates context. Five gRPC servers (statestore, archiegateway, archied harness, archiemessaging, channelext), six client dial sites with no shared helper, the webui handler chain and several bare HTTP servers.
Design
Modules (direct): go.opentelemetry.io/otel/sdk, otel/exporters/otlp/otlptrace, otlptrace/otlptracegrpc (or otlptracehttp) at v1.47.0; contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.72.0 (otelhttp v0.72.0 already resolved). New indirect only: cenkalti/backoff/v5, grpc-gateway/v2. otelgrpc and otlptracegrpc need grpc >= v1.83.2; v1.86.0-dev satisfies it (compile-checked).
Provider, built in servicekit before any server or client:
res, err := resource.Merge(resource.Default(), resource.NewWithAttributes(semconv.SchemaURL,
semconv.ServiceName(component), semconv.ServiceVersion(version), semconv.ServiceInstanceID(id)))
tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp), sdktrace.WithResource(res),
sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(ratio))))
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
Use semconv/v1.43.0 for the resource: resource.Default() uses that schema and Merge returns ErrSchemaURLConflict on a mismatch; fail startup on it. Disabled = never call SetTracerProvider (global falls back to no-op); OTEL_SDK_DISABLED is not implemented in v1.47.0. Exporters read OTEL_EXPORTER_OTLP_* natively but OTEL_EXPORTER_OTLP_PROTOCOL does not switch packages; [telemetry] passes endpoint, headers, TLS, sampler ratio explicitly.
Lifecycle: tp.Shutdown drains the batch queue then shuts the exporter; give it a deadline ctx. Register it on the servicekit.Logging closer (already first in each LIFO chain, so it flushes last). Diagnostics binaries and archie-agent need the same; archied offline commands skip setupLogging. Unreachable collector never blocks boot: export retries up to 1 min, queue of 2048 drops when full, errors go to otel.Handle.
Footgun: the global propagator defaults to an empty composite. Without SetTextMapPropagator no traceparent is sent by otelgrpc, otelhttp or NATS. Both instrumentations read globals once at construction, so set them first.
gRPC (stats handlers only; v0.72.0 has no interceptors):
grpc.StatsHandler(otelgrpc.NewServerHandler(otelgrpc.WithFilter(filters.HealthCheck()))) // via servicekit.ServerLimits() + channelext
grpc.WithStatsHandler(otelgrpc.NewClientHandler()) // gatewayrpc, staterpc, harnessrpc, daemonrpc, messagingrpc, tasktools
Server span exists before interceptors run, so credential interceptors can log ids. Client traceparent is injected after interceptors, so each must pass the ctx it receives. otelgrpc records no metadata or payloads; WithMessageEvents is a no-op. Streams: span ends at stream end, so a never-closing chat/task stream never exports; use one span per chat turn inside long streams.
HTTP: otelhttp.NewHandler outside requireToken(authorize(mux)) so spans rename to the ServeMux pattern; filter health. Outbound: otelhttp.NewTransport(base) outermost on the ~9 http.Client constructions. url.full keeps the query string (otelhttp internal/semconv/client.go), so move any token out of query strings first. traceparent goes to every host including third parties; filter if unwanted. Metrics are no-ops without a MeterProvider.
Open decision for the maintainer: [telemetry] as a bootstrap key following the [log]/[health] denied-key precedent (recommended) vs a full control-plane kind (~17 files, cf. tool-settings).
Acceptance criteria
With an exporter configured, a dashboard request that calls the state store and gateway produces one trace spanning all three processes; with none configured, nothing is exported and nothing fails.
Notes
Also verify fzb3x.1 end to end once the provider exists: a log call under a live span shows trace_id/span_id in the file log and dashboard viewer for all six binaries, including container logs relayed over NATS.
Close reason
Provider, [telemetry] config, otelgrpc on all servers/dial sites, otelhttp on archie-ui; verified one trace ui -> state store and gateway with a local OTLP receiver
Imported from beads
archie-core-fzb3x.3· GitHub #1854 · type feature · priority P2 · created 2026-10-25 · closed 2026-10-25