Skip to main content

Observability (Traces & Metrics)

:::info Status Implemented and verified against a live local stack. Source: fastapi-backend/core/otel.py, docker-compose.yml, otel-collector/config.yaml, tempo/tempo.yaml, prometheus/prometheus.yml, grafana/. :::

OpenTelemetry-based tracing and metrics for fastapi-backend, exported via OTLP to a self-hosted Tempo + Prometheus + Grafana stack. This page describes the implemented result; the full design record and revision history — including a real bug found and fixed during verification (see below) — lives at knowledge/outputs/observability-otel-architecture-20260829.md, outside this site's own generated docs.

Architecture​

fastapi-backend never talks to Tempo or Prometheus directly — the collector is the only thing it points at, and the only thing that knows where telemetry ultimately lands. Both Tempo and Prometheus are internal-only in production (no published host port, same posture as icecast/emqx) — Grafana is the one service in this stack a person needs to reach.

Instrumentation​

SourceMechanismModule
HTTP requestsAuto (opentelemetry-instrumentation-fastapi/-asgi)Instrumented at module level in main.py, right after app = FastAPI(...) — see a real bug this caused
DB queriesAuto (opentelemetry-instrumentation-asyncpg) — spans only, no metricscore/otel.py's _instrument()
Outbound HTTP (AlAdhan, Nominatim, EMQX scrape)Auto (opentelemetry-instrumentation-httpx)core/otel.py's _instrument()
Broadcast fan-out (broadcast.started/broadcast.ended)Manual span, opened in the _safe wrappers (not the raw functions — those aren't the actual entry point any caller uses)core/live_broadcast.py
FFmpeg spawn (ffmpeg.spawn)Manual span, mode attribute distinguishes live (pipe) vs. offline-azan (file)icecast_helper.py
MQTT publish (mqtt.publish)Manual span around core/mqtt.py's publish()core/mqtt.py

See Azaan Broadcast for what these three manual spans actually cover in the broadcast delivery path.

Metrics​

All custom metrics are prefixed azan360.* and use only bounded-cardinality labels (phase, outcome, mode, provider) — never a raw mosque_id or device_id, which would multiply cardinality by fleet size for no real gain. Per-mosque/per-device breakdowns belong in trace attributes instead.

MetricTypeLabels
azan360.broadcast.fanout.durationHistogramphase (started/ended), room_id, outcome, origin (started only)
azan360.ffmpeg.spawn.durationHistogrammode (pipe/file), outcome
azan360.mqtt.publish.durationHistogramoutcome
azan360.mqtt.publish.failuresCounter—
azan360.push.deliveryCounterprovider, outcome — additive to Broadcast Audit, not a replacement
azan360.broadcasts.activeObservable gauge— reads icecast_helper.active_broadcasters live, no separate state to keep in sync

HTTP/DB auto-instrumentation adds http.server.duration, http.client.duration, and related size/count metrics under their own OTel semantic-convention names (old semconv — http_target, not http_route, is the real label after Prometheus name-mangling; confirmed against a live instance, not assumed).

Sampling​

A custom sampler (_BroadcastAlwaysOnSampler, core/otel.py) applies two different rules depending on the trace:

  • The broadcast fan-out trace (broadcast.started/broadcast.ended) always samples at 100% — matched by span name, not a global ratio. Broadcast volume is inherently bounded (a handful of prayers/triggers a day per mosque), so full sampling here is cheap and it's exactly the trace this whole effort exists to make visible.
  • Everything else samples at OTEL_TRACES_SAMPLER_ARG (default 0.1, 10%) — a fixed launch value, not a placeholder pending measurement, chosen on the same footing DEPLOY.md already gives EMQX's own rate limits.

Child spans (e.g. mqtt.publish nested under a sampled broadcast.started) don't need their own rule — the sampler's ParentBased fallback already propagates the parent's sampled decision.

Configuration​

VariableDefaultPurpose
OTEL_ENABLEDtrueMaster switch — SDK always initializes. Not the inert-by-default gate (see below)
OTEL_EXPORTER_OTLP_ENDPOINTunsetThis is the real gate. Unset means the SDK is live but has nowhere to send — the intended state everywhere until a collector is deployed there
OTEL_EXPORTER_OTLP_PROTOCOLgrpcgrpc or http/protobuf — the collector's otlp receiver accepts both simultaneously either way
OTEL_TRACES_SAMPLER_ARG0.1Head-sampling ratio for non-broadcast spans, see Sampling
OTEL_SERVICE_NAMEazan360-backendResource attribute
OTEL_RESOURCE_ATTRIBUTESunsete.g. deployment.environment=production

Full reference: Configuration.

Dashboards​

Two Grafana dashboards, both auto-provisioned (grafana/provisioning/dashboards/, no manual setup):

  • Azan360 — Broadcast & Backend Observability — 8 panels against the metrics above: active broadcasts, fan-out duration/error-rate, FFmpeg spawn duration, MQTT publish duration/failures, push delivery by provider, HTTP request duration by target.
  • Azan360 — Traces — 3 Tempo TraceQL table panels (Recent, Error, Slow) for quick trace lookup without opening Explore. Click any Trace ID to open the full span waterfall.

Ad-hoc trace queries beyond the three built-in filters: Grafana Explore → Tempo datasource, TraceQL directly (e.g. {name="broadcast.started"}).

A real bug worth knowing about​

FastAPIInstrumentor.instrument_app() must be called at module level, immediately after app = FastAPI(...) — never from inside the app's lifespan. The first implementation put it in the lifespan (for code-organization reasons, alongside the rest of the OTel setup) and it silently produced zero HTTP server spans and zero http.server.* metrics — not a crash, not a warning, just nothing. Root cause: Starlette builds and caches its middleware stack before a lifespan-time mutation to it takes effect. Reproduced in isolation, fixed, and reconfirmed against the real app before being called done — see knowledge/outputs/observability-otel-architecture-20260829.md's ninth revision for the full isolation/verification trail if this pattern ever needs revisiting elsewhere in the codebase.