Skip to main content

ADR-0012: Self-Hosted Observability Backend (Tempo + Prometheus + Grafana)

Status​

Accepted

Context​

Observability needed a destination for traces and metrics exported via OTLP. The otel-collector sits between fastapi-backend and any destination specifically so the destination choice is a config change, not a code change — but a choice still had to be made before dashboards could exist at all.

Decision​

Self-hosted Grafana + Tempo (traces) + Prometheus (metrics), not a hosted vendor — same reasoning that already justifies self-hosting EMQX over a managed broker (ADR-0002): keep telemetry inside this infrastructure rather than send it to a third party.

Prometheus was chosen over Mimir for the metrics backend specifically, despite Mimir being the first name raised. Mimir's single-process mode still carries its full horizontal-scale architecture (distributor, ingester, compactor, store-gateway all in one binary) and realistically wants 512MB-1GB+ RAM regardless of actual ingestion volume — a cost this deployment's traffic doesn't earn back. Prometheus covers the same need at roughly 128-256MB.

Alternatives Considered​

  • Hosted vendor (e.g. Grafana Cloud) — genuinely the initial recommendation: zero new containers, native OTLP ingestion, no ops burden. Rejected in favor of keeping data in-house, consistent with the EMQX precedent.
  • Mimir instead of Prometheus — considered and explicitly deferred, not rejected outright. Migrating later is a backend swap behind the same Prometheus remote-write protocol the collector already speaks, not a rearchitecture — worth revisiting only if this platform's metrics volume or multi-tenancy needs genuinely outgrow a single Prometheus instance.
  • Jaeger (traces only, no metrics) and Datadog/New Relic (per-GB pricing) — ruled out early: the first can't cover the metrics half of this requirement at all, the second works directly against the cost-consciousness already driving the 10% default sampling ratio.

Consequences​

  • Three more containers to operate (tempo, prometheus, grafana) on a server DEPLOY.md itself describes as small — real, deliberately accepted operational cost, not overlooked. Confirmed workable against the actual server spec (4 vCPU, 8GB RAM): summed deploy.resources.limits across every service in docker-compose.yml comes to ~5.75GB, leaving real but not generous headroom.
  • Grafana is the one new service a person logs into, which makes it the one new place trace/metric data (including mosque_id/room_id/broadcast_id span attributes) becomes human-visible — its own auth stays on (GF_AUTH_ANONYMOUS_ENABLED=false, never relaxed), and the deploy config leaves room for an IP allowlist at the edge if password auth alone isn't enough.
  • No multi-tenancy or horizontal-scale headroom from Prometheus alone if metrics volume grows well beyond a single instance — accepted as a deferral with a known, low-friction upgrade path, not a permanent ceiling.