Monitoring

:::info Status Implemented. Source: DEPLOY.md. :::

Health endpoint

/health/detail (behind X-Monitoring-Token) is where the Super Admin Health page and external monitors look. It folds broker metrics in alongside application health:

curl -s -H "X-Monitoring-Token: $MONITORING_TOKEN" \
     https://azan360.com/health/detail | jq .broker

Requires EMQX_API_KEY / EMQX_API_SECRET on both the emqx and fastapi-backend services — the emqx service mints the key at first start, the backend authenticates with it (core/broker_metrics.py). Unset, the block reports not_configured — not an error, an honestly unmonitored broker.

Broker metrics to watch

Field	Means
mqtt_connections vs. the device registry's ACTIVE count	the gap is the fleet going dark
mqtt_messages_received_rate	far above the modeled ~18/s at 5,000 units suggests a firmware bug before it suggests growth
mqtt_authn_failures / mqtt_authz_denied	a device that can't connect reports nothing itself — this is the only place a broken credential rollout is visible
mqtt_memory_used_pct, mqtt_max_fds	the two ways a single-node broker dies quietly

Rates are derived from two samples: the first call after a backend restart reports null (not 0) — 0 would misleadingly read as "the fleet went silent." A counter that goes backwards (broker restarted) also reports null rather than a fake spike.

The scrape credential is administrator-scoped (per-key RBAC is EMQX Enterprise-only) — what limits blast radius is that the listener carrying it is never published, not the key's own scope.

What a broker outage costs

Scheduled azaan keeps playing from cached device config for as long as the entitlement lease holds; Bluetooth on a stream player is unaffected; app installs never touch the broker at all. What stops: live broadcast play commands, status/heartbeat, and config changes (the last of which queue in the outbox and drain on recovery). Retained state is rebuildable from devices.config via POST /api/v1/super-admin/republish-retained — which is why a single broker node is defensible as a first stage; see Device Deployment.

Traces and metrics

Observability covers fastapi-backend's own request/broadcast/MQTT tracing and metrics — a different surface from the broker metrics above (this section is about the EMQX broker specifically; that page is about the backend's own behavior). Two dashboards, both auto-provisioned in Grafana:

• Azan360 — Broadcast & Backend Observability — broadcast fan-out duration/errors, FFmpeg spawn time, MQTT publish duration/failures, push delivery by provider, HTTP request duration.
• Azan360 — Traces — quick-view trace tables (Recent/Error/Slow), click-through to the full span waterfall.

The broadcast fan-out trace (broadcast.started/broadcast.ended) is always 100% sampled regardless of the general 10% ratio — see Observability — Sampling for why.

Broadcast delivery

Broadcast Audit is the primary delivery-analytics surface — per-broadcast push send/failure counts and MQTT timing, queryable via /api/v1/broadcast-audit/* and visible on the Super Admin Broadcast Audit screen.

Rollout gates

Two staged rollouts expose their own monitoring queries before flipping to enforce — see Backend Deployment for the FANOUT_V2 and STREAM_AUTHZ_ENFORCE gate-check SQL.