Troubleshooting

:::info Status Implemented. Sources: DEPLOY.md, docs/azan-flow.md, docs/live-broadcast-flow.md. :::

Deployment

Symptom	Check
Deployment failed	GitHub Actions logs for the run
Containers won't start	docker compose logs --tail=50 on the server
Server disk space	docker system prune -af

Azaan broadcast (live and offline)

Symptom	Likely cause
Auto azan never fires	mosque is_active=false; master/prayer toggle off; no prayer times and no location (status → source:"none"); no recording uploaded
relation "azan_auto_settings" does not exist in logs	migration not run
Fired, but phones silent on answer	Icecast mount not live — check spawn_ffmpeg_file errors, confirm ffmpeg present in the backend image
Recording plays for wrong prayer	check audio_files.prayer value — 'fajr' vs NULL
Auto azan double-fired	both DB dedupe queries failed (audit tables missing?) — check logs around adhan time
Upload 413	nginx client_max_body_size missing (needs 20m on /api/)
Uploaded audio 404s	nginx missing location /admin/uploads/ → backend route
Recordings gone after deploy	uploads_data volume not mounted on /app/public/admin/uploads
Wrong/missing broadcast token	WebSocket closed 4403
Unknown room	WebSocket closed 4404
FFmpeg dies mid-stream	auto-respawns on the next chunk
No audio for BROADCAST_IDLE_TIMEOUT_SECS (default 60s)	broadcast auto-ends with full "ended" fan-out
MQTT broker down during a broadcast	logged + audit mqtt_failed; audio streaming itself is unaffected
Push provider errors	counted in the audit row, invalid tokens cleared; stream unaffected
Server restart mid-broadcast	FFmpeg processes are killed in lifespan shutdown; clients must reconnect

Full flow detail: Azaan Broadcast.

MQTT / devices

Symptom	Likely cause	Where to look
Device connects but every publish is refused	ACL row missing/wrong for that device_uid	SELECT username, action, topic FROM emqx_device_authz WHERE username = '<device_uid>'
A revoked device keeps publishing	authorization.cache turned on (should be off) — see Device Deployment	broker config
Fleet metrics show connections but registry shows fewer ACTIVE	devices connected but not properly claimed/approved	Device Lifecycle
Broker authenticates everyone after an upgrade	authz source failed to start silently	re-run scripts/emqx_acceptance.py
A box plays another mosque's azaan after a broker restore	retained state wasn't rebuilt	POST /api/v1/super-admin/republish-retained

Full protocol reference: MQTT Protocol.

Stream gate / recipient fan-out rollouts

See the gate-check queries and rollback steps in Backend Deployment — both FANOUT_V2 and STREAM_AUTHZ_ENFORCE roll back by environment variable alone, no restart.

General approach

1. Container-level issue → docker compose logs, see Logging.
2. Delivery/broadcast issue → query broadcast_audit_logs, see Broadcast Audit.
3. Device connectivity/authorization issue → /health/detail's broker block (Monitoring) and device_audit_log.
4. Still unclear → check whether a rollout flag (FANOUT_V2, STREAM_AUTHZ_ENFORCE) is in a non-default state.