Device Deployment — EMQX Broker

:::info Status Implemented. Source: DEPLOY.md, emqx/emqx.conf.template. :::

Azan360 runs a self-hosted EMQX broker, built from emqx/. This page covers deploying and maintaining the broker itself; the wire protocol devices speak against it is documented in MQTT Protocol, and the authorization model in Device Authorization.

Why self-hosted

See ADR: Self-hosted EMQX broker — per-device credentials from day one, no per-plan connection ceiling, at the cost that broker operations now belong to this team.

Authentication is Postgres, not the broker

Devices and the backend authenticate against Postgres, through two views (migrations 0013/0014) — emqx_device_authn, emqx_device_authz. There is no credential store inside the broker: the hash EMQX checks is devices.device_secret_hash, the same column the device registry clears on revoke, so revoking is a database write with nothing else to keep in sync.

One-time setup

CREATE ROLE emqx_auth LOGIN PASSWORD '<a strong password>';
GRANT USAGE ON SCHEMA public TO emqx_auth;
GRANT SELECT ON emqx_device_authn, emqx_device_authz TO emqx_auth;

Deliberately the whole grant — the broker's DB credential can read the hashes it must verify and nothing else (not users, not memberships, not the rest of devices):

SELECT has_table_privilege('emqx_auth','devices','SELECT')      AS should_be_false,
       has_table_privilege('emqx_auth','users','SELECT')        AS should_be_false,
       has_table_privilege('emqx_auth','emqx_device_authn','SELECT') AS should_be_true;

The backend gets its own broker password (bcrypt — the authn view is a single UNION ALL and can't mix algorithms):

from core.security import hash_password
# UPDATE mqtt_service_accounts SET password_hash = '<hash_password("...")>'
#  WHERE username = 'azan360-backend';

Set the same plaintext as MQTT_PASS in fastapi-backend/.env. Root .env needs EMQX_PG_PASSWORD, EMQX_DASHBOARD_PASSWORD, EMQX_NODE_COOKIE — the container refuses to start without the last two rather than falling back to public image defaults.

TLS

Device TLS is on port 8883, behind EMQX_TLS_ENABLE, reading ./emqx/certs/{cert.pem,key.pem} — off by default; enabling it without certificates present stops the broker from starting (fail closed on a security setting). Dev certs: scripts/emqx_dev_certs.sh (makes a private CA + server cert — firmware pins the CA, not the leaf, so routine renewal doesn't require re-flashing the fleet). emqx/certs/ is gitignored.

Renewing a certificate without an outage

Two steps, and only one is obvious:

# 1. replace the files (same paths)
cp new-cert.pem emqx/certs/cert.pem
cp new-key.pem  emqx/certs/key.pem

# 2. tell Erlang to forget the parsed copies
podman exec <emqx> emqx eval 'ssl:clear_pem_cache().'

Measured against EMQX 5.8.6:

Method	Live sessions	New certificate served
emqx eval 'ssl:clear_pem_cache().'	survive	yes
emqx ctl conf load (same paths)	survive	no
emqx ctl listeners restart ssl:default	dropped	no

Restarting the listener — the instinctive move — is both disruptive and ineffective: Erlang's ssl application caches parsed PEM files independent of the file on disk. Rehearse with scripts/emqx_tls_renewal.py (this table is version-specific) rather than trusting it blindly; re-run after any EMQX upgrade.

Verify what's actually being served:

openssl s_client -connect <host>:8883 -CAfile emqx/certs/ca.pem </dev/null 2>/dev/null \
  | openssl x509 -noout -serial -enddate -subject

Listener rate limits

Listener	Port	For	Messages	Bytes	Connections
tcp:default	1883	devices	200/s	512KB/s	10,000 @ 200/s
tcp:internal	1885	the backend	20,000/s	50MB/s	64

The split is about rate limits, not security — both listeners apply the same Postgres ACL. flapping_detect bans a client for 5 minutes after 10 connects in a minute; devices use jittered exponential backoff so a healthy unit recovering from an outage never trips it. These are starting values — capacity is a measured property re-established at each fleet milestone, per the transmitter roadmap.

The dashboard

Binds to the container's interfaces but is never published in production — the compose network is the boundary. It binds 0.0.0.0 (not loopback) only because the same listener also serves /api/v5/stats, scraped by the backend from a different container. Never set EMQX_DASHBOARD_BIND or publish the port in production — the dashboard authenticates with one shared password and can end any device's session. Reach it via an SSH tunnel if genuinely needed.

Upgrading the broker

A single node means a planned outage (see what an outage costs), not a rolling one.

1. Read release notes for config breakage — every non-default setting in emqx.conf.template (ACL cache off, deny_action=disconnect, both listeners' rate limits) is deliberate; a silently renamed key reverts to a permissive default.
2. Back up mnesia (see Operations — backup).
3. Bump the tag in emqx/Dockerfile, rebuild, docker compose up -d emqx.
4. Re-run scripts/emqx_acceptance.py (19 checks) — the only thing that catches an authz source that silently failed to start; a broker authenticating everyone looks healthy from every other angle. Point it at a throwaway broker, never production — it clears and restores a device secret.
5. Re-run scripts/emqx_tls_renewal.py if TLS is on (version-specific behavior).
6. POST /api/v1/super-admin/republish-retained, then confirm devices reconnect: compare /health/detail's broker.mqtt_connections against the registry's ACTIVE count.

Step 4 is the one people skip — it's the difference between finding a broken ACL now and finding it when a member's box plays another mosque's azaan.

Revoking a device

Order matters: publish deauthorize, kick the session, then clear device_secret_hash — reversed, the unit never receives the command telling it to wipe itself.

podman exec <emqx> emqx ctl clients kick <client_id>

UPDATE devices SET device_secret_hash = NULL WHERE device_uid = '<uid>';

authorization.cache is off in emqx.conf.template, so the database write alone revokes on the unit's next message even without the kick — if that cache is ever turned back on, the kick becomes required, not belt-and-braces. Full lifecycle: Device Lifecycle.