Backend Deployment — Migrations & Rollout Flags

:::info Status Implemented. Source: DEPLOY.md. :::

Database migrations

Schema changes are Alembic revisions in fastapi-backend/migrations/versions/, run by the one-shot migrate compose service — never at application startup. This is deliberate: a single Uvicorn process today would race migrations the moment it's given --workers N, and running migrations separately makes a failed migration stop the deploy rather than boot the API against a half-migrated schema.

Full schema ownership model: Database.

One-time setup on a pre-Alembic database

Azan360's schema predates Alembic (it came from the Supabase console and the idempotent ensure_* functions in core/db.py). Each existing environment is stamped at the baseline once, before the first deploy that includes migrate:

docker compose run --rm migrate alembic stamp 0001_baseline

Skipping this isn't destructive — every statement in the first revisions is conditional — but the stamp stops Alembic replaying revisions describing tables the database already has.

Writing a migration

cd fastapi-backend
alembic revision -m "short description"
alembic upgrade head
alembic current
alembic history

No --autogenerate — there's no ORM to diff against. Migrations are hand-written with op.execute(...), one statement per call (asyncpg prepares each statement it's given, and a prepared statement can't hold more than one command).

Review anything destructive before applying:

alembic upgrade head --sql   # emit SQL instead of running it

Rollback is by flag, not by migration

downgrade() raises NotImplementedError in these revisions, deliberately. Schema changes stay additive; behavior reverts by environment variable instead — see below. A migration that dropped a column on downgrade would destroy data that reverting the code doesn't bring back.

docker-compose.local.yml has no postgres/migrate service — run alembic upgrade head by hand there.

Recipient fan-out rollout (FANOUT_V2)

:::caution Status Partially implemented — staged rollout, currently off in production. :::

Push recipients are resolved by core/device_fanout.py. Governed by two env vars, both read on every call — no restart needed to change either.

Variable	Values	Default
FANOUT_V2	off · shadow · enforce	off
FANOUT_LEGACY_BRIDGE	on · off	on

• off — pre-migration behavior: one push endpoint per user.
• shadow — delivers exactly what off delivers, and additionally resolves the per-install set and records the difference on the broadcast's audit row. Use to gather evidence before flipping.
• enforce — delivers to the per-install set (a member with two phones is reached on both).

An unrecognised value logs a warning and is treated as off — a typo must not silently change who receives azaan.

Gate check before flipping to enforce:

SELECT count(*) FILTER (WHERE fanout_lost_users > 0)     AS losing_someone,
       count(*) FILTER (WHERE fanout_lost_users IS NOT NULL) AS measured,
       sum(fanout_bridged_endpoints)                     AS bridged_endpoints
FROM broadcast_audit_logs
WHERE started_at > now() - interval '7 days';

Flip only when losing_someone is 0 and measured covers real traffic across every active mosque. Rollback is FANOUT_V2=shadow (or off) — no restart, no migration, no data restore; rehearsed in fastapi-backend/test/test_rollback_rehearsal.py.

What rollback does not undo: a push token retired while enforce was on stays retired — it was retired because the provider reported it dead, which remains true regardless of recipient-set mode.

While FANOUT_LEGACY_BRIDGE=on, a member with an approved membership but no usable per-install token is resolved from the legacy users.*_token columns.

Stream gate rollout (STREAM_AUTHZ_ENFORCE)

:::caution Status Partially implemented — staged rollout, currently off in production. Do not flip yet (see below). :::

Listener authorization is core/playback_authorization.py, enforced at two points: Caddy's forward_auth on /stream/* (layer 1, the route) and Icecast's listener_add hook (layer 2, the mount).

Variable	Values	Default
STREAM_AUTHZ_ENFORCE	off · monitor · enforce	off
STREAM_GATEWAY_BASE_URL	a URL, or unset	unset
STREAM_TOKEN_SECRET	a key, or unset (falls back to JWT_SECRET)	unset

• off — both layers answer yes (pre-gate behavior).
• monitor — resolve, log and audit the denial that would have happened, then allow anyway.
• enforce — deny.

Why it isn't flipped yet

Every devices.device_secret_hash and app_installs.secret_hash on the live database is currently null — no subject can authenticate, so enforce today would deny every listener. Device/app enrollment is what populates those; until it has, the correct value stays off.

The flip is three steps

1. Set STREAM_GATEWAY_BASE_URL (e.g. https://azan360.com/stream). Until set, stream_url_public() addresses Icecast directly and Caddy can only authorize traffic on /stream/* — wiring forward_auth without this protects a path no listener uses.
2. STREAM_AUTHZ_ENFORCE=monitor, then read device_audit_log for stream_denied reasons (rate-limited to one row per subject/reason/minute):

   SELECT reason, count(*), max(created_at)
   FROM device_audit_log
   WHERE action = 'stream_denied' AND created_at > now() - interval '7 days'
   GROUP BY reason ORDER BY 2 DESC;

   no_membership / membership_not_approved are expected background noise; anything else needs an explanation first.
3. STREAM_AUTHZ_ENFORCE=enforce.

Rollback: STREAM_AUTHZ_ENFORCE=off (no restart, no migration); unsetting STREAM_GATEWAY_BASE_URL also moves listeners back off the gated URL.

Two operational surprises at enforce

• Icecast decides on a header, not the status code — icecast-auth-user: 1 admits the listener regardless of HTTP status. If layer 2 refuses everyone, check that header first.
• listener_add fires for /status-json.xsl — the container healthcheck's own poll. /stream-auth/icecast admits any mount that isn't {mosque_id}_{room}, which stops enforce from marking the Icecast container unhealthy.

Icecast has no published host port

docker-compose.yml deliberately publishes none — Icecast has no authentication of its own, so Caddy is the only thing authorizing a listener. docker-compose.local.yml publishes it for development only; never in production.