Skip to content

On-call & Rollback Runbook

Operational runbook for the Yam3at production stack. Keep it short, current, and rehearsed — a launch criterion (Scope & Roadmap): "Error monitoring, uptime alerting, and on-call rotation in place; rollback procedure tested."

The stack (at a glance)

  • Host: single Hetzner box <production-host>, shared with other projects — never disrupt co-tenants.
  • App root: /opt/yam3at (Docker Compose, docker-compose.prod.yml).
  • Containers: api (php-fpm), nginx, mysql, redis, meilisearch, worker, scheduler.
  • Edge: host nginx reverse-proxies api.yam3at.com → 127.0.0.1:8090 (stack nginx).
  • API base: https://api.yam3at.com/api/v1 · Web: yam3at.com · Docs: docs.yam3at.com.
  • Monitoring: Sentry (SENTRY_LARAVEL_DSN). Backups: nightly DB dump, /etc/cron.d/yam3at-backup → /var/backups/yam3at (14-day retention).

Deploy secrets live only in /opt/yam3at/apps/api/.env (never in git).

Health checks

Two endpoints, two different jobs. Uptime monitoring points at the readiness one.

Endpoint Checks Use it for
Readiness GET /api/v1/health MySQL and Redis uptime alerting, load-balancer membership, the deploy's own verification
Liveness GET /up nothing — only that PHP booted container restart policies

/up is Laravel's framework default and stays 200 straight through a total database outage, so it must never be the uptime monitor's target.

curl -i https://api.yam3at.com/api/v1/health
# 200 {"status":"ok","service":"yam3at-api","version":"v1","checks":{"database":"ok","redis":"ok"}}
ssh root@<production-host> 'cd /opt/yam3at && docker compose -f docker-compose.prod.yml ps'

Healthy is 200 with status: ok. Anything else means a dependency is down, and checks names which — database: unavailable or redis: unavailable. Losing Redis takes sessions, the cache and the queue with it, so it is a full outage, not a degradation. Both have a row in Common incidents below.

checks.redis reads not_configured only if this deployment does not put cache, sessions or the queue on Redis — production does all three, so in production it should always read ok or unavailable.

Alert on any non-200, not on the body. If Redis is what died, the API-wide rate limiter (also backed by Redis) can fail the request before the readiness check runs, so the response may be a bare 500 instead of the informative 503. Either way it is not a 200.

Deploy (normal)

# from a machine with the repo pushed to GitHub:
git push origin main
ssh root@<production-host> 'cd /opt/yam3at && ./deploy/deploy.sh'      # pull, build, migrate, restart
deploy.sh runs git pull --ff-only, rebuilds images, migrate --force, recreates nginx, caches config/routes/views, restarts workers, and then polls the readiness endpoint through the container nginx (127.0.0.1:8090) for up to ~60s. If the API never answers 200 {"status":"ok"} the script prints the last response plus the tail of the api logs and exits non-zero — "Deploy complete." is only ever printed by a deploy the app itself confirmed. A failed verification means the new code is already live and broken: go to Rollback.

Note — the local push quirk: pushing from the Windows dev box can hang on a Git Credential Manager popup. If it hangs, either approve the popup, or ship via bundle: git bundle create - main --not <server_HEAD> | ssh root@<host> 'cat > /tmp/x.bundle && cd /opt/yam3at && git pull --ff-only /tmp/x.bundle main' then ./deploy/deploy.sh --no-pull. Always push the same commit to GitHub afterwards so a later git pull deploy doesn't revert it.

Rollback

Deploys are just git + rebuild, so rollback = check out the previous good commit and redeploy.

ssh root@<production-host> 'cd /opt/yam3at && git log --oneline -5'      # find the last-good <sha>
ssh root@<production-host> 'cd /opt/yam3at && git checkout <sha> && ./deploy/deploy.sh --no-pull'
- --no-pull deploys the checked-out commit without pulling main. - Database: migrations are additive by design; a code rollback usually needs no DB change. If a bad migration must be undone: docker compose -f docker-compose.prod.yml exec api php artisan migrate:rollback --step=1 --force (only if the down() is safe — check first). - After the incident, return to main: git checkout main and redeploy the fixed commit. - Restore DB from backup (last resort): gunzip < /var/backups/yam3at/<file>.sql.gz | docker compose -f docker-compose.prod.yml exec -T mysql sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD" "$MYSQL_DATABASE"'.

Common incidents

Symptom Likely cause First action
/api/v1/health 502/504 api or nginx container down docker compose … ps; … up -d api nginx
/api/v1/health 503, database: unavailable mysql down/unhealthy … ps mysql; … restart mysql; check disk (df -h)
/api/v1/health 503, redis: unavailable redis down — sessions, cache and queue all gone … ps redis; … restart redis; … logs --tail=50 redis; df -h
/api/v1/health 500 with no useful body often redis: the rate limiter fails before the check runs treat as redis down (row above)
Deploy aborts with "the API never reported healthy" the release is live but broken read the checks line it printed, then Rollback (above)
All requests slow FPM saturation / a slow query check docker stats; FPM pm.max_children=20 (deploy/zz-fpm.conf); look at Sentry perf
5xx spike after a deploy bad release Rollback (above)
Payments not confirming webhook rejected in prod, unsigned webhooks are rejected — verify TAP_WEBHOOK_SECRET is set
Push/deploy won't reach server GCM popup hang use the bundle workaround (above)
Disk full backups, images, or an app log df -h; prune old backups; docker system prune (careful on a shared box). Container logs are capped (logging: in docker-compose.prod.yml, 3 × 10MB per service); storage/logs/laravel.log inside the volume is not — check it

On-call

  • Primary signal: Sentry alerts (errors + performance) → the on-call engineer.
  • Escalation: on-call → tech lead → founder. Target ack ≤ 15 min during launch hours.
  • Comms: post status + ETA in the team channel; if customer-facing, prepare a short bilingual (AR/EN) notice.
  • After every incident: write a 3-line post-mortem (what, why, fix) and file any follow-up as a task.

Pre-deploy checklist (for risky releases)

  • php artisan test green locally
  • Migration down() reviewed (reversible)
  • Note the current server HEAD (the rollback target) before deploying
  • Deploy in a low-traffic window; watch /api/v1/health + Sentry for 10 minutes after (deploy.sh verifies it once; it does not watch for a slow-burn failure)