Skip to content

On-call & Rollback Runbook

Operational runbook for the Yam3at production stack. Keep it short, current, and rehearsed — a launch criterion (Scope & Roadmap): "Error monitoring, uptime alerting, and on-call rotation in place; rollback procedure tested."

The stack (at a glance)

  • Host: single Hetzner box 89.167.17.109, shared with other projects — never disrupt co-tenants.
  • App root: /opt/yam3at (Docker Compose, docker-compose.prod.yml).
  • Containers: api (php-fpm), nginx, mysql, redis, meilisearch, worker, scheduler.
  • Edge: host nginx reverse-proxies api.yam3at.com → 127.0.0.1:8090 (stack nginx).
  • API base: https://api.yam3at.com/api/v1 · Web: yam3at.com · Docs: docs.yam3at.com.
  • Monitoring: Sentry (SENTRY_LARAVEL_DSN). Backups: nightly DB dump, /etc/cron.d/yam3at-backup → /var/backups/yam3at (14-day retention).

Deploy secrets live only in /opt/yam3at/apps/api/.env (never in git).

Health checks

curl -s https://api.yam3at.com/api/v1/health          # {"status":"ok","checks":{"database":"ok"}}
ssh root@89.167.17.109 'cd /opt/yam3at && docker compose -f docker-compose.prod.yml ps'
A healthy response is 200 with database: ok. If database is not ok, jump to MySQL down.

Deploy (normal)

# from a machine with the repo pushed to GitHub:
git push origin main
ssh root@89.167.17.109 'cd /opt/yam3at && ./deploy/deploy.sh'      # pull, build, migrate, restart
deploy.sh runs git pull --ff-only, rebuilds images, migrate --force, recreates nginx, caches config/routes/views, restarts workers. It is safe to re-run (idempotent).

Note — the local push quirk: pushing from the Windows dev box can hang on a Git Credential Manager popup. If it hangs, either approve the popup, or ship via bundle: git bundle create - main --not <server_HEAD> | ssh root@<host> 'cat > /tmp/x.bundle && cd /opt/yam3at && git pull --ff-only /tmp/x.bundle main' then ./deploy/deploy.sh --no-pull. Always push the same commit to GitHub afterwards so a later git pull deploy doesn't revert it.

Rollback

Deploys are just git + rebuild, so rollback = check out the previous good commit and redeploy.

ssh root@89.167.17.109 'cd /opt/yam3at && git log --oneline -5'      # find the last-good <sha>
ssh root@89.167.17.109 'cd /opt/yam3at && git checkout <sha> && ./deploy/deploy.sh --no-pull'
- --no-pull deploys the checked-out commit without pulling main. - Database: migrations are additive by design; a code rollback usually needs no DB change. If a bad migration must be undone: docker compose -f docker-compose.prod.yml exec api php artisan migrate:rollback --step=1 --force (only if the down() is safe — check first). - After the incident, return to main: git checkout main and redeploy the fixed commit. - Restore DB from backup (last resort): gunzip < /var/backups/yam3at/<file>.sql.gz | docker compose -f docker-compose.prod.yml exec -T mysql sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD" "$MYSQL_DATABASE"'.

Common incidents

Symptom Likely cause First action
/health 502/504 api or nginx container down docker compose … ps; … up -d api nginx
/health 200 but database not ok mysql down/unhealthy … ps mysql; … restart mysql; check disk (df -h)
All requests slow FPM saturation / a slow query check docker stats; FPM pm.max_children=20 (deploy/zz-fpm.conf); look at Sentry perf
5xx spike after a deploy bad release Rollback (above)
Payments not confirming webhook rejected in prod, unsigned webhooks are rejected — verify TAP_WEBHOOK_SECRET is set
Push/deploy won't reach server GCM popup hang use the bundle workaround (above)
Disk full logs/backups df -h; prune old backups; docker system prune (careful on a shared box)

On-call

  • Primary signal: Sentry alerts (errors + performance) → the on-call engineer.
  • Escalation: on-call → tech lead → founder. Target ack ≤ 15 min during launch hours.
  • Comms: post status + ETA in the team channel; if customer-facing, prepare a short bilingual (AR/EN) notice.
  • After every incident: write a 3-line post-mortem (what, why, fix) and file any follow-up as a task.

Pre-deploy checklist (for risky releases)

  • php artisan test green locally
  • Migration down() reviewed (reversible)
  • Note the current server HEAD (the rollback target) before deploying
  • Deploy in a low-traffic window; watch /health + Sentry for 10 minutes after