Перейти к содержанию

Disaster Recovery

RTO / RPO targets

Компонент RTO RPO
PostgreSQL 30 min 1 min
Redis (hot state) 5 min 0 (ephemeral OK)
realtime-service 5 min
Bot 10 min
MinIO/S3 1 h 0 (versioning)

NFR: NFR-007, NFR-008

Backup strategy

PostgreSQL

  • Continuous WAL archiving
  • Daily full backup
  • Test restore monthly

Redis

  • AOF enabled
  • Game state recoverable from PostgreSQL if Redis lost
  • Presence rebuilt on reconnect

MinIO

  • Bucket versioning
  • Cross-region replication (prod Phase 3)

Failure scenarios

FS-01: PostgreSQL down

  1. API returns 503
  2. Bot: «Сервис временно недоступен»
  3. Failover to replica
  4. RPO: last WAL

FS-02: Redis down

  1. Realtime degraded — poll fallback on web (Could)
  2. Game apply reads/writes PostgreSQL only (slower)
  3. Restart Redis, warm cache from PG

FS-03: NATS down

  1. Sync HTTP notify fallback bot (Phase 2)
  2. WS still via Redis Pub/Sub

FS-04: realtime down

  1. Web: long poll GET /games/{id}/state every 2s
  2. BPMN: bpmn-incident

FS-05: Bot token revoked

  1. Rotate BOT_TOKEN in env
  2. Redeploy bot
  3. Webhook re-register

Runbook contacts

Role Responsibility
On-call Restart services, escalate
DBA PostgreSQL failover
Lead Communication

Game in progress

  • State in PostgreSQL — survives Redis/NATS/realtime loss
  • Players reconnect → GET /games/{id}/state
  • No rollback of completed actions (audit immutable)