Disaster Recovery
RTO / RPO targets
| Компонент | RTO | RPO |
|---|---|---|
| PostgreSQL | 30 min | 1 min |
| Redis (hot state) | 5 min | 0 (ephemeral OK) |
| realtime-service | 5 min | — |
| Bot | 10 min | — |
| MinIO/S3 | 1 h | 0 (versioning) |
NFR: NFR-007, NFR-008
Backup strategy
PostgreSQL
- Continuous WAL archiving
- Daily full backup
- Test restore monthly
Redis
- AOF enabled
- Game state recoverable from PostgreSQL if Redis lost
- Presence rebuilt on reconnect
MinIO
- Bucket versioning
- Cross-region replication (prod Phase 3)
Failure scenarios
FS-01: PostgreSQL down
- API returns 503
- Bot: «Сервис временно недоступен»
- Failover to replica
- RPO: last WAL
FS-02: Redis down
- Realtime degraded — poll fallback on web (Could)
- Game apply reads/writes PostgreSQL only (slower)
- Restart Redis, warm cache from PG
FS-03: NATS down
- Sync HTTP notify fallback bot (Phase 2)
- WS still via Redis Pub/Sub
FS-04: realtime down
- Web: long poll
GET /games/{id}/stateevery 2s - BPMN: bpmn-incident
FS-05: Bot token revoked
- Rotate
BOT_TOKENin env - Redeploy bot
- Webhook re-register
Runbook contacts
| Role | Responsibility |
|---|---|
| On-call | Restart services, escalate |
| DBA | PostgreSQL failover |
| Lead | Communication |
Game in progress
- State in PostgreSQL — survives Redis/NATS/realtime loss
- Players reconnect →
GET /games/{id}/state - No rollback of completed actions (audit immutable)