Runbook: Postgres down / unreachable from engine¶

Prerequisites¶

Tools: maid-admin, psql, systemctl, journalctl, jq.
Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units and sudo -u postgres … shells, none of which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) grants (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root). psql -U maid_app_<inst> peer auth works (per packaging/postgres/).
Env file: /etc/maid/<inst>.env readable.
Paths to know:
/var/lib/pgsql/<version>/data/ or /var/lib/postgresql/<version>/main/ (distro-dependent)
/var/log/postgresql/ or journalctl -u postgresql
/var/lib/maid-engine/<inst>/
Audit access: tail -f /var/log/maid/ops-audit.jsonl.
Offline status: maid-admin status --instance <inst> (reports offline when the engine UDS is down).
Escalation: see ./escalation-contacts.md.template.

Symptoms¶

Summary¶

/readyz returns 503; status page goes red.
Engine journal full of asyncpg.exceptions.ConnectionDoesNotExistError, asyncpg.exceptions.CannotConnectNowError, or OSError: [Errno 111] Connection refused to 127.0.0.1:5432.
maid-admin status --instance <inst> prints db: unreachable or hangs.
New player connections are refused with service unavailable.

Detection¶

maid_db_connection_errors_total rate > 0 for > 60 s.
/readyz probe failure from any external uptime monitor.
No built-in engine webhook fires a named db_unreachable event (there is no MAID_WEBHOOK_* family; the observability webhook bridge MAID_OBSERVABILITY_WEBHOOK_* only POSTs periodic metric snapshots). Wire an external Prometheus/Alertmanager rule on maid_db_connection_errors_total or /readyz to page, and notify players manually with maid ops announce.
Doctor: maid-admin doctor --phase runtime --instance <inst> reports db_reachable: fail.

Blast radius¶

Players affected: ALL on this instance. Existing sessions still rendering but every command that writes (move, get, drop, combat resolution) is queued; the engine will stop accepting new sessions within ~30 s.
Data at risk: nothing already persisted. In-flight writes are in the EntityPersistenceManager dirty queue; they survive engine restart only if Postgres comes back before the engine exits.
AI/external systems: AI dialogue queue keeps draining (Redis is separate) but cannot persist memory; see ./redis_down.md for comparison.

Diagnostic Steps¶

First 5 minutes (LITERAL commands)¶

# 1. Notify connected players (functional) so they see a banner, not a stack
#    trace. `maid ops maintenance on` is an M9 stub — it does NOT gate logins;
#    see ../deployment/player_comms.md.
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"investigating DB"}'

# 2. Capture state (do BEFORE poking anything)
sudo journalctl -u maid-engine@<inst>.service -n 200 \
  > ./incident-engine-$(date -u +%Y%m%dT%H%M%SZ).log
sudo journalctl -u postgresql --since "-15 min" \
  > ./incident-pg-$(date -u +%Y%m%dT%H%M%SZ).log

# 3. Is Postgres even running?
sudo systemctl status postgresql --no-pager

# 4. Can we reach it?
sudo -u maid_app_<inst> psql -d maid_<inst> -c 'select 1' \
  || echo "psql failed exit=$?"

# 5. Disk OK on the PG data partition? (rule out disk_full first)
df -h /var/lib/pgsql /var/lib/postgresql 2>/dev/null | tail -n +1

Investigation¶

Decision tree:

systemctl status postgresql
├── active (running)?
│     └── yes → connectivity issue (auth, port, exhausted connections)
│         - check max_connections vs pg_stat_activity
│         - check pg_hba.conf and listen_addresses
│         - check `ss -tlnp | grep 5432`
│     └── no  → service is down; goto next branch
├── failed?
│     └── journalctl -u postgresql --since "-30 min"
│         - "out of memory" → see ./oom_loop.md (host OOM, not engine OOM)
│         - "could not write to file ... No space left" → see ./disk_full.md (DB volume section)
│         - "could not extend file" → disk_full
│         - "WAL replay" / "redo starts" / "recovering" → WAL replay slow; see RB18 host-loss DR
│         - "PANIC: corrupt" → STOP. Do NOT restart. See ./restore.md.
└── inactive (stopped)?
      └── someone or something stopped it; check audit and `last`

Useful queries once PG is back up:

# Active connection count vs limit
sudo -u postgres psql -c "
  select setting::int as max_conn from pg_settings where name='max_connections';
  select count(*) from pg_stat_activity;
"

# Long-running queries (>30s)
sudo -u postgres psql -c "
  select pid, now()-query_start as age, state, query
  from pg_stat_activity
  where now()-query_start > interval '30 seconds'
  order by age desc;
"

# Replication / WAL state
sudo -u postgres psql -c "select pg_is_in_recovery(), pg_current_wal_lsn();"

Resolution Steps¶

Mitigation¶

PG crashed cleanly (exited, no PANIC):

sudo systemctl restart postgresql
sleep 5
sudo systemctl status postgresql --no-pager

PG OOM-killed (dmesg | grep -i oom):
Reduce engine memory pressure first: see ./oom_loop.md.
Lower shared_buffers temporarily in postgresql.conf if the host is chronically tight, but file a follow-up to right-size the box.
PG out of disk: see ./disk_full.md DB volume section.
WAL replay too slow on restart:
This is expected after an unclean shutdown of a large DB. Let it finish; do NOT kill postgres during recovery.
If recovery exceeds the SLO (typically > 30 min for the size of MAID workload), declare extended maintenance and prepare for restore from backup per ./RB18_host_loss_dr.md.
PG PANIC / corruption: do NOT restart. Go directly to ./restore.md.
Connections exhausted (active = max_connections):
```
# Kick the engine; it will release pooled connections on exit.
sudo systemctl restart maid-engine@<inst>.service
```
Then investigate why pool was full (leak in a system, runaway script).

Recovery¶

sudo systemctl status postgresql reports active (running).
sudo -u maid_app_<inst> psql -d maid_<inst> -c 'select 1' returns 1.

Leader-lease entry refreshed within LEASE_TTL_S (30 s):

sudo -u maid_app_<inst> psql -d maid_<inst> -c "
  select singleton, holder_pid, generation,
         leased_at, lease_expires_at,
         lease_expires_at - now() as ttl_remaining
  from leader_lock;"

lease_expires_at must be in the future (i.e. ttl_remaining > 0). The engine renews ~ every LEASE_TTL_S/2 seconds; if ttl_remaining is consistently under 5 s after recovery the engine is starved or stuck — escalate. To confirm the holder PID is this host's engine, match it against pg_stat_activity:

sudo -u maid_app_<inst> psql -d maid_<inst> -c "
  select sa.pid, sa.application_name, sa.client_addr, sa.state
  from pg_stat_activity sa
  join leader_lock ll on ll.holder_pid = sa.pid
  where sa.application_name LIKE 'maid-engine:%';"

An empty result means the holder PID is on another host (or stale — see ./two_instances_detected.md).

maid-admin doctor --phase runtime --instance <inst> all green.
Post the all-clear to players via the admin broadcast (maid ops maintenance off is an M9 stub — no login gate to lift).
Players notified per ../deployment/player_comms.md.

Post-incident¶

File ticket with: PG version, distro, crash class (OOM / disk / PANIC / network), pg_log excerpt, time to detect, time to recover.
If WAL replay was the long pole, file a follow-up to tune checkpoint_timeout / max_wal_size.
If connections exhausted, file follow-up to add pool-saturation metric.

Escalation¶

Solo path: get PG back up; engine auto-recovers via reconnect loop.
Hosting console URL: see ./escalation-contacts.md.template.
DNS registrar URL: see ./escalation-contacts.md.template.
Comms channel URL: see ./escalation-contacts.md.template.
Peer operator: see ./escalation-contacts.md.template.

If you cannot recover within 30 minutes:

# `maid ops maintenance` is an M9 stub. Notify players via the functional
# in-game broadcast:
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"DB outage, ETA <ETA>"}'

and follow ./RB18_host_loss_dr.md if the host itself is suspected (disk failing, RAM ECC errors, etc.).

Alternate alerting path (no Prometheus)¶

Operators without a Prometheus / Alertmanager stack get the same coverage via the maid-doctor timer + alert-script pair shipped in PR-C. See deploy/monitoring/doctor-alert.sh and packaging/systemd/maid-doctor@.timer.

Enable per instance:

sudo systemctl enable --now maid-doctor@<inst>.timer

On every 5-minute tick the timer runs scripts/maid-doctor.sh --phase runtime --json, pipes the result into doctor-alert.sh, which then:

Writes /var/lib/node_exporter/textfile_collector/maid_doctor.prom (metrics maid_doctor_last_run, maid_doctor_last_status, maid_doctor_failed_checks_total) so any future scraper picks up the most recent doctor verdict without re-running it.
On warn or fail, emails the on-call address read from /etc/maid-engine/<inst>/oncall.env (ONCALL_EMAIL).

This is the default monitoring path for solo home-lab installs; Prometheus is the optional add-on for shops that want graphs/history.