Runbook: Disk full¶

Three sections — pick the one matching the alert/symptom:

DB volume full — Postgres refuses writes.
WAL archive volume full — WAL ship target out of space.
Backup volume full — local backup staging or remote target out of space.

Symptom (any section)¶

Engine refuses writes; players see "the world resists your change" errors.
Postgres journal: ERROR: could not extend file ... No space left on device.
disk_monitor (M1.2-B) fires maid_disk_free_bytes alert.
Backup job exits non-zero with disk-full class error (see ./backup_failed.md).

Detection¶

maid_disk_free_bytes{mount="<mount>"} < threshold for > 30 s.
df -h on the host shows ≥ 95 % used.
No built-in engine webhook fires a named disk_full event. The observability webhook bridge (MAID_OBSERVABILITY_WEBHOOK_*) only POSTs periodic metric snapshots; wire an external Prometheus/Alertmanager rule on maid_disk_free_bytes to page, and notify players manually with maid ops announce.

Blast radius¶

Players affected: ALL on this instance if DB or engine state volume.
Data at risk: writes since last successful checkpoint until disk is freed.
AI/external systems: AI dialogue persistence pauses; conversations continue in RAM but memory consolidation halts.

Prerequisites¶

Tools: maid-admin, df, du, find, pg_archivecleanup, systemctl, journalctl.
Access: root, or an equivalently broad sudo grant — this runbook runs sudo systemctl/sudo journalctl on the instance units, sudo -u maid-engine …, and sudo -u postgres … shells, none of which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) grants (it only covers fixed maid-admin <verb> calls). There is no maid-ops group in the packaging — only the maid-admin group plus the maid-engine service user (the restore/cleanup units run as root). Plus postgres peer auth for pg_archivecleanup.
Source checkout (MAID_SRC): deploy/scripts/backup.sh is repo-only and is not staged under /opt/maid/current until M1.4, so export MAID_SRC=/path/to/maid-checkout before running the backup steps below; they invoke "${MAID_SRC}/deploy/scripts/backup.sh" and forward MAID_SRC through sudo with --preserve-env=MAID_SRC.
Env file: /etc/maid/<inst>.env.
Paths to know:
DB data: /var/lib/pgsql/<ver>/data/ or /var/lib/postgresql/<ver>/main/
WAL inside data dir: <data>/pg_wal/
WAL archive: off by default. packaging/postgres/postgresql.tuning.conf.template ships archive_mode = off (WAL archiving is deferred to M1.4). The archive_command//var/lib/maid-wal-archive/<inst>/ path below only exists if you have manually enabled archive_mode = on plus a working archive_command (see packaging/postgres/README.md).
Backup staging: /var/lib/maid-backups/<BID>/ (no per-instance subdir; the instance is embedded in <BID>), pruned after upload.
Disk-monitor reservation file: /var/lib/maid-engine/<inst>/.disk_reserve
Audit access: tail -f /var/log/maid/ops-audit.jsonl.
Escalation: see ./escalation-contacts.md.template.

First 5 minutes (LITERAL commands — ANY section)¶

# 1. Confirm which volume is full
df -h

# 2. Notify connected players (functional) so they see a banner, not write
#    errors. `maid ops maintenance on` is an M9 stub — it does NOT gate
#    logins; see ../deployment/player_comms.md.
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"disk pressure, investigating"}'

# 3. Capture state
sudo journalctl -u maid-engine@<inst>.service -n 200 \
  > ./incident-engine-$(date -u +%Y%m%dT%H%M%SZ).log
sudo journalctl -u postgresql --since "-15 min" \
  > ./incident-pg-$(date -u +%Y%m%dT%H%M%SZ).log

# 4. Find the biggest offenders on the full mount
sudo du -xh --max-depth=2 /var/lib 2>/dev/null | sort -h | tail -30

Then pick the matching section.

Section 1: DB volume (Postgres data dir)¶

Investigation¶

# Where is PG actually writing?
sudo -u postgres psql -c "show data_directory;"
sudo -u postgres psql -c "show archive_command;"

# How big is pg_wal/?
sudo du -sh /var/lib/pgsql/*/data/pg_wal/ /var/lib/postgresql/*/main/pg_wal/ 2>/dev/null

# Any replication slots holding WAL?
sudo -u postgres psql -c "
  select slot_name, active, restart_lsn,
         pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as retained
  from pg_replication_slots;"

# Last successful archive
sudo -u postgres psql -c "
  select last_archived_wal, last_archived_time, last_failed_wal, last_failed_time
  from pg_stat_archiver;"

Mitigation¶

If disk_monitor reserved space, release it first (it created an intentional reservation so the engine could fail closed before the disk hit 0 bytes; freeing it gives you ~256 MB of headroom to do recovery):
```
sudo ls -la /var/lib/maid-engine/<inst>/.disk_reserve
sudo rm /var/lib/maid-engine/<inst>/.disk_reserve
df -h
```
If pg_wal/ is the culprit:
Confirm archiver is working (last_archived_wal is recent). If not, fix archive_command first (often the WAL archive volume is itself full — see Section 2).

Once archiver caught up, prune already-archived WAL:

# find the OLDEST WAL still required (= the last archived one)
LAST_ARCHIVED=$(sudo -u postgres psql -tAc \
  "select last_archived_wal from pg_stat_archiver")
echo "last archived: $LAST_ARCHIVED"

sudo -u postgres pg_archivecleanup -d \
  /var/lib/pgsql/*/data/pg_wal "$LAST_ARCHIVED"
# ...or for Debian:
# sudo -u postgres pg_archivecleanup -d /var/lib/postgresql/*/main/pg_wal "$LAST_ARCHIVED"

Never delete pg_wal/* files by hand. Always go through pg_archivecleanup.
If an abandoned replication slot is retaining WAL:
```
sudo -u postgres psql -c "select pg_drop_replication_slot('<slot_name>');"
```
Only drop slots you are sure are abandoned (no peer reading from them).
If PG already crashed because of disk full:
Free space first (above).
Then restart: sudo systemctl restart postgresql.
WAL replay may take a while; do NOT kill postgres during recovery.

Recovery¶

df -h shows < 80 % on the DB mount.
sudo -u postgres psql -c 'select 1' works.
maid-admin doctor --phase runtime --instance <inst> all green.
Post the all-clear to players via the admin broadcast (maid ops maintenance off is an M9 stub — no login gate to lift).

Section 2: WAL archive volume¶

Applies only if you enabled WAL archiving. Stock MAID ships archive_mode = off (deferred to M1.4), and backup.sh takes pg_basebackup --wal-method=stream full backups that bundle the WAL needed for their own consistency — so a stock install has no WAL archive volume and this section does not apply. It is relevant only if you have manually set archive_mode = on plus a working archive_command (see packaging/postgres/README.md).

Symptom (specific)¶

pg_stat_archiver.last_failed_wal is recent.
Postgres journal: archive command failed with exit code 1.
Disk-monitor alert on the WAL archive mount.
Backups may also be failing — see ./backup_failed.md.

Investigation¶

df -h /var/lib/maid-wal-archive
sudo ls -la /var/lib/maid-wal-archive/<inst>/ | head -20
sudo ls -la /var/lib/maid-wal-archive/<inst>/ | wc -l

# What LSN does the last full backup start from? The manifest records
# base_lsn (and wal_method), NOT a "min WAL" filename — full backups
# stream their own WAL, so the archive is only needed for PITR beyond a
# base. Backups all share the root; filter by the instance in the <BID>.
ls -1 /var/lib/maid-backups/ | grep -- "-<inst>-" | sort | tail -3

Mitigation¶

Prune WAL older than the oldest restore point:

# MANIFEST.json records base_lsn (e.g. "3/AB000028"), not a WAL
# filename. Convert the oldest retained backup's base_lsn to the WAL
# segment that contains it, then clean up everything older.
BASE_LSN=$(sudo jq -r '.base_lsn' \
  /var/lib/maid-backups/<oldest-retained-backup>/MANIFEST.json)
echo "oldest base_lsn needed: $BASE_LSN"
OLDEST_NEEDED=$(sudo -u postgres psql -tAX \
  -c "SELECT pg_walfile_name('${BASE_LSN}')")
echo "oldest WAL needed: $OLDEST_NEEDED"

sudo -u postgres pg_archivecleanup -d \
  /var/lib/maid-wal-archive/<inst> "$OLDEST_NEEDED"

Rotate to a new archive path (temporary, until you can grow the volume):
```
# Edit postgresql.conf archive_command to point at a new directory
sudo systemctl reload postgresql
```
File a follow-up to consolidate / move the historical archive.
Push the archive off-host if it isn't already; see the off-host backup transport in ./backup_failed.md.

Recovery¶

pg_stat_archiver.last_failed_wal is no longer growing.
df -h /var/lib/maid-wal-archive shows < 80 %.

Section 3: Backup volume¶

Symptom (specific)¶

backup.sh exits non-zero — typically rc=4 (cannot create staging dir) when the backup volume is full. backup.sh does not emit a disk_full failure class; diagnose from the journal and the rc in the OPS_BACKUP_COMPLETE audit row.
Disk-monitor alert on /var/lib/maid-backups/ mount.
The maid-backup-failure@ OnFailure hook fires (journal err line + maid_backup_last_status{instance}=0). Note the _success gauge is not emitted on success today (M9 packaging gap), so treat its absence as "no signal", not as failure.

Investigation¶

df -h /var/lib/maid-backups
sudo ls -la /var/lib/maid-backups/ | head
sudo du -xh --max-depth=1 /var/lib/maid-backups/ | sort -h | tail

Mitigation¶

Prune old local backups (keep most recent N per retention policy):

# List backup IDs oldest first (filter to this instance via the <BID>)
sudo ls -1t /var/lib/maid-backups/ | grep -- "-<inst>-" | tail -n +<KEEP+1>

# Delete one at a time, verifying remote copies exist FIRST.
# backup.sh reads MAID_DEPLOY_BACKUP_REMOTE from its OWN process env, so
# source the instance env with auto-export inside the maid-engine shell.
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance <inst> --json' \
  | jq -r '.[]'

# For each ID present remotely but oldest locally:
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" prune --instance <inst> --keep <KEEP>'

Use backup.sh prune rather than rm -rf; the script refuses to drop a backup whose remote copy hasn't been verified.

Rotate destination to a new mount (e.g. attached storage) and update /etc/maid/<inst>.env MAID_BACKUP_LOCAL_DIR.

Re-run backup manually after freeing space:

# NOTE: on a stock host this fails 203/EXEC until M1.4 stages
# bin/backup.sh; until then run ./deploy/scripts/backup.sh full from a
# checkout instead (see backup.md "Manual backup (interim)").
sudo systemctl start maid-backup@<inst>.service
sudo journalctl -u maid-backup@<inst>.service -n 100 --no-pager

Recovery¶

df -h /var/lib/maid-backups < 80 %.
The next backup's OPS_BACKUP_COMPLETE audit row appears in /var/log/maid/ops-audit.jsonl. (The maid_backup_last_success_timestamp_seconds gauge does not advance on a packaged host — the hardened unit cannot write the node_exporter textfile dir; M9 packaging fix pending.)
See ./backup_failed.md for full verification.

Post-incident (all sections)¶

File ticket with: which mount, what filled it, time to detect, time to recover.
Right-size the volume if this is recurring.
Verify disk-monitor thresholds in /etc/maid/<inst>.env (MAID_DISK_RESERVE_MB, MAID_DISK_ALERT_PCT) are tuned for your actual disk size — 5 % free on a 4 TB array is 200 GB; 5 % on 50 GB is 2.5 GB. Tune alert thresholds in absolute MB, not percent.
Update this runbook if any of the commands above did not work.

Escalation¶

Solo path: free space → restore service → file follow-up.
Hosting console URL: see ./escalation-contacts.md.template.
DNS registrar URL: see ./escalation-contacts.md.template.
Comms channel URL: see ./escalation-contacts.md.template.
Peer operator: see ./escalation-contacts.md.template.

If you cannot recover within 30 minutes:

# `maid ops maintenance` is an M9 stub. Notify players via the functional
# in-game broadcast:
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"disk full, ETA <ETA>"}'

Alternate alerting path (no Prometheus)¶

Operators without a Prometheus / Alertmanager stack get the same coverage via the maid-doctor timer + alert-script pair shipped in PR-C. See deploy/monitoring/doctor-alert.sh and packaging/systemd/maid-doctor@.timer.

Enable per instance:

sudo systemctl enable --now maid-doctor@<inst>.timer

On every 5-minute tick the timer runs scripts/maid-doctor.sh --phase runtime --json, pipes the result into doctor-alert.sh, which then:

Writes /var/lib/node_exporter/textfile_collector/maid_doctor.prom (metrics maid_doctor_last_run, maid_doctor_last_status, maid_doctor_failed_checks_total) so any future scraper picks up the most recent doctor verdict without re-running it.
On warn or fail, emails the on-call address read from /etc/maid-engine/<inst>/oncall.env (ONCALL_EMAIL).

This is the default monitoring path for solo home-lab installs; Prometheus is the optional add-on for shops that want graphs/history.