Runbook: Backup failed¶

Symptom¶

systemctl status maid-backup@<inst>.service shows failed for the most recent run, or inactive past its scheduled time.
The maid-backup-failure@<inst> OnFailure hook fired: a logger -t maid-backup-failure err-priority journal line, and (if node_exporter's textfile collector is installed) the metric maid_backup_last_status{instance}=0 plus maid_backup_last_failure_timestamp_seconds{instance}.

What is NOT shipped. backup.sh / the failure hook do not send a webhook, email, or Discord post, and there is no backup_failed alert payload or built-in pager. Paging is up to your alerting rule (e.g. maid_backup_last_status == 0) reading the node_exporter metric — MAID only writes the metric + journal line.

Detection¶

maid_backup_last_status == 0 (0 is written on failure by the root-owned maid-backup-failure@ hook — this is the reliable signal).
maid_backup_last_failure_timestamp_seconds recent.
journalctl -u maid-backup@<inst>.service non-empty for the last run.
The OPS_BACKUP_COMPLETE audit row with "result":"failure" and its rc in /var/log/maid/ops-audit.jsonl.
(Once the M9 gap below is fixed) a stale maid_backup_last_success_timestamp_seconds. Until then that gauge is never written on success — do not alert on its staleness alone.

Packaging limitation (M9-pending). The _success timestamp and the _status=1 value are not emitted on success today: the hardened maid-backup@.service (ProtectSystem=strict) has no write path to the node_exporter textfile directory, so backup.sh's success-metric write silently no-ops — only the failure hook (root) writes a metric. Until the M9 fix, treat a missing/stale success gauge as "no signal — check directly": rely on the maid-backup-failure@ alert, the journal, and the OPS_BACKUP_COMPLETE audit rows in /var/log/maid/ops-audit.jsonl for positive confirmation.

Blast radius¶

Players affected: none directly. But the safety net is down. A subsequent incident that needs restore will fail.
Data at risk: any new state created since the last successful backup is at increased risk.
AI/external systems: none.

If the most recent backup is older than the RPO (typically 24 h), declare this an internal-only incident even though players are unaffected. Fix backups before doing any risky operation.

Prerequisites¶

Tools: maid-admin, systemctl, journalctl, jq, psql, ssh, sha256sum.
Access: root, or an equivalently broad sudo grant. This runbook runs sudo systemctl/sudo journalctl on the backup units and sudo -u maid-engine bash -c '…' (arbitrary commands as the service user) — none of which the narrow maid-admin priv-helper allowlist (packaging/sudoers/maid-admin.template) permits. The maid-admin group only covers the fixed maid-admin <verb> wrapper surface; the raw service-user shells here require full sudo/root. Plus an SSH key to the remote backup destination (location documented in ./escalation-contacts.md.template).
Source checkout (MAID_SRC): deploy/scripts/backup.sh is repo-only — the M1.4 release tarball does not stage deploy/scripts/ under /opt/maid/current, so /opt/maid/current/deploy/scripts/… does not exist on an installed host. Check out MAID at the deployed release tag and export MAID_SRC=/path/to/maid-checkout; the commands below invoke "${MAID_SRC}/deploy/scripts/backup.sh".
Env file: /etc/maid/<inst>.env — MAID_BACKUP_LOCAL_DIR, MAID_DEPLOY_BACKUP_REMOTE (retention is set per-run via backup.sh prune --keep N, not an env var).
Paths to know:
/var/lib/maid-backups/<BID>/ — local staging (no per-instance subdir; the instance is embedded in <BID>), pruned after upload
/var/log/maid/ — backup logs (also via journalctl)
Remote: per MAID_DEPLOY_BACKUP_REMOTE
Audit access: tail -f /var/log/maid/ops-audit.jsonl.
Escalation: see ./escalation-contacts.md.template.

First 5 minutes (LITERAL commands)¶

# 1. What broke and when?
sudo systemctl status maid-backup@<inst>.service --no-pager
sudo journalctl -u maid-backup@<inst>.service -n 200 --no-pager \
  > ./incident-backup-$(date -u +%Y%m%dT%H%M%SZ).log

# 2. Why did it fail? backup.sh does NOT emit a "failure class" string —
#    it exits with a stable rc and logs to the journal. Read the journal
#    tail and the rc from the audit row:
grep -E 'ERROR|FATAL|die|pg_basebackup|transport|verify' \
  ./incident-backup-*.log | tail -20
sudo tail -5 /var/log/maid/ops-audit.jsonl | grep OPS_BACKUP_COMPLETE
# The OPS_BACKUP_COMPLETE row carries "result":"failure" and "rc":<n>.
# Exit codes (deploy/scripts/lib/common.sh, stable):
#   2 usage/env (e.g. MAID_DEPLOY_BACKUP_REMOTE unset)
#   3 missing dependency   4 permission / disk (cannot create staging dir)
#   5 lock held            6 PostgreSQL error (pg_basebackup)
#   7 transport error      8 verify failure (sha256 mismatch / missing file)
# The sub-sections below group remediation by SYMPTOM; those names are
# this runbook's own grouping, not strings backup.sh prints.

# 3. What backups do we currently have on hand?
# `backup.sh list --json` prints a JSON array of backup-id strings.
# backup.sh reads MAID_DEPLOY_BACKUP_REMOTE from its OWN process env, so
# source the instance env with auto-export inside the maid-engine shell
# (a bare sudo does not load EnvironmentFile=). MAID_SRC is your source
# checkout (deploy/scripts/ is NOT staged under /opt/maid/current).
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance <inst> --json' \
  | jq -r '.[]'

# 4. Is the disk OK?
df -h /var/lib/maid-backups

Investigation — by symptom¶

These sub-headings group remediation by likely cause. They are not failure-class strings emitted by backup.sh (it has none); map them to the exit code (rc) from the OPS_BACKUP_COMPLETE audit row above: PG=6, transport=7, verify=8, permission/disk=4.

PostgreSQL error (rc=6)¶

Likely cause: Postgres unreachable, hit lock timeout, or schema change mid-dump.

Check:

sudo -u postgres psql -c 'select 1' || echo "PG unreachable -> see ./db_down.md"
sudo journalctl -u postgresql --since "-2 hours" | tail -50

Mitigation: get PG healthy, then re-run (below).

WAL archive error (rc=6, only if you enabled archiving)¶

Likely cause: WAL archive volume full or PG archive_command failing. Only applies if you manually set archive_mode = on — stock MAID ships archive_mode = off (M1.4), and backup.sh streams WAL into the base backup, so a stock install has no WAL archive.

Check:

df -h /var/lib/maid-wal-archive
sudo -u postgres psql -c "select last_failed_wal, last_failed_time from pg_stat_archiver;"

Mitigation: see ./disk_full.md WAL archive section.

Transport error (rc=7)¶

Likely cause: SSH to remote destination failing, network outage, destination disk full, key revoked.

Check:

# Pull connection target from env
source /etc/maid/<inst>.env; echo "$MAID_DEPLOY_BACKUP_REMOTE"

# Try the SSH path backup.sh uses — same key it uses: MAID_DEPLOY_SSH_KEY
# (default /var/lib/maid-engine/.ssh/id_ed25519), as the maid-engine user.
# The remote path is the transport root (backups are <root>/<BID>).
sudo -u maid-engine ssh -i "${MAID_DEPLOY_SSH_KEY:-/var/lib/maid-engine/.ssh/id_ed25519}" \
    -o BatchMode=yes \
    <backup-user>@<remote-host> 'df -h /srv/maid-backups; echo READY'

Mitigation:
If destination disk full → see ./disk_full.md Backup volume section (prune old backups to the newest N to free space; run it env-sourced — sudo -u maid-engine --preserve-env=MAID_SRC bash -c 'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" prune --instance <inst> --keep <N>').
If SSH key revoked → restore from key escrow per ./escalation-contacts.md.template.
If network outage → wait, retry, or rotate to the secondary transport documented in escalation-contacts.

Verify failure (rc=8)¶

Likely cause: sha256 of a fetched file doesn't match MANIFEST.json. This is the scary one — possible corruption in transit or at rest.
Check — run the real verifier and gate on its exit code. It re-downloads every file listed in MANIFEST.json and sha256-checks each one, then exits 0 (all files verified) or 8 (any file missing / mismatched, or the manifest itself failed its sidecar check):
```
# Env-sourced (backup.sh reads MAID_DEPLOY_BACKUP_REMOTE from its own
# process env); MAID_SRC points at a source checkout (deploy/scripts/ is
# NOT staged under /opt/maid/current — see backup.md "Manual backup").
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance <inst> --backup-id <BID>'
echo "verify rc=$?"   # 0 = every listed file verified; 8 = corrupt/missing
```
Do not substitute a local sha256sum -c MANIFEST.json.sha256: that only validates the manifest file's own checksum, not the payload files it lists — a corrupt base/base.tar.gz under an intact manifest passes it. Gate on the verify exit code (0/8), not on any printed text.
If verify exits 8 → the backup is corrupt or incomplete; the previous good backup is your latest restorable point. Do NOT trust this backup.

Mitigation: quarantine the bad backup only after proving an older backup verifies clean (gate on verify exit 0), and quarantine it by moving the specific <BID> aside by hand — never with prune:

# 1. Confirm an OLDER candidate verifies clean (exit 0). Never trust text.
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance <inst> --json' \
  | jq -r '.[]' | sort -r
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance <inst> --backup-id <OLDER_BID>'
# Proceed only when that verify exits 0.

# 2. Quarantine the corrupt <BID> by MOVING it OUT of the transport root.
#    Do NOT use `backup.sh prune`: prune applies GFS tiered retention
#    (keeps the newest N as a MINIMUM, drops the rest by age), so it
#    CANNOT target a specific backup-id — it may delete good older backups
#    or leave the corrupt recent one in place. `backup.sh delete` is not
#    yet implemented (M1.3), so relocate the directory manually so it no
#    longer appears in `list`/restore selection:
#      local_path:// transport (on this host):
#        sudo -u maid-engine mkdir -p /var/lib/maid-backups-quarantine
#        sudo -u maid-engine mv "<transport-root>/<BID>" \
#             "/var/lib/maid-backups-quarantine/<BID>"
#      ssh:// transport (on the REMOTE host, outside the backup root):
#        mv "<remote-root>/<BID>" "<remote-root>/../quarantine/<BID>"

Permission / disk full (rc=4)¶

See ./disk_full.md Backup volume section.

Authentication failure (rc=6 PG role, or rc=7 transport key)¶

Likely cause: PG role password changed, or remote-host key revoked.
Check /etc/maid/<inst>.env matches the current PG maid_backup_<inst> role password; if mismatched, fix env and systemctl daemon-reload.

Retention prune failed¶

Likely cause: the (separate) maid-backup-prune@ step failed (remote returned an error). The backup itself usually succeeded; check the log to confirm. Re-run prune:

sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" prune --instance <inst> --keep <N>'

Mitigation — re-run manually¶

# Re-run the backup unit immediately.
# NOTE: on a stock host this fails 203/EXEC until M1.4 stages
# bin/backup.sh; until then run "${MAID_SRC}/deploy/scripts/backup.sh" full
# from a source checkout instead (see backup.md "Manual backup (interim)").
sudo systemctl start maid-backup@<inst>.service

# Watch it
sudo journalctl -u maid-backup@<inst>.service -f
# Ctrl-C when you see a success line, or a die/ERROR line + non-zero exit.

# Confirm exit code
sudo systemctl show maid-backup@<inst>.service --property=ExecMainStatus

If the manual re-run also fails, capture the new log and follow the matching symptom section above.

Recovery¶

sudo systemctl status maid-backup@<inst>.service shows inactive (dead) since ... result 'success'.
The run's OPS_BACKUP_COMPLETE audit row appears in /var/log/maid/ops-audit.jsonl. Use that row — not maid_backup_last_success_timestamp_seconds — as the confirmation: under ProtectSystem=strict the backup unit has no write path to the node_exporter textfile directory, so backup.sh's success-metric write silently no-ops until the M9 packaging fix (only the root-owned maid-backup-failure@ hook can emit a metric today).
Verification passes — backup.sh verify exits 0 (gate on the exit code, not on printed text; rc=8 means a listed file is corrupt/missing). Run it env-sourced (a bare sudo aborts rc=2 because backup.sh reads MAID_DEPLOY_BACKUP_REMOTE from its own process env):

sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance <inst> --backup-id <BID>'
echo "verify rc=$?"   # 0 = verified; 8 = corrupt/missing

- The remote copy is present — the new id appears in the env-sourced listing:

sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance <inst> --json' | jq -r '.[]'

- No player comms required.

Post-incident¶

File ticket with: exit code (rc) + root cause, time since last good backup, time to recover, was a restore drill scheduled this period.
If a verify failure (rc=8): file a high-priority follow-up to root-cause the corruption and to add an end-to-end restore drill to the next maintenance window.
Update this runbook if a new failure mode showed up.

Escalation¶

Solo path: identify the exit code → fix root cause → re-run manually.
Hosting console URL: see ./escalation-contacts.md.template.
DNS registrar URL: see ./escalation-contacts.md.template.
Off-host backup destination: see ./escalation-contacts.md.template.
Comms channel URL: see ./escalation-contacts.md.template.
Peer operator: see ./escalation-contacts.md.template.
If you cannot produce a verified backup within 24 h, post an internal-only notice in the comms channel and do NOT perform any destructive ops (restore drill, schema migration, mass purge) until backups are healthy again.

Alternate alerting path (no Prometheus)¶

Operators without a Prometheus / Alertmanager stack get the same coverage via the maid-doctor timer + alert-script pair shipped in PR-C. See deploy/monitoring/doctor-alert.sh and packaging/systemd/maid-doctor@.timer.

Enable per instance:

sudo systemctl enable --now maid-doctor@<inst>.timer

On every 5-minute tick the timer runs scripts/maid-doctor.sh --phase runtime --json, pipes the result into doctor-alert.sh, which then:

Writes /var/lib/node_exporter/textfile_collector/maid_doctor.prom (metrics maid_doctor_last_run, maid_doctor_last_status, maid_doctor_failed_checks_total) so any future scraper picks up the most recent doctor verdict without re-running it.
On warn or fail, emails the on-call address read from /etc/maid-engine/<inst>/oncall.env (ONCALL_EMAIL).

This is the default monitoring path for solo home-lab installs; Prometheus is the optional add-on for shops that want graphs/history.