Skip to content

Runbook: MAID Backup Timer

Audience: operators installing, verifying, or troubleshooting the scheduled backup pipeline on a MAID host.

Failure runbook: backup_failed.md covers the on-call playbook for a maid-backup-failure@<inst> alert. This document covers the install/verify/maintenance lifecycle of the timer pair itself.

⚠️ Packaging gap (M1.4 pending) — the scheduled pipeline is not yet functional. The units below invoke @@MAID_HOME@@/bin/backup.sh (and the restore drill invokes @@MAID_HOME@@/bin/restore.sh), but scripts/install.sh does not install those scripts — it only creates an empty ${MAID_HOME}/bin/. Populating bin//lib/ is owned by the not-yet-implemented M1.4 release tarball, so on a freshly installed host the timers and the restore drill fail every run (ExecStart file not found → the OnFailure alert fires). Take backups manually from a repo checkout — deploy/scripts/backup.sh self-locates its lib/ — as shown under Manual backup (interim) below. Do not rely on the timers until packaging is fixed.

Components

A complete MAID host stages six systemd unit files for the backup pipeline (all copied in by scripts/install.sh, but non-functional until the M1.4 gap above is closed):

Unit Purpose
maid-backup@<inst>.service Daily oneshot: backup.sh full --instance <inst>
maid-backup@<inst>.timer OnCalendar=daily, RandomizedDelaySec=30m
maid-backup-prune@<inst>.service Weekly retention prune
maid-backup-prune@<inst>.timer OnCalendar=Sun *-*-* 04:30:00
maid-backup-failure@<inst>.service OnFailure hook for daily backup + restore drill
maid-backup-prune-failed@<inst>.service OnFailure hook for retention prune (R1 HIGH 1)

The transport is selected by MAID_DEPLOY_BACKUP_REMOTE in the per-instance env file (/etc/maid/<inst>.env). The bash pipeline (deploy/scripts/backup.sh, the one these units run) supports exactly two URL schemes: local_path:// and ssh://. A separate Python backup package (maid_engine.backup) has its own MAID_BACKUP_TRANSPORT=local|ssh|s3|webhook discriminator; it is not driven by deploy/scripts/backup.sh and does not add s3/webhook to the bash pipeline. See docs/deploy/ssh-bootstrap.md for the SSH side.

Install

  1. Install the unit files (the installer at scripts/install.sh does this automatically; for manual installs):
sudo cp packaging/systemd/maid-backup@.service              /etc/systemd/system/
sudo cp packaging/systemd/maid-backup@.timer                /etc/systemd/system/
sudo cp packaging/systemd/maid-backup-prune@.service        /etc/systemd/system/
sudo cp packaging/systemd/maid-backup-prune@.timer          /etc/systemd/system/
sudo cp packaging/systemd/maid-backup-failure@.service      /etc/systemd/system/
sudo cp packaging/systemd/maid-backup-prune-failed@.service /etc/systemd/system/

Copy all six files: maid-backup-prune@.service declares OnFailure=maid-backup-prune-failed@%i.service, so omitting the prune-failed handler leaves that reference dangling (systemd logs Failed to enqueue OnFailure job and no prune-failure metric/alert is written).

The @@MAID_HOME@@, @@MAID_USER@@, @@MAID_GROUP@@, and @@ETC_DIR@@ substitution markers must be replaced before the units are loaded (the installer uses sed -i).

These units call @@MAID_HOME@@/bin/backup.sh / @@MAID_HOME@@/bin/restore.sh, which are not installed yet (M1.4 gap, see the warning at the top). Copying the units is therefore not sufficient to make the schedule work — until M1.4 stages bin/, use Manual backup (interim) below.

  1. Create the per-instance env file at /etc/maid/<inst>.env, mode 0640 owned by root:maid-engine:
# /etc/maid/prod.env  — example
MAID_DEPLOY_BACKUP_REMOTE=ssh://maid@backup.example.com/srv/maid-backups
MAID_DEPLOY_BACKUP_KEEP=14
MAID_DEPLOY_AUDIT_FILE=/var/log/maid/ops-audit.jsonl

# SSH transport (Python pipeline) — opt-in destructive --delete.
# OFF by default as of R1 BLOCK 3: rsync only ADDS/updates files
# on the remote, never deletes. Flip to "true" only when you want
# local deletions to propagate to the remote mirror (and accept
# that a corrupted local snapshot can wipe the remote on rsync).
# Accepted truthy values: true, 1, yes, on (case-insensitive).
# MAID_BACKUP_SSH_DELETE=false
  1. Reload systemd to pick up the new units:
sudo systemctl daemon-reload
  1. Enable + start the timers (services are NOT enabled directly — the timers fire them on schedule):

⚠️ On a stock host this only schedules a broken pipeline: the units call @@MAID_HOME@@/bin/backup.sh, which is not staged until M1.4 (see the warning at the top), so every fire fails 203/EXEC and trips the OnFailure alert. Enable the timers now if you want the schedule wired for when M1.4 lands, but until then rely on Manual backup (interim) below for real backups.

sudo systemctl enable --now maid-backup@prod.timer
sudo systemctl enable --now maid-backup-prune@prod.timer

Repeat per instance (prod, staging, …).

  1. If using the ssh:// transport, follow docs/deploy/ssh-bootstrap.md to stage the SSH key and accept the remote host key. Do this before the first scheduled run.

Manual backup (interim — until the M1.4 packaging gap is closed)

Because the timers cannot run bin/backup.sh yet (see the top warning), take backups by running the bash driver directly from a MAID source checkout on the engine host. It self-locates its lib/, but reads MAID_DEPLOY_BACKUP_REMOTE from its own process env, so source the instance env with auto-export first:

# From the root of a MAID source checkout on the engine host:
sudo -u maid-engine bash -c \
  'set -a; . /etc/maid/prod.env; set +a; exec ./deploy/scripts/backup.sh full --instance prod'

Capture the printed backup_id, then verify that exact id (see Verify a specific backup). This is the only working backup path until M1.4 stages bin/backup.sh.

Verification

Confirm timers are active

systemctl list-timers --all | grep maid-backup

You should see two lines per instance — one each for maid-backup@<inst>.timer and maid-backup-prune@<inst>.timer — with a NEXT column showing the upcoming firing time and a LEFT column showing the wait.

Force a one-shot run

Works only once bin/backup.sh is staged (M1.4). On a stock install this start fails immediately with 203/EXEC ("no such file"); use Manual backup (interim) above instead.

sudo systemctl start maid-backup@prod.service
sudo journalctl -u maid-backup@prod.service -n 50 --no-pager

The journal should end with Deactivated successfully and a final audit row should appear in /var/log/maid/ops-audit.jsonl (event type OPS_BACKUP_COMPLETE) — that audit row, not the maid_backup_last_success_timestamp_seconds metric, is the signal to check. Under ProtectSystem=strict the unit has no write path to the node_exporter textfile directory, so backup.sh's success-metric write silently no-ops; only the root-owned maid-backup-failure@ hook can emit a metric today, and the positive signal lands with the M9 packaging fix.

List remote backups

The list/verify examples below run deploy/scripts/backup.sh from a source checkout, because the M1.4 release tree does not stage it under /opt/maid/current. export MAID_SRC=/path/to/maid-checkout first; the commands invoke "${MAID_SRC}/deploy/scripts/backup.sh" and forward MAID_SRC through sudo with --preserve-env=MAID_SRC (sudo's env_reset would otherwise drop it before the inner shell expands the path). The arguments are identical to the interim manual backup above.

# backup.sh reads MAID_DEPLOY_BACKUP_REMOTE from its OWN process env; a
# bare sudo does not load EnvironmentFile=, so source the instance env
# with auto-export inside the maid-engine shell (otherwise
# source_transport aborts: "MAID_DEPLOY_BACKUP_REMOTE is not set").
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/prod.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" list --instance prod'

For the local_path:// transport this lists the staging dir; for ssh:// it sshs to the destination and runs ls. Only local_path:// and ssh:// are supported — the bash pipeline has no s3:// transport.

Verify a specific backup

# Backup ids are <UTC-compact>-<instance>-<8 hex>, e.g.
# 20250115T030000Z-prod-a1b2c3d4 — get real ids from the JSON listing
# (`... list --instance prod --json | jq -r '.[]'`). verify re-downloads
# the manifest and every file, then sha256-checks each one, fail-closed.
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/prod.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance prod --backup-id 20250115T030000Z-prod-a1b2c3d4'

Exit code 8 indicates verify failure.

Maintenance

Change retention

Edit MAID_DEPLOY_BACKUP_KEEP in the per-instance env file. The prune service reads it fresh from EnvironmentFile= on each run, so the new value takes effect automatically on the next scheduled timer run (maid-backup-prune@<inst>.timer, Sun *-*-* 04:30). Editing the file is all that is required — there is nothing to reload for the timer to pick it up.

Do NOT systemctl restart maid-backup-prune@<inst>.service to "apply it at the next run". That unit is Type=oneshot, so restart runs the prune immediately and destructively, deleting every backup beyond the newest MAID_DEPLOY_BACKUP_KEEP. Start it by hand only if you intend to prune right now with the new retention:

# Deliberate, immediate destructive prune with the new retention:
sudo systemctl start maid-backup-prune@prod.service

Pause backups (e.g. during a planned outage)

sudo systemctl stop --no-block maid-backup@prod.timer
sudo systemctl stop --no-block maid-backup-prune@prod.timer

To resume:

sudo systemctl start maid-backup@prod.timer maid-backup-prune@prod.timer

(The enable --now from install is idempotent — re-running it re-arms the timers.)

Rotate the SSH transport key

  1. Generate a new key on the engine host: 
    sudo -u maid-engine ssh-keygen -t ed25519 -f /var/lib/maid-engine/.ssh/id_ed25519.new -N ""
  2. Push the new pubkey to the backup destination's ~maid/.ssh/authorized_keys (via the bootstrap-ssh.sh flow on the destination host).
  3. Atomically swap: 
    sudo -u maid-engine mv /var/lib/maid-engine/.ssh/id_ed25519     /var/lib/maid-engine/.ssh/id_ed25519.old
sudo -u maid-engine mv /var/lib/maid-engine/.ssh/id_ed25519.new /var/lib/maid-engine/.ssh/id_ed25519
  4. Force a test run (see Force a one-shot run above).
  5. Once verified, remove the old key from the destination's authorized_keys and /var/lib/maid-engine/.ssh/id_ed25519.old.

Common Failure Modes

Symptom Most likely cause / fix
ConditionPathExists=/etc/maid/<inst>.env not met Env file missing; create it (see step 2) then systemctl daemon-reload.
MAID_DEPLOY_BACKUP_REMOTE is not set; expected … Env file present but variable unset; add the line per step 2.
rsync: connection unexpectedly closed Host key not in known_hosts. Re-run the destination bootstrap in ssh-bootstrap.md §7.
maid-backup-failure@<inst> fires Follow backup_failed.md — that runbook owns the on-call response.
maid-backup-prune-failed@<inst> fires Prune escalation — see Prune failure below. Distinct alert series so it does not page the same as a daily-backup failure.
Timer says n/a in NEXT column Persistent=true couldn't read /var/lib/systemd/timers/; check disk space and that the host has booted at least once after enable.
Prune deletes a backup you needed Raise MAID_DEPLOY_BACKUP_KEEP; if a restore is in flight, snapshot the destination FIRST.

Prune failure

When maid-backup-prune@<inst>.service exits non-zero, systemd invokes maid-backup-prune-failed@<inst>.service via OnFailure= (R1 HIGH 1). That handler writes a dedicated node_exporter metric pair to /var/lib/node_exporter/textfile_collector/maid-backup-prune-<inst>.prom:

maid_backup_prune_last_status{instance="<inst>"} 0
maid_backup_prune_last_failure_timestamp_seconds{instance="<inst>"} <ts>

These are separate series from the daily-backup failure metric (maid_backup_last_status / maid_backup_last_failure_timestamp_seconds) on purpose: a failed daily backup risks data loss; a failed prune risks the backup volume filling up. Different incident classes, different runbooks, different paging policies. Suggested alerting:

maid_backup_prune_last_status == 0 for 1h

The handler also writes an err-priority journal line (logger -t maid-backup-prune-failed) so log shippers pick it up unconditionally. Inspect the actual failure with:

sudo journalctl -u maid-backup-prune@<inst>.service -n 200 --no-pager

Most common cause: the configured backup destination (remote rsync target or MAID_BACKUP_LOCAL_DIR) is full / read-only / unreachable. Fix the destination, then retry with systemctl start maid-backup-prune@<inst>.service.

References

  • Plan: §6 Backup/Restore, R10.A.10 (retention).
  • Failure runbook: backup_failed.md.
  • SSH bootstrap: ../deploy/ssh-bootstrap.md.
  • Bash backup driver: deploy/scripts/backup.sh.
  • Python backup pipeline: packages/maid-engine/src/maid_engine/backup/.