Skip to content

Bare-Metal Deployment — 14-Step Procedure

Audience: the operator who has just been handed a fresh Linux VM and a release tarball, and needs MAID running, hardened, and verified in under two hours.

Format: every step has a one-sentence preamble, the exact commands to run, the expected output (or how to test for it), a one-line rollback, and a pointer to the runbook or security-checklist row that owns the failure mode.

Time budget: ~90 minutes if every command works on the first try (Steps 3 and 7 are the longest at ~10 min each).

References:

Prerequisites — set substitution variables (run this first)

What this does: declares the four shell variables that every subsequent step references. Set these once at the top of your session; every fenced block below pastes against them. The validation test (tests/integration/test_deploy_guide_commands.sh) sources this block first so bash -n parses the rest of the guide.

This is a prerequisite, not a numbered deployment step — the 14 numbered steps below all assume these variables are set.

# REQUIRED — set before running any subsequent command.
INST=prod                                  # instance name; ^[a-z][a-z0-9_]{0,30}$
RELEASE=v0.9.0                             # release tag; matches the git tag
OPERATOR="${SUDO_USER:-${USER}}"           # your interactive login (NOT root)
ORG=your-github-org                        # only used in the git-clone URL
HOSTNAME_FQDN=maid-prod-01.example.com     # what DNS resolves to this host
export INST RELEASE OPERATOR ORG HOSTNAME_FQDN

Expected output: none (assignments are silent). Run echo "${INST}" to confirm.

Rollback: unset INST RELEASE OPERATOR ORG HOSTNAME_FQDN.

Reference: install.sh enforces INST regex per R10.A.8 / R11.A.11.

Step 1 — Provision host

What this does: boots a clean Linux VM with the resources MAID needs and a stable DNS record so TLS issuance works.

# Run on your provisioning console (cloud panel / VM tooling) — not the host.
# Required:
#   - Debian 12 (bookworm) OR Ubuntu 22.04 LTS (jammy) — both are tier-1.
#   - 4 vCPU / 8 GB RAM / 100 GB SSD (minimum; doubles for >500 concurrent).
#   - Static IPv4 (the firewall rules in Step 11 assume one).
#   - DNS A record (HOSTNAME_FQDN -> public IP). TLS issuance in Step 12
#     needs this to resolve from off-host.
# Verify from your laptop:
dig +short "${HOSTNAME_FQDN}" || echo "expected: DNS not propagated yet, retry in 5 min"

Expected output: dig returns the host's public IP. If it doesn't, DNS is still propagating; wait 5 minutes and retry before continuing.

Rollback: destroy the VM in your provider console.

Reference: security_checklist.md → "Network exposure".

Step 2 — Set hostname

What this does: gives the host a stable, FQDN-resolvable name so journal entries, audit logs, and TLS certs all agree on what this box is called.

SHORT_NAME="${HOSTNAME_FQDN%%.*}"
sudo hostnamectl set-hostname "${SHORT_NAME}"
# Add the FQDN to /etc/hosts so `hostname -f` works even without DNS:
printf '127.0.1.1 %s %s\n' "${HOSTNAME_FQDN}" "${SHORT_NAME}" \
    | sudo tee -a /etc/hosts
hostname -f

Expected output: the FQDN (maid-prod-01.example.com), not just the short name.

Rollback: sudo hostnamectl set-hostname <previous-name> and remove the line you appended to /etc/hosts.

Reference: security_checklist.md → "Filesystem and accounts" (the audit log embeds hostname -f).

Step 3 — Apply OS updates

What this does: brings the kernel and userland to current patch levels before MAID is installed; the reboot ensures the new kernel is actually running.

sudo apt update
sudo apt full-upgrade -y
# Reboot is mandatory if the kernel was upgraded; the firewall and
# systemd-hardening directives in later steps assume a current kernel.
sudo reboot
# After the host comes back, log back in and verify:
uptime
uname -r

Expected output: uptime reports < 5 minutes; uname -r reports the upgraded kernel (compare to dpkg -l linux-image-generic or linux-image-amd64).

Rollback: none — you cannot un-patch a kernel. If the host fails to boot, revert from the provider snapshot taken in Step 1.

Reference: runbooks/RB18_host_loss_dr.md (if the reboot does not return).

Step 4 — Install OS dependencies

What this does: installs every package install.sh and the runtime require — Postgres 16, Redis, nginx (for the admin UI), and the operator-tooling base (jq, nftables, stunnel4).

sudo apt update
sudo apt install -y \
    postgresql-16 postgresql-contrib postgresql-client-16 \
    redis-server \
    nginx \
    git curl ca-certificates jq \
    stunnel4 nftables \
    util-linux systemd
# Confirm postgres started cleanly:
sudo systemctl status postgresql@16-main.service --no-pager | head -6 \
    || echo "expected: investigate via 'journalctl -u postgresql@16-main.service'"
redis-cli ping

Expected output: postgresql@16-main.service is active (running), and redis-cli ping returns PONG.

Rollback: sudo apt remove --purge <pkg> && sudo apt autoremove.

Reference: security_checklist.md → "systemd hardening"; runbooks/db_down.md if postgres won't start.

Step 5 — Create users and groups

What this does: creates the unprivileged service account maid-engine (no shell, no home login) and the operator group maid-admin (members can run the priv-helper via sudoers).

# Service user (R10.A.7 — owns /opt/maid, runs the engine).
sudo useradd -r -s /usr/sbin/nologin -d /opt/maid -m maid-engine \
    || echo "expected: user already exists (idempotent, OK)"
# Operator group (R10.A.7 — sudoers allowlist target).
sudo groupadd -f maid-admin
# Add your interactive operator login to the group:
sudo usermod -aG maid-admin "${OPERATOR}"
# Verify:
getent passwd maid-engine
getent group maid-admin

Expected output: maid-engine exists with shell /usr/sbin/nologin and home /opt/maid; maid-admin group lists ${OPERATOR}. The new group does not apply until you log out and back in (or start a new login shell).

Rollback: sudo userdel maid-engine && sudo groupdel maid-admin (safe only if no files are owned by them yet).

Reference: security_checklist.md → "Filesystem and accounts".

Step 6 — Clone the engine source

What this does: pulls the MAID repository at the release tag into the source checkout directory. On an M1.1–M1.3 host this checkout is the only place the application code exists.

⛔ The stock /opt/maid/venv is empty (M1.1–M1.3). install.sh creates the venv with uv venv --no-project (scripts/install.sh:518) and its stage_app_code step is a documented M1.1 placeholder that only creates directories (scripts/install.sh:527) — it does not install maid_engine. The release tarball that would populate the venv and /opt/maid/current/{bin/backup.sh,scripts/,deploy/,packages/} is deferred to M1.4. Until M1.4 lands:

  • /opt/maid/venv/bin/maid does not exist and maid_engine is not importable from /opt/maid/venv;
  • the maid-engine@<inst> systemd unit cannot start — its ExecStartPre/ExecStart run /opt/maid/venv/bin/python -m maid_engine.cli.app against that empty venv (see Step 13); and
  • every engine / backup / restore / doctor command runs from this checkout via MAID_SRC, using an interim venv built with uv sync (Step 10). That ${MAID_SRC}/.venv is deliberately separate from the empty /opt/maid/venv.

On an M1.1–M1.3 host the installer only stages the operator wrapper + priv-helper under /opt/maid/current/{bin,libexec}; the full release tarball is deferred to M1.4 as described above. The commands below reference this checkout via MAID_SRC.

export MAID_SRC=/opt/maid/src
sudo mkdir -p "${MAID_SRC}"
sudo chown maid-engine:maid-engine "${MAID_SRC}"
sudo -u maid-engine git clone "https://github.com/${ORG}/MAID" "${MAID_SRC}" \
    || echo "expected: clone target already exists (idempotent if same remote)"
cd "${MAID_SRC}"
sudo -u maid-engine git fetch --tags
sudo -u maid-engine git checkout "${RELEASE}"
sudo -u maid-engine git rev-parse --short HEAD

Expected output: the short SHA matches the release tag you intended to deploy. Cross-check against the release notes.

Rollback: sudo -u maid-engine git checkout <previous-release>.

Reference: runbooks/rollback.md for the code-rollback procedure once the engine is running.

Step 7 — Run install.sh (bootstrap stage)

What this does: installs Python 3.12 (via uv), creates the venv at /opt/maid/venv empty (interpreter only — no maid_engine; see the warning in Step 6), copies the systemd units, installs the priv-helper at /usr/local/sbin/maid-priv-helper, drops the sudoers allowlist at /etc/sudoers.d/maid-admin, runs all preflight gates, but stops short of starting the engine (--no-start).

cd "${MAID_SRC}"
# --no-start so we can configure secrets (Step 9) before the engine boots.
# --manage-system-postgresql is OMITTED here because we installed
# postgresql-16 directly via apt in Step 4 (Debian/Ubuntu native package).
sudo bash scripts/install.sh "${INST}" --no-start
echo "install.sh exit: $?  (0 = success, 4 = outcome unknown, 64 = usage,"
echo "                   71 = missing tool, 75 = contention, 78 = config,"
echo "                   124 = timeout)"

Expected output: exit code 0; the script's log ends with STAGE pg: done (it stopped before start per --no-start). The venv exists at /opt/maid/venv/ (interpreter only — it holds no maid_engine until M1.4; see Step 6); the env file exists at /etc/maid/${INST}.env mode 0640 root:maid-engine.

Rollback: sudo bash scripts/install.sh "${INST}" --rollback --yes (undoes the per-instance install in reverse order). Verify with sudo bash scripts/install.sh "${INST}" --doctor after rollback.

Reference: security_checklist.md → "Filesystem and accounts"; runbooks/RB22_python_uv.md if uv bootstrap fails.

Step 8 — Run doctor (post-install sanity)

What this does: runs the install-phase doctor to confirm every file permission, role, sudoers entry, and systemd unit is exactly as expected; this is the gate before secrets and migrations.

# /opt/maid/current/scripts/ is NOT staged until M1.4 — run the doctor from
# the source checkout (MAID_SRC, exported in Step 6).
sudo "${MAID_SRC}/scripts/maid-doctor.sh" --phase install --instance "${INST}"
echo "doctor exit: $?  (0 = clean, 1 = warnings, 2 = failures)"

Expected output: exit code 0 (clean) or 1 (warnings — review, do not ignore). Exit 2 means a hard-fail; do not proceed.

Rollback: no state change to roll back; re-run after fixing the flagged item.

Reference: security_checklist.md — the doctor checks the same items.

Step 9 — Configure secrets

What this does: writes the runtime secrets that must NOT be in git (DB password, admin secret key, AI provider API keys). The env file was created with 0640 root:maid-engine in Step 7; you only need to edit values.

# Required keys (referenced by name in security_checklist.md):
#   - PGPASSWORD                  (set by install.sh; rotate per RB secret_rotation)
#   - MAID_ADMIN_SECRET_KEY        (set by install.sh; ≥32-char HMAC secret.
#                                   Signs PLAYER access/refresh JWTs always
#                                   (HS256), and admin JWTs only when
#                                   MAID_ADMIN_ALGORITHM=HS256. Under the
#                                   default RS256, admin JWTs use an
#                                   auto-generated RSA keypair instead, but
#                                   player JWTs still use this secret. CSRF
#                                   tokens are random, not derived from it.
#                                   Engine refuses to start in prod if this
#                                   is unset or <32 chars.)
#   - MAID_AI_ANTHROPIC_API_KEY    (operator-supplied; leave blank to disable AI)
#   - MAID_AI_OPENAI_API_KEY       (operator-supplied; optional)
#   - MAID_DEPLOY_BACKUP_REMOTE    (operator-supplied; off-host backup URI)
# NOTE: backup.sh does NOT encrypt archives, and there is no
# MAID_DEPLOY_BACKUP_KEY. If you need encryption-at-rest, apply it at
# the destination/storage layer (an encrypted filesystem or bucket on
# the backup host) — see security_checklist.md.
sudo -e "/etc/maid/${INST}.env"       # uses sudoedit; preserves 0640 root:maid-engine
# Verify perms did not regress:
stat -c '%a:%U:%G' "/etc/maid/${INST}.env"

Expected output: 640:root:maid-engine. Anything else is a finding — fix before continuing.

Rollback: restore the previous env file from your secrets backup (sudo cp /etc/maid/${INST}.env.bak /etc/maid/${INST}.env && sudo chmod 0640 /etc/maid/${INST}.env && sudo chown root:maid-engine /etc/maid/${INST}.env).

Reference: security_checklist.md → "Secrets and backups"; runbooks/secret_rotation.md for the rotation procedure.

Step 10 — Apply engine schema migrations

What this does: runs the engine's migration framework (maid db migrate) to create and upgrade the application schema in the per-instance database maid_${INST}. This is distinct from the bootstrap SQL migrations in packaging/postgres/migrations/ (leader lock, grants, role hardening, tripswitch, GDPR erasure) that install.sh's pg stage already applied in Step 7. The framework tracks applied migrations per namespace (engine, stdlib, and any enabled content packs) in its own _migration_history table and takes a Postgres advisory lock so concurrent runs are safe.

⛔ Blocked on a stock install — build the interim venv first (M1.1–M1.3). /opt/maid/venv/bin/maid does not exist yet (Step 6). Build a runnable venv inside the source checkout with uv sync and run migrations from it. ${MAID_SRC}/.venv is deliberately separate from the empty /opt/maid/venv and is only needed until M1.4 stages the application.

# Interim runtime (M1.1–M1.3): build ${MAID_SRC}/.venv from the checkout.
# uv was bootstrapped by install.sh at /opt/maid/current/.uv/bin/uv (Step 7).
# --frozen installs exactly what uv.lock pins; --project targets the checkout
# regardless of cwd; -H gives uv a writable cache under the service account
# home. This venv is DISTINCT from the empty /opt/maid/venv.
sudo -u maid-engine -H /opt/maid/current/.uv/bin/uv sync --frozen --project "${MAID_SRC}"
sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" --version   # expect: MAID Engine version <x.y.z>

# The db CLI has no --instance flag; it reads its target from the MAID_DB_*
# variables in the instance env file (the same file the systemd unit loads
# via EnvironmentFile=). Run as the maid-engine service user and source that
# file so the CLI connects to maid_${INST}.
sudo -u maid-engine bash -c 'set -a; . "/etc/maid/'"${INST}"'.env"; set +a; exec "'"${MAID_SRC}"'/.venv/bin/maid" db migrate'
# Verify (expect 0 pending migrations across all namespaces):
sudo -u maid-engine bash -c 'set -a; . "/etc/maid/'"${INST}"'.env"; set +a; exec "'"${MAID_SRC}"'/.venv/bin/maid" db status --show-pending'

After M1.4 stages the application into /opt/maid/venv, drop the checkout venv and run the same db migrate / db status --show-pending commands via /opt/maid/venv/bin/maid.

Expected output: maid db migrate reports each namespace advancing to its head sequence (or that it is already up to date); maid db status --show-pending then reports no pending migrations for any namespace.

Rollback: see runbooks/failed_migration.md. Migrations are forward-only by design; rollback is a restore-from-backup.

Reference: runbooks/failed_migration.md.

Step 11 — Apply firewall

What this does: loads the nftables ruleset that allows :22 (SSH from admin source), :4001 (TLS-fronted telnet, see Step 12), :8080 (admin UI behind nginx), and blocks everything else — including loopback :9090 from being reachable off-host.

# Review the rules before loading. If the file does not exist yet (the
# bundle in deploy/firewall/ is delivered per release), copy the template:
test -r deploy/firewall/nftables.example.conf \
    || echo "expected: review packaging/etc/ for the shipped template"
sudo cp deploy/firewall/nftables.example.conf /etc/nftables.conf
sudo -e /etc/nftables.conf       # whitelist your SSH source CIDR
sudo nft -c -f /etc/nftables.conf  # parse-check WITHOUT loading
sudo nft -f /etc/nftables.conf     # load
sudo systemctl enable --now nftables
sudo nft list ruleset | head -20

Expected output: nft list ruleset shows your rule set; SSH still works from your allow-listed source; ss -tlnp | grep :9090 shows 127.0.0.1:9090 only.

Rollback: sudo nft flush ruleset && sudo systemctl disable --now nftables (returns the host to "no firewall" state; do this BEFORE you lock yourself out).

Reference: runbooks/RB17_firewall_fallback.md.

Step 12 — Configure TLS termination (stunnel for telnet)

What this does: installs the stunnel front-end so MUD clients can reach :4001/TCP (TLS), with cleartext :4000 firewalled to loopback only.

# Provision the cert before this step — letsencrypt via certbot, or
# your internal CA. The cert MUST cover HOSTNAME_FQDN from Step 1.
sudo cp deploy/stunnel/maid-telnet.conf.example /etc/stunnel/maid-telnet.conf
sudo -e /etc/stunnel/maid-telnet.conf       # set cert/key paths, FQDN
# Permissions for the config (per security_checklist):
sudo chmod 0640 /etc/stunnel/maid-telnet.conf
sudo chown root:stunnel4 /etc/stunnel/maid-telnet.conf
sudo systemctl enable --now stunnel-maid-telnet.service \
    || sudo systemctl enable --now stunnel4.service
# Verify the listener is up:
ss -tlnp | grep -E ':4001\b' || echo "expected: stunnel listener on :4001"

Expected output: ss -tlnp shows *:4001 LISTEN stunnel. A TLS probe from another host (run from your laptop, not the host: openssl s_client -connect "${HOSTNAME_FQDN}:4001" -servername "${HOSTNAME_FQDN}" </dev/null 2>&1 | head -3) returns a valid certificate.

Rollback: sudo systemctl disable --now stunnel-maid-telnet.service (disables TLS termination; the plain :4000 remains firewalled).

Reference: security_checklist.md → "Network exposure"; ssl_configuration.md for cert provisioning.

Step 13 — Start the engine

What this does: brings the engine up and waits for the [READY] signal. On an M1.1–M1.3 host the systemd unit cannot serve — you run an interim foreground process from the checkout venv instead; the supported systemd path lands with M1.4.

⛔ Do not systemctl enable --now maid-engine@ on an M1.1–M1.3 install. The unit runs /opt/maid/venv/bin/python -m maid_engine.cli.app — its ExecStartPre=… db migrate-check and ExecStart=… server start (packaging/systemd/maid-engine@.service:120,130). Because the stock venv is empty (Step 6), ExecStartPre fails with ModuleNotFoundError: No module named 'maid_engine', systemd marks the unit failed, and it never serves. Starting it only produces a broken, flapping unit — leave it disabled until M1.4.

Interim runtime (M1.1–M1.3) — run the engine in the foreground from the ${MAID_SRC}/.venv you built in Step 10. This is a non-systemd, bring-up-validation process (no watchdog, no auto-restart); use it to confirm the install serves, then stop it with Ctrl-C. It is distinct from the /opt/maid/venv systemd path.

# Foreground run as the service user; reads MAID_* from the instance env
# file (same file the unit loads). No --instance flag: server start takes
# its instance from MAID_INSTANCE / the sourced env. Ctrl-C to stop.
sudo -u maid-engine -H bash -c 'set -a; . "/etc/maid/'"${INST}"'.env"; export MAID_INSTANCE="'"${INST}"'"; set +a; exec "'"${MAID_SRC}"'/.venv/bin/maid" server start'
# In a second shell, confirm the readiness sentinel appears:
ls -l "/run/maid-engine/${INST}/READY"

After M1.4 stages the application into /opt/maid/venv, the supported systemd path is:

sudo systemctl daemon-reload
sudo systemctl enable --now "maid-engine@${INST}.service"
sudo systemctl enable --now "maid-watchdog@${INST}.service"
# Watch for the [READY] marker (Ctrl-C once you see it):
sudo journalctl -u "maid-engine@${INST}.service" -f --no-pager

Expected output:

  • Interim (M1.1–M1.3): the foreground process prints [READY] instance=${INST} tick_rate=4Hz ... within 60 seconds and /run/maid-engine/${INST}/READY exists.
  • Post-M1.4 (systemd): the journal shows the same [READY] line and systemctl is-active maid-engine@${INST}.service returns active.

Readiness signalling (R1 CAVEAT 1, M2 update): the engine emits two readiness channels: 1. File sentinel — /run/maid-engine/${INST}/READY (the canonical signal; used by maid doctor --runtime and ConditionPathExists=). 2. sd_notify READY=1 — sent on $NOTIFY_SOCKET when systemd is supervising us. As of M2 the shipped unit at packaging/systemd/maid-engine@.service uses Type=notify with NotifyAccess=main. The two channels are emitted together so an interim mix of ConditionPathExists= and Type=notify consumers both work; long-term migration to pure Type=notify is tracked in M1.2-B wave-3.

If you maintain a custom unit override, ensure it carries Type=notify and NotifyAccess=main. WatchdogSec= must NOT be set until the tick-loop watchdog pet lands (separate M2 follow-up); a WatchdogSec= with no notify_watchdog() pet will mark the unit failed after the configured timeout.

Rollback: sudo systemctl disable --now "maid-engine@${INST}.service" "maid-watchdog@${INST}.service" (the DB and state on disk are untouched; safe to re-start once the underlying issue is fixed).

Reference: runbooks/db_down.md if startup hangs at the DB-connect line; runbooks/RB22_python_uv.md for venv failures; runbooks/two_instances_detected.md if the leader lock is held by an old PID.

Step 14 — Verify

What this does: runs the three end-to-end smoke checks (HTTP health, telnet banner, runtime doctor) that prove the install is actually serving players, not just running. On an M1.1–M1.3 host these validate the interim foreground runtime started in Step 13 (post-M1.4 they validate the systemd unit); either way the engine must be running before you run them.

# 1. HTTP /health (admin UI / web back-end on :8080).
#    (The engine's /healthz, /readyz, /livez probes live on the internal
#     observability server at 127.0.0.1:9090 — loopback only, see Step 11.)
curl -sf http://localhost:8080/health \
    || echo "expected: 200 OK when engine is ready"
# 2. Telnet banner (cleartext, loopback; TLS proxy verified separately).
echo quit | timeout 5 telnet localhost 4000 2>&1 | head -5
# 3. Runtime doctor (requires a running engine). /opt/maid/current/scripts/
#    is NOT staged until M1.4 — run it from the source checkout (MAID_SRC).
sudo "${MAID_SRC}/scripts/maid-doctor.sh" --phase runtime --instance "${INST}"
echo "runtime doctor exit: $?  (0 = clean, 1 = warnings, 2 = failures)"

Expected output:

  1. curl prints {"status":"ok"} and exits 0.
  2. telnet shows the MAID greeting banner (e.g. Welcome to MAID...).
  3. maid-doctor.sh --phase runtime exits 0.

If any of the three fail: stop, read the runbook for the symptom, fix, re-run from Step 13.

Rollback: if smoke fails after a deploy, see runbooks/rollback.md for the symlink-swap procedure; for a full uninstall, see the Rollback section below.

Reference: SLO targets; Operator handbook.

Post-install hardening

Before you announce the host to players, walk through these in order. Each links to the authoritative source — do NOT trust this section to be complete (it summarizes; the cited docs are the spec).

  1. Walk the security checklist top to bottom. maid-doctor.sh --phase install covers the machine-verifiable subset; the rest (sudoers review, ex-team-member cleanup, key rotation cadence) is operator judgement.
  2. Print the must-print runbook bundle and put it in the rack. If the host is unreachable you cannot read these in vim.
  3. Wire monitoring. The engine exposes Prometheus metrics on 127.0.0.1:9090/metrics (loopback-only per Step 11). Pull from your metrics aggregator over a private channel or a tunnel; do not expose :9090 publicly. The metrics to alert on are documented in OPERATOR_HANDBOOK.md → "Reading metrics".
  4. Configure the backup destination. MAID_DEPLOY_BACKUP_REMOTE (Step 9) is the off-host URI. The scheduled backup timers are not functional on a stock host — the maid-backup@ unit calls @@MAID_HOME@@/bin/backup.sh, which the installer does not yet stage (M1.4 packaging gap), so every scheduled run fails 203/EXEC and fires the OnFailure alert. Until M1.4 lands, take backups manually from a repo checkout per runbooks/backup.md → "Manual backup (interim)", and verify that run by confirming an OPS_BACKUP_COMPLETE row in /var/log/maid/ops-audit.jsonl (or journalctl -t maid-backup). The maid_backup_last_success_timestamp_seconds gauge is not a reliable signal on a packaged host: the hardened maid-backup@.service (ProtectSystem=strict) has no write path to the node_exporter textfile directory, so backup.sh's success-metric write silently no-ops until the M9 packaging fix (only the root-owned failure hook can currently emit a metric). The destination must be physically OFF this host; see durability matrix and runbooks/backup_failed.md.
  5. Fill the escalation contacts template and save the filled copy to /etc/maid/escalation-contacts.md (mode 0640 root:maid-admin). Do NOT commit it.
  6. Run a restore drill. Per SLO targets, restore cadence is 1× per quarter; the first drill should be within 7 days of the first production deploy. The automated maid-restore-drill@ timer and the maid-admin restore verb are not functional yet (both are M1.4: the drill unit calls an unstaged bin/restore.sh, and the priv-helper restore verb is an M1.3 shim that returns "deferred"). Run the drill manually with the functional deploy/scripts/restore.sh staging from a checkout, as documented in runbooks/restore.md.
  7. Read privacy.md and privacy-policy-template.md. If you collect player data and serve EU residents you have GDPR obligations; the templates are not legal advice but cover the operator-visible surface.

Rollback

Two modes, depending on what you need to undo.

Per-version rollback (revert a bad release)

Use when: a release has been deployed (Step 6+7 ran for a new version), it is misbehaving, and you want /opt/maid/current swapped back to the previous version. The DB, secrets, and state on disk are preserved.

sudo bash scripts/install.sh "${INST}" --rollback-version --yes

This atomically repoints the symlink, restarts the engine and watchdog units, and reverts the symlink swap if the engine fails to confirm ready within 60s. See runbooks/rollback.md for the migration matrix (some releases require DB restore, not just symlink swap).

Full uninstall (decommission an instance)

Use when: you are decommissioning the host or the instance. This removes systemd units, the priv-helper, sudoers entries, and the venv, but DOES NOT delete /var/lib/maid-engine/${INST}/ or the DB. Add --purge to also wipe state (not implemented yet — manual dropdb + rm -rf for now).

sudo bash scripts/install.sh "${INST}" --rollback --yes
# Verify nothing remains:
sudo bash scripts/install.sh "${INST}" --doctor

Expected output: the doctor reports every stage as [SKIP] stage: not_done — confirming the install is gone.

Validation

This document is exercised by tests/integration/test_deploy_guide_commands.sh (in the repo root), which:

  • Greps every fenced ```bash block out of this file.
  • Runs bash -n on each to catch syntax errors before the operator hits them at 3am.
  • Asserts every curl http://localhost... line is paired with an || echo "expected ..." fallback so doc-build CI (offline) does not hang.

Run locally:

bash tests/integration/test_deploy_guide_commands.sh