Upgrade and Rollback

There are three escalating change classes. Pick the lightest one that satisfies your change. Each section below is a literal runbook — copy/paste the commands as written, substituting <inst> for your systemd instance (e.g. default) and <NEW_VERSION> for the release tag.

Class	Operator action	Player-visible impact
1. Hot reload (content only)	sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" ops hot-reload --templates '<glob>' --instance <inst> (via ops UDS — not systemctl reload)	In-place; sessions kept open
2. Restart (engine code, no schema)	Stage new release (manual until M1.4) + systemctl restart	≤ 60s disconnect; players reconnect automatically
3. Migration (schema change)	Maintenance window + maid db migrate	Planned outage; broadcast required

⚠️ Tooling status — read before copy/pasting. Parts of this runbook depend on automation that is not implemented yet. Verify against your installed version:

• install.sh --upgrade <ver> is not available. The installer has no --upgrade flag and rejects it (unknown option). Automated forward-upgrade (release tarball staging + atomic /opt/maid/current symlink swap) is deferred to M1.4. Until it lands there is no supported in-place upgrade command — stage the new release with your provisioning process (see bare_metal.md). The rollback flags install.sh --rollback-version --yes (swap back to the previous version) and install.sh --rollback (full uninstall) do exist.
• systemctl reload does NOT do an in-place reload. The unit's ExecReload sends SIGHUP, which the engine does not handle — the process terminates and systemd restarts it (Restart=on-failure), the same disconnect as a Class-2 restart. For a true in-place content reload that keeps sessions open, use maid ops hot-reload (Class 1 below).
• maid ops maintenance on|off is an M9 stub — it prints NOT YET IMPLEMENTED (M9) and does not gate logins. There is no working login gate today. To notify players during a window, combine maid ops announce (webhooks) with an in-game admin broadcast (POST /api/v1/admin/broadcast); see player_comms.md.
• maid ops doctor is an M9 stub. For a health check today use systemctl status maid-engine@<inst>.service, curl -sf http://localhost:8080/health, and the repo-only doctor script scripts/maid-doctor.sh --phase runtime (see bare_metal.md → Step 14). The installer does not stage scripts/ under /opt/maid/current (staging is deferred to M1.4), so run the repo scripts (scripts/maid-doctor.sh, scripts/install.sh) from a source checkout. Set this once for the session before running the class runbooks below: export MAID_SRC=/path/to/maid-checkout, then invoke e.g. sudo "${MAID_SRC}/scripts/maid-doctor.sh" --phase runtime --instance <inst>.
• maid ops announce (webhook-only; --channels is metadata, not routing), maid ops hot-reload, systemctl restart/stop/start, maid db migrate, and deploy/scripts/backup.sh are functional.
• The same MAID_SRC checkout also provides the maid CLI (there is no system-wide maid until M1.4, when install.sh will stage the app). Build its venv once — sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}" — and the commands below run it as "${MAID_SRC}/.venv/bin/maid".

1. Hot reload (preferred for content)

Use hot reload for content-only edits that do not change persistence schema, command protocol, or engine internals.

Suitable for:

• NPC dialogue prompt/template tweaks
• Item template adjustments (numeric stats, descriptions)
• Room descriptions and atmosphere text
• Quest text and dialogue scripting

NOT suitable for:

• Engine code (anything under packages/maid-engine/src/)
• Persistence schema changes (new columns, new tables)
• Command protocol changes (new args, removed args)
• Configuration shape changes (new sub-settings)
• Anything that adds or removes a content pack

# In-place reload through the ops UDS (gated by maid-ops group membership).
# Reloads templates/packs/systems/modules in the RUNNING engine without
# dropping sessions. Do NOT use `systemctl reload` — that SIGHUPs the
# engine, which is unhandled and causes a full restart (disconnect).
sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" ops hot-reload --templates 'items/*' --instance <inst>
# ...or reload a whole content pack:
#   sudo -u maid-engine "${MAID_SRC}/.venv/bin/maid" ops hot-reload --pack <pack_name> --instance <inst>

# Verify runtime health (maid ops doctor is an M9 stub; use these instead).
curl -sf http://localhost:8080/health && echo OK
sudo "${MAID_SRC}/scripts/maid-doctor.sh" --phase runtime --instance <inst>

If the reload reports FAILED, or the health check fails, escalate immediately to Class 2 — restart. Do not leave a hot-reload in a half-applied state.

2. Restart (engine code, no schema)

Use restart for engine binary upgrades that do not require a database migration.

# 0. Wire the webhook target into THIS shell's environment (announce reads
# its own process env; it is not the engine's EnvironmentFile).
export MAID_BRIDGES_WEBHOOK_URLS="discord:https://discord.com/api/webhooks/<id>/<token>"

# 1a. Pre-announce to webhooks (≥ 15 minutes). --channels is metadata only.
sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --instance <inst> \
  --channels webhook \
  --severity info --eta "$(date -u -d '+15 minutes' --iso-8601=seconds)" \
  --message "Scheduled restart for v<NEW_VERSION>; expected outage ~60s."

# 1b. Notify CONNECTED players in-game (announce does not reach them).
# maid ops maintenance/broadcast are M9 stubs; use the admin REST API.
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"Restart for v<NEW_VERSION> in ~15 min; auto-reconnect when the server returns."}'

# 2. There is NO working login gate (maid ops maintenance is an M9 stub).
# Proceed directly; the restart itself is the only enforcement.

# 3. Stage the new release.
# PLANNED (M1.4): the automated forward-upgrade command
# `install.sh --upgrade <NEW_VERSION>` (release tarball staging + atomic
# /opt/maid/current symlink swap) is NOT implemented yet — the installer
# rejects `--upgrade`. Until M1.4 lands there is no supported in-place
# upgrade command; stage the new release using your provisioning process
# (see docs/deployment/bare_metal.md) before continuing.

# 4. Restart the service.
sudo systemctl restart maid-engine@<inst>.service

# 5. Verify runtime health (maid ops doctor is an M9 stub).
curl -sf http://localhost:8080/health && echo OK
sudo "${MAID_SRC}/scripts/maid-doctor.sh" --phase runtime --instance <inst>

If verification fails:

1. Do not announce "all clear".
2. Swap the current symlink back to the previous release: sudo bash "${MAID_SRC}/scripts/install.sh" <inst> --rollback-version --yes. (Do not use bare --rollback — that flag is a destructive full uninstall, not a version swap-back.)
3. sudo systemctl restart maid-engine@<inst>.service.
4. Re-run the health checks.
5. Open a postmortem; do not retry the upgrade until the failure mode is understood.

3. Migration (schema change)

Use this path whenever any migration in the release diff touches the database schema. Take a fresh backup first, always. A failed migration is recoverable only if you have a backup taken AFTER the engine stopped writing.

# 0. Wire the webhook target into THIS shell's environment.
export MAID_BRIDGES_WEBHOOK_URLS="discord:https://discord.com/api/webhooks/<id>/<token>"

# 1a. Pre-announce to webhooks (≥ 1 hour for any migration; ≥ 24h if
# irreversible). Valid severities are info|warn|critical (not "warning").
sudo -u maid-engine --preserve-env=MAID_BRIDGES_WEBHOOK_URLS "${MAID_SRC}/.venv/bin/maid" ops announce \
  --instance <inst> \
  --channels webhook \
  --severity warn --eta "$(date -u -d '+1 hour' --iso-8601=seconds)" \
  --message "Scheduled maintenance for v<NEW_VERSION>; planned outage ~10m."

# 1b. Notify CONNECTED players in-game via the admin REST API.
curl -fsS -X POST http://127.0.0.1:8080/api/v1/admin/broadcast \
  -H "X-API-Key: ${MAID_ADMIN_API_KEY}" -H 'Content-Type: application/json' \
  -d '{"prefix":"[MAINTENANCE]","message":"Maintenance for v<NEW_VERSION> in ~1h; planned outage ~10m."}'

# 2. There is NO working login gate (maid ops maintenance is an M9 stub).
# The stop in step 3 is the enforcement.

# 3. Stop the engine (so the backup captures a quiescent state).
sudo systemctl stop maid-engine@<inst>.service

# 4. Take a fresh backup and verify THAT backup, fail-closed. Capture the
# id the backup actually emitted instead of listing the newest one:
# `backup.sh full --json` prints exactly one JSON object on success
# ({"backup_id":...,"result":"success"}); all logs go to stderr. On ANY
# failure `full` exits non-zero and prints no id, so BID stays empty and
# we abort — we must NOT fall back to verifying an older, pre-upgrade
# backup (that would give false confidence). verify needs an explicit
# --backup-id (no --latest/--label); it re-downloads and sha256-checks
# every file and is fail-closed (non-zero exit on any mismatch).
# backup.sh reads MAID_DEPLOY_BACKUP_REMOTE from its OWN process env; a
# bare sudo does NOT load the instance EnvironmentFile, so source it with
# auto-export inside the maid-engine shell (else source_transport aborts:
# "MAID_DEPLOY_BACKUP_REMOTE is not set").
# MAID_SRC must point at a MAID source checkout on the host: backup.sh is
# repo-only until M1.4 (it is NOT staged under /opt/maid/current). Set it
# first — `export MAID_SRC=/path/to/maid-checkout` — and forward it through
# sudo with --preserve-env=MAID_SRC (env_reset would otherwise drop it
# before the inner shell expands the path).
BID="$(sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
        'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" full --instance <inst> --json' \
      | jq -r '.backup_id // empty')"
test -n "${BID}" || { echo "backup did not complete — aborting upgrade"; exit 1; }
export BID
sudo -u maid-engine --preserve-env=BID,MAID_SRC bash -c \
        'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/deploy/scripts/backup.sh" verify --instance <inst> --backup-id "${BID}"' \
    || { echo "verify of ${BID} failed — aborting upgrade"; exit 1; }

# 5. Stage the new release.
# PLANNED (M1.4): `install.sh --upgrade <NEW_VERSION>` is NOT implemented
# yet (the installer rejects `--upgrade`); stage the new release via your
# provisioning process (docs/deployment/bare_metal.md) before continuing.

# 6. Run migrations under the service account. The db CLI has no --instance
# flag; it reads its MAID_DB_* target from the instance env file (the same
# file the systemd unit loads via EnvironmentFile=), so source it first.
# NOTE (M1.1–M1.3): /opt/maid/venv is created empty by install.sh (no
# maid_engine); build a runnable venv in the checkout first with
# `sudo -u maid-engine -H uv sync --frozen --project "${MAID_SRC}"`
# (use the host's uv, e.g. /opt/maid/current/.uv/bin/uv on installer
# hosts), then run the CLI from "${MAID_SRC}/.venv/bin/maid". Forward
# MAID_SRC through sudo because env_reset drops it (see backup.sh above).
sudo -u maid-engine --preserve-env=MAID_SRC bash -c 'set -a; . /etc/maid/<inst>.env; set +a; exec "${MAID_SRC}/.venv/bin/maid" db migrate'

# 7. Restart engine.
sudo systemctl start maid-engine@<inst>.service

# 8. Verify (maid ops doctor is an M9 stub).
curl -sf http://localhost:8080/health && echo OK
sudo "${MAID_SRC}/scripts/maid-doctor.sh" --phase runtime --instance <inst>

# 9. Announce all-clear to webhooks. `maid ops announce` reads
# MAID_BRIDGES_WEBHOOK_URLS from its own process environment, so source
# the instance env with auto-export (a bare --preserve-env forwards
# nothing unless the var was already exported into this shell):
sudo -u maid-engine --preserve-env=MAID_SRC bash -c \
  'set -a; . /etc/maid/<inst>.env; set +a;
   exec "${MAID_SRC}/.venv/bin/maid" ops announce --instance <inst> --severity info \
     --message "Maintenance complete; v<NEW_VERSION> is live."'

If maid db migrate fails, follow docs/runbooks/failed_migration.md.

Rollback matrix

Migration kind	Rollback path
Additive (new nullable column or table)	Code rollback only; new column ignored by old binary.
Backfill with default	Code rollback; column tolerated by old binary.
Rename phase 1 (add new name, dual-write)	Code rollback; both names tolerated.
Rename phase 2 (drop old name)	Restore from backup. See docs/runbooks/restore.md.
Type change in place	Forbidden. Must be expressed as expand-contract (add new column, dual-write, backfill, drop old).
Irreversible (irreversible=True)	Restore from backup. Requires explicit operator confirmation. See migration policy.

The migration framework refuses to run an irreversible=True migration unless all of: (a) MAID_MIGRATION_ALLOW_IRREVERSIBLE=1 is set in the environment, (b) a fresh backup manifest exists with mtime ≤ 1 hour, and © the migration advisory lock is held. See docs/contributing/migrations.md for the runtime preconditions and the CI gate that enforces reviewer sign-off.

Player communications

See docs/deployment/player_comms.md for the literal broadcast cadence (T-24h / T-1h / T-15m), the in-game banner contents, the Discord webhook payload, and the post-incident summary template.