Skip to content

Runbook: RB22 — Python 3.12 / uv toolchain

Symptom

You hit one of: - install.sh aborts inside stage_python with one of the messages (see scripts/install.sh:485 / 498 / 503): - uv installer failed (see RB22) - uv python install 3.12 failed (RB22) - uv python install 3.12 produced no usable interpreter (RB22) - Engine fails to start with ModuleNotFoundError after a deploy, suggesting /opt/maid/venv/ is missing, stale, or pointing at the wrong interpreter. - maid-admin doctor --phase runtime --instance <inst> reports python_version: fail or venv_path: fail. - You are in ./RB18_host_loss_dr.md and the fresh host has no managed Python yet.

This runbook documents how install.sh stage_python (scripts/install.sh:460-512) bootstraps uv and a managed CPython 3.12, what the on-disk layout looks like afterwards, how to re-validate the engine venv, and how to recover by hand.

Detection

  • install.sh non-zero exit with an RB22 marker in the message.
  • /opt/maid/current/.uv/bin/uv --version exits non-zero (uv missing).
  • /opt/maid/current/.uv/bin/uv python list --only-managed shows no 3.12.x entry (managed CPython missing).
  • cat /opt/maid/current/.python-interpreter is missing, points at a non-existent path, or names a non-3.12 binary.
  • /opt/maid/venv/bin/python --version is missing or wrong major.minor.

Blast radius

  • Players affected: none until the engine fails to start; then ALL.
  • Data at risk: none (toolchain-only).
  • AI/external systems: irrelevant.

Prerequisites

  • Tools: curl, sudo, the maid-engine system user.
  • Access: root for the package and install.sh invocations; maid-engine UID owns /opt/maid/current/{.uv,python} and /opt/maid/venv.
  • Network egress to astral.sh (uv installer) and github.com/indygreg/python-build-standalone (uv pulls managed CPython tarballs from this release stream).
  • Audit access: tail -f /var/log/maid/ops-audit.jsonl.
  • Escalation: see ./escalation-contacts.md.template.

On-disk layout (after a healthy install.sh stage_python)

Path Owner Purpose
/opt/maid/current/.uv/bin/uv maid-engine uv tool binary (one per release dir)
/opt/maid/current/python/ maid-engine managed CPython 3.12 install root (--install-dir)
/opt/maid/current/.python-interpreter maid-engine recorded full path to the active 3.12 interpreter
/opt/maid/venv/ maid-engine engine venv — lives OUTSIDE current/ so blue-green symlink swaps don't disturb it (cross-cut #1, scripts/install.sh:61)

The exact interpreter path resolves via uv python find 3.12 --python-preference only-managed; it is typically /opt/maid/current/python/cpython-3.12.<patch>-linux-<arch>-gnu/bin/python3.12 but you should always read /opt/maid/current/.python-interpreter rather than hard-coding.

First 5 minutes (LITERAL commands)

# 1. What does the install think it has? (uv + managed Python + recorded interp)
ls -la /opt/maid/current/.uv/bin/uv 2>&1
/opt/maid/current/.uv/bin/uv --version 2>&1
sudo -u maid-engine /opt/maid/current/.uv/bin/uv python list --only-managed 2>&1
cat /opt/maid/current/.python-interpreter 2>&1

# 2. Is the venv usable?
ls -la /opt/maid/venv/bin/python 2>&1
/opt/maid/venv/bin/python --version 2>&1
# Expect: Python 3.12.x

# 3. Is the recorded interpreter still resolvable?
PYEXE="$(sudo -u maid-engine /opt/maid/current/.uv/bin/uv python find 3.12 \
         --python-preference only-managed 2>/dev/null || true)"
echo "uv resolves 3.12 to: ${PYEXE:-<none>}"

# 4. Capture state
{
  echo "=== uv ==="
  /opt/maid/current/.uv/bin/uv --version 2>&1
  echo "=== uv python list (managed only) ==="
  sudo -u maid-engine /opt/maid/current/.uv/bin/uv python list --only-managed 2>&1
  echo "=== recorded interpreter ==="
  cat /opt/maid/current/.python-interpreter 2>&1
  echo "=== venv ==="
  ls -la /opt/maid/venv/bin/python 2>&1
  /opt/maid/venv/bin/python --version 2>&1
  echo "=== distro ==="
  . /etc/os-release; echo "$NAME $VERSION_ID"
} > ./incident-toolchain-$(date -u +%Y%m%dT%H%M%SZ).log

How install.sh provisions Python 3.12

stage_python (scripts/install.sh:460-512) does NOT rely on the distro's Python at all. It always bootstraps the same way:

  1. Create owner-aligned directories:
  2. ${MAID_HOME}/.uv/ and ${MAID_HOME}/.uv/bin/ (maid-engine:maid-engine 0750)
  3. ${MAID_HOME}/python/ (maid-engine:maid-engine 0750)
  4. ${VENV_DIR} parent (root:root 0755), then ${VENV_DIR} itself (maid-engine 0750)
  5. If ${MAID_HOME}/.uv/bin/uv does not already exist, run the official uv installer as the maid-engine user with UV_INSTALL_DIR=${MAID_HOME}/.uv/bin and UV_NO_MODIFY_PATH=1 (scripts/install.sh:480-486): 
    sudo -u maid-engine env \
  UV_INSTALL_DIR=/opt/maid/current/.uv/bin \
  UV_NO_MODIFY_PATH=1 \
  bash -c 'curl -LsSf --max-time 60 https://astral.sh/uv/install.sh | sh'
  6. Install managed CPython 3.12 into the release-local prefix (scripts/install.sh:497-498): 
    sudo -u maid-engine /opt/maid/current/.uv/bin/uv python install 3.12 \
  --install-dir /opt/maid/current/python
  7. Resolve the freshly-installed interpreter and create the engine venv pointed at it (scripts/install.sh:500-507): 
    PYEXE="$(sudo -u maid-engine /opt/maid/current/.uv/bin/uv python find 3.12 \
         --python-preference only-managed)"
sudo -u maid-engine /opt/maid/current/.uv/bin/uv venv \
  --python "$PYEXE" --no-project /opt/maid/venv
  8. Record the chosen interpreter path so subsequent stages and doctor can audit it (scripts/install.sh:509): 
    echo "$PYEXE" > /opt/maid/current/.python-interpreter
chmod 0644 /opt/maid/current/.python-interpreter

There is intentionally no apt/dnf/pacman fallback and no install.sh --check-python flag — install.sh either succeeds end-to-end in stage_python or dies with one of the three RB22 messages above. Distro Python (system python3) is never used and is not on the engine's PATH at runtime.

Manual recovery

Use the manual sequence below when stage_python failed partway and you cannot re-run install.sh cleanly (e.g. mid-RB18 recovery). The shell commands replay what stage_python would have done.

export MAID_HOME=/opt/maid/current
export VENV_DIR=/opt/maid/venv
export MAID_USER=maid-engine

# 1. Directory skeleton (already correct on a vanilla install — these are idempotent)
sudo install -d -o root        -g root        -m 0755 "$(dirname "$MAID_HOME")"
sudo install -d -o $MAID_USER  -g $MAID_USER  -m 0750 "$MAID_HOME"
sudo install -d -o $MAID_USER  -g $MAID_USER  -m 0750 "$MAID_HOME/.uv" "$MAID_HOME/.uv/bin"
sudo install -d -o $MAID_USER  -g $MAID_USER  -m 0750 "$MAID_HOME/python"
sudo install -d -o root        -g root        -m 0755 "$(dirname "$VENV_DIR")"

# 2. Install uv as the maid-engine user, into the release-local .uv dir
sudo -u $MAID_USER env \
  UV_INSTALL_DIR="$MAID_HOME/.uv/bin" \
  UV_NO_MODIFY_PATH=1 \
  bash -c 'curl -LsSf --max-time 60 https://astral.sh/uv/install.sh | sh'
sudo -u $MAID_USER "$MAID_HOME/.uv/bin/uv" --version

# 3. Install managed CPython 3.12 (always uses python-build-standalone)
sudo -u $MAID_USER "$MAID_HOME/.uv/bin/uv" python install 3.12 \
  --install-dir "$MAID_HOME/python"

# 4. Resolve the interpreter uv just installed
PYEXE="$(sudo -u $MAID_USER "$MAID_HOME/.uv/bin/uv" python find 3.12 \
         --python-preference only-managed)"
test -x "$PYEXE" || { echo "no managed 3.12 found"; exit 1; }
echo "managed 3.12 at: $PYEXE"

# 5. (Re)create the engine venv outside current/
sudo install -d -o $MAID_USER -g $MAID_USER -m 0750 "$VENV_DIR"
sudo -u $MAID_USER "$MAID_HOME/.uv/bin/uv" venv \
  --python "$PYEXE" --no-project "$VENV_DIR"

# 6. Record the chosen interpreter
echo "$PYEXE" | sudo -u $MAID_USER tee "$MAID_HOME/.python-interpreter" > /dev/null
sudo chmod 0644 "$MAID_HOME/.python-interpreter"

If step 2 fails (network egress blocked, certificate chain broken, etc.): - Confirm outbound HTTPS to astral.sh and github.com/indygreg/python-build-standalone. - uv honours HTTPS_PROXY / NO_PROXY env vars — set them in the sudo -u $MAID_USER env ... invocation if you are behind a proxy.

If step 3 fails with a build-standalone download error, you can pre-stage a .python-build-standalone-3.12.X.tar.gz to the local uv cache (path under $HOME/.cache/uv/python/) and re-run step 3 — uv will reuse it instead of fetching.

Re-validating the engine venv with uv sync

After fixing the toolchain you typically still need to (re-)hydrate the venv from uv.lock: 

cd /opt/maid/current
sudo -u maid-engine /opt/maid/current/.uv/bin/uv sync --frozen
# Recreates /opt/maid/venv/ deterministically from the release's uv.lock.

If uv sync reports an error: - lockfile mismatch → you are on the wrong release tarball. Match /opt/maid/current/uv.lock to the release archive. - network error to PyPI → check the host's outbound proxy / DNS; retry. uv honours HTTPS_PROXY / NO_PROXY env vars. - missing system library (e.g. libffi, openssl) → install the matching -dev / -devel package and retry. The managed CPython is statically linked for most stdlib modules but a small number (sqlite3, readline) benefit from system libs being present.

Verifying the engine venv

⚠️ M1.1–M1.3: the stock /opt/maid/venv is empty. install.sh creates it with uv venv --no-project (scripts/install.sh:518) and its stage_app_code step is a documented M1.1 placeholder that installs no packages (scripts/install.sh:527). Until the M1.4 release tarball lands, maid_engine is not installed here, so checks #2 and #4 below fail by design and the maid-engine@ unit cannot start. The interim runtime is a source checkout venv built with uv sync (see docs/deployment/bare_metal.md, Step 10) — point these checks at ${MAID_SRC}/.venv/bin/python on such hosts. Checks #1 and #3 (the interpreter and the unit wiring) remain valid against /opt/maid/venv.

# 1. The venv exists and is the right interpreter
ls -la /opt/maid/venv/bin/python
/opt/maid/venv/bin/python --version
# Expect: Python 3.12.x

# 2. Key packages are importable (POST-M1.4 only; empty venv fails pre-M1.4).
#    Pre-M1.4, run this against ${MAID_SRC}/.venv/bin/python instead.
sudo -u maid-engine /opt/maid/venv/bin/python -c \
  "import maid_engine, asyncpg, redis, pydantic; print('ok')"

# 3. The systemd unit references the venv correctly
sudo systemctl cat maid-engine@<inst>.service \
  | grep -E 'ExecStart|EnvironmentFile'
# ExecStart should invoke /opt/maid/venv/bin/python -m maid_engine
# (NOT /opt/maid/current/venv/...; the venv is intentionally outside current/.)

# 4. py-spy (used by oom_loop / RB19) is installed
sudo -u maid-engine /opt/maid/venv/bin/python -m pip show py-spy 2>/dev/null \
  || sudo -u maid-engine /opt/maid/venv/bin/pip install py-spy
# POST-M1.4 only: py-spy ships with the release tarball. Pre-M1.4 the venv
# has neither py-spy nor pip (uv venv --no-project omits pip); use the
# checkout venv (`uv sync` then `${MAID_SRC}/.venv/bin/python -m ...`).

Recovery

  • /opt/maid/current/.uv/bin/uv --version works.
  • /opt/maid/current/.uv/bin/uv python list --only-managed lists a 3.12.x entry whose install path is under /opt/maid/current/python/.
  • cat /opt/maid/current/.python-interpreter prints a path that is executable and reports Python 3.12.x for --version.
  • /opt/maid/venv/bin/python --version is 3.12.x.
  • maid-admin doctor --phase runtime --instance <inst> python_version: pass.
  • Engine starts and maid-admin status --instance <inst> reports engine_state=running.

Post-incident

  • File ticket with: distro, exit message from stage_python, whether the failure was the uv installer download, the uv python install step, the interpreter resolution, or the venv creation.
  • If the failure was network-related, file a follow-up to either pre-mirror the python-build-standalone tarballs onto a local artifact host or document the HTTPS_PROXY requirement.
  • Update this runbook if stage_python semantics change in a future install.sh revision.

Escalation

  • Solo path: detect what's missing → re-run stage_python via sudo ./install.sh --resume-from python (if available) or the manual recovery sequence above → uv sync → verify.
  • Hosting console URL: see ./escalation-contacts.md.template.
  • Comms channel URL: see ./escalation-contacts.md.template.
  • Peer operator: see ./escalation-contacts.md.template.
  • If outbound HTTPS to astral.sh / github.com is blocked and cannot be unblocked within 30 min, you cannot complete a fresh install on this host. Either pre-stage uv + the python-build-standalone tarball from a working host (copy to ~/.cache/uv/) before retrying, or choose a different host with egress.