Durability Matrix¶
Each gameplay event class has an explicit Recovery Point Objective (RPO) and a
mechanism that achieves it. This document is the source of truth for what
survives an OOM kill, a SIGKILL, or a power loss, and what does not.
The matching code lives in packages/maid-engine/src/maid_engine/persistence/
(future, M4 — see below). For each row the test cited in the right-hand
column is planned (M4) and does not yet exist.
Matrix¶
| Class | RPO target | Mechanism | Test |
|---|---|---|---|
| Currency mutation | ≤ 1s | critical_write("currency") |
tests/integration/test_critical_writes.py::test_currency (planned, M4) |
| Inventory transfer | ≤ 1s | critical_write("inventory_transfer") |
tests/integration/test_critical_writes.py::test_inventory (planned, M4) |
| Quest completion | ≤ 1s | critical_write("quest_completion") |
tests/integration/test_critical_writes.py::test_quest (planned, M4) |
| Builder commit | ≤ 2s | critical_write("builder_commit") |
tests/integration/test_critical_writes.py::test_builder (planned, M4) |
| Combat HP/MP | ≤ save_interval (300s / 5 min default) |
normal batch via EntityPersistenceManager |
tests/integration/test_critical_writes.py::test_normal_class (planned, M4) |
| NPC autonomy | ≤ save_interval |
normal batch | (n/a in M1; system not active in beta) |
| Memory / relationship | ≤ save_interval |
normal batch | (n/a in M1; system gated by MAID_MEMORY_ENABLED) |
| Ephemeral (party, channel, online presence) | none — recreated post-restart | by design | tests/integration/test_critical_writes.py::test_ephemeral_lost (planned, M4) |
Per-class rationale¶
Currency mutation — ≤ 1s¶
Currency loss is the single most damaging recoverable bug in any MUD: it
breaks player trust and is impossible to silently audit away. The
critical_write("currency") wrapper synchronously flushes the affected entity
via EntityPersistenceManager.flush_now([entity_id]) before the command
returns ACK to the player. Combined with R4.1's operation-ledger envelope, an
operator can prove for any specific transaction whether the COMMIT landed on
disk before the crash.
Inventory transfer — ≤ 1s¶
Inventory transfer is two coupled writes (source loses, destination gains).
The critical_write("inventory_transfer") block wraps both writes in a single
DB transaction (per R5.B.2's single-transaction Option A) so that either both
sides commit or neither does. The synchronous flush on commit ensures the
gameplay invariant count(item) is conserved across crash holds.
Quest completion — ≤ 1s¶
Quest completion delivers rewards (XP, items, currency) that are visible to
the player and often advance world state (faction reputation, follow-up
quests). The critical_write("quest_completion") wrapper batches reward
delivery and quest-state update in the same critical block; the synchronous
flush guarantees the reward is durable before the player sees the "quest
complete" message.
Builder commit — ≤ 2s¶
Builder edits (@create, @destroy, @dig, @describe, @set) represent
creative work that has no upstream source — losing five minutes of building
because of a crash is unacceptable. The slightly looser 2s target reflects
the fact that builder operations often touch larger entity graphs (a new
room, exits, attached templates), and bounding their flush at 2s avoids
blocking the tick loop for outlier writes.
Combat HP/MP — ≤ save_interval¶
Combat state churns far too fast for synchronous flush at the per-tick
granularity. Losing the last save_interval window of combat (up to the
300s / 5-minute default, unless MAID_PERSISTENCE_SAVE_INTERVAL is lowered)
is an acceptable cost: combat encounters are short, restartable, and rarely
the source of permanent progression.
NPC autonomy / Memory / Relationship — ≤ save_interval¶
These systems are either disabled in M1 beta (memory gated by
MAID_MEMORY_ENABLED=false) or run as periodic batch workloads. Their data
is regenerable from observation in the worst case, so the normal save batch
is sufficient.
Ephemeral — by design¶
Party membership, voice-chat channels, and "who is online right now" are
intentionally non-durable. The cost of persisting them exceeds the value;
players re-form parties post-restart in seconds. Tested as the negative
case: test_ephemeral_lost asserts that after SIGKILL+restart the party
list is empty.
How critical_write actually works¶
critical_write is a context-manager / decorator implemented in
packages/maid-engine/src/maid_engine/persistence/critical_write.py
(planned, M4). The runtime behaviour:
- Mark. On entry, the wrapped callable is associated with a write class
(e.g.
"currency") and the set of dirty entity IDs it intends to touch. - Mutate. The callable runs to completion inside a single PG transaction (per R5.B.2 — "single-transaction COMMITTED + reconciliation"). All gameplay effects, including entity state changes and ledger inserts, are in the same transaction.
- Drain-then-re-mark. On successful exit,
flush_now(ids, timeout=...)is called onEntityPersistenceManager. The persistence scheduler drains any in-flight normal-class batch that conflicts with the critical write, performs the critical flush, then re-marks any entities that were re-dirtied during the drain window (per plan.mdscheduler.pynotes "oldest_pending_age, drain-then-re-mark", line 713). - Emit metric. On flush success, emit
critical_write.flushed{class="..."}. On timeout, emitcritical_write.timeout{class="..."}and DO NOT ACK the operation to the player — the command surfaces an error.
The ops_ledger is observational only¶
Earlier revisions (R4.1 / R5.B.2) proposed using the ops_ledger table for
strong consistency machinery: a deferred-constraint trigger
(enforce_ledger_entity_ids_consistency), an _maid_write_audit table, and
per-entity write tracking. All of this was CUT in R10.B.11. The
remaining ops_ledger is purely observational: it records what
critical_write did and when, for after-the-fact audit and incident
forensics. It does not gate any runtime behaviour, and consistency relies on
the single-transaction COMMITTED + reconciliation pattern from R5.B.2
(plan.md lines 3262–3334).
Why the 50ms p99 budget from R4 is now 1–2s¶
Earlier drafts targeted 50ms p99 for critical_write. R4.1 explicitly
rejected this as unrealistic once the ledger + per-entity locking +
single-transaction reconciliation were folded in. The shipped target is the
RPO column above (1s for atomic writes, 2s for builder writes). The
operator-facing SLO is the RPO, not the wall-clock command latency budget.