MAID World Data Loaders — Final Design Specification¶

Document Version: 3.4 Date: July 2025 Status: Final (revised with review round 5 improvements) Authors: MAID Development Team

1. Executive Summary¶

MAID currently requires Python code to define game content. Existing data files (rooms, NPCs, items, spells) are loaded by bespoke per-pack code with no shared infrastructure. Each loader re-implements file discovery, YAML parsing, validation, and indexing. This blocks non-programmers from creating content and makes it impossible to round-trip between in-game builder commands and declarative data files.

This document specifies a unified World Data Loader framework that:

Provides a composable phase pipeline: discover → lint → parse → validate → resolve references → instantiate — each phase is standalone and independently testable
Uses YAML 1.2 via ruamel.yaml as the sole parser — no PyYAML fallback — eliminating the Norway Problem and all YAML 1.1 coercion bugs
Validates data against Pydantic component schemas already defined in the engine and stdlib, with auto-discovered component types
Supports single-inheritance templates with variable substitution and a maid content resolve debugging command
Resolves cross-entity references by symbolic name with pre-assigned deterministic UUIDs to avoid chicken-and-egg ordering problems
Integrates with the ContentPack lifecycle and DocumentStore for persistence using a definition vs. instance state machine model
Reports errors with file path, line number, field path, unified error codes, and "Did you mean…" suggestions
Provides CLI tooling for validation, linting, preview, and export
Includes security hardening: resource limits, path jail, no YAML aliases, no pickle, no recursive expansion

Design Goals¶

#	Goal	Rationale
G1	Non-programmer authoring	Content creators should only need to learn YAML
G2	Single source of truth	Data files are the canonical representation
G3	Definition-state export	Export captures author-defined state, not runtime
G4	Incremental loading	Only changed files are reprocessed on reload
G5	Error locality	Every error reports file, line, and field path
G6	Zero runtime cost	Loaded entities are indistinguishable from code-created ones
G7	Safe hot reload	Reload refuses if entities have active interactions

Scope — Minimum Viable Loader (MVL)¶

The implementation is scoped to a Minimum Viable Loader for v1. The following features ship in v1; everything else is deferred to v2+:

v1 (Ship)	v2+ (Deferred)
YAML parsing with `ruamel.yaml` (no fallback)	VS Code extension
Pydantic validation against component schemas	Migration tool (AST/runtime)
Single-level template inheritance (`_extends`)	Multiple inheritance / mixins
Pre-assigned UUID reference resolution	Three-way definition/instance merge
ContentPack `on_load()` integration	Lazy zone loading
`maid data validate` + `maid data lint` CLI	Parallel file parsing
EntityAssembler with EntityTypeConfig	Conditional content loading
Typed SemanticRule protocol with `--skip-rule`	Localization framework
Definition vs. instance state machine	Compiled binary cache (msgpack+zstd)
Atomic load with rollback (LoadTransaction)
Security hardening (resource limits, path jail)
Unified TemplateRegistry (loader + builder)
No-Code declarative content pack loading
Generic `_append` block for all list fields
Unified error codes (MAID-V/S/R/T###)

2. Problem Statement¶

2.1 Current State¶

Two systems in maid-classic-rpg have stubbed data-loading methods:

# EcosystemSystem (ecosystem.py, lines 72–83)
async def _load_definitions(self) -> None:
    logger.warning("_load_definitions() not yet implemented")

# WorldEventSystem (events.py, lines 71–81)
async def _load_definitions(self) -> None:
    logger.warning("_load_definitions() not yet implemented")

Existing YAML files across the monorepo use bespoke loaders:

Package	Files	Loader
maid-classic-rpg	9 spell files, 3 skill files, 3 ability files	`SpellLoader`, `SkillLoader`, `AbilityLoader`
maid-tutorial-world	`rooms.yaml`, `npcs.yaml`, `items.yaml`	Bespoke code in `pack.py`

Each loader independently implements discovery, parsing, validation, and indexing — duplicating ~200 lines of infrastructure per content type.

2.2 Content Creation Paths¶

Path	Audience	Limitation
Builder commands (`@create`, `@dig`)	In-game builders	No export to files
ContentPack `on_load()`	Python developers	Requires code
`@batch` command	Admins	Imperative, not declarative
Bespoke YAML loaders	Mixed	Fragmented, no shared infra
Data Loader (new)	Non-programmers	This design

2.3 Infrastructure to Build On¶

ComponentRegistry: Maps type names to Python classes via component_type ClassVar — component types are auto-discovered, no manual registration
EntityTemplate / TemplateRegistry: In maid-stdlib/commands/building/create.py, stored in World.custom_data["stdlib:template_registry"]
ContentPack protocol: on_load() is the natural integration point
DocumentStore: Abstract persistence with typed collections
ReloadManager: Hot-reload with ReloadScope enum

Naming: The data loader and builder commands share a single TemplateRegistry instance. An abstract TemplateRegistry Protocol is defined in maid-engine; the concrete implementation lives in maid-stdlib (in commands/building/create.py). GameEngine owns the instance via dependency injection and injects it into both the data pipeline and builder commands (see §16, Q7).

3. Architecture Overview¶

3.1 Loader Pipeline — Composable Phase Architecture¶

The data loading pipeline is decomposed into standalone Phase objects coordinated by a Pipeline. Each phase is independently testable, and CLI tools compose subsets of phases for different workflows.

class Phase(Protocol):
    """A single step in the data loading pipeline."""

    @property
    def name(self) -> str: ...

    async def execute(self, context: LoaderContext) -> PhaseResult: ...

class Pipeline:
    """Composes Phase objects into an executable sequence."""

    def __init__(
        self,
        phases: list[Phase],
        timeout: float = 60.0,
    ) -> None: ...

    async def run(
        self,
        context: LoaderContext,
        cancellation_token: CancellationToken | None = None,
    ) -> PipelineResult: ...

class CancellationToken:
    """Cooperative cancellation for long-running pipelines.

    Phases check ``token.is_cancelled`` between units of work. The
    pipeline checks after each phase completes.
    """

    def __init__(self) -> None:
        self._cancelled: bool = False

    @property
    def is_cancelled(self) -> bool: ...

    def cancel(self) -> None:
        self._cancelled = True

Pipeline timeout (default: 60s): If a pipeline exceeds timeout seconds, it is aborted with a PipelineTimeoutError. Individual phases respect the CancellationToken for cooperative cancellation. The timeout is configurable via LoaderConfig.pipeline_timeout.

Phase inventory:

Phase 1: DiscoverPhase    → Find .yaml/.json files in data/
Phase 2: LintPhase        → YAML pitfall detection on raw text
Phase 3: ParsePhase       → YAML 1.2 to Python dicts (ruamel.yaml)
Phase 4: SchemaRoutePhase → Read _meta, route to EntityAssembler
Phase 5: RegisterIDsPhase → Collect _id values, pre-assign UUIDs
Phase 6: ValidatePhase    → Pydantic component validation
Phase 7: SemanticPhase    → Gameplay-logic semantic checks
Phase 8: ResolveRefsPhase → Replace @ref: strings with UUIDs
Phase 9: InstantiatePhase → Create ECS entities in World (atomic)
Phase 10: PostLoadPhase   → Content pack callbacks

CLI tools compose phase subsets:

CLI Command	Phases
`maid data lint`	Discover, Lint
`maid data validate`	Phases 1–8 (no instantiation)
`maid data load`	All phases
`maid data preview`	Phases 1–8 + dry-run summary

Each phase receives a shared LoaderContext and returns a PhaseResult containing errors, warnings, and produced artifacts. CRITICAL errors are per-file, not per-pipeline — a CRITICAL error in one file causes that file's remaining entities to be skipped, but other files continue processing. The pipeline collects all errors across all files for maximum diagnostic output per run.

@dataclass
class LoaderContext:
    """Shared mutable state threaded through all phases."""

    world: World
    config: LoaderConfig
    reference_registry: ReferenceRegistry   # Engine-owned, injected
    template_registry: TemplateRegistry     # Unified registry
    discovered_files: list[DiscoveredFile] = field(default_factory=list)
    parsed_docs: dict[Path, ParsedDocument] = field(default_factory=dict)
    entity_definitions: dict[str, EntityDefinition] = field(default_factory=dict)
    errors: list[LoadError] = field(default_factory=list)

3.2 Phase Details¶

Phase 1 — Discover: Recursively scans the content pack's data/ directory. Respects load_order in _meta.yaml to ensure templates load before entities that reference them. Symlinks are ignored and path traversal outside the pack root is rejected (see §15).

_meta.yaml failure modes:

Condition Behavior

Missing _meta.yaml Use convention defaults (directory-based schema inference)

Unparseable _meta.yaml CRITICAL error — entire pack fails to load

References a missing file ERROR, skip that file, continue with others

Files exist but not in load_order Load after ordered files, emit INFO diagnostic

Phase 2 — Lint (pre-parse): Runs the YAML pitfall linter (§4) on raw text BEFORE parsing. Catches boolean coercion, octals, and other issues that disappear after parsing.

Phase 3 — Parse: Uses ruamel.yaml in YAML 1.2 safe mode. Source line/column positions are preserved on parsed nodes for error reporting. YAML aliases are disabled to prevent billion-laughs expansion attacks (see §15). TOCTOU mitigation: Before reading each file, the phase re-verifies path safety (is_safe_path()) immediately prior to open(), guarding against symlink-swap attacks between DiscoverPhase and ParsePhase.

Phase 4 — Schema Route: Extracts _meta.schema and routes to the EntityAssembler with the appropriate EntityTypeConfig. Unknown types produce a "Did you mean…" error. When _meta.schema is absent, schema is inferred from the directory name or top-level key (see §4.4). Template resolution and variable substitution (_extends, _use, _vars) are resolved in this phase — the output is a fully-expanded entity definition with all template inheritance and ${var} placeholders replaced by concrete values. This ensures that Phase 6 (Validate) validates the final expanded form, not the pre-substitution template.

Phase 5 — Register IDs: All _id values are collected. A deterministic UUID is pre-assigned via uuid5(NAMESPACE_DATA, f"{pack}:{scoped_id}"). This solves the chicken-and-egg problem — references resolve to real UUIDs before any entity exists. Collisions are checked against the current load, dependency packs, and existing world entities (see §4.6).

Phase 6 — Validate: Component blocks are validated against Pydantic models via model_validate(). Resource limits are enforced: max 1000 entities per file, max 50 components per entity (see §15).

Phase 7 — Semantic Validate: Runs typed SemanticRule objects against validated entities (see §3.4). Rules receive the full LoaderContext for cross-entity validation.

Phase 8 — Resolve References: @ref: strings are replaced with pre-assigned UUIDs. Unresolved refs produce errors with suggestions.

Phase 9 — Instantiate (atomic): Non-template entities are created inside a LoadTransaction using the resolve_load_action() state machine (§8.3) to determine the correct action per entity: create from definition, load from instance store, quarantine, or skip. For each entity, the phase checks whether an instance already exists in the DocumentStore, compares definition hashes, and selects CREATE_FROM_DEFINITION, LOAD_FROM_INSTANCE, LOAD_INSTANCE_WARN_STALE, or QUARANTINE_LOAD_DEFINITION accordingly.

Engine prerequisite (addresses review round 5, critical #1): World.create_entity() does NOT currently accept a suppress_events parameter. The LoadTransaction uses a staging EntityManager pattern instead: entities are assembled in a detached staging area (an EntityManager instance not connected to the EventBus), then bulk-transferred into the live World at commit time. This avoids requiring changes to the World.create_entity() signature. See §8.5 for the full staging API.

Each entity receives a DataProvenanceComponent tracking its origin file and definition hash. If any entity fails post-instantiation reference integrity checks, the entire transaction rolls back (see §8.5).

Phase 10 — Post-Load: Content packs run callbacks for cross-entity relationship setup and index building.

3.3 Error Handling Strategy¶

The loader uses a collect-all-errors strategy with structured error codes:

class ErrorSeverity(Enum):
    WARNING = "warning"   # Non-fatal; entity still created
    ERROR = "error"       # Fatal for this entity; others continue
    CRITICAL = "critical" # Fatal for entire file; remaining entities skipped

@dataclass(frozen=True)
class LoadError:
    code: str                       # e.g., "MAID-V001", "MAID-S003"
    file_path: Path
    line: int | None
    field_path: str
    message: str
    severity: ErrorSeverity
    suggestion: str | None = None

Unified error code ranges:

Prefix	Domain	Example
`MAID-Y###`	YAML lint (pre-parse)	`MAID-Y001` unquoted boolean
`MAID-V###`	Schema validation	`MAID-V001` unknown component
`MAID-S###`	Semantic validation	`MAID-S001` HP exceeds max
`MAID-R###`	Reference resolution	`MAID-R001` unresolved ref
`MAID-T###`	Template resolution	`MAID-T001` circular inheritance

Severity	Entity Created?	File Continues?
WARNING	Yes	Yes
ERROR	No	Yes
CRITICAL	No	No (remaining entities in file skipped)

Error cap: To prevent denial-of-service from malformed files flooding logs, the pipeline enforces a max_errors_per_file limit (default: 50). When the limit is reached, the file is abandoned with a summary diagnostic: "50 errors reached for this file; stopping. Fix the above errors first." This limit is configurable via LoaderConfig.max_errors_per_file.

3.4 Semantic Validation¶

Pydantic validates structure but not semantics. The dedicated SemanticPhase (Phase 7) runs typed SemanticRule objects that catch gameplay logic errors.

Check	Severity	Example
`current <= maximum` for resource components	Error	HP 200/100
Aggressive NPC must have `CombatComponent`	Error	Fighter with no combat stats
Weapon items should have damage-related fields	Warning	Sword with no attack_power
Rooms should have at least one exit	Warning	Isolated room
`wander_radius > 0` requires adjacent rooms	Warning	NPC can't actually wander
Merchant NPC should have `InventoryComponent`	Warning	Empty shop

SemanticRule Protocol¶

Rules are typed objects (not lambdas) that declare metadata for discovery, filtering, and CLI integration:

class SemanticRule(Protocol):
    """A single semantic validation rule."""

    @property
    def id(self) -> str:
        """Unique rule identifier, e.g. 'MAID-S001'."""
        ...

    @property
    def description(self) -> str:
        """Human-readable description of what this rule checks."""
        ...

    @property
    def severity(self) -> ErrorSeverity: ...

    @property
    def applies_to(self) -> set[str]:
        """Entity types this rule applies to, e.g. {'npc', 'room'}."""
        ...

    def check(self, entity: EntityDefinition, context: LoaderContext) -> list[LoadError]:
        """Validate entity, returning errors. Empty list = pass.

        Implementations MUST complete within the configured timeout
        (default: 5s). The ``SemanticPhase`` executor wraps each call in
        ``asyncio.wait_for(asyncio.to_thread(rule.check, ...), timeout=config.semantic_rule_timeout)``.
        Rules that exceed the timeout are skipped with a WARNING diagnostic
        and the rule ID is logged for investigation.
        """
        ...

The context parameter enables cross-entity validation — rules can inspect other entities in the same load batch (e.g., verifying that a merchant's shop room actually exists, or that faction references are consistent).

Example rule implementation:

class AggressiveNeedsCombat:
    id = "MAID-S002"
    description = "Aggressive NPCs must have a CombatComponent"
    severity = ErrorSeverity.ERROR
    applies_to = {"npc"}

    def check(self, entity: EntityDefinition, context: LoaderContext) -> list[LoadError]:
        npc = entity.components.get("NPCComponent", {})
        if npc.get("behavior_type") == "aggressive" and "CombatComponent" not in entity.components:
            return [LoadError(
                code=self.id,
                file_path=entity.source_file,
                line=entity.line,
                field_path="components",
                message="Aggressive NPCs must have a CombatComponent.",
                severity=self.severity,
                suggestion="Add a CombatComponent with attack_power and defense.",
            )]
        return []

Content packs register additional rules:

class MyContentPack:
    def get_semantic_rules(self) -> list[SemanticRule]:
        return [AggressiveNeedsCombat(), MerchantNeedsInventory(), ...]

CLI integration: Rules can be skipped by ID:

$ maid data validate data/ --skip-rule MAID-S003 --skip-rule MAID-S007
$ maid data validate data/ --list-rules   # Show all registered rules

4. YAML Schema Specification¶

4.1 Parser Choice: `ruamel.yaml` Only (No Fallback)¶

ruamel.yaml is a hard dependency. There is no PyYAML fallback path.

Criteria	`ruamel.yaml` (YAML 1.2)	PyYAML (YAML 1.1)
`NO` value	`"NO"` (string ✅)	`False` (bool ❌)
`0755` value	`"0755"` (string ✅)	`493` (octal int ❌)
`1:30` value	`"1:30"` (string ✅)	`90` (sexagesimal ❌)
Line tracking	✅ Preserved	❌ Lost after parse
Boolean literals	Only `true`/`false`	22 variants

Design Decision (addresses P0-1 from critique): A single YAML semantics is worth the dependency. The dual-parser "Two YAML" problem — where content validated with one parser fails with another — is eliminated entirely. ruamel.yaml is pure Python with an optional C extension for performance.

4.2 Safe Loading¶

All YAML parsing MUST use safe loading:

# ✅ CORRECT — ruamel.yaml with YAML 1.2
from ruamel.yaml import YAML

yaml = YAML(typ="safe")
yaml.version = (1, 2)
data = yaml.load(path)

# ❌ NEVER — arbitrary code execution
data = yaml.load(f, Loader=yaml.FullLoader)

4.3 The Norway Problem and Mitigation¶

YAML 1.1 (PyYAML) treats 22 bare words as booleans. YAML 1.2 (ruamel.yaml) only recognizes true and false. Since MAID mandates YAML 1.2, the Norway Problem is eliminated at the parser level.

The linter (§4.7) still flags these patterns because:

Content may be edited with tools that show YAML 1.1 previews
Authors moving from other YAML-based games expect 1.1 behavior
It catches confusing-to-humans syntax regardless of parser

# All of these are STRINGS in YAML 1.2 — no boolean coercion
country_code: NO        # → "NO" ✅
enabled: YES            # → "YES" ✅ (use true/false for actual booleans)
flag: On                # → "On" ✅
answer: Off             # → "Off" ✅

# Booleans in YAML 1.2 — ONLY these two values
pvp_enabled: true       # → True
safe_zone: false        # → False

4.4 Schema Version Header¶

Every data file SHOULD begin with a _meta block:

_meta:
  schema: "maid:rooms:v1"
  pack: "tutorial-world"           # Optional: owning content pack
  description: "Village area rooms"  # Optional: human description
  author: "World Builder Team"       # Optional: attribution

The schema field format is namespace:type:version:

Component	Examples
`namespace`	`maid`, `classic-rpg`, `my-pack`
`type`	`rooms`, `npcs`, `items`, `templates`, `zones`
`version`	`v1`, `v2`

Implicit Schema Inference¶

When _meta.schema is absent, the loader infers schema from context:

Signal	Inferred Schema	Example
Directory name	`maid:{dir}:v1`	`data/rooms/` → `maid:rooms:v1`
Top-level key	`maid:{key}:v1`	`rooms:` at top level → `maid:rooms:v1`
Both match	Use directory name	Consistent
Conflict	ERROR with suggestion	`npcs:` in `rooms/` directory

This eliminates boilerplate for standard entity types while preserving explicit control for custom schemas.

Schema Evolution Policy: Schemas are append-only — new fields get defaults, existing fields are never removed or renamed. If a breaking change is needed, a new version is created (e.g., v2). Both versions are supported simultaneously for one major release cycle.

4.5 Directory Structure¶

my-content-pack/
├── src/my_pack/
│   ├── data/
│   │   ├── _meta.yaml          # Pack-level metadata and load order
│   │   ├── zones/
│   │   │   └── village.yaml
│   │   ├── rooms/
│   │   │   ├── village_rooms.yaml
│   │   │   └── dungeon_rooms.yaml
│   │   ├── npcs/
│   │   │   └── merchants.yaml
│   │   ├── items/
│   │   │   └── weapons.yaml
│   │   └── templates/
│   │       └── archetypes.yaml
│   └── pack.py
└── pyproject.toml

4.6 `_id` Scoping, Uniqueness, and UUID Overrides¶

Entity _id values are scoped to their directory path to prevent namespace collisions:

# These are DISTINCT IDs — no collision
rooms/village/guard     # → uuid5("mypack:rooms/village/guard")
rooms/dungeon/guard     # → uuid5("mypack:rooms/dungeon/guard")

Within a single file, _id values must be unique. The linter enforces uniqueness at the directory level and flags potential collisions.

Deterministic UUID Collision Handling¶

Pre-assigned UUIDs are checked for collisions at three levels:

Current load batch — within the same Pipeline.run() invocation
Dependency packs — against entities from already-loaded packs
Existing world entities — against entities already in the World

If a collision is detected, the loader emits an ERROR with both entity definitions for comparison. Pack names are required to be globally unique (enforced by the ContentPackLoader).

Explicit UUID Overrides¶

For refactoring safety (renaming _id without breaking external references), entities may declare an explicit UUID:

village_square_renamed:
  _uuid: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"  # Preserves old UUID
  components:
    DescriptionComponent:
      name: "Village Square"

When _uuid is present, it is used directly instead of the deterministic uuid5() derivation. The linter warns if _uuid duplicates a deterministically-assigned UUID from another entity.

4.7 Linting Rules¶

The pre-parse linter operates on raw YAML text. It catches problems that disappear after parsing.

Rule	Sev.	Description	Example
`MAID-Y001`	Warn	Unquoted YAML 1.1 boolean literal	`country: NO`
`MAID-Y002`	Warn	Unquoted sexagesimal value	`version: 1:30`
`MAID-Y003`	Warn	Integer with leading zero (octal in 1.1)	`room: 0755`
`MAID-Y004`	Error	Empty/null in required string field	`name:`
`MAID-Y005`	Warn	Special float (`.inf`, `.nan`)	`value: .inf`
`MAID-Y006`	Warn	Inconsistent block scalar indentation	Mixed indent in `>`
`MAID-Y007`	Error	YAML merge key (`<<:`)	Not YAML 1.2
`MAID-Y008`	Error	Duplicate mapping key	Two `HealthComponent:` blocks
`MAID-Y009`	Error	Unresolvable `@ref:` reference	Typo in entity name
`MAID-Y010`	Warn	Cross-file `@ref:` without `depends_on`	Missing dependency
`MAID-Y011`	Warn	Runtime-only field set in data file	`in_combat: true`
`MAID-Y012`	Error	Unknown component type name	`HelthComponent`

Merge key change (addresses P0-1): MAID-Y007 is now Error (not Info) because YAML merge keys (<<:) are not part of YAML 1.2. Use _extends or _use instead.

Example linter output:

$ maid data lint data/npcs/

data/npcs/merchants.yaml
  line 12: MAID-Y001 [warn] Unquoted value 'NO' matches YAML 1.1 boolean.
           Use "NO" (quoted) if you mean the string.
  line 28: MAID-Y012 [error] Unknown component 'HelthComponent'.
           Did you mean 'HealthComponent'?
  line 45: MAID-Y008 [error] Duplicate key 'HealthComponent' at this level.
           Only the last definition will be used.

3 issues (2 errors, 1 warning) in 1 file

5. Validation¶

5.1 Pydantic Component Validation¶

Components are validated against their existing Pydantic models via model_validate(). The ComponentRegistry maps type name strings to classes, auto-discovered by scanning all Component subclasses using the component_type ClassVar.

# Auto-discovery — no manual registration needed
for cls in Component.__subclasses_recursive__():
    registry.register(cls.component_type, cls)

5.2 Schema Derivation Considerations¶

Components are not pure data bags — they carry methods (HealthComponent.damage(), InventoryComponent.add_item()) and custom validators (DialogueComponent.sanitize_prompt_text). The loader accounts for this:

Concern	Mitigation
Union types (`str \\| TranslatableText`)	Pydantic discriminated union handles it; error messages include both accepted forms
Runtime fields (`in_combat`, `target_id`)	Lint rule MAID-Y011 warns; fields are valid but semantically meaningless in data files
Custom validators (prompt injection)	Error messages rewritten to explain validation to content authors
Complex nested types (`UUID`, `datetime`)	UUIDs use `@ref:` syntax; datetimes use ISO 8601 strings
`extra="forbid"` on all definitions	Typos like `compnents:` produce immediate errors with suggestions

5.3 Cross-Reference Validation¶

References are validated in Phase 7. Each @ref: string must resolve to a pre-assigned UUID from Phase 5:

# Reference syntax
@ref:rooms/village_square       # By type and scoped name
@ref:stdlib:templates/base_npc  # Cross-pack namespaced reference

Unresolved references produce errors with fuzzy-match suggestions:

ERROR in data/npcs/merchants.yaml, line 15:
  Field 'location' references '@ref:rooms/vilage_square'
  which does not exist.
  Did you mean '@ref:rooms/village_square'?

Fuzzy-match rate limiting: To prevent pathological performance on large registries, the fuzzy matcher caps the candidate pool at 5,000 entries. Above this threshold, fuzzy matching is disabled and the error reports only "reference not found" with no suggestion. The threshold is configurable via LoaderConfig.fuzzy_match_max_candidates.

5.4 Validation Modes¶

Mode	`extra` fields	Unknown refs	Missing optional	Use case
Strict	Error	Error	Warning	CI/CD, production
Lenient	Warning	Warning	Silent	Development, iteration

Lenient mode caution: When a skipped entity is referenced by a valid entity, the valid entity's reference becomes unresolvable. The loader builds an explicit SkipGraph — a directed graph where each node is an entity and edges represent reference dependencies. When an entity is skipped, the SkipGraph propagates the skip to all transitive dependents with a clear error chain showing the full path. To prevent runaway cascades, a max_cascade_depth config (default: 10) caps the cascade chain length. If the cascade limit is reached, the pipeline emits a CRITICAL error listing the full cascade chain for diagnosis. The SkipGraph is available for inspection in PipelineResult.skip_graph for debugging.

6. Content Pack Integration¶

6.1 ContentPack `on_load()` Integration¶

class MyContentPack:
    async def on_load(self, engine: GameEngine) -> None:
        pipeline = engine.create_data_pipeline(
            config=LoaderConfig(strict_mode=True),
        )
        result = await pipeline.run_for_pack(
            self, data_path=self._data_path,
        )
        if not result.success:
            for error in result.errors:
                logger.error("  %s", error)

The GameEngine owns the Pipeline factory method and injects shared resources (ReferenceRegistry, TemplateRegistry, World).

EntityAssembler + EntityTypeConfig¶

Instead of per-type SchemaHandler classes, a single generic EntityAssembler handles all entity types using lightweight EntityTypeConfig declarations:

@dataclass(frozen=True)
class EntityTypeConfig:
    """Declarative config for how an entity type is assembled."""

    type_name: str                          # e.g., "room", "npc", "item"
    required_components: set[str]           # Must be present
    default_components: dict[str, dict]     # Added if absent
    default_tags: set[str]                  # e.g., {"room"}, {"npc"}
    allowed_top_level_fields: set[str]      # e.g., {"exits", "location"}

# Adding a new entity type is a 5-line declaration:
ROOM_CONFIG = EntityTypeConfig(
    type_name="room",
    required_components={"DescriptionComponent"},
    default_components={"ExtendedRoomComponent": {}},
    default_tags={"room"},
    allowed_top_level_fields={"exits", "zone"},
)

NPC_CONFIG = EntityTypeConfig(
    type_name="npc",
    required_components={"DescriptionComponent", "NPCComponent"},
    default_components={},
    default_tags={"npc"},
    allowed_top_level_fields={"location"},
)

class EntityAssembler:
    """Generic assembler that uses EntityTypeConfig to create entities."""

    def __init__(self, configs: dict[str, EntityTypeConfig]) -> None: ...

    def assemble(
        self, definition: EntityDefinition, config: EntityTypeConfig, world: World,
    ) -> Entity: ...

Content packs register additional type configs:

class MyContentPack:
    def get_entity_type_configs(self) -> list[EntityTypeConfig]:
        return [ROOM_CONFIG, NPC_CONFIG, ITEM_CONFIG]

Protocol extension (DD-25, updated): get_entity_type_configs() and get_semantic_rules() are optional extensions to the ContentPack protocol. They are NOT added to the base ContentPack protocol in maid-engine (which must remain content-agnostic). Instead, a DataLoaderPack(Protocol) with full method signatures is defined in maid-engine for type-safe opt-in. The loader uses isinstance() — not hasattr() — to detect packs that support data loading:
@runtime_checkable
class DataLoaderPack(Protocol):
    """Optional protocol extension for packs that use data loading."""

    def get_entity_type_configs(self) -> list[EntityTypeConfig]: ...
    def get_semantic_rules(self) -> list[SemanticRule]: ...
    def get_data_paths(self) -> list[Path]: ...
if isinstance(pack, DataLoaderPack):
    configs.update(pack.get_entity_type_configs())
    rules.extend(pack.get_semantic_rules())

6.2 Engine-Owned ReferenceRegistry¶

The ReferenceRegistry is owned by GameEngine, not created by the data loader. This eliminates the process-global singleton pattern and makes the system multi-world safe.

Concurrency safety: The ReferenceRegistry uses an asyncio.Lock to synchronize concurrent access during hot reload. A pipeline must acquire the lock before mutating the registry. Read-heavy paths (reference resolution) use a copy-on-write snapshot to avoid blocking:

class ReferenceRegistry:
    """Maps symbolic names to pre-assigned UUIDs. Thread/reload-safe."""

    def __init__(self) -> None:
        self._entries: dict[str, UUID] = {}
        self._lock: asyncio.Lock = asyncio.Lock()
        self._snapshot: MappingProxyType[str, UUID] = MappingProxyType({})

    async def register(self, name: str, entity_id: UUID) -> None:
        async with self._lock:
            self._entries[name] = entity_id
            # Eagerly rebuild snapshot under lock to prevent
            # readers from observing stale state (race fix).
            self._snapshot = MappingProxyType(dict(self._entries))

    def snapshot(self) -> Mapping[str, UUID]:
        """Return an immutable snapshot for lock-free reads.

        Always consistent — rebuilt eagerly on every register().
        """
        return self._snapshot

class GameEngine:
    def __init__(
        self,
        settings: Settings,
        template_registry: TemplateRegistry | None = None,
    ) -> None:
        self.world = World()
        self.reference_registry = ReferenceRegistry()   # Engine-owned
        # TemplateRegistry: abstract Protocol in maid-engine, concrete
        # implementation injected from maid-stdlib (or a default no-op).
        self.template_registry: TemplateRegistry = (
            template_registry or NullTemplateRegistry()
        )
        ...

    def create_data_pipeline(self, config: LoaderConfig) -> Pipeline:
        """Factory: creates a pipeline with injected shared resources."""
        context = LoaderContext(
            world=self.world,
            config=config,
            reference_registry=self.reference_registry,
            template_registry=self.template_registry,
        )
        return Pipeline(phases=DEFAULT_PHASES, context=context)

When maid-classic-rpg depends on maid-stdlib, the engine's ReferenceRegistry is already populated with stdlib's entity IDs before classic-rpg's pipeline runs. This enables cross-pack references:

# In classic-rpg data file:
_extends: "@ref:stdlib:templates/base_humanoid"

For multi-world scenarios, each GameEngine instance carries its own registry — no shared global state.

6.3 Builder Command Export (Definition-State Only)¶

The @export command serializes the definition layer, not runtime state:

@export rooms village --output data/rooms/village.yaml
@export npcs --zone village --output data/npcs/village_npcs.yaml

An NPC at 20/100 HP mid-combat exports with its original authored HP. Runtime state is managed by the instance layer (§8).

6.4 Batch Command Relationship¶

Aspect	`@batch`	Data Pipeline
Format	Imperative (command sequences)	Declarative (YAML schemas)
Use case	One-time setup, migrations	Persistent definitions
Reload	Re-execute commands	Incremental file reload
Audience	Builders / admins	Content creators

6.5 No-Code Content Pack Loading¶

Community content packs from untrusted sources must not require importing arbitrary Python code. A declarative entry point enables "No-Code" packs that consist entirely of data files and a TOML manifest:

# manifest.toml — No-Code content pack
[pack]
name = "community-tavern"
version = "1.0.0"
description = "A cozy tavern with NPCs, items, and quests."
no_code = true                    # Declarative-only mode

[pack.dependencies]
stdlib = "*"

[pack.data]
paths = ["data/"]                 # Directories to scan for YAML
entity_types = ["room", "npc", "item"]  # Allowed entity types (whitelist)

When no_code = true:

The engine creates a DeclarativeContentPack wrapper that implements ContentPack + DataLoaderPack without importing any Python modules from the pack.
Only standard EntityTypeConfig declarations from dependency packs are available — the pack cannot register custom assemblers, semantic rules, or systems.
The allowed_entity_types whitelist restricts which types can be defined (prevents a tavern pack from defining admin-level entities).
No on_load() / on_unload() callbacks execute.

class DeclarativeContentPack:
    """Auto-generated ContentPack for no-code data-only packs.

    Created by ContentPackLoader when manifest declares ``no_code = true``.
    No Python is imported from the pack directory.
    """

    def __init__(self, manifest: ContentPackManifest, data_paths: list[Path]) -> None: ...

    async def on_load(self, engine: GameEngine) -> None:
        pipeline = engine.create_data_pipeline()
        await pipeline.run_for_pack(self, data_paths=self._data_paths)

Security: No-Code packs are the recommended distribution format for community content. They receive all the security protections of §15 (path jail, resource limits, no aliases) without any Python execution surface. The entity_types whitelist is enforced during the SchemaRoutePhase — entities of unlisted types produce an ERROR.

7. Hot Reload¶

7.1 Strategy: Refuse-If-Active (v1) with Force-Reload Protocol¶

Design Decision (addresses P0-2 from critique): The drain-phase protocol is deferred to v2. The v1 reload strategy is simpler and safer: refuse to reload files with active entities.

$ maid data reload data/npcs/merchants.yaml

ERROR: Cannot reload — 2 entities from this file have active interactions:
  • merchant_elara (entity a1b2c3): Active trade session with player "Kira"
  • guard_aldric (entity d4e5f6): In combat with player "Thorn"

Options:
  • Wait for interactions to complete, then retry
  • Use --force to destroy entities immediately (DANGEROUS)
  • Use --skip-active to reload only inactive entities

Rationale: The drain-phase requires every game system (Combat, Trade, Dialogue) to implement drain handlers — a cross-cutting concern that touches the entire engine. Refuse-if-active is safe, simple, and covers the development workflow where hot reload matters most.

Force-Reload Protocol (`--force`)¶

When --force is used, the loader performs a graceful teardown:

Cancel active interactions: Each entity's active systems receive an on_entity_force_destroyed(entity_id, reason) callback. Systems must handle this gracefully (e.g., CombatSystem ends combat, TradeSystem cancels trade).
Notify connected players: Players interacting with the entity receive a notification: "[System] The merchant shimmers and reforms."
Destroy and recreate: The entity is destroyed and recreated from the updated definition.
Post-reload verify: References to the entity are re-validated.

@runtime_checkable
class ReloadableSystem(Protocol):
    """Optional protocol for systems that manage active interactions.

    Systems that hold active state (combat, trade, dialogue) implement
    this protocol to participate in force-reload teardown. The reload
    manager uses ``isinstance(system, ReloadableSystem)`` to discover
    capable systems — the base ``System`` ABC is NOT modified.
    """

    async def on_entity_force_destroyed(
        self, entity_id: UUID, reason: str,
    ) -> None:
        """Called before an entity is force-destroyed during reload.
        Systems should cancel interactions gracefully.
        """
        ...

Design note: on_entity_force_destroyed is NOT added to the base System ABC (which is content-agnostic). Only systems that manage stateful interactions need to implement ReloadableSystem. This avoids widening the core protocol for a reload-only concern.

7.2 Unload / Pack Deletion Strategy¶

The unload_pack() workflow removes a content pack's entities:

async def unload_pack(engine: GameEngine, pack_name: str) -> UnloadResult:
    """Remove all entities from a content pack."""
    ...

Workflow:

Identify entities: Find all entities with DataProvenanceComponent.pack_name == pack_name.
Check active references: If entities from OTHER packs reference this pack's entities, refuse with an error listing the dependents.
Check active interactions: Same as reload — refuse if active, unless --force.
Destroy entities: Remove from World.
Clean registries: Remove IDs from ReferenceRegistry and templates from TemplateRegistry.

$ maid data unload classic-rpg

ERROR: Cannot unload 'classic-rpg' — 3 entities are referenced by other packs:
  • @ref:classic-rpg:templates/base_warrior (used by: expansion-pack)

7.3 DataFileWatcher¶

class DataFileWatcher:
    """Watches data directories for changes and triggers reloads.

    Uses OS file events (watchfiles) with mtime+size+content_hash
    fast-path. Includes partial-write protection and re-entrancy guard.

    Re-entrancy safety: An ``asyncio.Lock`` prevents concurrent reloads.
    File change events arriving during an active reload are queued and
    processed after the current reload completes. This prevents
    inconsistent state from overlapping pipeline runs.

    Args:
        pipeline_factory: Callable to create a Pipeline for reload.
        watch_paths: Directories to watch.
        debounce_seconds: Minimum time between reloads (default: 2.0).
    """

    def __init__(self) -> None:  # ... args ...
        self._reload_lock: asyncio.Lock = asyncio.Lock()
        self._pending_changes: asyncio.Queue[set[Path]] = asyncio.Queue()
        ...

    async def _on_change(self, changed_paths: set[Path]) -> None:
        """Handle file change events with TOCTOU-safe coalescing.

        All events are unconditionally queued and processed under
        the reload lock. This eliminates the TOCTOU race where
        ``self._reload_lock.locked()`` could change between the
        check and the ``async with`` acquisition.
        """
        async with self._reload_lock:
            # Coalesce: drain any events queued during our wait
            all_changed = set(changed_paths)
            while not self._pending_changes.empty():
                all_changed |= await self._pending_changes.get()
            await self._reload(all_changed)

    async def start(self) -> None: ...
    async def stop(self) -> None: ...

Partial-write protection:

Editors write files non-atomically (write → truncate → flush). The watcher guards against loading half-written files:

Protection	Mechanism
Stability check	Require two consecutive polls (2.0s apart) with identical mtime+size and content hash before triggering reload. Content hash guards against partial-write scenarios where mtime/size stabilize before the write is complete.
Parse-before-destroy	Parse the new file successfully before destroying old entities
Last-good checksum	Track `last_good_checksum` per file; if parse fails, keep previous entities
Debounce	2.0s minimum between reloads (increased from 0.5s)

OS file events: Uses the watchfiles library (already a project dependency) for cross-platform file watching via Rust-based notify backend. Falls back to polling only when watchfiles is unavailable. Tracking mtime+size+content_hash as a fast-path avoids unnecessary full-file reparsing.

7.4 ReloadScope Extension¶

class ReloadScope(Enum):
    MODULE = auto()
    CONTENT_PACK = auto()
    SYSTEM = auto()
    DATA_FILES = auto()  # NEW
    ALL = auto()

7.5 UUID Stability¶

Pre-assigned UUIDs are deterministic: uuid5(NAMESPACE_DATA, f"{pack}:{scoped_id}"). This means the same _id always produces the same UUID, even across reloads. Systems holding UUID references (not Python object references) survive reload.

8. Definition vs. Instance Model¶

8.1 Problem¶

When a data-loaded entity is modified at runtime (NPC takes damage, merchant sells an item), which state takes priority on restart?

Startup Authority Boundary (cross-reference: Doc 01 — Durable Persistence): This document (Doc 04) owns the loading of defined entities — those with YAML source files as their canonical representation. Doc 01 owns the loading and persistence of dynamic entities — those created at runtime via builder commands, combat drops, player actions, etc., which have no YAML source file.

Entity Kind Canonical Source Loaded By Example

Defined YAML data file Doc 04 pipeline Town NPCs, rooms, quest items

Dynamic DocumentStore Doc 01 persistence Player-crafted items, spawned mobs

At startup, the load order is: (1) Doc 04 pipeline loads defined entities from YAML, then (2) Doc 01 persistence restores dynamic entities from the DocumentStore. Defined entities that have instance state in the DocumentStore use the state machine in §8.3 to reconcile. Dynamic entities are fully outside this document's scope.

8.2 Two-Layer Design¶

┌──────────────────────────────────────────────┐
│  Instance Layer (DocumentStore)              │
│  Runtime modifications: HP, inventory, pos   │
│  Priority: HIGH (takes precedence on load)   │
├──────────────────────────────────────────────┤
│  Definition Layer (YAML data file)           │
│  Author-defined base state: stats, desc,     │
│  initial position, template reference        │
│  Priority: LOW (baseline for new entities)   │
└──────────────────────────────────────────────┘

8.3 Load State Machine¶

The definition/instance decision logic is an explicit state machine with enum types. A pure function maps state to action — no implicit decision matrix.

class EntityLoadState(Enum):
    """Observed state of an entity at load time."""

    NEW = "new"                       # No instance, first load
    INSTANCE_CLEAN = "instance_clean" # Instance exists, definition unchanged
    INSTANCE_STALE = "instance_stale" # Instance exists, definition changed
    INSTANCE_CORRUPT = "instance_corrupt"  # Instance fails validation
    RESET_REQUESTED = "reset_requested"    # --reset flag active

class EntityLoadAction(Enum):
    """Action to take for an entity."""

    CREATE_FROM_DEFINITION = "create_from_definition"
    LOAD_FROM_INSTANCE = "load_from_instance"
    LOAD_INSTANCE_WARN_STALE = "load_instance_warn_stale"
    QUARANTINE_LOAD_DEFINITION = "quarantine_load_definition"
    DELETE_INSTANCE_RELOAD = "delete_instance_reload"

State → Action mapping (pure function):

State	`--reset`?	Action
`NEW`	—	`CREATE_FROM_DEFINITION`
`INSTANCE_CLEAN`	No	`LOAD_FROM_INSTANCE`
`INSTANCE_CLEAN`	Yes	`DELETE_INSTANCE_RELOAD`
`INSTANCE_STALE`	No	`LOAD_INSTANCE_WARN_STALE`
`INSTANCE_STALE`	Yes	`DELETE_INSTANCE_RELOAD`
`INSTANCE_CORRUPT`	—	`QUARANTINE_LOAD_DEFINITION`
`RESET_REQUESTED`	—	`DELETE_INSTANCE_RELOAD`

def resolve_load_action(state: EntityLoadState, reset: bool) -> EntityLoadAction:
    """Pure function: maps observed state to action. No side effects."""
    if state == EntityLoadState.INSTANCE_CORRUPT:
        return EntityLoadAction.QUARANTINE_LOAD_DEFINITION
    if reset or state == EntityLoadState.RESET_REQUESTED:
        return EntityLoadAction.DELETE_INSTANCE_RELOAD
    ...

Corrupt instance handling: When an instance fails Pydantic validation (e.g., incompatible schema change, data corruption), the loader:

Moves the corrupt instance to a _quarantine collection in DocumentStore
Logs a WARNING with the entity ID, corruption details, and quarantine path
Creates the entity from the current definition instead
The quarantined data is available for manual inspection/recovery

Quarantine cap: The _quarantine collection is bounded by LoaderConfig.max_quarantine_entries (default: 500). When the cap is reached, the oldest quarantined entries are evicted (FIFO). An INFO diagnostic is emitted when entries are evicted.

Three-way merge deferred to v2: The v1 system offers only two modes — "always load definition" and "always load instance." Field-level dirty tracking is required for intelligent merging, and this does not yet exist in the engine (addresses P2-2 from critique).

8.4 Provenance and Instance State Components¶

The original DataSourceComponent is split into two components with distinct mutability semantics:

class DataProvenanceComponent(Component):
    """Immutable record of an entity's origin in the data loading system.

    This component is set once at load time and never modified at runtime.
    It tracks WHERE an entity came from.
    """

    source_file: str              # Relative path to source YAML
    pack_name: str                # Owning content pack name
    definition_id: str            # The _id from the data file
    definition_hash: str          # SHA-256 of canonicalized definition (see below)
    definition_version: str = "1.0"
    loaded_at: datetime           # Timestamp of load

Definition hash canonicalization (DD-24): The definition_hash is computed from a canonicalized dict representation, not the raw YAML text. The parsed dict is sorted recursively by key, serialized to JSON with sort_keys=True and ensure_ascii=True, then SHA-256 hashed. This makes the hash semantically stable — reformatting YAML (changing indentation, quoting style, comment content, key order) does not change the hash, preventing spurious stale detections on cosmetic edits.

class InstanceStateComponent(Component):
    """Mutable tracking of runtime modifications to a data-loaded entity.

    Dirty tracking is **automatically enforced** — systems do not need to
    remember to mark entities as modified. The engine provides two
    mechanisms:

    1. ``World.set_component(entity, component)`` auto-sets ``modified=True``
       and appends the component type to ``modified_components`` when the
       target entity has an ``InstanceStateComponent``.
    2. **Pydantic ``model_validator(mode='wrap')``** on tracked components
       intercepts field mutations and marks dirty automatically. This
       replaces the original ``Component.__setattr__`` hook design, which
       is incompatible with Pydantic's ``__setattr__`` machinery. The
       ``TrackedComponent`` mixin applies the validator:

       ```python
       class TrackedComponent(Component):
           """Mixin that auto-marks InstanceStateComponent dirty on mutation."""

           @model_validator(mode="wrap")
           @classmethod
           def _track_mutation(cls, values: Any, handler: Any) -> Any:
               instance = handler(values)
               # Notification dispatched via World.component_mutated()
               return instance
       ```

       Components opt in by inheriting from ``TrackedComponent`` instead
       of ``Component``. The data loader automatically applies this mixin
       to all components on entities that have an ``InstanceStateComponent``.

    Manual opt-in via ``instance_state.mark_modified(component_type)``
    remains available for batch operations where the validator is
    bypassed for performance.
    """

    modified: bool = False
    modified_components: set[str] = Field(default_factory=set)
    last_modified_at: datetime | None = None
    modified_by: str | None = None   # System or player that modified

    def mark_modified(self, component_type: str, actor: str | None = None) -> None:
        """Explicitly mark a component as modified."""
        self.modified = True
        self.modified_components.add(component_type)
        self.last_modified_at = datetime.now(tz=UTC)
        if actor:
            self.modified_by = actor

Rationale: Immutable provenance data (where did this entity come from?) is conceptually distinct from mutable instance tracking (has this entity been changed?). Separating them:

Prevents accidental mutation of provenance during runtime
Allows efficient queries: "find all entities from file X" uses only the immutable component
Makes serialization cleaner — provenance is mostly static, but definition_hash MUST be persisted to enable cross-restart stale detection (comparing the stored hash against the current definition hash to determine INSTANCE_CLEAN vs INSTANCE_STALE in the load state machine)

8.5 Atomic Load with Rollback (Phase 9)¶

Phase 9 (Instantiate) wraps entity creation in a LoadTransaction to ensure atomicity — either all entities from a file are created successfully, or none are.

Engine prerequisite: World.create_entity() does not accept a suppress_events parameter, and EventBus.emit() is async. The LoadTransaction uses a staging pattern: entities are assembled in a detached EntityManager (not connected to the live EventBus), then bulk-transferred into the World on commit(). This avoids requiring engine API changes and naturally suppresses events during the transaction.

class LoadTransaction:
    """Atomic entity creation using a staging EntityManager.

    Entities are assembled in a detached staging area. On ``commit()``,
    they are bulk-transferred into the live ``World`` and creation events
    are emitted. On ``rollback()``, the staging area is discarded — no
    side effects reach the live world.

    Both ``commit()`` and ``rollback()`` are ``async`` because
    ``EventBus.emit()`` is async.
    """

    def __init__(self, world: World) -> None:
        self._world = world
        self._staging: EntityManager = EntityManager()  # Detached from EventBus
        self._staged_ids: list[UUID] = []
        ...

    def stage_entity(self, entity_id: UUID, **kwargs: Any) -> Entity:
        """Create entity in the staging area (no events fired)."""
        entity = self._staging.create(entity_id=entity_id, **kwargs)
        self._staged_ids.append(entity_id)
        return entity

    def verify_references(self) -> list[LoadError]:
        """Post-staging: verify all @ref: targets actually exist."""
        ...

    async def commit(self) -> None:
        """Transfer staged entities to live World, then emit events."""
        for entity_id in self._staged_ids:
            entity = self._staging.get(entity_id)
            self._world.adopt_entity(entity)
        for entity_id in self._staged_ids:
            await self._world.event_bus.emit(
                EntityCreatedEvent(entity_id=entity_id),
            )
        self._staged_ids.clear()

    async def rollback(self) -> None:
        """Discard staged entities. No events, no side effects."""
        self._staging.clear()
        self._staged_ids.clear()

Engine prerequisites for staging pattern:

EntityManager: A lightweight container for entities without event dispatch. May be extracted from World internals or implemented as a new class in maid_engine.core.ecs.

World.adopt_entity(entity): Transfers an entity from an external EntityManager into the live world. Must be implemented as a new method on World.

These are scoped as Sprint 0 / Phase 1 deliverables in the implementation plan (§18).

Workflow:

Begin transaction (create staging EntityManager)
Stage all entities for the current file/pack
Run post-staging reference integrity check (verify that all @ref: targets resolved to entities that actually exist in the staging area or the live World)
If integrity check passes → await commit()
If any check fails → await rollback() and report errors

This prevents the "half-loaded pack" failure mode where some entities reference others that failed to create.

9. Error Reporting for Content Authors¶

Error messages are designed for non-programmer content creators. Every error includes the file, line, field path, a plain-English explanation, and (where possible) a suggested fix.

9.1 Error Message Format¶

[SEVERITY] file:line — field_path
  Problem description in plain English.
  Suggestion: how to fix it.

9.2 Example Error Output¶

$ maid data validate data/

Validating 12 files in data/...

ERROR data/npcs/merchants.yaml:28 — components.HelthComponent
  Unknown component type 'HelthComponent'.
  Did you mean 'HealthComponent'?

ERROR data/npcs/merchants.yaml:35 — components.HealthComponent.current
  Value 200 exceeds maximum (100). Current HP cannot exceed max HP.
  Set 'current' to a value ≤ 'maximum', or increase 'maximum'.

WARNING data/rooms/village.yaml:12 — exits
  Room 'old_well' has no exits. Players cannot leave this room.
  Add at least one exit, or mark the room as a dead-end with
  tag 'no_exit' to suppress this warning.

ERROR data/items/weapons.yaml:45 — location
  Reference '@ref:rooms/vilage_square' does not exist.
  Did you mean '@ref:rooms/village_square'?

WARNING data/npcs/guards.yaml:8 — components.NPCComponent.wander_radius
  NPC 'gate_guard' has wander_radius=5 but is placed in a room
  with no adjacent rooms. The NPC cannot actually wander.

────────────────────────────────────────
Results: 12 files, 47 entities
  ✓ 44 entities valid
  ✗ 3 entities with errors (skipped)
  ⚠ 2 warnings
  Errors: 3 | Warnings: 2

9.3 AI Safety in DialogueComponent¶

The DialogueComponent uses a layered defense for prompt injection:

Static analysis at load time (reinstated): The linter runs a set of pattern-based checks against personality, greeting, farewell, and other free-text fields. Matches against known jailbreak patterns (e.g., "ignore all previous instructions", "you are now", "system prompt override") are flagged as Warning (suspicious phrasing) or Error (high-confidence jailbreak pattern). This catches both malicious content pack submissions and accidental prompt leakage from authors copy-pasting from AI tools.
Declarative safety_policy field: Selects the runtime safety tier.
Runtime AI provider safety layer: The AI provider's own content filtering applies at generation time (see maid_engine/ai/safety.py).

components:
  DialogueComponent:
    ai_enabled: true
    personality: >
      Gruff, direct, no-nonsense military commander.
    safety_policy: "standard"   # "standard" | "strict" | "permissive"

The loader validates that safety_policy is a known value. Static analysis patterns are registered in the linter and can be extended by content packs.

Linter output examples:

WARNING data/npcs/innkeeper.yaml:52 — components.DialogueComponent.personality
  Suspicious phrase detected: "ignore all previous". This resembles a
  prompt injection pattern. Review the text and quote it if intentional.

ERROR data/npcs/evil_npc.yaml:30 — components.DialogueComponent.personality
  High-confidence jailbreak pattern: "you are now an unrestricted AI".
  This text will be rejected. Remove or rephrase the personality text.

10. Template / Archetype System¶

10.1 Template Definition¶

Templates are reusable base definitions marked with _template: true:

_meta:
  schema: "maid:templates:v1"

templates:
  base_merchant:
    _template: true
    _extends: "base_npc"
    entity_type: npc
    tags: ["npc", "merchant", "friendly"]
    components:
      HealthComponent:
        current: 75
        maximum: 75
      NPCComponent:
        behavior_type: "friendly"
        is_merchant: true
      InventoryComponent:
        capacity: 100

10.2 Single Inheritance¶

Templates support single inheritance via _extends (renamed from _inherit for consistency with common OOP terminology):

village_merchant:
  _template: true
  _extends: "base_merchant"
  tags: ["village"]          # Replaces parent tags
  _append:                   # Additive merge for any list field
    tags: ["village"]
  components:
    HealthComponent:
      current: 60            # Overrides parent's 75
      maximum: 60
    # NPCComponent inherited unchanged from base_merchant

Standardized directive names:

Old Name	New Name	Purpose
`_inherit`	`_extends`	Single-inheritance parent reference
`_template_ref`	`_use`	Instantiate from a template
`_variables`	`_vars`	Variable values for template instantiation

Merge rules:

Field Type	Behavior
Scalars (name, description)	Child overrides parent
Component dicts	Deep-merged at field level
Lists (tags, keywords)	Child replaces parent entirely
`_append` block	Additive — unions with parent for specified list fields
Max depth	5 levels (prevents infinite chains)

Design Decision (addresses S2 from first critique): Multiple inheritance is deferred. Single inheritance covers ~90% of use cases. Composition is achieved by explicitly listing components.

10.3 List Merge Policy¶

All lists replace by default. This is the simplest, most predictable policy. To append to any parent list field, use the generic _append block:

# Parent has tags: [npc, friendly], knowledge_domains: ["village"]
child:
  # Replace entirely:
  tags: [npc, friendly, merchant]

  # Or append selectively to any list field:
  _append:
    tags: [merchant]
    knowledge_domains: ["trade", "prices"]

The _append block supports all list fields, not just tags. Each key in _append must correspond to a list field on the entity or component. Values are unioned with the parent's values (duplicates removed).

10.4 Variable Substitution¶

guard_template:
  _template: true
  _vars:
    title:
      type: str
      required: true
    max_hp:
      type: int
      default: 100
  components:
    DescriptionComponent:
      name: "${title}"
    HealthComponent:
      current: "${max_hp}"
      maximum: "${max_hp}"

# Instantiation:
north_gate_guard:
  _use: guard_template
  _vars:
    title: "North Gate Guard"
    max_hp: 120

Type inference for defaults:

Default Pattern	Type	Example
Integer literal	`int`	`${hp:100}` → `100`
Float literal	`float`	`${speed:1.5}` → `1.5`
`true` / `false`	`bool`	`${mortal:true}` → `True`
Everything else	`str`	`${name:Guard}` → `"Guard"`

10.5 Template Debugging¶

$ maid content resolve base_merchant

Resolved template: base_merchant
  Inheritance chain: base_npc → base_merchant
  Effective definition:
    entity_type: npc
    tags: [npc, merchant, friendly]
    components:
      HealthComponent:
        current: 75        # overridden (parent: 50)
        maximum: 75        # overridden (parent: 50)
      NPCComponent:
        behavior_type: "friendly"  # defined here
        is_merchant: true          # defined here
        respawn_time: 300.0        # inherited from base_npc

11. Reference Resolution¶

11.1 Syntax¶

Syntax	Description	Example
`@ref:type/name`	By type and scoped name	`@ref:rooms/village_square`
`@ref:pack:type/name`	Cross-pack namespaced	`@ref:stdlib:templates/base_npc`

Design Decision (addresses P2-4 from second critique): The # prefix for name lookups and the $ prefix for templates are removed. All name-based references use the @ref:type/name syntax. Direct UUID references are rarely needed and use the full @ref:uuid:<uuid> form.

11.2 Pre-Assigned UUIDs¶

Canonical identity (cross-doc): UUIDs are the canonical identity mechanism across the entire MAID engine, not just the data loader. Entity IDs, reference resolution, DocumentStore keys, and hot-reload stability all use uuid.UUID. This document's pre_assign_uuid() function produces UUIDs that are fully interoperable with engine-created entities. Doc 01 (Durable Persistence) uses the same UUID type for dynamic entities. No secondary identity scheme exists.

UUIDs are deterministic and stable across reloads:

import uuid
NAMESPACE_DATA = uuid.UUID("a1b2c3d4-e5f6-7890-abcd-ef1234567890")

def pre_assign_uuid(pack_name: str, scoped_id: str) -> uuid.UUID:
    return uuid.uuid5(NAMESPACE_DATA, f"{pack_name}:{scoped_id}")

This means @ref:rooms/village_square in pack tutorial-world always resolves to the same UUID, regardless of file processing order.

11.3 Reference Scope¶

References are resolved against:

The current content pack's registered IDs (Phase 5)
IDs from dependency packs (populated before current pack loads)
The engine-owned ReferenceRegistry (injected via LoaderContext)

12. Component Schema Catalog¶

12.1 Quick Reference¶

Entity Type	Typical Components
Room	`DescriptionComponent`, `ExtendedRoomComponent`
NPC	`DescriptionComponent`, `NPCComponent`, `HealthComponent`, `CombatComponent`, `DialogueComponent`, `StatsComponent`, `InventoryComponent`
Merchant	All NPC components + `GoldComponent`
Item (weapon)	`DescriptionComponent`, `ItemComponent`, `EquipmentComponent`
Item (consumable)	`DescriptionComponent`, `ItemComponent`

12.2 Common Components¶

HealthComponent¶

components:
  HealthComponent:
    current: 100              # int — required (Field(ge=0, le=100_000))
    maximum: 100              # int — required (Field(ge=1, le=100_000), must be ≥ current)
    regeneration_rate: 1.0    # float (default: 1.0, Field(ge=0.0, le=100.0))

DescriptionComponent¶

components:
  DescriptionComponent:
    name: "Iron Shortsword"           # str — required (Field(min_length=1, max_length=200))
    short_desc: "an iron shortsword"  # str (default: "", Field(max_length=500))
    long_desc: >                      # str (default: "", Field(max_length=10_000))
      A well-forged shortsword with a
      double-edged iron blade.
    keywords:                         # list[str] (default: [], max 50 items, each max_length=50)
      - sword
      - iron

NPCComponent¶

components:
  NPCComponent:
    behavior_type: "passive"    # passive | friendly | aggressive | patrol
    respawn_time: 300.0         # float (default: 300.0, Field(ge=0.0, le=86_400.0))
    wander_radius: 0            # int (default: 0, Field(ge=0, le=100))
    faction_id: null            # str | null (Field(max_length=100))
    is_merchant: false          # bool (default: false)
    is_quest_giver: false       # bool (default: false)

DialogueComponent¶

components:
  DialogueComponent:
    ai_enabled: true
    personality: >                                      # str (Field(max_length=2000))
      Warm and motherly. Loves village gossip.
    speaking_style: "casual"                            # str (Field(max_length=200))
    knowledge_domains: ["village history", "local rumors"]  # list[str] (max 20 items, each max_length=200)
    secret_knowledge: ["Knows about the hidden cave"]       # list[str] (max 20 items, each max_length=500)
    will_discuss: ["food", "lodging", "rumors"]             # list[str] (max 50 items, each max_length=100)
    wont_discuss: ["politics"]                              # list[str] (max 50 items, each max_length=100)
    greeting: "Hello there, traveler!"                      # str (Field(max_length=500))
    farewell: "Safe travels!"                               # str (Field(max_length=500))
    max_response_tokens: 200    # int | null (Field(ge=50, le=1000))
    temperature: 0.7            # float | null (Field(ge=0.0, le=1.0))

ExtendedRoomComponent¶

components:
  ExtendedRoomComponent:
    descriptions:
      base_description: "A cobblestone square with a central fountain."
      time_variants:
        morning: "Golden morning light bathes the square."
        evening: "Lanterns flicker to life around the fountain."
      time_mode: "append"       # "append" | "replace"
      season_variants:
        spring: "Cherry blossoms drift through the air."
        winter: "Frost covers the cobblestones."
      weather_effects:
        rain: "Rain patters on the cobblestones."
      random_details:
        - text: "A stray cat slinks between the market stalls."
          weight: 0.3
      mood: "peaceful"

Enum values: - TimeOfDay: dawn, morning, noon, afternoon, dusk, evening, night, midnight - Season: spring, summer, autumn, winter - Weather: clear, cloudy, rain, storm, snow, fog, wind

ExitDefinition (Room `exits` Field)¶

The exits field on room entities uses a structured Pydantic model:

class ExitDefinition(BaseModel):
    """Schema for room exit declarations in data files."""

    direction: str = Field(max_length=50)        # e.g., "north", "up", "portal"
    destination: str = Field(max_length=200)      # @ref: string resolved in Phase 8
    description: str = Field(default="", max_length=500)
    hidden: bool = False
    locked: bool = False
    key_id: str | None = Field(default=None, max_length=200)  # @ref: to key item

exits:
  - direction: north
    destination: "@ref:rooms/village_square"
  - direction: east
    destination: "@ref:rooms/armory"
    description: "A heavy iron door leads east."
    locked: true
    key_id: "@ref:items/armory_key"

12.3 Runtime-Only Components (Not for Data Files)¶

Component	Why Runtime-Only
`CorpseComponent`	Created by combat system on death
`GhostComponent`	Created alongside corpse for player deaths
`RestStateComponent`	Managed by rest system

13. Testing Strategy¶

13.1 Unit Tests¶

Area	Tests
Phase protocol	Each phase tested in isolation with mock `LoaderContext`
Pipeline composition	CLI subsets (lint, validate, load) produce correct phase sequences
Schema parsing	`SchemaRef.parse()`, `DataFileMeta.model_validate()`
Validation	Component validation, unknown types, missing `_meta`, implicit schema inference
References	Symbolic resolution, pre-assigned UUIDs, UUID collisions, `_uuid` overrides
Templates	`_extends`, deep merge, circular detection, max depth, `_append`
Linting	MAID-Y001 through MAID-Y012
Semantic rules	Each `SemanticRule` tested individually; `--skip-rule` integration
EntityAssembler	Entity creation from `EntityTypeConfig` declarations
State machine	`resolve_load_action()` for all `EntityLoadState` × `--reset` combinations
LoadTransaction	Commit and rollback with reference integrity verification
Security	Resource limits, path jail, alias rejection, symlink handling

13.2 Integration Tests¶

Full pipeline tests loading actual data files:

tests/fixtures/data/
    valid/rooms.yaml
    valid/npcs.yaml
    valid/items.yaml
    valid/templates.yaml
    invalid/missing_meta.yaml
    invalid/circular_extends.yaml
    cross_ref/rooms.yaml + npcs.yaml
    large/stress_1000.yaml

13.3 Property-Based Tests¶

Using Hypothesis to verify: - Deep merge is associative and preserves all parent keys - Reference resolution is idempotent - Template resolution terminates for any valid chain

13.4 Performance Benchmarks¶

Scenario	Target
100 entities	< 200 ms
1,000 entities	< 2 seconds
10,000 entities	< 20 seconds
Incremental reload (1 file)	< 500 ms

14. Performance¶

14.1 Benchmark Expectations¶

Scale	Total Entities	Target Time	Memory
Small	350	< 0.5s	~50 MB
Medium	3,500	< 2s	~200 MB
Large	17,000	< 8s	~800 MB
Massive	40,000	< 20s	~2 GB

14.2 Optimization Strategy (v1)¶

ruamel.yaml C extension (ruamel.yaml.clib): ~2× faster than pure Python mode. Installed automatically if a C compiler is available.
Sequential parsing with ThreadPoolExecutor for I/O: Avoids ProcessPoolExecutor pickle serialization issues with CommentedMap.
Checksum-based incremental reload: Only re-parse changed files.
Release parsed dicts after entity creation: GC reclaims immediately.

Design Decision (addresses P0-3): ProcessPoolExecutor is not used. ruamel.yaml's CommentedMap objects are not safely picklable, and the process spawn overhead exceeds parsing time for < 5,000 entities. ThreadPoolExecutor is used instead.

14.3 Compiled Binary Cache (Deferred to v2)¶

Scope change (DD-23): The compiled binary cache is deferred to v2. The JSON cache (DD-8) is sufficient for v1 workloads. This section documents the v2 design for reference.

For large worlds, a msgpack + zstd compiled cache enables skip-to-instantiate when data files are unchanged:

@dataclass
class CompiledCache:
    """Binary cache of parsed and validated entity definitions."""

    cache_dir: Path                    # .maid_cache/ in pack root
    format: str = "msgpack+zstd"       # Compact binary + compression

    def is_valid(self, source_files: list[Path]) -> bool:
        """Check if cache is newer than all source files."""
        ...

    def load(self) -> list[EntityDefinition]:
        """Deserialize cached definitions (skip phases 1-7)."""
        ...

    def save(self, definitions: list[EntityDefinition]) -> None:
        """Serialize validated definitions to cache."""
        ...

Cache invalidation: The cache stores a manifest of source file paths and their SHA-256 hashes. If any source file changes, the entire cache is invalidated. The cache file is ignored by version control (.gitignore).

Security: msgpack cannot execute arbitrary code (unlike pickle). The cache is local-only and never distributed with content packs.

JSON cache remains as fallback: The JSON cache format (DD-8) is retained for debugging and environments where msgpack is unavailable. The binary cache is a transparent acceleration layer.

14.3 Memory Per Entity¶

Component	Size
`Entity` object + metadata	~400 bytes
`DescriptionComponent`	~600 bytes
`HealthComponent`	~100 bytes
Average entity (4 components)	~1.5 KB
10,000 entities	~15 MB

15. Security Model¶

The data loader processes untrusted content (community content packs). The following protections apply to all phases.

15.1 Resource Limits and Bomb Protection¶

Limit	Default	Rationale
`max_file_size`	10 MB	Prevents memory exhaustion from giant files
`max_entities_per_file`	1,000	Bounds per-file processing time
`max_components_per_entity`	50	Prevents entity bloat
`max_template_depth`	5	Already enforced; prevents stack overflow
`max_ref_chain_depth`	20	Already enforced for circular references
`max_errors_per_file`	50	Prevents DoS from malformed files flooding logs
`max_cascade_depth`	10	Caps cascade skip chains in lenient mode
`max_quarantine_entries`	500	Bounds quarantine collection growth
`semantic_rule_timeout`	5s	Prevents hung semantic rules from blocking pipeline
`pipeline_timeout`	60s	Prevents runaway pipelines from blocking engine startup
`fuzzy_match_max_candidates`	5,000	Prevents pathological fuzzy matching on large registries

YAML alias/anchor expansion is disabled. The ruamel.yaml parser is configured to reject YAML aliases (*anchor / &anchor), preventing billion-laughs-style expansion attacks:

yaml = YAML(typ="safe")
yaml.version = (1, 2)
yaml.allow_duplicate_keys = False
# Aliases are not supported in safe mode — no explicit config needed

Variable substitution is single-pass. The ${var} expansion does NOT recursively expand — a variable's value is always a literal. This prevents recursive expansion bombs like ${a} → ${b} → ${a} → ....

15.2 File Discovery Hardening¶

Threat	Protection
Symlink escape	Symlinks are ignored during discovery; only regular files are processed
Path traversal	All resolved paths are jail-checked against the content pack root; `../../etc/passwd` is rejected
Hidden files	Files starting with `.` are skipped (except `_meta.yaml`)
TOCTOU symlink swap	`is_safe_path()` re-verified in `ParsePhase` immediately before `open()`

def is_safe_path(pack_root: Path, candidate: Path) -> bool:
    """Verify candidate is inside pack_root after symlink resolution."""
    resolved = candidate.resolve()
    return resolved.is_relative_to(pack_root.resolve())

15.3 Content Safety¶

No pickle anywhere. Cache uses msgpack (binary) or JSON. Never pickle.
No yaml.full_load(). Always typ="safe".
No code execution in data files. YAML custom tags that invoke Python constructors are not supported.
AI safety: Layered defense — static pattern analysis at load time flags suspicious prompt injection patterns as Warning/Error (see §9.3), with runtime content filtering by the AI provider as a second layer.

16. Open Questions¶

Q1: Should templates support multiple inheritance?¶

Recommendation: Defer. Single inheritance + explicit component listing covers ~90% of use cases. Add _mixins syntax in v2 if demand warrants.

Q2: Should the loader support conditional loading?¶

Recommendation: Defer. Add _meta.conditions in v2 for seasonal content, difficulty modes, and feature flags.

Q3: Should variable substitution use Jinja2?¶

Recommendation: No. The ${var:default} system handles parameterization without adding a heavy dependency.

Q4: How should cross-pack references be namespaced?¶

Decision: Explicit prefix: @ref:tutorial-world:rooms/village_square. Import declarations as syntactic sugar are a v2 feature.

Q5: Should quality variants be a first-class feature?¶

Recommendation: Store variants as metadata. A QualitySystem creates concrete instances at runtime.

Q6: Should `_id` use `!ref` YAML tags instead of `@ref:` strings?¶

Decision: Keep @ref: strings. Custom tags are not supported by JSON Schema validators or the Red Hat YAML VS Code extension.

Q7: Unify data loader and builder TemplateRegistry?¶

Decision: Yes — unify, but with layered ownership. An abstract TemplateRegistry interface (Protocol) lives in maid-engine, owned by GameEngine and injected via dependency injection. The concrete implementation remains in maid-stdlib (where it currently exists in commands/building/create.py). Both data loader pipelines and builder commands (@create) receive the same registry instance via GameEngine. This eliminates the confusion of two parallel template systems while preserving the package layering constraint that maid-engine contains no game-specific logic.

Q8: How should large worlds handle lazy zone loading?¶

Recommendation: Defer to v2. When implemented, only "eager" zones (spawn, tutorial) load at startup; others load as players approach. Cross-zone references resolve to pre-assigned UUIDs even for unloaded zones.

17. Design Decisions Log¶

This section records key decisions and their rationale, including responses to the two design critique rounds.

DD-1: `ruamel.yaml` as sole parser (no PyYAML fallback)¶

Source: P0-1 from second critique Decision: Hard dependency on ruamel.yaml. No fallback. Rationale: A single YAML semantics eliminates the "Two YAML" problem where content validated with YAML 1.2 fails silently under YAML 1.1.

DD-2: Refuse-if-active hot reload (no drain phase in v1)¶

Source: C1 from first critique, P0-2 from second critique Decision: v1 refuses to reload files with active entities. Drain phase deferred to v2. Rationale: Drain phase requires every game system to implement handlers. Refuse-if-active is safe and covers the development workflow.

DD-3: Single inheritance only¶

Source: S2 from first critique Decision: No multiple inheritance. Single _extends field. Rationale: Multiple inheritance creates debugging nightmares for non-programmer content creators. Single inheritance + composition is clear.

DD-4: Directory-scoped `_id` values¶

Source: S3 from first critique Decision: IDs are scoped to directory path within the pack. Rationale: Prevents namespace collisions in large content packs without requiring verbose naming conventions.

DD-5: Pre-assigned deterministic UUIDs¶

Source: C4 from first critique Decision: uuid5(NAMESPACE, f"{pack}:{scoped_id}") before entity creation. Rationale: Eliminates chicken-and-egg ordering. References resolve to real UUIDs that Pydantic can validate before any entity exists.

DD-6: Definition-state export (not round-trip)¶

Source: S4 from first critique Decision: Export captures authored state, not runtime state. Rationale: True round-trip is impossible (runtime fields, UUID stability, template provenance). Definition-state export is achievable and useful.

DD-7: `extra="forbid"` on entity definitions¶

Source: S6 from first critique Decision: Entity definitions reject unknown fields in strict mode. Rationale: Prevents silent typos like compnents: that create entities with missing components.

DD-8: JSON cache format (not pickle)¶

Source: P1-3 from second critique Decision: Definition cache uses JSON, not pickle. Rationale: pickle.load() executes arbitrary code. Community content packs could exploit this for RCE.

DD-9: ThreadPoolExecutor (not ProcessPoolExecutor)¶

Source: P0-3 from second critique Decision: Parallel parsing uses threads, not processes. Rationale: ruamel.yaml's CommentedMap objects are not picklable. Process spawn overhead exceeds parsing time for typical workloads.

DD-10: VS Code extension is a separate project¶

Source: P1-5 from second critique Decision: VS Code extension extracted from this design document. Rationale: A VS Code extension with live preview, Go to Definition, and Find References is 4–6 weeks of work alone — a separate product with its own release cycle.

DD-11: Migration tool is "best effort / alpha"¶

Source: P1-4 from second critique Decision: Runtime interception migration is documented as alpha quality. Rationale: Monkey-patching World.create_entity() is fragile. The tool is a developer aid, not a supported feature.

DD-12: Schema evolution is append-only¶

Source: P2-3 from second critique Decision: Schemas are append-only (new fields get defaults, old fields never removed). Rationale: Avoids need for migration functions between schema versions. Breaking changes require a new version number.

DD-13: Localization at render layer (not component mutation)¶

Source: P2-5 from second critique Decision: Locale translations are applied at the rendering layer, not by mutating component fields in place. Rationale: In-place mutation (setattr of str → TranslatableText) breaks Pydantic model contracts, serialization, and type checking.

DD-14: Semantic validator phase added¶

Source: P1-1 from second critique Decision: Added SemanticValidator between Phase 6 and Phase 7. Rationale: Pydantic validates structure but not gameplay logic. current > maximum passes schema validation but produces broken entities.

DD-15: Sprint 0 before full implementation¶

Source: P2-6 from second critique Decision: Begin with a Sprint 0 that loads existing tutorial-world data. Rationale: Validates YAML format decisions against real content before building the full pipeline.

DD-16: Composable Phase pipeline (replaces DataLoader monolith)¶

Source: Review round 3, critical fix #1 Decision: The loader pipeline is decomposed into standalone Phase objects coordinated by a Pipeline. Each phase is independently testable. Rationale: A monolithic DataLoader class is hard to test, hard to compose for CLI tools, and hard to extend. Phase objects enable maid data lint to run only [Discover, Lint] without instantiating the full pipeline.

DD-17: Engine-owned ReferenceRegistry (not global singleton)¶

Source: Review round 3, critical fix #2 Decision: ReferenceRegistry is created and owned by GameEngine, injected into pipelines via LoaderContext. Rationale: A process-global singleton breaks multi-world scenarios and makes testing difficult. Engine ownership follows dependency injection.

DD-18: Split DataSourceComponent into provenance + instance state¶

Source: Review round 3, critical fix #3 Decision: Immutable DataProvenanceComponent + mutable InstanceStateComponent. Rationale: Provenance (where from?) and instance tracking (changed?) have different mutability semantics and query patterns.

DD-19: Typed SemanticRule protocol (replaces lambda rules)¶

Source: Review round 3, critical fix #4 Decision: Semantic rules are typed objects with id, description, severity, applies_to, and check(entity, context). Rationale: Lambda rules can't be discovered, listed, skipped by CLI, or optimized by entity type filtering.

DD-20: EntityAssembler with EntityTypeConfig (replaces SchemaHandlers)¶

Source: Review round 3, architecture improvement #5 Decision: A single generic EntityAssembler uses lightweight EntityTypeConfig declarations instead of per-type handler classes. Rationale: Adding a new entity type should be a 5-line config declaration, not a new class file.

DD-21: Unified TemplateRegistry¶

Source: Review round 3, architecture improvement #7 Decision: Abstract TemplateRegistry Protocol in maid-engine; concrete implementation in maid-stdlib. GameEngine owns the instance via DI. Rationale: Two parallel template systems create confusion. An abstract interface in maid-engine preserves package layering (engine has no game logic) while the concrete implementation lives in maid-stdlib where the existing code already resides.

DD-22: Atomic load with rollback¶

Source: Review round 3, reliability improvement #9 Decision: Phase 9 wraps entity creation in LoadTransaction with staging EntityManager and async commit/rollback semantics. Rationale: Prevents half-loaded packs where some entities reference others that failed to create. Staging pattern avoids requiring a suppress_events parameter on World.create_entity().

DD-23: Binary cache format (msgpack+zstd) — deferred to v2¶

Source: Review round 3, performance improvement #23; review round 5 scope review Decision: Compiled binary cache using msgpack+zstd deferred to v2. JSON retained as only cache format in v1. Rationale: Binary cache is a performance optimization, not a correctness requirement. v1 scope focuses on pipeline correctness. JSON cache (DD-8) is sufficient for typical workloads (< 5,000 entities).

DD-24: Semantic definition hash (canonicalized dict, not raw YAML)¶

Source: Review round 4, high fix #9 Decision: definition_hash is computed from a canonicalized dict representation (sorted keys, JSON serialization), not the raw YAML text. Rationale: Raw YAML hashing causes spurious stale detections on cosmetic edits (reindentation, comment changes, key reordering). Semantic hashing ensures only meaningful content changes trigger stale detection.

DD-25: DataLoaderPack protocol extension (isinstance, not hasattr)¶

Source: Review round 4, high fix #13; review round 5, high fix #1 Decision: DataLoaderPack(Protocol) is @runtime_checkable with full method signatures. Detection uses isinstance(), not hasattr(). Rationale: hasattr() duck-typing is fragile — typos in method names silently fail, and there's no IDE support for discovering the expected interface. A formal Protocol provides type safety, discoverability, and isinstance() checks.

DD-26: Staging EntityManager for LoadTransaction¶

Source: Review round 4, critical fix #1; review round 5, critical #1 Decision: LoadTransaction uses a staging EntityManager pattern. Entities are assembled in a detached staging area, then bulk-transferred into the live World on commit. Replaces the previous suppress_events parameter design. Rationale: World.create_entity() does not accept suppress_events, and adding it would be a cross-cutting engine change. The staging pattern naturally prevents EventBus side effects during the transaction without requiring engine API modifications. commit() and rollback() are async def because EventBus.emit() is async.

DD-27: Enforced dirty tracking via TrackedComponent mixin¶

Source: Review round 4, critical fix #2; review round 5, critical #2 Decision: InstanceStateComponent dirty tracking is automatically enforced via World.set_component() and a TrackedComponent mixin using Pydantic model_validator(mode='wrap'). Replaces the previous Component.__setattr__ hook design. Rationale: Pydantic BaseModel uses its own __setattr__ for validation, making a custom __setattr__ hook incompatible without careful ordering. The model_validator(mode='wrap') approach is a documented Pydantic extension point that integrates cleanly with model validation and serialization.

DD-28: ReferenceRegistry concurrency safety (eager snapshot)¶

Source: Review round 4, critical fix #3; review round 5, high fix #3 Decision: ReferenceRegistry uses asyncio.Lock for writes and eagerly rebuilds the copy-on-write snapshot inside the lock on every register() call. Rationale: The previous design invalidated the snapshot on write and lazily rebuilt it on read. This created a TOCTOU race where a reader could observe None between invalidation and rebuild, or two concurrent readers could race to rebuild. Eager rebuild under the lock guarantees the snapshot is always consistent.

DD-29: Reinstated prompt injection static analysis¶

Source: Review round 4, critical fix #6 Decision: Reinstate pattern-based prompt injection detection at load time with Warning/Error severity (not Informational). Rationale: Runtime-only defense is insufficient — malicious community content packs should be caught at validation time, before any AI provider interaction. Demoting to INFO effectively disables the defense.

DD-30: Staging EntityManager replaces suppress_events¶

Source: Review round 5, critical #1 Decision: LoadTransaction uses a detached EntityManager for staging instead of passing suppress_events=True to World.create_entity(). Rationale: World.create_entity() has no suppress_events parameter in the current engine. Adding one would be an invasive cross-cutting change. Staging naturally isolates the transaction from the live EventBus.

DD-31: TrackedComponent Pydantic mixin replaces setattr hook¶

Source: Review round 5, critical #2 Decision: Dirty tracking uses TrackedComponent(Component) mixin with model_validator(mode='wrap') instead of Component.__setattr__ override. Rationale: Pydantic BaseModel.__setattr__ handles field validation and coercion. Overriding it risks breaking Pydantic's internal machinery. model_validator(mode='wrap') is a documented Pydantic extension point.

DD-32: TemplateRegistry — abstract in engine, concrete in stdlib¶

Source: Review round 5, critical #3 Decision: TemplateRegistry Protocol defined in maid-engine; concrete implementation stays in maid-stdlib. GameEngine receives it via DI. Rationale: The existing TemplateRegistry lives in maid-stdlib (commands/building/create.py). Moving the concrete class to maid-engine would add game-specific logic to the infrastructure layer. DI preserves the package layering.

DD-33: Startup Authority boundary with Doc 01¶

Source: Review round 5, critical #4 Decision: Doc 04 owns defined entities (YAML source); Doc 01 owns dynamic entities (runtime-created, DocumentStore-only). Startup order: Doc 04 pipeline first, then Doc 01 persistence restore. Rationale: Without an explicit boundary, both documents claim authority over entity loading at startup, creating ambiguity about which system "wins" for entities that appear in both YAML and the DocumentStore.

DD-34: No-Code declarative content pack mode¶

Source: Review round 5, critical #5 Decision: Content packs with no_code = true in manifest are loaded via DeclarativeContentPack without importing any Python from the pack. Rationale: Community content packs should not require Python execution. A declarative-only mode provides the full data loading pipeline with no code execution surface, enabling safe distribution of untrusted content.

DD-35: ReloadableSystem Protocol (separate from System)¶

Source: Review round 5, high #5 Decision: on_entity_force_destroyed() is defined on a separate ReloadableSystem(Protocol), not added to the base System ABC. Rationale: The System ABC in maid-engine is content-agnostic. Adding reload-specific methods widens the interface for all systems, even those that never participate in reload. isinstance() detection keeps the concern isolated.

DD-36: Pipeline timeout and cooperative cancellation¶

Source: Review round 5, high #4 Decision: Pipeline accepts a timeout (default: 60s) and optional CancellationToken. Phases check cancellation cooperatively. Rationale: A pipeline processing a malformed or enormous content pack could run indefinitely. Timeout ensures the engine remains responsive.

DD-37: Eager ReferenceRegistry snapshot rebuild¶

Source: Review round 5, high #3 Decision: ReferenceRegistry.register() eagerly rebuilds the immutable snapshot inside the write lock. Rationale: Lazy invalidate-on-write / rebuild-on-read creates a race where concurrent readers may observe None or trigger duplicate rebuilds. Eager rebuild under the lock is simple and correct.

DD-38: ExitDefinition Pydantic model¶

Source: Review round 5, medium #5 Decision: Room exits field uses a structured ExitDefinition Pydantic model with direction, destination, description, hidden, locked, and key_id. Rationale: The exits field was previously unschematized — a plain list of dicts with no validation. A Pydantic model provides type safety, documentation, and error messages for malformed exits.

18. Implementation Plan¶

Sprint 0: Validation Spike (1 week)¶

Take existing maid-tutorial-world/data/rooms.yaml, write a minimal loader, parse it, validate it. Confirm format decisions work for real content. Identify issues before building infrastructure.

Dependency: Add ruamel.yaml>=0.18 to maid-engine pyproject.toml dependencies (replacing pyyaml for the data loader path). pyyaml remains for any legacy code that still imports it.

Phase 1: Core Pipeline (4 weeks)¶

Week	Deliverable
1	`Phase` protocol, `Pipeline` coordinator (with `CancellationToken` + timeout), `LoaderContext`, `LoadError` with error codes
2	`DiscoverPhase`, `LintPhase`, `ParsePhase` with security hardening and TOCTOU re-verification
3	`EntityAssembler` + `EntityTypeConfig` for rooms, NPCs, items; `EntityManager` staging class; `World.adopt_entity()`
4	`ValidatePhase`, `SemanticPhase` with typed `SemanticRule` protocol

Phase 2: Templates + References (3 weeks)¶

Week	Deliverable
5	`TemplateResolver` with `_extends`/`_use`/`_vars`/`_append` directives
6	`ReferenceRegistry` (engine-owned, eager snapshot), `ResolveRefsPhase` with pre-assigned UUIDs, `DataLoaderPack(Protocol)`
7	`maid content resolve` CLI, unified `TemplateRegistry` (abstract in engine, concrete in stdlib), `--skip-rule` / `--list-rules`

Phase 3: Integration (3 weeks)¶

Week	Deliverable
8	`GameEngine` pipeline factory, `DataProvenanceComponent`/`InstanceStateComponent` (with `TrackedComponent` mixin), staging `LoadTransaction` with async commit/rollback
9	`EntityLoadState` state machine, refuse-if-active reload with `ReloadableSystem(Protocol)`, force-reload protocol
10	`DataFileWatcher` with TOCTOU-safe coalescing, `unload_pack()`, `DeclarativeContentPack` (No-Code mode)

Phase 4: CLI Tooling + Polish (2 weeks)¶

Week	Deliverable
11	`maid data validate`, `maid data lint`, `maid data preview`, `maid data unload`
12	`ExitDefinition` model, resource limit enforcement, `maid data schema`, `SkipGraph` diagnostics

Files created:

packages/maid-engine/src/maid_engine/data/
    __init__.py
    models.py          # Data models, EntityTypeConfig, LoaderContext, ExitDefinition
    pipeline.py        # Pipeline coordinator + Phase protocol + CancellationToken
    protocols.py       # DataLoaderPack, ReloadableSystem protocols
    staging.py         # EntityManager staging for LoadTransaction
    tracked.py         # TrackedComponent mixin (Pydantic model_validator)
    skip_graph.py      # SkipGraph for cascading skip propagation
    declarative.py     # DeclarativeContentPack (No-Code mode)
    phases/
        __init__.py
        discover.py    # DiscoverPhase
        lint.py        # LintPhase
        parse.py       # ParsePhase (with TOCTOU re-verification)
        schema_route.py # SchemaRoutePhase
        register_ids.py # RegisterIDsPhase
        validate.py    # ValidatePhase
        semantic.py    # SemanticPhase + SemanticRule protocol
        resolve_refs.py # ResolveRefsPhase
        instantiate.py # InstantiatePhase + LoadTransaction (staging)
        post_load.py   # PostLoadPhase
    assembler.py       # EntityAssembler
    entity_types.py    # EntityTypeConfig declarations
    rules/             # Semantic rule implementations
        __init__.py
        health_rules.py
        npc_rules.py
        room_rules.py
    resolver.py        # TemplateResolver + ReferenceRegistry (eager snapshot)
    cache.py           # JSON cache (binary cache deferred to v2)
    watcher.py         # DataFileWatcher with TOCTOU-safe coalescing
    security.py        # Resource limits, path jail, alias rejection

19. Complete Example — Thornwall Outpost¶

A realistic content pack demonstrating all features.

19.1 Pack Manifest¶

# data/thornwall/manifest.toml
# NOTE: The canonical manifest filename is manifest.toml, matching the
# ContentPackLoader requirement. The data loader discovers data files
# from the data/ subdirectory; manifest.toml lives at the pack root.

# data/thornwall/manifest.toml
[pack]
name = "thornwall-outpost"
version = "1.0.0"
description = """
A frontier military outpost on the edge of the Darkwood Forest.
Features a garrison, armory, and patrol quests for levels 5-10."""

[pack.dependencies]
stdlib = "*"

19.2 NPC Template¶

# data/thornwall/npcs/_templates.yaml
_meta:
  schema: "maid:templates:v1"

templates:
  thornwall_soldier:
    _template: true
    _extends: "base_humanoid"
    tags: [garrison_member, military]
    components:
      HealthComponent:
        current: 80
        maximum: 80
      CombatComponent:
        attack_power: 18
        defense: 22
        accuracy: 75
      NPCComponent:
        behavior_type: "patrol"
        respawn_time: 600.0
        wander_radius: 3
        faction_id: "thornwall_garrison"

19.3 NPC with AI Dialogue¶

# data/thornwall/npcs/thornwall/garrison.yaml
_meta:
  schema: "maid:npcs:v1"

npcs:
  commander_vex:
    _use: thornwall_soldier
    location: "@ref:rooms/thornwall/commanders_quarters"
    tags: [quest_giver, unique]
    components:
      DescriptionComponent:
        name: "Commander Vex"
        short_desc: "Commander Vex, a battle-scarred veteran"
        long_desc: >
          A lean, weathered woman in her fifties with close-cropped
          grey hair and a missing left ear. Her pale blue eyes evaluate
          everything with cold tactical precision.
      HealthComponent:
        current: 150
        maximum: 150
      DialogueComponent:
        ai_enabled: true
        personality: >
          Gruff, direct, no-nonsense military commander. Cares deeply
          about her soldiers but hides it behind brusque professionalism.
        speaking_style: "clipped military speech"
        knowledge_domains:
          - "Darkwood Forest threats"
          - "Military tactics"
        secret_knowledge:
          - "Suspects a traitor in the garrison"
        greeting: "State your business. I don't have time for idle chatter."
        farewell: "Dismissed."

19.4 Room with Extended Descriptions¶

# data/thornwall/rooms/thornwall/outpost_gate.yaml
_meta:
  schema: "maid:rooms:v1"

rooms:
  thornwall_gate:
    zone: "@ref:zones/thornwall"
    tags: [room, outdoors, zone_boundary]
    components:
      DescriptionComponent:
        name: "Outpost Gate"
        long_desc: >
          Massive iron-banded oak gates stand between two squat stone
          towers. The walls bristle with sharpened stakes — the thorns
          that give this outpost its name.
      ExtendedRoomComponent:
        descriptions:
          base_description: ""
          time_variants:
            morning: "The morning watch changes shifts."
            night: "The gate is barred. A single lantern burns."
          weather_effects:
            fog: >
              Thick fog rolls up from the valley.
              Guards peer nervously into the white void.
          random_details:
            - text: "A supply wagon rumbles through the gate."
              weight: 0.2
          mood: "vigilant"
    exits:
      - direction: north
        destination: "@ref:rooms/thornwall/barracks"
      - direction: east
        destination: "@ref:rooms/thornwall/armory"
      - direction: up
        destination: "@ref:rooms/thornwall/watchtower"

19.5 Loading the Pack¶

class ThornwallContentPack:
    def get_dependencies(self) -> list[str]:
        return ["stdlib"]

    async def on_load(self, engine: GameEngine) -> None:
        pipeline = engine.create_data_pipeline()
        result = await pipeline.run_for_pack(self)
        if not result.success:
            for error in result.errors:
                logger.error("Thornwall: %s", error)

Expected output:

Content Statistics for data/thornwall/:
  Files:       12
  Templates:    5 (1 NPC, 4 item)
  Zones:        1
  Rooms:        5
  NPCs:         3 (1 merchant, 1 quest giver, 1 sentry)
  Items:       10
  Quests:       2
  References:  48 (all resolved)

Appendix A: Glossary¶

Term	Definition
Content Pack	Pluggable module implementing `ContentPack` protocol
Data File	YAML file containing declarative entity definitions
Phase	Standalone pipeline step implementing the `Phase` protocol
Pipeline	Coordinator that composes `Phase` objects into a sequence
Schema	Expected structure identified by `_meta.schema`
EntityAssembler	Generic assembler that creates entities using `EntityTypeConfig`
EntityTypeConfig	Lightweight declaration of how an entity type is assembled
SemanticRule	Typed validation rule with id, description, severity, and check method
Template	Reusable base definition (`_template: true`)
Archetype	Top-level template with no parent
Reference	Symbolic link using `@ref:` syntax
ReferenceRegistry	Engine-owned registry mapping symbolic names to UUIDs
Deep Merge	Recursive dict merge; child overrides at leaf level
Variable Substitution	`${var:default}` placeholder replacement
DataProvenanceComponent	Immutable record of entity origin (file, hash); `definition_hash` is persisted
InstanceStateComponent	Auto-enforced tracking of runtime modifications
LoadTransaction	Atomic entity creation with staging and async commit/rollback
Definition Layer	Author-defined base state from YAML
Instance Layer	Runtime-modified state in DocumentStore
Pre-Assigned UUID	Deterministic UUID from pack name + scoped ID
Semantic Validation	Gameplay-logic checks beyond schema structure
LoaderContext	Shared state threaded through all pipeline phases
MVL	Minimum Viable Loader — v1 implementation scope
Staging EntityManager	Detached entity container used by `LoadTransaction` for side-effect-free assembly
TrackedComponent	Pydantic mixin that auto-marks `InstanceStateComponent` dirty on mutation
DataLoaderPack	Optional `@runtime_checkable` Protocol for packs that use data loading
ReloadableSystem	Optional Protocol for systems that manage active interactions during reload
DeclarativeContentPack	Auto-generated ContentPack wrapper for No-Code data-only packs
No-Code Pack	Content pack with `no_code = true` that requires no Python execution
SkipGraph	Directed graph of entity dependencies used to propagate cascading skips
CancellationToken	Cooperative cancellation mechanism for long-running pipelines
ExitDefinition	Pydantic model for structured room exit declarations in data files
Startup Authority	Boundary defining which document/system loads which category of entities

Appendix B: Risks and Mitigations¶

#	Risk	Impact	Mitigation
R1	`ruamel.yaml` performance	High	C extensions; thread-based parallelism; binary cache deferred to v2 (DD-23)
R2	Schema evolution	High	Append-only policy; version field in `_meta.schema`
R3	Circular references	Medium	Visited-set detection; max depth 20; clear errors
R4	Cross-pack reference stability	Medium	Namespaced refs; lint warns on renamed entities; `_uuid` overrides
R5	Hot reload race conditions	Medium	Refuse-if-active; 2.0s debounce; stability check with content hash; parse-before-destroy; asyncio.Lock on watcher and registry
R6	Community content security	High	Safe YAML; no pickle; no aliases; resource limits; path jail; prompt injection static analysis (§15); No-Code mode (§6.5)
R7	Large world startup time	Medium	Incremental reload; binary cache (v2); lazy zones (v2)
R8	Half-loaded pack on error	Medium	`LoadTransaction` with staging `EntityManager` and async rollback (§8.5)
R9	Corrupt instance data	Medium	Quarantine + fallback to definition; bounded quarantine (§8.3)
R10	Partial file writes during reload	Medium	Stability check with content hash; parse-before-destroy; last-good checksum (§7.3)
R11	Malformed file DoS	Medium	`max_errors_per_file` cap; `max_cascade_depth` limit; pipeline timeout (§3.1)
R12	TOCTOU in file loading	Low	Re-verify `is_safe_path()` in ParsePhase immediately before `open()` (§3.2)
R13	`ruamel.yaml` not in dependencies	Low	Added as Sprint 0 prerequisite in §18; replaces `pyyaml` for loader path
R14	Startup authority conflict with Doc 01	Medium	Explicit boundary defined in §8.1; defined vs. dynamic entity scoping

Appendix C: Source Migration Guide¶

Scope clarification: This appendix covers source migration — converting existing bespoke loaders to use the unified data pipeline. For data migration (schema evolution, field renames, version upgrades), see the schema evolution policy in §4.4 and DD-12. For database migration (DocumentStore schema changes), see Doc 02 — Database Migrations.

Migrating SpellLoader to Data Pipeline¶

Before (bespoke loader):

spells:
  - internal_name: fireball
    name: "Fireball"
    school: evocation
    level: 3

After (unified loader):

_meta:
  schema: "classic-rpg:spells:v1"

spells:
  fireball:
    name: "Fireball"
    school: "evocation"
    level: 3

Steps:

Add _meta block to each YAML file
Register a SpellTypeConfig with the EntityAssembler
Replace SpellLoader.load_all() with pipeline.run_for_pack()
Run maid data validate to verify
Remove the bespoke loader

End of document.

Condition	Behavior
Missing `_meta.yaml`	Use convention defaults (directory-based schema inference)
Unparseable `_meta.yaml`	CRITICAL error — entire pack fails to load
References a missing file	ERROR, skip that file, continue with others
Files exist but not in `load_order`	Load after ordered files, emit INFO diagnostic

Entity Kind	Canonical Source	Loaded By	Example
Defined	YAML data file	Doc 04 pipeline	Town NPCs, rooms, quest items
Dynamic	DocumentStore	Doc 01 persistence	Player-crafted items, spawned mobs