World Data Loaders — Implementation Plan¶

Design Document: docs/designs/v3.1/04-world-data-loaders.md (v3.4) Estimated Timeline: 12 weeks (Sprint 0 + 4 phases)

Summary¶

This plan implements a unified World Data Loader framework for MAID that provides shared infrastructure for declarative content authoring via YAML files. Data-loading infrastructure does not yet exist in the codebase; existing content is loaded via bespoke Python code in content pack on_load() methods. This framework provides a composable pipeline for non-programmer content creation, validated against existing Pydantic component schemas, with templates, cross-entity references, and atomic loading with rollback.

Key deliverables: - Composable 6-phase pipeline (discover → parse+lint → prepare → resolve refs → instantiate → post-load) - YAML 1.2 parsing via ruamel.yaml (coexists with PyYAML for legacy code) - Single-inheritance templates with variable substitution - Pre-assigned deterministic UUIDs for reference resolution - Definition vs. instance state machine for persistence reconciliation - CLI tooling: maid data validate, maid data lint, maid data preview, maid data load, maid data resolve - Security hardening (resource limits, path jail, no aliases)

Cross-document dependencies: - Doc 01 (Durable Persistence): DocumentStore integration for instance state, quarantine collection, startup authority boundary (Doc 04 loads defined entities first, Doc 01 restores dynamic entities after) - Doc 02 (Database Migrations): Schema evolution for DataProvenanceComponent / InstanceStateComponent persistence

Sprint 0: Validation Spike (Week 0)¶

Validate YAML format decisions against real content before building infrastructure.

0.1 Dependency Setup¶

Package: maid-engine | Priority: P0 | Dependencies: None

[ ] Add ruamel.yaml>=0.18 to maid-engine/pyproject.toml dependencies
[ ] Verify watchfiles is already present in maid-engine/pyproject.toml (line 59: "watchfiles>=0.21.0")
[ ] Run uv sync to verify dependency resolution
[ ] Verify ruamel.yaml C extension (ruamel.yaml.clib) installs when C compiler is available

0.2 Spike: Load Tutorial World Data¶

Package: maid-engine, maid-tutorial-world | Priority: P0 | Dependencies: 0.1

[ ] Write minimal spike script that parses existing maid-tutorial-world data files with ruamel.yaml in YAML 1.2 safe mode
[ ] Verify line/column tracking is preserved on parsed nodes
[ ] Validate parsed data against existing Pydantic component models (HealthComponent, DescriptionComponent, NPCComponent)
[ ] Identify any format incompatibilities between existing YAML files and the design spec
[ ] Produce a MIGRATION_GAPS.md documenting required changes to maid-tutorial-world YAML files and maid-classic-rpg data files, including any list-based vs. keyed-entity format differences
[ ] Document findings and any required YAML format adjustments
[ ] Delete spike script after findings are documented

Phase 1: Core Pipeline Infrastructure (Weeks 1–4)¶

1.1 Data Models and Error Types¶

Package: maid-engine | Priority: P0 | Dependencies: 0.1

[ ] Create packages/maid-engine/src/maid_engine/loader/__init__.py
[ ] Create packages/maid-engine/src/maid_engine/loader/models.py with:
[ ] ErrorSeverity enum (WARNING, ERROR, CRITICAL)
[ ] LoadError frozen dataclass with fields: code, file_path, line, field_path, message, severity, suggestion
[ ] DiscoveredFile dataclass (path, relative_path, file_size, mtime)
[ ] ParsedDocument dataclass (path, data dict, source_map for line tracking)
[ ] EntityDefinition dataclass (id, entity_type, components dict, tags, source_file, line, template_ref, raw_data)
[ ] DataFileMeta Pydantic model for _meta block parsing (schema, pack, description, author, load_order)
[ ] SchemaRef Pydantic model with parse() classmethod for namespace:type:version format
[ ] LoaderConfig dataclass with all configurable limits:
- strict_mode: bool = True
- max_file_size: int = 10_485_760 (10 MB)
- max_entities_per_file: int = 1000
- max_components_per_entity: int = 50
- max_template_depth: int = 5
- max_ref_chain_depth: int = 20
- max_errors_per_file: int = 50
- max_quarantine_entries: int = 500
- pipeline_timeout: float = 60.0
- fuzzy_match_max_candidates: int = 5000
[ ] LoaderContext dataclass with fields: world, config, reference_registry, template_registry, discovered_files, parsed_docs, entity_definitions, errors
[ ] EntityTypeConfig frozen dataclass with fields: type_name, required_components, default_components, default_tags, allowed_top_level_fields
[ ] Write unit tests for SchemaRef.parse() and DataFileMeta.model_validate()
[ ] Write unit tests for LoaderConfig defaults and validation

1.1a ComponentRegistry¶

Package: maid-engine | Priority: P0 | Dependencies: None

[ ] Create ComponentRegistry in packages/maid-engine/src/maid_engine/core/ecs/registry.py:
[ ] Central registry mapping component type names to Python classes
[ ] Populate via Component.__init_subclass__ hook or runtime subclass scan
[ ] get(type_name) -> type[Component] | None method
[ ] list() -> dict[str, type[Component]] method
[ ] Used by the loader's PreparePhase to resolve component names from YAML to classes
[ ] Write unit tests for ComponentRegistry discovery and lookup

1.2 Pipeline Coordinator and Phase Protocol¶

Package: maid-engine | Priority: P0 | Dependencies: 1.1

[ ] Create packages/maid-engine/src/maid_engine/loader/pipeline.py with:
[ ] Phase Protocol with name property and async execute(context) -> PhaseResult
[ ] PhaseResult dataclass (errors, warnings, artifacts dict, phase_name, duration_ms)
[ ] PipelineResult dataclass (success, errors, warnings, phase_results, entity_count, duration_ms)
[ ] CancellationToken class with is_cancelled property and cancel() method
[ ] Pipeline class:
- Constructor accepting phases: list[Phase], timeout: float = 60.0
- async run(context, cancellation_token=None) -> PipelineResult — executes phases sequentially, checks cancellation between phases, enforces timeout
- run_for_pack(pack, data_path=None, data_paths=None) -> PipelineResult convenience method
[ ] DEFAULT_PHASES list constant defining the standard 6-phase ordering
[ ] Create packages/maid-engine/src/maid_engine/loader/phases/__init__.py
[ ] Write unit tests for Pipeline coordinator:
[ ] Test sequential phase execution
[ ] Test pipeline abort on CRITICAL error (per-file, not per-pipeline)
[ ] Test cancellation token halts pipeline between phases
[ ] Test pipeline timeout enforcement
[ ] Test error collection across phases

1.3 DiscoverPhase (Phase 1)¶

Package: maid-engine | Priority: P0 | Dependencies: 1.2

[ ] Create packages/maid-engine/src/maid_engine/loader/phases/discover.py with DiscoverPhase:
[ ] Recursively scan content pack data/ directory for .yaml and .json files
[ ] Parse _meta.yaml at directory root for load_order if present
[ ] Respect load_order: templates load before entities that reference them
[ ] Handle _meta.yaml failure modes per design:
- Missing _meta.yaml → use convention defaults
- Unparseable _meta.yaml → CRITICAL error, entire pack fails
- References missing file → ERROR, skip that file, continue
- Files not in load_order → load after ordered files, emit INFO
[ ] Skip hidden files (starting with .) except _meta.yaml
[ ] Ignore symlinks (security: symlink escape prevention)
[ ] Reject paths that traverse outside pack root via is_safe_path()
[ ] Enforce max_file_size limit
[ ] Populate context.discovered_files
[ ] Create packages/maid-engine/src/maid_engine/loader/security.py with:
[ ] is_safe_path(pack_root, candidate) -> bool function
[ ] Resource limit validation helpers
[ ] Write unit tests:
[ ] Test recursive directory scanning
[ ] Test load_order respect from _meta.yaml
[ ] Test all four _meta.yaml failure modes
[ ] Test symlink rejection
[ ] Test path traversal rejection (../../etc/passwd)
[ ] Test hidden file skipping
[ ] Test max file size enforcement

1.4 ParsePhase (Phase 2: Parse + Lint)¶

Package: maid-engine | Priority: P0 | Dependencies: 1.3

[ ] Create packages/maid-engine/src/maid_engine/loader/phases/parse.py with ParsePhase:
[ ] Use ruamel.yaml with typ="safe" and version=(1, 2)
[ ] Disable YAML aliases (safe mode does this by default)
[ ] Set allow_duplicate_keys = False
[ ] Preserve source line/column positions on parsed nodes for error reporting
[ ] Re-verify is_safe_path() immediately before open() (TOCTOU mitigation)
[ ] Populate context.parsed_docs with ParsedDocument instances
[ ] Handle parse errors with file path and line information
[ ] Integrated pre-parse lint rules (operate on raw YAML text before parsing):
- [ ] MAID-Y001 (Warn): Unquoted YAML 1.1 boolean literal (NO, YES, On, Off, etc.)
- [ ] MAID-Y002 (Warn): Unquoted sexagesimal value (1:30)
- [ ] MAID-Y003 (Warn): Integer with leading zero (octal in 1.1) (0755)
- [ ] MAID-Y005 (Warn): Special float (.inf, .nan)
- [ ] MAID-Y006 (Warn): Inconsistent block scalar indentation
- [ ] MAID-Y007 (Error): YAML merge key (<<:)
- [ ] MAID-Y008 (Error): Duplicate mapping key
[ ] Enforce max_errors_per_file limit per file
[ ] Write unit tests:
[ ] Test valid YAML parsing with line tracking
[ ] Test YAML 1.2 semantics (NO → string, not bool)
[ ] Test duplicate key rejection
[ ] Test parse error reporting with file/line
[ ] Test TOCTOU re-verification
[ ] Test each lint rule (MAID-Y001 through MAID-Y008)
[ ] Test error cap enforcement

1.5 PreparePhase (Phase 3: Route + Register IDs + Validate + Semantic)¶

Package: maid-engine | Priority: P0 | Dependencies: 1.4

[ ] Create packages/maid-engine/src/maid_engine/loader/phases/prepare.py with PreparePhase:
Schema routing:
- [ ] Extract _meta.schema from parsed documents
- [ ] Implement implicit schema inference when _meta.schema is absent:
- Directory name → maid:{dir}:v1
- Top-level key → maid:{key}:v1
- Conflict between directory and top-level key → ERROR with suggestion
- [ ] Route to EntityAssembler with appropriate EntityTypeConfig
- [ ] Unknown types produce "Did you mean…" error with fuzzy matching
- [ ] Resolve template directives (_extends, _use, _vars, _append) in this phase
- [ ] Perform variable substitution (${var}, ${var:default})
- [ ] Output fully-expanded entity definitions (no unresolved templates/variables)
ID registration:
- [ ] Collect all _id values from entity definitions
- [ ] Scope IDs to directory path within the pack
- [ ] Pre-assign deterministic UUIDs via uuid5(NAMESPACE_DATA, f"{pack}:{scoped_id}")
- [ ] Define NAMESPACE_DATA constant UUID
- [ ] Implement pre_assign_uuid(pack_name, scoped_id) -> UUID function
- [ ] Support explicit _uuid overrides
- [ ] Check UUID collisions at three levels:
- Current load batch
- Dependency packs (via ReferenceRegistry)
- Existing world entities
- [ ] Register IDs in context.reference_registry
Validation:
- [ ] Validate component blocks against Pydantic models via model_validate()
- [ ] Auto-discover component types from ComponentRegistry using component_type ClassVar
- [ ] Support validation modes:
- Strict: extra fields → Error, unknown refs → Error, missing optional → Warning
- Lenient: extra fields → Warning, unknown refs → Warning, missing optional → Silent
- [ ] Enforce resource limits: max_entities_per_file, max_components_per_entity
- [ ] Rewrite Pydantic validation errors for content author readability
- [ ] Handle union types (str | TranslatableText), complex nested types (UUID, datetime)
- [ ] Post-parse lint rules (require parsed structure):
- [ ] MAID-Y004 (Error): Empty/null in required string field
- [ ] MAID-Y009 (Error): Unresolvable @ref: reference (basic check)
- [ ] MAID-Y010 (Warn): Cross-file @ref: without depends_on
- [ ] MAID-Y011 (Warn): Runtime-only field set in data file (in_combat: true)
- [ ] MAID-Y012 (Error): Unknown component type name with "Did you mean…" suggestion
Semantic checks (v1: minimal rule set):
- [ ] SemanticRule Protocol with properties: id, description, severity, applies_to; and method: check(entity, context) -> list[LoadError] (synchronous — pure functions over in-memory data, no timeout needed)
- [ ] Filter rules by applies_to entity type
- [ ] Support --skip-rule IDs (via LoaderConfig.skip_rules: set[str])
- [ ] V1 rules:
- [ ] MAID-S001: current <= maximum for HealthComponent (Error)
- [ ] MAID-S005: Rooms should have at least one exit (Warning)
[ ] Create packages/maid-engine/src/maid_engine/loader/assembler.py with EntityAssembler:
[ ] Constructor accepting configs: dict[str, EntityTypeConfig]
[ ] assemble(definition, config, world) -> Entity method
[ ] Apply required_components validation
[ ] Apply default_components for absent components
[ ] Apply default_tags
[ ] Validate against allowed_top_level_fields
[ ] Create packages/maid-engine/src/maid_engine/loader/entity_types.py with standard configs:
[ ] ROOM_CONFIG — required: DescriptionComponent; defaults: ExtendedRoomComponent; tags: room; top-level: exits, zone
[ ] NPC_CONFIG — required: DescriptionComponent, NPCComponent; tags: npc; top-level: location
[ ] ITEM_CONFIG — required: DescriptionComponent, ItemComponent; tags: item; top-level: location
[ ] TEMPLATE_CONFIG — for template definitions
[ ] Create packages/maid-engine/src/maid_engine/loader/rules/__init__.py
[ ] Create packages/maid-engine/src/maid_engine/loader/rules/builtin.py with v1 semantic rules
[ ] Write unit tests:
[ ] Test explicit schema routing
[ ] Test implicit schema inference (directory, top-level key, conflict)
[ ] Test "Did you mean…" suggestions for unknown types
[ ] Test EntityAssembler with each EntityTypeConfig
[ ] Test required component validation
[ ] Test default component injection
[ ] Test default tag application
[ ] Test deterministic UUID generation
[ ] Test directory-scoped ID uniqueness
[ ] Test UUID collision detection at all three levels
[ ] Test _uuid explicit override
[ ] Test cross-reload UUID stability (same input → same UUID)
[ ] Test valid component validation
[ ] Test unknown component type detection with suggestions
[ ] Test strict vs. lenient mode behavior
[ ] Test resource limit enforcement
[ ] Test error message rewriting for readability
[ ] Test each semantic rule
[ ] Test --skip-rule integration
[ ] Test post-parse lint rules (MAID-Y004, Y009–Y012)

Phase 2: Templates and References (Weeks 5–7)¶

2.1 Template Resolver¶

Package: maid-engine | Priority: P0 | Dependencies: 1.6

[ ] Create packages/maid-engine/src/maid_engine/loader/resolver.py with TemplateResolver:
[ ] Single inheritance via _extends directive
[ ] Template instantiation via _use directive
[ ] Variable substitution via _vars / ${var} / ${var:default} syntax
[ ] Generic _append block for additive list merging on any list field
[ ] Deep merge logic:
- Scalars: child overrides parent
- Component dicts: deep-merged at field level
- Lists: child replaces parent entirely (unless _append)
- _append block: unions with parent for specified list fields (duplicates removed)
[ ] Circular inheritance detection with visited-set
[ ] Max template depth enforcement (max_template_depth = 5)
[ ] Variable type inference for defaults (int, float, bool, str)
[ ] Single-pass variable substitution (no recursive expansion — security)
[ ] Template entities marked with _template: true are NOT instantiated as world entities
[ ] Write unit tests:
[ ] Test single-level _extends inheritance
[ ] Test multi-level inheritance chain (3+ levels)
[ ] Test _use template instantiation
[ ] Test variable substitution with typed defaults
[ ] Test _append for tags and other list fields
[ ] Test deep merge at component field level
[ ] Test circular inheritance detection
[ ] Test max depth enforcement (6 levels → error)
[ ] Test _template: true entities are skipped during instantiation

2.2 TemplateRegistry Integration¶

Package: maid-engine, maid-stdlib | Priority: P0 | Dependencies: 2.1

[ ] The existing TemplateRegistry in packages/maid-stdlib/src/maid_stdlib/commands/building/create.py already provides register(), get(), unregister(), list(), clear(). Use it directly — no Protocol abstraction needed until a second implementation exists.
[ ] The data loader accepts a TemplateRegistry instance via constructor injection
[ ] Update GameEngine.__init__() to accept optional template_registry parameter and store as self.template_registry
[ ] Default to None when not provided (loader skips template features if absent)
[ ] Write unit tests for loader integration with existing TemplateRegistry

2.3 ReferenceRegistry (Engine-Owned)¶

Package: maid-engine | Priority: P0 | Dependencies: 1.7

[ ] Add ReferenceRegistry class to packages/maid-engine/src/maid_engine/loader/resolver.py:
[ ] _entries: dict[str, UUID] mapping symbolic names to UUIDs
[ ] _lock: asyncio.Lock for write synchronization
[ ] _snapshot: MappingProxyType[str, UUID] copy-on-write snapshot for lock-free reads
[ ] async register(name, entity_id) — acquires lock, stores entry, eagerly rebuilds snapshot under lock (DD-28/DD-37)
[ ] snapshot() -> Mapping[str, UUID] — returns immutable snapshot (no lock needed)
[ ] async unregister(name) -> bool
[ ] async clear()
[ ] Add self.reference_registry = ReferenceRegistry() to GameEngine.__init__()
[ ] Write unit tests:
[ ] Test register and snapshot retrieval
[ ] Test concurrent register calls (asyncio.Lock correctness)
[ ] Test eager snapshot rebuild (no stale reads)
[ ] Test unregister and clear

2.4 ResolveRefsPhase (Phase 4)¶

Package: maid-engine | Priority: P0 | Dependencies: 2.3

[ ] Create packages/maid-engine/src/maid_engine/loader/phases/resolve_refs.py with ResolveRefsPhase:
[ ] Parse @ref:type/name syntax for same-pack references
[ ] Parse @ref:pack:type/name syntax for cross-pack references
[ ] Parse @ref:uuid:<uuid> syntax for direct UUID references
[ ] Replace @ref: strings with pre-assigned UUIDs from ReferenceRegistry
[ ] Unresolved references produce errors with fuzzy-match suggestions
[ ] Fuzzy match rate limiting: cap candidate pool at fuzzy_match_max_candidates
[ ] Resolve references in exit destination and key fields (validated against the engine's existing exit structure, no separate ExitDefinition model)
[ ] Reference resolution scope: current pack → dependency packs → engine registry
[ ] In lenient mode, unresolved references produce flat error messages listing the failed entity and all entities that reference it (no dependency graph needed)
[ ] Write unit tests:
[ ] Test same-pack reference resolution
[ ] Test cross-pack namespaced reference resolution
[ ] Test unresolved reference error with "Did you mean…" suggestion
[ ] Test fuzzy match rate limiting
[ ] Test @ref:uuid:<uuid> direct references
[ ] Test lenient mode cascade error reporting

2.5 DataLoaderPack Protocol Extension¶

Package: maid-engine | Priority: P0 | Dependencies: 1.5, 2.1

[ ] Add to packages/maid-engine/src/maid_engine/loader/protocols.py:
[ ] @runtime_checkable DataLoaderPack(Protocol) with methods:
- get_entity_type_configs() -> list[EntityTypeConfig]
- get_semantic_rules() -> list[SemanticRule]
- get_data_paths() -> list[Path]
[ ] @runtime_checkable ReloadableSystem(Protocol) with method:
- async on_entity_force_destroyed(entity_id: UUID, reason: str) -> None
[ ] Write unit tests:
[ ] Test isinstance() detection for packs implementing DataLoaderPack
[ ] Test isinstance() detection for systems implementing ReloadableSystem
[ ] Test that base ContentPack does NOT satisfy DataLoaderPack

Phase 3: Integration (Weeks 8–10)¶

3.1 Provenance and Instance State Components¶

Package: maid-engine (or maid-stdlib) | Priority: P0 | Dependencies: 1.1

[ ] Create DataProvenanceComponent(Component):
[ ] Immutable fields: source_file: str, pack_name: str, definition_id: str, definition_hash: str, definition_version: str = "1.0", loaded_at: datetime
[ ] definition_hash computed from canonicalized dict (sorted keys, JSON with sort_keys=True, ensure_ascii=True, SHA-256) — DD-24
[ ] Create InstanceStateComponent(Component):
[ ] Mutable fields: modified: bool = False, modified_components: set[str], last_modified_at: datetime | None, modified_by: str | None
[ ] mark_modified(component_type, actor=None) method — manual dirty tracking only for v1 (automatic TrackedComponent mixin deferred to v2)
[ ] Implement compute_definition_hash(definition_dict) -> str utility function
[ ] Write unit tests:
[ ] Test DataProvenanceComponent immutability semantics
[ ] Test InstanceStateComponent modification tracking via mark_modified()
[ ] Test definition hash canonicalization (reordered keys → same hash, changed values → different hash)

3.2 Staging EntityManager and LoadTransaction¶

Package: maid-engine | Priority: P0 | Dependencies: 1.5, 3.1

[ ] Create packages/maid-engine/src/maid_engine/loader/staging.py with:
[ ] Verify existing EntityManager in packages/maid-engine/src/maid_engine/core/ecs/entity.py can be used as detached staging area (no EventBus dependency) — it already can, EntityManager is standalone
[ ] Add EntityManager.adopt(entity: Entity) -> None public method for transferring entities between managers
[ ] Add World.adopt_entity(entity: Entity) -> None method to transfer entity from external EntityManager into live world:
- Register entity in world's EntityManager via adopt()
- Update all component/tag indexes
- Update RoomIndex if entity has PositionComponent
- Duplicate UUID behavior: If entity UUID already exists, replace-with-warning (log warning, destroy old entity, adopt new). This supports hot reload use case.
- Missing room: If entity references a room that doesn't exist, raise ValueError with descriptive message
[ ] LoadTransaction class:
- Constructor: __init__(self, world: World, reference_registry: ReferenceRegistry)
- _staging: EntityManager — detached staging area
- _staged_ids: list[UUID]
- _registered_refs: list[str] — track refs registered during this transaction
- stage_entity(entity_id, **kwargs) -> Entity — create in staging area
- verify_references() -> list[LoadError] — post-staging integrity check
- async commit() -> None — transfer staged entities to live World via adopt_entity(), then emit EntityCreatedEvent for each
- async rollback() -> None — discard staging area, clear staged IDs, clean up ReferenceRegistry entries added during this transaction (remove _registered_refs from registry)
[ ] Write unit tests:
[ ] Test World.adopt_entity() transfers entity correctly
[ ] Test World.adopt_entity() replaces existing entity with same UUID (with warning)
[ ] Test World.adopt_entity() raises on missing room reference
[ ] Test staging entity creation does NOT emit events
[ ] Test commit transfers entities and emits events
[ ] Test rollback discards all staged entities
[ ] Test rollback cleans up ReferenceRegistry entries
[ ] Test reference verification catches broken refs
[ ] Test partial failure → full rollback

3.3 InstantiatePhase (Phase 5)¶

Package: maid-engine | Priority: P0 | Dependencies: 3.2

[ ] Create packages/maid-engine/src/maid_engine/loader/phases/instantiate.py with InstantiatePhase:
[ ] Create LoadTransaction for each file or pack
[ ] For each non-template entity definition:
- Stage entity with pre-assigned UUID
- Add all validated components
- Apply tags from EntityTypeConfig
- Attach DataProvenanceComponent with source file, pack name, definition hash
- Attach InstanceStateComponent
[ ] Run verify_references() after staging all entities
[ ] On success: await transaction.commit()
[ ] On failure: await transaction.rollback() and report errors
[ ] Skip entities marked with _template: true
[ ] Write unit tests:
[ ] Test successful instantiation creates entities in World
[ ] Test DataProvenanceComponent is attached correctly
[ ] Test template entities are skipped
[ ] Test reference verification failure triggers rollback
[ ] Test atomic behavior (all-or-nothing per file)

3.4 PostLoadPhase (Phase 6)¶

Package: maid-engine | Priority: P1 | Dependencies: 3.3

[ ] Create packages/maid-engine/src/maid_engine/loader/phases/post_load.py with PostLoadPhase:
[ ] Call content pack post-load callbacks for cross-entity relationship setup
[ ] Build indexes (room connections, NPC location mapping)
[ ] Emit summary statistics (files processed, entities created, templates, references resolved)
[ ] Log pipeline results via logging.getLogger("maid.loader") with structured context (for production on_load() calls where CLI output isn't visible)
[ ] Write unit tests

3.5 GameEngine Pipeline Factory¶

Package: maid-engine | Priority: P0 | Dependencies: 3.3, 2.3, 2.2

[ ] Add create_data_pipeline(config: LoaderConfig | None = None) -> Pipeline factory method to GameEngine:
[ ] Create LoaderContext with injected shared resources (world, reference_registry, template_registry, config)
[ ] Collect EntityTypeConfig from all loaded packs implementing DataLoaderPack
[ ] Collect SemanticRule instances from all loaded packs implementing DataLoaderPack
[ ] Return Pipeline with DEFAULT_PHASES and collected configs/rules
[ ] Write integration test:
[ ] Test full pipeline execution from GameEngine.create_data_pipeline()
[ ] Test cross-pack reference resolution (stdlib entities referenced by classic-rpg)

3.6 Entity Load State Machine¶

Package: maid-engine | Priority: P0 | Dependencies: 3.3

[ ] Add to packages/maid-engine/src/maid_engine/loader/models.py:
[ ] EntityLoadState enum: NEW, INSTANCE_CLEAN, INSTANCE_STALE, INSTANCE_CORRUPT, RESET_REQUESTED
[ ] EntityLoadAction enum: CREATE_FROM_DEFINITION, LOAD_FROM_INSTANCE, LOAD_INSTANCE_WARN_STALE, QUARANTINE_LOAD_DEFINITION, DELETE_INSTANCE_RELOAD
[ ] resolve_load_action(state, reset) -> EntityLoadAction pure function implementing the state → action mapping table
[ ] Integrate state machine into InstantiatePhase:
[ ] Check DocumentStore for existing instance
[ ] Compare definition hashes to determine state (INSTANCE_CLEAN vs INSTANCE_STALE)
[ ] Handle corrupt instances: move to _quarantine collection, log WARNING, create from definition
[ ] Enforce quarantine cap (max_quarantine_entries, FIFO eviction)
[ ] Support --reset flag for DELETE_INSTANCE_RELOAD action
[ ] Write unit tests for ALL state × reset combinations:
[ ] NEW → CREATE_FROM_DEFINITION
[ ] INSTANCE_CLEAN + no reset → LOAD_FROM_INSTANCE
[ ] INSTANCE_CLEAN + reset → DELETE_INSTANCE_RELOAD
[ ] INSTANCE_STALE + no reset → LOAD_INSTANCE_WARN_STALE
[ ] INSTANCE_STALE + reset → DELETE_INSTANCE_RELOAD
[ ] INSTANCE_CORRUPT → QUARANTINE_LOAD_DEFINITION
[ ] RESET_REQUESTED → DELETE_INSTANCE_RELOAD
[ ] Write unit tests for quarantine mechanics

3.7 Hot Reload Integration¶

Package: maid-engine | Priority: P1 | Dependencies: 3.3, 3.6

[ ] Add DATA_FILES = auto() to existing ReloadScope enum in packages/maid-engine/src/maid_engine/reload/manager.py
[ ] Implement refuse-if-active reload strategy:
[ ] Query entities from target file(s) via DataProvenanceComponent.source_file
[ ] Check for active interactions by querying systems implementing ReloadableSystem
[ ] If active: refuse with error listing active entities and their interactions
[ ] Support --force flag:
- Call on_entity_force_destroyed(entity_id, reason) on each ReloadableSystem
- Notify connected players
- Destroy and recreate from updated definition
- Post-reload reference re-validation
[ ] Support --skip-active flag to reload only inactive entities
[ ] Implement unload_pack(engine, pack_name) -> UnloadResult:
[ ] Identify entities via DataProvenanceComponent.pack_name
[ ] Check cross-pack references (refuse if other packs reference this pack's entities)
[ ] Check active interactions (refuse unless --force)
[ ] Destroy entities from World
[ ] Clean ReferenceRegistry and TemplateRegistry
[ ] Write unit tests:
[ ] Test refuse-if-active behavior
[ ] Test --force teardown flow
[ ] Test --skip-active partial reload
[ ] Test unload_pack with cross-pack reference checking
[ ] Test unload_pack registry cleanup

Phase 4: CLI Tooling and Polish (Weeks 11–12)¶

4.1 `maid data` CLI Command Group¶

Package: maid-engine | Priority: P0 | Dependencies: 3.5

[ ] Create maid data command group in CLI (likely add to packages/maid-engine/src/maid_engine/cli/app.py or new data.py submodule):
[ ] maid data validate <path> — run Phases 1–4 (no instantiation):
- --strict / --lenient mode flag
- --skip-rule <RULE_ID> (repeatable)
- --list-rules — show all registered semantic rules
- Output: file-by-file error/warning report with summary
[ ] maid data lint <path> — run Phase 1 + lint portion of Phase 2 (Discover + Parse with lint):
- Output: lint issues per file with line numbers
[ ] maid data preview <path> — run Phases 1–4 + dry-run summary:
- Output: statistics (files, entities, templates, references) without creating entities
[ ] maid data load <path> — run all 6 phases:
- --reset flag for instance state reset
- --strict / --lenient mode flag
[ ] maid data resolve <template> — display template inheritance chain and effective definition (moved from maid content resolve)
[ ] maid data reload <path> — hot reload specific data files:
- --force flag for active entity teardown
- --skip-active flag
[ ] maid data unload <pack_name> — unload all entities from a content pack:
- --force flag
[ ] Write CLI integration tests for each command

4.2 `maid data schema` CLI Command¶

Package: maid-engine | Priority: P2 | Dependencies: 4.1

[ ] Add maid data schema command:
[ ] maid data schema list — list all known component types with descriptions
[ ] maid data schema show <component_type> — show component fields, types, defaults, and constraints
[ ] Output should be content-author friendly (not raw Pydantic schema)
[ ] Write CLI integration tests

4.3 `@export` Builder Command¶

Package: maid-stdlib | Priority: P1 | Dependencies: 3.1

[ ] Implement @export in-game builder command:
[ ] @export rooms <filter> --output <path> — export rooms to YAML data file
[ ] @export npcs <filter> --output <path> — export NPCs
[ ] @export <entity_type> --zone <zone> --output <path> — filter by zone
[ ] Export captures definition-state only (original authored values from DataProvenanceComponent, not runtime state)
[ ] Generate valid YAML with _meta block, _id fields, proper component structure
[ ] Preserve @ref: syntax for entity references
[ ] Write unit tests for export correctness
[ ] Write round-trip test: load entities from YAML → export to YAML → re-load exported YAML → verify entity equality

Phase 5: Content Pack Migration and Integration Tests (Ongoing)¶

5.1 Tutorial World Migration¶

Package: maid-tutorial-world | Priority: P1 | Dependencies: 3.5

[ ] Add _meta blocks to existing tutorial world YAML data files
[ ] Convert tutorial world pack.py to use data pipeline in on_load():
[ ] Implement DataLoaderPack protocol
[ ] Return EntityTypeConfig declarations for rooms, NPCs, items
[ ] Remove bespoke loading code
[ ] Run maid data validate against tutorial world data
[ ] Write integration test: load tutorial world via pipeline, verify entity creation

5.2 Classic RPG Migration¶

Package: maid-classic-rpg | Priority: P1 | Dependencies: 5.1

[ ] Create YAML data files for spells, skills, and abilities (these currently exist as empty directories or Python-only definitions — no bespoke loader classes exist to replace)
[ ] Add _meta blocks to new YAML data files
[ ] Register SpellTypeConfig, SkillTypeConfig, AbilityTypeConfig via get_entity_type_configs()
[ ] Register classic-rpg-specific semantic rules via get_semantic_rules()
[ ] Update EcosystemSystem._load_definitions() and WorldEventSystem._load_definitions() (both fully implemented, loading JSON from data/ directories) to use pipeline
[ ] Run maid data validate against classic-rpg data
[ ] Write integration test: load classic-rpg data via pipeline

5.3 Full Integration Test Suite¶