Hot Reload Guide¶

Hot reload allows you to update Python code, content packs, and ECS systems without restarting the MAID server. This is invaluable during development for rapid iteration and testing.

Table of Contents¶

Overview
CLI Commands
In-Game Commands
Watch Mode
Module Tracking
Rollback System
Rollback Event Handler Limitation
State Checkpointing
Best Practices
Troubleshooting

Overview¶

What Can Be Hot Reloaded?¶

Type	Description	Reload Command
Python Modules	Any Python module in the codebase	`@reload module <name>`
Content Packs	Entire content packs with their systems	`@reload pack <name>` or `@reload <name>`
ECS Systems	Individual game systems	`@reload system <name>`
Templates	Content templates (items, NPCs, etc.)	`@reload templates [pattern]`
All Tracked	All modules being watched	`@reload all`

How It Works¶

Content pack reloads (@reload pack) use the engine's built-in HotReloadManager which:

Pauses the tick loop
Snapshots the current pack state for rollback
Unloads the old pack (systems, commands, event handlers)
Reimports the pack's Python modules
Reloads the pack (re-registers systems, commands, events)
Resumes the tick loop

Module-level reloads (@reload module, @reload system, @reload all) use a shared ReloadManager stored in world data. This manager handles dependency tracking, cascade reloading, and snapshot-based rollback for individual Python modules.

Limitations¶

State Loss: In-memory state is lost during reload (use persistent storage)
Active References: Old object references may persist in some cases
Database Connections: Connection pools may need to be re-established
Running Tasks: Async tasks from the old module continue until complete
Event Handler Restoration on Rollback: See Rollback Event Handler Limitation below
Security Modules: Modules in the security blocklist (auth, rate limiting, safety) cannot be hot-reloaded

CLI Commands¶

Start Server with Watch Mode¶

# Start server with file watching enabled
uv run maid server start --watch

# Start with custom debounce delay (default 0.5s)
uv run maid server start --watch --watch-debounce 1.0

Manual Reload via CLI¶

# Reload a specific module
uv run maid dev reload maid_stdlib.commands.building --type module

# Reload a content pack
uv run maid dev reload classic-rpg --type pack

# Reload a system
uv run maid dev reload CombatSystem --type system

# Reload with watch mode (auto-reload on file changes)
uv run maid dev reload maid_stdlib --type module --watch

# Watch specific paths
uv run maid dev reload maid_stdlib --type module --watch --watch-path packages/maid-stdlib

# Disable cascading to dependents
uv run maid dev reload maid_stdlib --type module --no-cascade

# Check reload manager status
uv run maid dev reload-status

In-Game Commands¶

@reload¶

Reload modules, packs, or systems from within the game.

Syntax:

@reload <type> <target>
@reload <pack_name>          # shorthand for @reload pack <pack_name>

Types: - module - Python module by import path - pack - Content pack by name (uses engine hot-reload system) - system - ECS system by name - templates - Reload templates from content pack data directories - all - All tracked modules (skips security-blocked modules) - status - Show hot-reload status overview - list - List tracked modules with dependency info - track - Add a module to the explicitly tracked set - untrack - Remove a module from the explicitly tracked set

Examples:

@reload module maid_stdlib.commands.building.create
@reload pack classic-rpg
@reload classic-rpg
@reload system CombatSystem
@reload templates items/*
@reload all
@reload status
@reload list
@reload track maid_classic_rpg.systems.combat

@reload module¶

Reload a Python module by its import path.

@reload module maid_stdlib.components.health

Output:

Reloading module: maid_stdlib.components.health...
Reload successful! (took 0.125s)
Changes:
  - Reloaded: maid_stdlib.components.health
Rollback available with: @rollback

@reload pack¶

Reload an entire content pack using the engine's hot-reload system. The tick loop is paused during the operation.

@reload pack stdlib

Output:

Reloading content pack 'stdlib'...
Successfully reloaded 'stdlib' in 0.450s

You can also use the shorthand syntax:

@reload stdlib

Note: When a pack reload fails, the engine automatically rolls back to the previous state. However, event handlers may not be fully restored after rollback - see Rollback Event Handler Limitation.

@reload system¶

Reload a specific ECS system by finding its module automatically.

@reload system CombatSystem

Output:

Reloading system: CombatSystem (module: maid_classic_rpg.systems.combat)...
Reload successful! (took 0.075s)
Changes:
  - Reloaded: maid_classic_rpg.systems.combat
Rollback available with: @rollback

@reload status¶

Show a hot-reload status overview including reload counts, loaded content packs (from the engine), tracked modules, and AutoReloader state.

@reload status

Output:

=== Hot Reload Status ===
  Enabled:          True
  Total reloads:    12
  Successes:        11
  Failures:         1
  Last reload:      2024-01-15 10:35:00
  Rollback avail:   True
  Snapshots:        5
  Tracked modules:  45
  Content packs:    stdlib, classic-rpg
  AutoReloader:     active, watching 2 path(s)

@reload list¶

List all tracked modules with their loaded status and dependency counts. Security-blocked modules are marked as [blocked].

@reload list

Output:

Tracked modules (45):
  maid_stdlib.commands.building.create  [loaded] deps=3 dependents=1
  maid_stdlib.commands.building.destroy [loaded] deps=2 dependents=0
  maid_stdlib.components.health         [loaded] deps=1 dependents=5
  maid_engine.auth                      [blocked] deps=0 dependents=0
  ...

@reload track / @reload untrack¶

Manage explicitly tracked modules. Tracked modules are included when running @reload all.

Track a module:

@reload track maid_classic_rpg.systems.combat

Output (success):

Now tracking module: maid_classic_rpg.systems.combat

Output (not loaded):

Module 'maid_classic_rpg.systems.combat' is not loaded in sys.modules.

Output (already tracked):

Module 'maid_classic_rpg.systems.combat' is already explicitly tracked.

Untrack a module:

@reload untrack maid_classic_rpg.systems.combat

Output:

Stopped tracking module: maid_classic_rpg.systems.combat

Watch Mode¶

Watch mode automatically reloads modules when file changes are detected.

Enabling Watch Mode¶

Via CLI:

uv run maid server start --watch

Via Configuration:

# .env file
MAID_RELOAD_WATCH_ENABLED=true
MAID_RELOAD_WATCH_DIRS=packages/
MAID_RELOAD_WATCH_PATTERNS=*.py

Watch Mode Behavior¶

File Change Detected: inotify/fsevents detects a file save
Debounce: Waits 500ms for additional changes (configurable)
Module Mapping: Determines which module(s) the file belongs to
Auto Reload: Triggers reload for affected modules
Notification: Sends message to connected admins

Watch Configuration¶

# In your .env file

# Enable/disable watch mode
MAID_RELOAD_WATCH_ENABLED=true

# Directories to watch (comma-separated)
MAID_RELOAD_WATCH_DIRS=packages/,content/

# File patterns to watch
MAID_RELOAD_WATCH_PATTERNS=*.py

# Debounce delay in seconds
MAID_RELOAD_WATCH_DEBOUNCE=0.5

# Enable/disable cascade reload to dependents
MAID_RELOAD_CASCADE_RELOAD=true

# Maximum snapshots to keep for rollback
MAID_RELOAD_MAX_SNAPSHOTS=10

Additional hot reload settings are available via the MAID_HOT_RELOAD_ prefix:

MAID_HOT_RELOAD_ENABLED=true
MAID_HOT_RELOAD_FILE_WATCH=false
MAID_HOT_RELOAD_WATCH_PATHS=
MAID_HOT_RELOAD_DEBOUNCE_DELAY=0.5
MAID_HOT_RELOAD_ENABLE_ROLLBACK=true
MAID_HOT_RELOAD_MAX_HISTORY=10

@watch¶

Manage the file watcher (AutoReloader) for automatic hot-reload on file changes. The AutoReloader is stored in world data and created on first @watch start.

Syntax:

@watch <subcommand> [args]

Subcommands:

Subcommand	Description
`start`	Start the file watcher (defaults to watching `packages/`)
`stop`	Stop the file watcher
`status`	Show watcher state, paths, patterns, backend, and reload count
`add <path>`	Add a directory or file path to watch (must exist)
`remove <path>`	Remove a watched path
`ignore <pattern>`	Add a glob ignore pattern (e.g., `*.pyc`)

Examples:

@watch start
@watch status
@watch add packages/maid-classic-rpg/src
@watch remove packages/maid-classic-rpg/src
@watch ignore "*.bak"
@watch stop

@watch start¶

Start the file watcher. On first start, it automatically watches the packages/ directory. If already running, the command reports the current state.

@watch start

Output:

File watcher started, watching 1 path(s).

@watch stop¶

Stop the file watcher.

@watch stop

Output:

File watcher stopped.

@watch status¶

Show detailed watcher state. Does not create a watcher if one hasn't been started yet.

@watch status

Output (running):

=== File Watcher Status ===
  Running:          True
  Watched paths:    2
    - packages/maid-stdlib/src
    - packages/maid-classic-rpg/src
  Match patterns:   *.py
  Ignore patterns:  *.pyc, __pycache__
  Backend:          native
  Reload count:     5

Output (not initialized):

=== File Watcher Status ===
  Running:          False
  (not initialized - use @watch start)

@watch add / @watch remove¶

Add or remove watched paths at runtime. The path must exist on disk. If the watcher is running, it is automatically restarted so the native file-system backend picks up the change.

@watch add packages/maid-tutorial-world/src

Output:

Added watch path: packages/maid-tutorial-world/src (watcher restarted)

@watch remove packages/maid-tutorial-world/src

Output:

Removed watch path: packages/maid-tutorial-world/src (watcher restarted)

@watch ignore¶

Add a glob ignore pattern so the watcher skips matching files.

@watch ignore "*.bak"

Output:

Added ignore pattern: *.bak

Module Tracking¶

The hot reload system tracks modules for reloading. Root packages are discovered dynamically from loaded content packs.

Automatic Tracking¶

Modules are automatically tracked if they belong to any loaded content pack's root package (e.g., maid_engine, maid_stdlib, maid_classic_rpg, maid_tutorial_world, maid_registry).

Manual Tracking¶

Track additional modules for hot-reload using in-game commands or programmatically.

In-game commands:

@reload track maid_classic_rpg.systems.combat
@reload untrack maid_classic_rpg.systems.combat
@reload list

See @reload track / @reload untrack and @reload list above for full details and example output.

Programmatically:

from maid_engine.reload import ReloadManager

manager = ReloadManager()
manager.track_module("my_module.path")
manager.untrack_module("my_module.path")

Track Status¶

View all tracked modules and their status with @reload list:

@reload list

Output:

Tracked modules (45):
  maid_stdlib.commands.building.create  [loaded] deps=3 dependents=1
  maid_stdlib.commands.building.destroy [loaded] deps=2 dependents=0
  maid_stdlib.components.health         [loaded] deps=1 dependents=5
  maid_engine.auth                      [blocked] deps=0 dependents=0
  maid_stdlib.systems.combat            [not loaded] deps=0 dependents=0
  ...

Rollback System¶

The rollback system allows reverting to previous module states.

How Rollback Works¶

Before each reload, a snapshot is created
Snapshots store serialized module state
Rollback restores the previous state
Limited number of snapshots are kept (default: 10)

@rollback¶

Rollback to the previous snapshot.

@rollback

Output:

Rolling back to snapshot #3...
  Restored: maid_stdlib.commands.building.create
  Rollback complete (95ms)

List Snapshots¶

@rollback list

Output:

Available Snapshots:
  #5  2024-01-15 10:35:00  maid_stdlib.commands.building.create
  #4  2024-01-15 10:30:00  maid_classic_rpg.systems.combat
  #3  2024-01-15 10:25:00  maid_stdlib.components.health
  #2  2024-01-15 10:20:00  [multiple modules]
  #1  2024-01-15 10:15:00  maid_stdlib.systems.movement

Rollback to Specific Snapshot¶

@rollback 3

Clear Snapshots¶

@rollback clear         # Clear all snapshots

Snapshot Configuration¶

# Maximum snapshots to keep
MAID_RELOAD_MAX_SNAPSHOTS=10

# Enable rollback support
MAID_HOT_RELOAD_ENABLE_ROLLBACK=true

Rollback Event Handler Limitation¶

This is a documented limitation of the hot reload system.

When a hot reload fails and the system attempts to rollback, event handlers cannot be fully restored. This may result in degraded event handling functionality.

Why This Happens¶

Event handlers in the snapshot are stored by their IDs only, not by their actual function references. This is intentional for the following technical reasons:

Stale References: Handler functions capture references to module-level state (closures, globals, class definitions) at the time they were defined. After a reload attempt - even a failed one - the old module may have been partially or fully unloaded. Restoring handlers that reference this stale state would cause AttributeError, NameError, or subtle bugs.
Memory Safety: Storing strong references to handler functions would prevent garbage collection of old module objects, leading to memory leaks across multiple reload cycles.
Module Identity: Python's reload mechanism creates NEW function objects in the new module. Even if we stored function references, we couldn't safely "transplant" the old function's bytecode - it would still reference the old module's globals dict.

Why This Is Difficult to Fix¶

Cannot serialize/deserialize arbitrary Python functions (closures, C extensions, etc.)
Cannot safely update a function's __globals__ without breaking module consistency
The event bus may have lost track of handler-to-event mappings during the failed reload
The only reliable restoration method is on_load(), which depends on the pack being in a consistent state

Symptoms¶

After a failed reload with rollback, you may notice: - Events not triggering expected handlers - Missing functionality that depends on event subscriptions - Log warnings about "event handlers may not be fully restored"

Workaround¶

If event handling appears degraded after a failed reload with rollback:

# Full server restart is the only reliable fix
uv run maid server restart

This ensures all content packs are cleanly loaded and all event handlers are properly registered.

Mitigations¶

The hot reload system provides these safety measures:

Warning Logs: When rollback cannot restore handlers, warnings are logged
on_load() Recovery: The pack's on_load() is called during restore to attempt re-registration
Event Emission: HotReloadFailedEvent is emitted so monitoring can alert operators

State Checkpointing¶

The state checkpoint system preserves combat and session state during hot reload operations, ensuring players in active combat don't end up in undefined states.

Overview¶

When a reload is triggered, the checkpoint system:

Captures combat state - Active fights, targets, cooldowns, PvP flags
Captures session state - Player connections, authentication, metadata
Pauses combat (optional) - Prevents race conditions during reload
Restores state - After successful reload, state is restored
Handles failures - On reload failure, checkpoint is discarded and rollback handles module state

Using Checkpoints¶

Automatic (Recommended):

The checkpoint manager integrates with ReloadManager via hooks:

from maid_engine.reload import ReloadManager, CheckpointManager

# Create managers
reload_manager = ReloadManager(root_packages=["maid_engine", "maid_stdlib"])
checkpoint_manager = CheckpointManager(
    world=engine.world,
    session_manager=engine.session_manager,
    pause_combat=True,  # Pause combat during reload
)

# Register as hooks - checkpoints are created/restored automatically
reload_manager.add_async_pre_reload_hook(checkpoint_manager.pre_reload_hook)
reload_manager.add_async_post_reload_hook(checkpoint_manager.post_reload_hook)

Manual (Advanced):

from maid_engine.reload import CheckpointManager, ReloadScope

checkpoint_manager = CheckpointManager(world, session_manager)

# Before reload
checkpoint = await checkpoint_manager.create_checkpoint(
    scope=ReloadScope.MODULE,
    modules=["maid_classic_rpg.systems.combat"],
)

# ... perform reload ...

# After successful reload
await checkpoint_manager.restore_checkpoint(checkpoint)

# Or on failure
await checkpoint_manager.discard_checkpoint(checkpoint)

What Gets Checkpointed¶

Combat State (per entity): - in_combat - Whether entity is in active combat - target_id - Current combat target - cooldowns - Ability cooldown timers - pvp_flag_state - PvP flags, protection, kill streaks - active_effects - Status effects with remaining durations - tactical_position - Grid position for tactical combat

Session State (per session): - player_id - Associated player entity - account_id - Authenticated account - state - Connection state (PLAYING, etc.) - metadata - Session-specific data - last_activity - Activity timestamp

System State (StatefulSystem): Systems implementing the StatefulSystem protocol have their state captured:

from maid_engine.plugins.migration import StatefulSystem

class CombatQueueSystem(System, StatefulSystem):
    def __init__(self, world: World):
        super().__init__(world)
        self._pending_attacks: list[Attack] = []
        self._cooldowns: dict[UUID, float] = {}

    def capture_state(self) -> dict[str, Any]:
        """Called before reload."""
        return {
            "pending_attacks": [a.to_dict() for a in self._pending_attacks],
            "cooldowns": dict(self._cooldowns),
        }

    def restore_state(self, state: dict[str, Any]) -> None:
        """Called after reload."""
        self._pending_attacks = [
            Attack.from_dict(a) for a in state.get("pending_attacks", [])
        ]
        self._cooldowns = state.get("cooldowns", {})

Combat Pause During Reload¶

When pause_combat=True, the checkpoint manager:

Sets world.get_data("combat_paused_for_reload") to True
Combat systems should check this flag and skip processing
After reload, the flag is cleared

In your combat system:

class CombatSystem(System):
    async def update(self, delta: float) -> None:
        # Skip if paused for reload
        if self.world.get_data("combat_paused_for_reload"):
            return

        # Normal combat processing...

Configuration¶

checkpoint_manager = CheckpointManager(
    world=engine.world,
    session_manager=engine.session_manager,
    pause_combat=True,      # Pause combat during reload (default: True)
    max_checkpoints=5,      # Keep last 5 checkpoints (default: 5)
)

Checkpoint History¶

Access previous checkpoints for debugging:

# Get checkpoint history (most recent first)
history = checkpoint_manager.checkpoint_history

for checkpoint in history:
    print(f"Checkpoint {checkpoint.checkpoint_id}")
    print(f"  Age: {checkpoint.age:.2f}s")
    print(f"  Combat states: {checkpoint.combat_count}")
    print(f"  Sessions: {checkpoint.session_count}")
    print(f"  In combat: {checkpoint.entities_in_combat}")

Serialization¶

Checkpoints can be serialized for debugging or persistence:

# Serialize
data = checkpoint.to_dict()

# Deserialize
from maid_engine.reload import StateCheckpoint
restored = StateCheckpoint.from_dict(data)

Best Practices¶

1. Design for Reloadability¶

Do:

# Store state externally
class CombatSystem(System):
    def __init__(self, world: World):
        self.world = world
        # State in world, not in system

    def get_combatants(self):
        # Query world each time
        return self.world.query(CombatComponent)

Don't:

# State stored in module/class
_combatants = {}  # Module-level state is lost!

class CombatSystem(System):
    combatants = []  # Class-level state is lost!

2. Use Lazy Initialization¶

_cached_data = None

def get_data():
    global _cached_data
    if _cached_data is None:
        _cached_data = load_expensive_data()
    return _cached_data

def on_reload():
    """Called during reload to reset state."""
    global _cached_data
    _cached_data = None

3. Implement Reload Hooks¶

# In your module
def __reload_prepare__():
    """Called before module is reloaded."""
    save_state_to_persistent_storage()

def __reload_complete__():
    """Called after module is reloaded."""
    restore_state_from_persistent_storage()

4. Test Reloads¶

# In tests
async def test_combat_system_reload():
    engine = create_test_engine()

    # Initial state
    system = engine.world.get_system(CombatSystem)
    initial_state = system.get_state()

    # Reload
    await engine.reload_system("CombatSystem")

    # Verify state preserved
    new_system = engine.world.get_system(CombatSystem)
    assert new_system.get_state() == initial_state

5. Use Version Checks¶

# For data migrations during reload
__version__ = "2.0.0"

def migrate_from_v1(old_data):
    """Migrate data from v1 format."""
    return {
        "new_field": old_data.get("old_field", "default"),
        **old_data
    }

Troubleshooting¶

"Module not found" Error¶

Problem: Module import path is incorrect.

Solution:

# Verify the import path is correct, then try reloading
@reload module maid_stdlib.commands.building

# Check reload manager status via CLI
uv run maid dev reload-status

"Cannot reload: dependent packs loaded"¶

Problem: Trying to reload a pack that other packs depend on.

Solution:

# Reload all packs in order
@reload all

# Or reload the dependent pack first
@reload pack classic-rpg
@reload pack stdlib

"State lost after reload"¶

Problem: Module-level state was lost.

Solution: - Store state in the World or document store - Implement __reload_prepare__ and __reload_complete__ hooks - Use the rollback system to recover

"Old code still running"¶

Problem: References to old objects persist.

Solution:

# Use weak references
import weakref
_handlers = weakref.WeakValueDictionary()

# Or clear registries on reload
def __reload_complete__():
    event_bus.clear_handlers(prefix="my_module")

"Watch mode not detecting changes"¶

Problem: File changes not triggering reload.

Solution:

# Manual trigger to force reload
@reload module <module_name>

# Or use the CLI watch mode
uv run maid dev watch --pack <pack_name>

Performance Issues After Reload¶

Problem: Server slower after multiple reloads.

Solution:

# Restart with fresh state (when convenient)
uv run maid server stop
uv run maid server start

# Check for memory leaks
@memory compare

Event Handlers Not Working After Failed Reload¶

Problem: After a reload failed and was rolled back, some events aren't being handled correctly. You may see warnings like "event handlers may not be fully restored" in the logs.

Cause: This is a documented limitation of the rollback system. Event handlers cannot be directly restored from snapshots because storing function references would be unsafe - they would point to stale module state.

Solution:

# The only reliable fix is a full server restart
uv run maid server stop && uv run maid server start

Prevention: - Test reload changes on a development server first - Ensure content pack on_load() methods properly register all event handlers - Monitor for HotReloadFailedEvent in your logging/alerting system

Rollback Succeeded But Events Still Broken¶

Problem: The rollback reported success, but event handling is still degraded.

Explanation: A "successful" rollback means: 1. Systems were re-registered 2. Commands were re-registered 3. The pack's on_load() was called

However, on_load() may not have re-registered all event handlers, especially if: - The handler registration happens in a separate initialization step - The pack has conditional handler registration based on state - There was a subtle error in on_load() that didn't raise an exception

Solution:

# Check the logs for warnings about handler restoration
@logs search "event handlers"

# Full server restart is the reliable fix
uv run maid server restart

Hot Reload Guide¶

Table of Contents¶

Overview¶

What Can Be Hot Reloaded?¶

How It Works¶

Limitations¶

CLI Commands¶

Start Server with Watch Mode¶

Manual Reload via CLI¶

In-Game Commands¶

@reload¶

@reload module¶

@reload pack¶

@reload system¶

@reload status¶

@reload list¶

@reload track / @reload untrack¶

Watch Mode¶

Enabling Watch Mode¶

Watch Mode Behavior¶

Watch Configuration¶

@watch¶

@watch start¶

@watch stop¶

@watch status¶

@watch add / @watch remove¶

@watch ignore¶

Module Tracking¶

Automatic Tracking¶

Manual Tracking¶

Track Status¶

Rollback System¶

How Rollback Works¶

@rollback¶

List Snapshots¶

Rollback to Specific Snapshot¶

Clear Snapshots¶

Snapshot Configuration¶

Rollback Event Handler Limitation¶

Why This Happens¶

Why This Is Difficult to Fix¶

Symptoms¶

Workaround¶

Mitigations¶

State Checkpointing¶

Overview¶

Using Checkpoints¶

What Gets Checkpointed¶

Combat Pause During Reload¶

Configuration¶

Checkpoint History¶

Serialization¶

Best Practices¶

1. Design for Reloadability¶

2. Use Lazy Initialization¶

3. Implement Reload Hooks¶

4. Test Reloads¶

5. Use Version Checks¶

Troubleshooting¶

"Module not found" Error¶

"Cannot reload: dependent packs loaded"¶

"State lost after reload"¶

"Old code still running"¶

"Watch mode not detecting changes"¶

Performance Issues After Reload¶

Event Handlers Not Working After Failed Reload¶

Rollback Succeeded But Events Still Broken¶

See Also¶