4.0 KiB
Recovery and verified backups
Bot recovery
The minute operations job supplies fresh, managed quest/explore intentions to
ale/azerothai_recovery.lua. A native 30-second sampler independently observes
position, map, level and health. Ten minutes without moving over five yards or
making observable progress makes a resident eligible for recovery. Combat,
casting, taxi travel, death, roots, movement impairment, a human master, or a
party containing humans/offline members reset that observation. Rest, trade and
social intentions are excluded. Inputs expire after two minutes if telemetry or
publication stops. Every action checks the exact active native identity.
Recovery clears a stale flight movement controller only when the character is
not actually on a taxi, then resets AI caches and asks native AI for a short
collision/path-checked walk. It never teleports, revives, changes health, resets
levels, deletes memories, or increases the population cap. A persistent SQL log
acore_auth.azerothai_recovery_log records attempts and enforces a thirty-minute
per-resident cooldown. A walking request succeeding is not proof the bot remains
unstuck: subsequent sampling observes actual progress. Repeated failures remain
visible in the existing System Health stalled-bot list.
Backups
azerothai-backup.timer runs daily at 06:00 UTC, with up to five minutes of
jitter, and catches missed runs after boot. The last seven verified snapshots
are retained under ~/azerothai-pc-lab/backups. Backup folders and credentials
are accessible only to the realm owner. Check latest-status.json or System
Health maintenance status; failures also appear in the user systemd journal.
Each snapshot includes:
- Transactional MySQL dumps of auth, characters, and playerbots databases, including lifecycle/recovery ledgers and transport data.
- Online SQLite snapshots including bot memories, relationships and guild goals.
- Runtime configuration, deployed Lua, compose configuration and AzerothAI units.
- A Git bundle, commit identifier, file hashes, and verification manifest.
Each run imports the SQL into a disposable MySQL container using the same image, with no network, no published ports and no live database volumes. It checks the restored table inventory and character/lifecycle data. SQLite backups are copied, reopened and integrity-checked; the restored realm state, memories and bonds are read. Configuration archives and the Git bundle are checked as well. The test container and its volume are removed afterward. A failed verification never replaces a verified snapshot or triggers pruning of good backups.
These are live, independently consistent database snapshots, not one atomic cross-database timestamp. Static AzerothCore world data, client assets and model weights are not duplicated; reinstall the matching versions when rebuilding. Snapshots currently live inside the Ubuntu VM on the same physical PC: they protect against application mistakes and corruption, not loss of that PC/disk.
Operation and recovery
Install as the realm owner:
python3 deploy/playable/install-safety-services.py
Reload ALE after installing the guard. Existing operations and user lingering services provide the intention updates and reboot startup. Trigger a backup:
systemctl --user start azerothai-backup
Read verification results:
cat ~/azerothai-pc-lab/backups/latest-status.json
journalctl --user -u azerothai-backup --no-pager -n 30
For disaster recovery, use a fresh isolated realm first: verify manifest hashes,
restore the source bundle at the recorded commit, recreate matching database
containers, import realm.sql.gz, restore each application SQLite file to the
path recorded in the manifest, and restore configuration with owner-only access.
Stop the app, relay/autonomy workers and world/auth servers before replacing any
live databases. Keep the previous live data until the recovered realm has been
checked. Never import a backup into an actively running game. The automatic
restore check does not overwrite the live realm.