Add guarded bot recovery and daily restore-verified realm backups

docs/recovery-and-backups.md

@@ -0,0 +1,82 @@
+# Recovery and verified backups
+
+## Bot recovery
+
+The minute operations job supplies fresh, managed quest/explore intentions to
+`ale/azerothai_recovery.lua`. A native 30-second sampler independently observes
+position, map, level and health. Ten minutes without moving over five yards or
+making observable progress makes a resident eligible for recovery. Combat,
+casting, taxi travel, death, roots, movement impairment, a human master, or a
+party containing humans/offline members reset that observation. Rest, trade and
+social intentions are excluded. Inputs expire after two minutes if telemetry or
+publication stops. Every action checks the exact active native identity.
+
+Recovery clears a stale flight movement controller only when the character is
+not actually on a taxi, then resets AI caches and asks native AI for a short
+collision/path-checked walk. It never teleports, revives, changes health, resets
+levels, deletes memories, or increases the population cap. A persistent SQL log
+`acore_auth.azerothai_recovery_log` records attempts and enforces a thirty-minute
+per-resident cooldown. A walking request succeeding is not proof the bot remains
+unstuck: subsequent sampling observes actual progress. Repeated failures remain
+visible in the existing System Health stalled-bot list.
+
+## Backups
+
+`azerothai-backup.timer` runs daily at 06:00 UTC, with up to five minutes of
+jitter, and catches missed runs after boot. The last seven **verified** snapshots
+are retained under `~/azerothai-pc-lab/backups`. Backup folders and credentials
+are accessible only to the realm owner. Check `latest-status.json` or System
+Health maintenance status; failures also appear in the user systemd journal.
+
+Each snapshot includes:
+
+- Transactional MySQL dumps of auth, characters, and playerbots databases,
+  including lifecycle/recovery ledgers and transport data.
+- Online SQLite snapshots including bot memories, relationships and guild goals.
+- Runtime configuration, deployed Lua, compose configuration and AzerothAI units.
+- A Git bundle, commit identifier, file hashes, and verification manifest.
+
+Each run imports the SQL into a disposable MySQL container using the same image,
+with no network, no published ports and no live database volumes. It checks the
+restored table inventory and character/lifecycle data. SQLite backups are copied,
+reopened and integrity-checked; the restored realm state, memories and bonds are
+read. Configuration archives and the Git bundle are checked as well. The test
+container and its volume are removed afterward. A failed verification never
+replaces a verified snapshot or triggers pruning of good backups.
+
+These are live, independently consistent database snapshots, not one atomic
+cross-database timestamp. Static AzerothCore world data, client assets and model
+weights are not duplicated; reinstall the matching versions when rebuilding.
+Snapshots currently live inside the Ubuntu VM on the same physical PC: they
+protect against application mistakes and corruption, not loss of that PC/disk.
+
+## Operation and recovery
+
+Install as the realm owner:
+
+```sh
+python3 deploy/playable/install-safety-services.py
+```
+
+Reload ALE after installing the guard. Existing operations and user lingering
+services provide the intention updates and reboot startup. Trigger a backup:
+
+```sh
+systemctl --user start azerothai-backup
+```
+
+Read verification results:
+
+```sh
+cat ~/azerothai-pc-lab/backups/latest-status.json
+journalctl --user -u azerothai-backup --no-pager -n 30
+```
+
+For disaster recovery, use a fresh isolated realm first: verify manifest hashes,
+restore the source bundle at the recorded commit, recreate matching database
+containers, import `realm.sql.gz`, restore each application SQLite file to the
+path recorded in the manifest, and restore configuration with owner-only access.
+Stop the app, relay/autonomy workers and world/auth servers before replacing any
+live databases. Keep the previous live data until the recovered realm has been
+checked. Never import a backup into an actively running game. The automatic
+restore check does not overwrite the live realm.