83 lines
4.0 KiB
Markdown
83 lines
4.0 KiB
Markdown
# Recovery and verified backups
|
|
|
|
## Bot recovery
|
|
|
|
The minute operations job supplies fresh, managed quest/explore intentions to
|
|
`ale/azerothai_recovery.lua`. A native 30-second sampler independently observes
|
|
position, map, level and health. Ten minutes without moving over five yards or
|
|
making observable progress makes a resident eligible for recovery. Combat,
|
|
casting, taxi travel, death, roots, movement impairment, a human master, or a
|
|
party containing humans/offline members reset that observation. Rest, trade and
|
|
social intentions are excluded. Inputs expire after two minutes if telemetry or
|
|
publication stops. Every action checks the exact active native identity.
|
|
|
|
Recovery clears a stale flight movement controller only when the character is
|
|
not actually on a taxi, then resets AI caches and asks native AI for a short
|
|
collision/path-checked walk. It never teleports, revives, changes health, resets
|
|
levels, deletes memories, or increases the population cap. A persistent SQL log
|
|
`acore_auth.azerothai_recovery_log` records attempts and enforces a thirty-minute
|
|
per-resident cooldown. A walking request succeeding is not proof the bot remains
|
|
unstuck: subsequent sampling observes actual progress. Repeated failures remain
|
|
visible in the existing System Health stalled-bot list.
|
|
|
|
## Backups
|
|
|
|
`azerothai-backup.timer` runs daily at 06:00 UTC, with up to five minutes of
|
|
jitter, and catches missed runs after boot. The last seven **verified** snapshots
|
|
are retained under `~/azerothai-pc-lab/backups`. Backup folders and credentials
|
|
are accessible only to the realm owner. Check `latest-status.json` or System
|
|
Health maintenance status; failures also appear in the user systemd journal.
|
|
|
|
Each snapshot includes:
|
|
|
|
- Transactional MySQL dumps of auth, characters, and playerbots databases,
|
|
including lifecycle/recovery ledgers and transport data.
|
|
- Online SQLite snapshots including bot memories, relationships and guild goals.
|
|
- Runtime configuration, deployed Lua, compose configuration and AzerothAI units.
|
|
- A Git bundle, commit identifier, file hashes, and verification manifest.
|
|
|
|
Each run imports the SQL into a disposable MySQL container using the same image,
|
|
with no network, no published ports and no live database volumes. It checks the
|
|
restored table inventory and character/lifecycle data. SQLite backups are copied,
|
|
reopened and integrity-checked; the restored realm state, memories and bonds are
|
|
read. Configuration archives and the Git bundle are checked as well. The test
|
|
container and its volume are removed afterward. A failed verification never
|
|
replaces a verified snapshot or triggers pruning of good backups.
|
|
|
|
These are live, independently consistent database snapshots, not one atomic
|
|
cross-database timestamp. Static AzerothCore world data, client assets and model
|
|
weights are not duplicated; reinstall the matching versions when rebuilding.
|
|
Snapshots currently live inside the Ubuntu VM on the same physical PC: they
|
|
protect against application mistakes and corruption, not loss of that PC/disk.
|
|
|
|
## Operation and recovery
|
|
|
|
Install as the realm owner:
|
|
|
|
```sh
|
|
python3 deploy/playable/install-safety-services.py
|
|
```
|
|
|
|
Reload ALE after installing the guard. Existing operations and user lingering
|
|
services provide the intention updates and reboot startup. Trigger a backup:
|
|
|
|
```sh
|
|
systemctl --user start azerothai-backup
|
|
```
|
|
|
|
Read verification results:
|
|
|
|
```sh
|
|
cat ~/azerothai-pc-lab/backups/latest-status.json
|
|
journalctl --user -u azerothai-backup --no-pager -n 30
|
|
```
|
|
|
|
For disaster recovery, use a fresh isolated realm first: verify manifest hashes,
|
|
restore the source bundle at the recorded commit, recreate matching database
|
|
containers, import `realm.sql.gz`, restore each application SQLite file to the
|
|
path recorded in the manifest, and restore configuration with owner-only access.
|
|
Stop the app, relay/autonomy workers and world/auth servers before replacing any
|
|
live databases. Keep the previous live data until the recovered realm has been
|
|
checked. Never import a backup into an actively running game. The automatic
|
|
restore check does not overwrite the live realm.
|