tws-zombies-scripts/t5/scripts/README.md

# TWS T5 Zombies scripts

Custom HUD + AFK toggle + immediate join + lobby lock + game telemetry + map records for Plutonium T5
(Black Ops 1) zombies dedicated servers. Split across eight `.gsc` files,
with `tws_t5_common` as the shared hub. Lobby lock also requires the HUD module.

Platform quirks and deploy commands: [NOTES.md](NOTES.md).

## Files

| File | Role |
|---|---|
| [tws_t5_common.gsc](tws_t5_common.gsc) | Shared config, HUD helpers, `.help`, connect/spawn registries, single `connected` listener + dispatcher |
| [tws_t5_hud.gsc](tws_t5_hud.gsc)       | Top-left Discord URL, timers, zombie count, health bar |
| [tws_t5_afk.gsc](tws_t5_afk.gsc)       | `.afk` / `.back` chat toggle + `[AFK]` indicator |
| [tws_t5_join.gsc](tws_t5_join.gsc)     | `.join` lets newcomers spawn before the current round ends |
| [tws_t5_lock.gsc](tws_t5_lock.gsc)     | Lobby `g_password` lock at a milestone round |
| [tws_t5_telemetry.gsc](tws_t5_telemetry.gsc) | Direct HTTP game start, round, heartbeat, and game end reports |
| [tws_t5_leaderboard.gsc](tws_t5_leaderboard.gsc) | Private map high-round record and all tied holders on connection |
| [tws_t5_bank.gsc](tws_t5_bank.gsc) | Persistent shared GUID bank and private balance/deposit/withdraw commands |

## Install / upgrade

Copy `tws_t5_*.gsc` to `storage/t5/scripts/sp/`, then restart the map.
When upgrading an existing installation, first remove these old scripts from
that directory:

- `tws_bank.gsc` and `tws_db.gsc` (removed modules)
- `tws_common.gsc`, `tws_hud.gsc`, `tws_afk.gsc`, `tws_join.gsc`,
  `tws_lock.gsc`, `tws_telemetry.gsc` and `tws_leaderboard.gsc` (renamed modules)

Do not leave both common scripts installed: duplicate connection listeners
break T5's connect handshake. Existing dvar names are unchanged.

## Player bank

The optional `tws_t5_bank.gsc` adds `.b`, `.d <amount|all>` and
`.w <amount|all>` and registers them in help. It requires common and telemetry
`1.0.0` or newer, the TWS t5 plugin `1.1.0`, Electric Cherry's bank API and
a private per-server credential. Follow [bank setup and recovery](../../README.md#player-bank-t5-and-t6),
including a full process restart for the plugin and a map restart for scripts.

## Persistent Discord text

The HUD shows `https://discord.thewrightserver.net` in small light-gray text
at the top-left, including for newcomers waiting in spectator mode. The label
is created once per connection and persists across death and respawn. The
gameplay HUD starts one line lower, keeping timers, help and records clear.
Install HUD and common `1.1.0` together, then restart the map; no plugin update
or full server-process restart is required.

## Map record announcements

Copy `tws_t5_leaderboard.gsc` and the updated `tws_t5_common.gsc` to
`storage/t5/scripts/sp/`, then restart the map. Leaderboard requires a
[t5-gsc-utils](https://github.com/alicealys/t5-gsc-utils) build exposing
`http::request`, `jsonParse`, `typeof` and `formatString`. A plugin upgrade
requires a full server-process restart. To omit leaderboard, remove its script;
the enable dvar does not bypass native linking dependencies.

Records are read from the public Electric Cherry API with no credentials and
work independently of telemetry and HUD. Spectator newcomers receive
the same private announcement as playable joins. T5 matches its engine map to
the API's map list. [Shared configuration, caching and validation](../../README.md#map-record-announcements)
apply to both games.

## Player commands

Type `.help` or `.commands` in chat to see available player commands and a short description.
The response is visible only to the requesting player and works while spectating,
before the first spawn. Commands are case-insensitive.
Help appears below the health bar for 12 seconds, clear of the timers, zombie
counter and health display. Typing either command again restarts that display time.

| Command | Effect |
|---|---|
| `.help`, `.commands` | List available commands |
| `.afk` | Pause with protection |
| `.back` | Return from AFK |
| `.join` | Join mid-round (newcomers only) |
| `.leaders` | Show the map record again (leaderboard module required) |

Each module registers its own help entries; removing AFK or join also removes
those entries from `.help` after a map restart.
The enabled leaderboard module registers `.leaders`, which replays the latest
cached map record privately. Repeating it restarts the record from the first page;
help temporarily hides the record until its own display ends.

## Dependency graph

Arrows read "depends on" — solid means the source names the target with
`scripts\sp\tws_t5_common::...`, dashed means the source waits on a level
notify fired by the target.

```mermaid
graph BT
    common[tws_t5_common<br/><i>config + helpers + dispatcher</i>]

    hud[tws_t5_hud]
    afk[tws_t5_afk]
    join[tws_t5_join]
    lock[tws_t5_lock]
    telemetry[tws_t5_telemetry]
    leaderboard[tws_t5_leaderboard]

    hud --> common
    afk --> common
    join --> common
    lock --> common
    telemetry --> common
    leaderboard --> common

    lock -. hud_round_changed<br/>hud_game_over .-> hud
```

- `tws_t5_common` is the only module named by other files at compile time.
- All other cross-module coupling flows through level notifies
  (`hud_round_changed`, `hud_match_started`,
  `hud_game_over`) or through spawn handlers registered with
  `tws_t5_common::register_spawn_handler` / `register_connect_handler`.
- Any file except `tws_t5_common` can be removed; the rest still load.
  `tws_t5_afk` has no downstream consumers — pull it independently.
  `tws_t5_lock` needs `tws_t5_hud` to receive round and game-over notifications.

## Runtime notifies

| Notify | Fired by | Read by |
|---|---|---|
| `hud_round_changed` | `tws_t5_hud` | `tws_t5_hud` (round-timer reset), `tws_t5_lock` |
| `hud_match_started` | `tws_t5_hud` | `tws_t5_hud` (match-timer reveal) |
| `hud_game_over`     | `tws_t5_hud` | `tws_t5_hud` (freeze timers), `tws_t5_lock` (clear password) |

---

## tws_t5_common.gsc

Hub file. Shared config, HUD helpers, spawn-handler registry, and the
single `level waittill("connected")` in the fleet. Adding a second
connect listener anywhere else crashes the T5 SP handshake.

**Features**
- Shared HUD constants: `level.hud_x`, `hud_y`, `hud_row_*`, `hud_col_*`,
  `hud_font_*`, `hud_bar_*`, `hud_c_*`, `hud_rate`, `hud_health_threshold`
- Spawn-handler registry
- Connection-handler registry for optional modules that serve spectator newcomers
- `.help` listener and help registry populated by installed command modules
- Connect listener + per-player dispatch: iterates
  `level.tws_spawn_handlers` after `spawned_player` and threads each on
  the player

**Provides**
| Function | Purpose |
|---|---|
| `common_config()`                  | Populate every `level.hud_*` (idempotent) |
| `ensure_config()`                  | No-op if config already set; called by every module's `init` |
| `register_spawn_handler(fn)`       | Appends `::fn` to `level.tws_spawn_handlers` |
| `register_connect_handler(fn)`     | Threads `::fn` after connection dispatch yields, without waiting for playable spawn |
| `register_help_command(command, description)` | Adds a player command to `.help`; call from the owning module's `init()` |
| `make_text_elem(p, x, y, ax, s, c)`| `NewClientHudElem` text element with black-glow outline |
| `make_shader_elem(p, x, y)`        | `NewClientHudElem` shader element |
| `destroy_on_disconnect(elems)`     | Waits on `disconnect`, destroys the array |
| `int_dvar_or(name, fallback)`      | Parse an integer dvar with default |
| `format_mmss(elapsed_ms)`          | Format milliseconds as `M:SS` |

**Depends on**: nothing (leaf module).

**Live validation (requires a T5 server)**

1. Type `.help` and `.HELP` while playing and while spectating before the first
   spawn. Verify all four descriptions appear below the health bar, only for the
   requesting player. Take damage so health numbers are visible and check that
   they do not cover the help text.
2. Remove AFK or join, restart the map, and verify its entries disappear while
   `.help` and the remaining commands still work. Restore the scripts afterward.
3. Repeat `.help` before 12 seconds elapse; verify one list stays visible for
   12 seconds after the latest request, then hides. Reopen it after expiry,
   die/respawn and reconnect; verify no duplicate text or leftover help elements.

---

## tws_t5_hud.gsc

Bottom-left cluster. Per-player elements spawn via `::hud_start`.

**Features**
- Match timer + round timer (round resets each round, both freeze on game over)
- Player-name row (only visible in co-op)
- Zombies-remaining counter
- Health bar with green / amber / red ramp; numbers fade in below full HP
- Round-transition and game-over notify machinery

**Provides**
| Notify / level var | Meaning |
|---|---|
| notify `hud_round_changed` | Fires each time `level.round_number` transitions |
| notify `hud_match_started` | Fires once when the match clock seeds |
| notify `hud_game_over`     | Fires on the first `end_game` / `game_ended` / `intermission` notify |
| `level.round_start_time`   | Anchored at first observed round transition |
| `level.match_start_time`   | Anchored at first observed round-1 (or on first player spawn) |

Spawn handler registered: `::hud_start` — threads `hud_timer_row`,
`hud_zombie_counter`, `hud_health_display`.

**Depends on**
- `scripts\sp\tws_t5_common::` — `ensure_config`, `make_text_elem`,
  `make_shader_elem`, `destroy_on_disconnect`, `format_mmss`,
  `register_spawn_handler`
- Every `level.hud_*` config value from `tws_t5_common`
- Stock BO1 state: `level.round_number`, `level.zombie_total`,
  `level.players`, `self.health`, `self.maxhealth`, `self.name` /
  `self.playername`, AI entities' `.animname` and `.ignore_enemy_count`

---

## tws_t5_afk.gsc

Self-invoked pause mode. Owns the `[AFK]` indicator element entirely — HUD
module has zero knowledge of AFK.

**Features**
- Chat commands `.afk` and `.back` (via `level waittill("say")` — not
  `callbackPlayerSay`, which never fires on T5 SP dedicated)
- On enter: `enableInvulnerability`, `freezeControls(true)`,
  `self.ignoreme = true` (drops out of zombie targeting)
- On exit: reverse, then a 2s grace tail keeps invuln + ignoreme through
  the target-reacquisition window
- `[AFK]` HUD element created per-player via spawn handler; alpha-toggled
  by enter / exit

**Provides**
- Per-player: `self.hud_afk_on` (bool), `self.hud_afk_elem` (HUD element),
  writes `self.ignoreme` (stock BO1 targeting flag)
- Level var: `level.afk_grace_seconds` (default 2)

**Depends on**
- `scripts\sp\tws_t5_common::` — `ensure_config`, `make_text_elem`,
  `destroy_on_disconnect`, `register_spawn_handler`, `register_help_command`
- Shared config: `level.hud_x`, `level.hud_row_match`,
  `level.hud_font_small`, `level.hud_c_afk`
- Stock BO1: `enableInvulnerability`, `disableInvulnerability`,
  `freezeControls`, `.ignoreme`, `level waittill("say", message, player)`

Spawn handler registered: `::afk_spawn`.

---

## tws_t5_join.gsc

Type `.join` in chat while spectating after joining a match in progress to
spawn immediately. Install with `tws_t5_common.gsc`; no HUD dependency.

**Eligibility**

- Only before the player's first playable spawn on the current connection.
- A normal spawn or successful `.join` permanently closes eligibility for
  that connection. Getting downed, bleeding out, or changing rounds never
  restores it. Dead players must wait for the normal round-end respawn.
- Requests during connection setup, without a living teammate, or after
  game over cannot spawn a player. Repeated `.join` requests cannot create
  overlapping spawns.
- This is connection-local tracking; reconnecting creates a new player and
  resets eligibility. It is not an account-level reconnect restriction.

Uses the stock `level.spawnPlayer` entry point and preserves map-specific
`level.custom_spawnPlayer` callbacks. If absent, installs the stock zombies
spectator callback, as the round-end routine does. Missing spectator spawn
locations are seeded from the map's `initial_spawn_points`.

Spawn handler registered: `::join_spawn`. Stock `self.initialized` provides
an additional first-spawn guard; `hasSpawned` cannot distinguish spectators.
See the [stock zombies lifecycle](https://github.com/plutoniummod/t5-scripts/blob/main/ZM/Common/maps/_zombiemode.gsc)
and [spawn entry point](https://github.com/plutoniummod/t5-scripts/blob/main/ZM/Common/maps/_callbackglobal.gsc).

**Live validation (requires a T5 server)**

1. Join during an active round, type `.join`, and verify one playable spawn
   with normal weapons, controls, zombie targeting, and HUD initialization.
   Check both round 1 and a later round, including each deployed map.
2. Repeat `.join` rapidly; verify no duplicate spawns. Try it alive, while
   downed, and after bleeding out; verify no revive or respawn.
3. Join and wait for the normal round-end spawn instead; after dying, verify
   `.join` remains blocked, including on subsequent rounds.
4. Try during round transition and game over; verify no duplicate spawn or
   restart of an ended match. Check that `.afk` / `.back` still work.

---

## tws_t5_lock.gsc

**Features**
- Sets `g_password` to a random per-lobby value when the round crosses a
  milestone (default: round 20)
- Existing players are unaffected; `g_password` is only checked at connect
- Clears `g_password` on `hud_game_over` so the next lobby opens unlocked
- Trigger round is dvar-driven — retune without redeploying

**Provides**
| Level var | Meaning |
|---|---|
| `level.lock_round`  | Round at which the lobby locks (read at init) |
| `level.lock_active` | `true` after the password has been set |

**Depends on**
- `scripts\sp\tws_t5_common::` — `ensure_config`, `int_dvar_or`
- `tws_t5_hud` — waits on `hud_round_changed` and `hud_game_over`
- Stock BO1: `level.round_number`
- Dvar: `tws_lock_round` (int, default `20`)

---

## Dvars

| Dvar | Consumer | Default | Purpose |
|---|---|---|---|
| `zombiemode`     | every module | — | Gate: every `init()` returns early unless `"1"` |
| `tws_lock_round` | `tws_t5_lock`   | `20` | Round at which the lobby locks |
| `tws_server_id`  | `tws_t5_telemetry` | Required for telemetry | Exact registered Electric Cherry server name |
| `tws_telemetry_enabled` | `tws_t5_telemetry` | `1` | Set to `0` to disable telemetry at init |
| `tws_telemetry_url` | `tws_t5_telemetry` | `https://ec.thewrightserver.net/api/telemetry` | Electric Cherry telemetry endpoint for JSON POSTs |
| `tws_telemetry_log_level` | `tws_t5_telemetry` | `1` | Read live: `0` warnings/errors/recovery only, `1` lifecycle/request/health logs, `2` also capture/queue logs and response previews |

## tws_t5_telemetry.gsc

Uses `tws-t5-gsc-utils` HTTP/JSON components to POST directly to Electric Cherry's
`/api/telemetry`. Install alongside `tws_t5_common.gsc` in `storage/t5/scripts/sp/`.
No HUD dependency. Reports run whether or not a beta is active.
Follow the shared [authentication setup](../../README.md#telemetry-authentication-t5-and-t6)
to configure the service before upgrading. The reporter token is embedded in
`level.telemetry_token`; it must match the service's `EC_TELEMETRY_TOKEN`. Short
embedded tokens and non-HTTPS URLs disable reporting at init.

Example server config (except log level, read at script init; restart the map after changes):

```cfg
set tws_server_id "t5_zombie_prototype_1"
set tws_telemetry_enabled "1"
set tws_telemetry_url "https://ec.thewrightserver.net/api/telemetry"
set tws_telemetry_log_level "1"
```

`tws_server_id` must match the registered server name exactly. If it is
unset, the script logs a configuration error and does not start reporting.
An unknown or destroyed server is rejected by Electric Cherry with HTTP 404.

The reporter emits exactly four event types:

| Event | When | Payload |
|---|---|---|
| `game_start` | First playable spawn, with the stock `begin_spawning` flag as a fallback | `server`, `map`, `players` with `name` and `guid` |
| `round_change` | First observed round after start, then each change (checked every 0.5 seconds) | `server`, `map`, `round`, full player stats |
| `heartbeat` | Every 30 seconds after `game_start`, until game end | Same fields as `round_change`, with current round and player stats |
| `game_end` | First stock `end_game`, `game_ended`, or `intermission` notification | `server`, `map`, final `round`, full player stats |

All four events also include `match` and `seq` (see delivery below).

Each player in round/heartbeat/end reports has `name`, `guid`, `kills`, `headshots`,
`downs`, `revives`, `score`, and `score_total`. Names and GUIDs remain strings; GUIDs never
pass through numeric conversion. Round and stats are nonnegative integers,
with missing or negative stat values reported as zero. The four counters
come from T5's `player.stats`; `score` is the player's current spendable
`player.score`. `score_total` reads the stock `player.score_total` cumulative
match score used by the final scoreboard; spending points does not reduce it.
It follows stock scoring rules, including grants excluded from the total.
Missing or negative totals report as zero, independently of the balance.
`game_start` includes only player identity. `map` is `mapname`.

Stock `end_game` yields before `player_intermission` replaces `player.score`
with `player.score_total`. The reporter captures both immediately at the first
end signal without yielding; retries and replay retain that exact snapshot.
Custom scripts that overwrite the score before emitting any supported end
signal must preserve this ordering to retain a spendable final balance.
Deploy receiver support for `score_total` before the reporters. Old payloads
remain valid; the receiver treats their missing totals as unknown, not zero.

Every report includes everyone returned by `getPlayers()`, including
spectators, downed players, and AFK players. Empty rosters are sent as `[]`.
Undefined players and players with missing/empty names or GUIDs are skipped
with a warning containing their roster index; a later snapshot can include
them once their identity is available.
The script builds the JSON array delimiters explicitly because the plugin
serializes an empty GSC array as `null`; player objects and strings still
use `jsonSerialize` for escaping.

```json
{"event":"game_start","match":"process-token-1","seq":1,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","players":[{"name":"Josh","guid":"12345678901234567"}]}
{"event":"round_change","match":"process-token-1","seq":2,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":420,"headshots":160,"downs":1,"revives":3,"score":32000,"score_total":62000}]}
{"event":"heartbeat","match":"process-token-1","seq":3,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":430,"headshots":165,"downs":1,"revives":3,"score":32500,"score_total":62500}]}
{"event":"game_end","match":"process-token-1","seq":4,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":18,"players":[{"name":"Josh","guid":"12345678901234567","kills":445,"headshots":170,"downs":2,"revives":3,"score":33500,"score_total":65000}]}
```

Events are captured immediately into an in-memory queue. One HTTP worker
POSTs snapshots in gameplay order, with `Content-Type:
application/json` and `Authorization: Bearer <token>`, waiting for completion
before sending the next. Gameplay and round detection continue while requests
are pending. Repeated joins or
end notifications do not duplicate start/end reports. At game over, the
reporter captures any round change since the last poll, stops detection,
and lets the sender drain through `game_end`.

The first heartbeat is captured 30 seconds after match start, then every
30 seconds even when the round is unchanged or the roster is empty. Round
changes and later spawns do not reset the interval or add heartbeat workers.
Heartbeats use the same snapshot builder and ordered sender as round reports;
pending HTTP requests can delay transmission. Game end stops the heartbeat
worker. Only the newest waiting heartbeat is retained; round reports and the
in-flight request retain their order before `game_end`.

Each report includes a native-generated `match` key and a positive `seq`, starting
at 1 for `game_start`. Start/end snapshots are written to disk before sending and
replayed before a new match after map/process restarts. Retryable failures retain
the queue head with 1–30 second backoff; other 4xx are terminal. Acceptance requires
a 2xx response containing the JSON boolean `ok: true`, and logs include Electric
Cherry's `telemetry.timing.matchId`. The native 5-second connect / 15-second total
transfer limits still apply. Ordinary round/heartbeat failures are best effort.

**Deploy Electric Cherry #144 and the updated TWS plugin before these scripts.**
See the shared [delivery and rollout notes](../../README.md#telemetry-delivery-and-rollout-t5-and-t6)
for outbox location, persistence requirements, capacity/error handling and tests.

**Diagnosing a lobby that stops reporting**

Default logging includes initialization, match start/end, each request's
event/round and local ID, queue wait and request duration in milliseconds,
transport failures/recovery, HTTP rejection/status/timings, and the final queue drain.
Each log identifies the original match key and payload sequence, including replays. Every log includes
the server, map, and game time in milliseconds.

An independent watchdog prints health every 30 seconds, even before match
start and while HTTP is pending or the game-end queue is draining. It shows
the sender stage, current snapshot capture, pending queue size (including
the retained in-flight head), total captured, accepted reports, completed transports, transport failures,
HTTP rejections, and
time since the last completion (`-1` means none). Warnings remain enabled
at log level `0`:

- `HTTP stalled`: a request has been pending for at least 30 seconds. The
  warning repeats on watchdog ticks with its ID, event, age, and queue size.
- `sender is not consuming`: the oldest queued report has waited at least
  30 seconds with no in-flight request outside an intentional retry backoff.
- `round watcher has stopped advancing`: no polling progress for over
  5 seconds, detected on the next watchdog tick.
- `no snapshot captured`: an active match has gone at least 65 seconds
  without a new snapshot, despite its 30-second heartbeat.

Set `tws_telemetry_log_level "2"` in the server console while investigating;
no map restart is needed. This adds capture/queue stages and API response
previews, escaped onto one line and limited to the first 256 characters.
Look for `ok: false` and the rejection reason in those previews; HTML may
indicate a proxy error. Previews can include player identities echoed by
the API. Restore `1` afterward, or use `0` to retain only warnings, errors,
and transport recovery messages.

Missing handles, transport errors, empty/invalid acknowledgments and retryable
HTTP responses keep start/end pending. The watchdog never launches overlapping
requests. A request pending beyond 30 seconds warrants checking the installed
plugin version, worker-queue delay and completion delivery. Native deadlines,
diagnostics and the telemetry outbox/acknowledgment helpers require the updated
[TWS plugin build](../plugins/README.md).
The upstream baseline contains
[`http::request`](https://github.com/alicealys/t5-gsc-utils/blob/main/src/component/http.cpp)
and [`jsonSerialize`](https://github.com/alicealys/t5-gsc-utils/blob/main/src/component/json.cpp).
Disabling the dvar does not remove that compile-time dependency; remove the
script when using a plugin build without those functions.

Stat and lifecycle references: [stock T5 zombies](https://github.com/plutoniummod/t5-scripts/blob/main/ZM/Common/maps/_zombiemode.gsc)
and [score tracking](https://github.com/plutoniummod/t5-scripts/blob/main/ZM/Common/maps/_zombiemode_score.gsc).

**Live validation (requires a T5 server)**

1. Start a match and verify one `game_start`, then an initial `round_change`
   and one per new round. Check the exact server name, map, string GUIDs,
   all player identities, and all six stats against the game. Earn points, then
   spend some: the balance should decrease while the total stays unchanged.
2. Join again in the same round, go AFK, become downed, and spectate. Verify
   the next report includes the full roster without another start event.
3. End a match and verify one `game_end` with both scores captured before the
   scoreboard overwrite; compare `score_total` to the final scoreboard. Verify
   the final stats arrive after all round reports, including if more than one
   stock end notification fires. Check
   a round-0 intro and an empty roster, including last-player disconnect.
4. Delay HTTP completion while rounds advance and the match ends. Verify
   requests remain ordered and retain the stats captured at each event.
   With the API unreachable, verify transport-error logging and continued
   gameplay. Repeat without an active beta and with the HUD module removed.
5. Stay in one round for over a minute. Verify heartbeats at 30 and 60 seconds
   after start, with current stats and the same fields as `round_change`.
   Check that round changes and joins do not reset or duplicate the interval,
   and no heartbeat follows `game_end`, including with requests pending.
6. Change log level between `0`, `1`, and `2` without a map restart. Check
   request IDs pair sends with completions and level `2` shows bounded,
   escaped response previews. Use an unregistered server name on a test
   server and confirm the API rejection is visible in the response even
   though transport completes. Also check an empty or HTML response.
7. Point a test server at an endpoint that accepts a connection but does
   not reply. With the TWS plugin, verify a native timeout near 15 seconds and
   a retained start/end retry with bounded backoff while gameplay continues.
   Restore the endpoint and verify the queue resumes in order and
   the watchdog stops after the final drain. Test a refused connection
   separately to check failure counts and the recovery log once restored.

## External dependencies

[t5-gsc-utils](https://github.com/alicealys/t5-gsc-utils) is required for telemetry
and leaderboard HTTP, JSON, type and string helpers. Install a build exposing
the natives listed in each module's documentation, then restart the server
process. No database is required.