tws-zombies-scripts/README.md

# tws-zombies-scripts

Combined working repo for the TWS Plutonium Zombies scripts and native plugins across two games.

## Layout

- [t5/scripts/](t5/scripts/) — T5 GSC modules, [README](t5/scripts/README.md), and [notes](t5/scripts/NOTES.md).
- [t5/plugins/](t5/plugins/) — `tws-t5-gsc-utils`, based on [alicealys/t5-gsc-utils](https://github.com/alicealys/t5-gsc-utils), with TWS HTTP changes.
- [t6/scripts/](t6/scripts/) — T6 GSC modules, [README](t6/scripts/README.md), [notes](t6/scripts/NOTES.md), and compiler checks.
- [t6/plugins/](t6/plugins/) — `tws-t6-gsc-utils`, based on [alicealys/t6-gsc-utils](https://github.com/alicealys/t6-gsc-utils), with the same HTTP changes.

The scripts still deploy to the game's existing `storage/t5/scripts/sp/` or
`storage/t6/scripts/zm/` directories; the new source layout does not change GSC namespaces.

## Building plugins

Plugin source is vendored as regular tracked files so local modifications travel
with this repository. Each plugin's `upstream.json` records its upstream revision
and exact Git dependency revisions. Dependencies and build outputs are ignored.
The dependency loader replaces upstream's submodule setup; nested upstream
`.github` files are retained for reference and do not run as this repo's CI.

Use Windows with Visual Studio 2022 C++ build tools, a Windows SDK, Git, and
Python 3.9+. From a Developer PowerShell with `msbuild` on PATH, at the repository root:

```powershell
.\tools\build_plugins.ps1 -Game all -Configuration Release
```

This restores pinned dependencies, generates both projects, and builds x86 DLLs.
Upstream's Premake setup also downloads its MySQL client dependency on first use.
Outputs are `t5/plugins/tws-t5-gsc-utils/build/bin/Release/tws-t5-gsc-utils.dll` and
`t6/plugins/tws-t6-gsc-utils/build/bin/Release/tws-t6-gsc-utils.dll`.
Remove the old `t5-gsc-utils.dll` or `t6-gsc-utils.dll` before installing the
corresponding `tws-` DLL in `Plutonium/plugins/`; load only one version per game.
**Restart the server process**;
restarting only the map cannot load an updated plugin.

Use `python tools/fetch_plugin_deps.py all --check` to verify dependency revisions
without fetching. Use `python3 tools/test_plugin_http.py` on macOS/Linux with a
C++20 compiler and libcurl development files to compile the real HTTP utilities
and test local stalled connections, response deadlines, status reporting and recovery.
These portable tests do not build the Windows DLLs or exercise Plutonium's VM.
Run `python3 tools/test_plugin_component.py` to compile the completion/option
handling against host engine shims with address and undefined-behavior sanitizers.

Both plugins now default to a 5-second connection timeout and 15-second total
transfer timeout. Request options can override these with positive millisecond
values. Telemetry explicitly sets those limits and logs native HTTP status and
timings. See the [T5](t5/plugins/README.md) and [T6](t6/plugins/README.md) plugin notes.
Telemetry retries and durable replay require the updated plugins; see the
[delivery and rollout notes](#telemetry-delivery-and-rollout-t5-and-t6).

## Telemetry delivery and rollout (T5 and T6)

**Deploy Electric Cherry [#144](https://gitea.thewrightserver.net/josh/electric-cherry/issues/144)
before these reporters.** Replay requires the receiver to associate reports with
`match` and deduplicate `seq`; the legacy receiver can apply an old end to a newer
match. Build and install both updated TWS plugins, restart the server processes,
then deploy the corresponding scripts. The new `telemetryNewMatch`,
`telemetryLoad`, `telemetryStore`, `telemetryRemove` and `telemetryResponse`
functions are required even if telemetry is disabled by dvar.

Every captured report includes a `match` key (random native process token plus
an increasing match counter, at most 64 characters) and a positive `seq`.
`game_start` is sequence 1. Retries and replay send the exact original JSON;
sequence numbers are never reassigned. Player snapshots still reflect capture
time; no client timestamp has been added and API receipt times retain their
existing meaning.

A single sender keeps `game_start` and `game_end` at the queue head on transport
failure, native timeout, HTTP 429 or 5xx. Delays are 1, 2, 4, 8, 16, then 30 seconds
between attempts, capped at 30 seconds without a retry-count limit. Other 4xx
(and redirects) are logged and dropped. A 2xx with a JSON boolean `"ok": true`
acknowledges delivery; empty, malformed or negative 2xx replies and missing HTTP
diagnostics are unacknowledged and retried for boundaries. Accepted reports log
Electric Cherry's `telemetry.timing.matchId` as `ec_match_id` when supplied.
Heartbeats and round changes remain best effort. Waiting heartbeats are coalesced
to the newest snapshot, preserving round reports and any in-flight request.
Intentional coalescing can produce gaps in `seq`.

The native outbox saves start/end snapshots synchronously before sending, and
removes them only after acknowledgment or a terminal HTTP rejection. Files live
under `fs_homepath/tws-telemetry/t5/` or `t6/`, scoped to the exact server name and
endpoint. Keep this directory on persistent writable storage across process or
container restarts; it contains player snapshots but no authorization token.
Use a single process per server/endpoint and storage directory. Endpoint/server
changes use a separate outbox, so restore the original configuration to replay
its backlog. Keep `tws_server_id` stable between maps. On map/process startup,
recovered boundaries precede any new `game_start`. An accepted report whose
acknowledgment/removal was lost is safely replayed with the same identity.

Writes replace the committed file only after a complete temporary write. A
storage error blocks the affected boundary before sending and logs an error;
unsaved in-memory reports cannot survive a restart. Unreadable/corrupt saved data
disables telemetry until storage is repaired and the map restarted, preserving
the file for recovery. Each outbox is limited to 4,096 reports / 16 MiB, with
individual report bodies below 20 KiB; reaching a limit fails visibly without
discarding the saved backlog. Disk removal failures retry cleanup without sending
an already resolved report again during the same map. The plugin cannot recover
a `game_end` that was never captured (for example, a process crash during play).

Run `node tools/check_telemetry.mjs` for both reporters' queue/retry fixtures and
`python3 tools/test_plugin_telemetry.py` for native storage, acknowledgment and
process-restart checks with sanitizers. The native test uses each restored JSON
dependency; alternatively pass `--json-header '/path/{game}-json.hpp'` with the
single headers from each plugin's pinned revision. These tests use engine shims.
Validate Windows DLL linking and both games in Plutonium before rollout: force a
timeout/429/503 at start and end, confirm permanent 400/401 failures do not retry,
restart a map and then a process with pending boundaries, and confirm their
original match IDs complete before the next start. Also verify a 2xx without
`ok: true` stays pending, an outage retains only the latest waiting heartbeat,
and a disk failure is visible without advancing past an unsaved boundary.

## Parity

T5 and T6 are separate source trees, not a shared library — GSC namespaces, includes, and stock APIs differ enough between the two engines that a shared file is not practical. Feature parity is enforced pairwise: a change landed in `t5/` should be considered for a companion change in `t6/`, adapted to T6 conventions.

Modules currently in each tree:

| Module     | T5                            | T6                              |
|------------|-------------------------------|---------------------------------|
| common     | `t5/scripts/tws_t5_common.gsc`           | `t6/scripts/tws_t6_common.gsc`          |
| hud        | `t5/scripts/tws_t5_hud.gsc`              | `t6/scripts/tws_t6_hud.gsc`             |
| afk        | `t5/scripts/tws_t5_afk.gsc`              | `t6/scripts/tws_t6_afk.gsc`             |
| lock       | `t5/scripts/tws_t5_lock.gsc`             | `t6/scripts/tws_t6_lock.gsc`            |
| telemetry  | `t5/scripts/tws_t5_telemetry.gsc`        | `t6/scripts/tws_t6_telemetry.gsc` — queued `game_start` / `heartbeat` / `round_change` / `game_end` matching T5 |
| join       | `t5/scripts/tws_t5_join.gsc`             | `t6/scripts/tws_t6_join.gsc`            |
| leaderboard | `t5/scripts/tws_t5_leaderboard.gsc`    | `t6/scripts/tws_t6_leaderboard.gsc`        |

## Map record announcements

Install `tws_t5_leaderboard.gsc` for T5 or `tws_t6_leaderboard.gsc` for T6,
**along with the updated common module** for that game.
The new module privately announces the current map's fleet-wide highest recorded
round and every tied holder once per connection, including spectator newcomers.
Type `.leaders` to replay the record privately from the latest usable cache.
Repeating the command restarts the announcement from its first page. It appears
in `.help` when the leaderboard module is enabled. If no usable record is
available, the command asks you to try again shortly.
It reads Electric Cherry's public `/api/leaderboard` endpoint without a token;
only completed matches contribute. Ties may come from the same run or separate
runs. Names are the API's latest player names, with players identified by GUID.

Both versions default to `https://ec.thewrightserver.net/api/leaderboard`.
T5 discovers the map by engine identifier. T6 selects the exact variant using
the active mode/location, so Town Survival, TranZit and Town Grief stay separate.
See the game READMEs for plugin requirements and installation paths.

| Dvar | Default | Purpose |
| --- | --- | --- |
| `tws_leaderboard_enabled` | enabled | `0` disables announcements and requests |
| `tws_leaderboard_url` | endpoint above | Full HTTPS API endpoint, without query parameters |
| `tws_leaderboard_map_key` | automatic | Exact, decoded API map key when automatic selection is unavailable |

For example, `set tws_leaderboard_map_key "t6:zm_transit:town"` explicitly
selects Town Survival. Normally this setting is unnecessary. An override must
exist in the API's map list and match the current game and engine map. Clear it
when changing variants; restart the map after changing these settings.

The announcement starts five seconds after player readiness and waits for the
initial record fetch if necessary. It appears in the same HUD area as `.help`,
below the health bar, with a heading such as `Record: Round 23` and holder names
underneath. Each page shows up to three holder lines for 12 seconds. Opening
`.help` hides the record and pauses its display time until help closes.
All holders are included; colour/control codes are removed, and unusually long
names are shortened to keep messages readable. HUD elements are removed when
the announcement finishes, the player disconnects, or the game ends.

One level worker fetches the map list and pages of three players, stopping when
the highest-round tie ends. Successful snapshots refresh every five minutes.
Failures retry after 30 seconds and retain the last complete snapshot for up to
15 minutes. An empty selection says “No recorded high round yet”; unavailable,
malformed or ambiguous results are logged and skipped when no usable cache exists.
Partial page results never replace the cache. The API has no pagination snapshot
token: changed totals/record rounds reject the refresh, but concurrent renames can
still move entries between pages. Refreshes deduplicate GUIDs.

Requests have a 15-second GSC wait limit. The TWS plugins also enforce native
5-second connection and 15-second transfer limits. The leaderboard waits for native
completion before starting another request;
player threads never wait directly on HTTP. Responses at the plugins' 20 KiB
truncation boundary are rejected. The complete JSON grammar, UTF-8, string escapes
and response shape are checked before consuming data. These guards also reject
HTML/error bodies. Leaderboard still uses body validation; the new native HTTP
status diagnostics are consumed by telemetry.

Run the portable data/structure checks with `node tools/check_leaderboard.mjs`.
They execute the scripts' data helpers with engine-value shims, not a game VM.
Live checks: join while playing and spectating; die/respawn (no repeat); reconnect
(one new announcement); compare Town/TranZit/Grief; check one holder, ties across
pages, no records, and a slow/unavailable/malformed API. Disconnect or end the game
mid-announcement and confirm messages stop. Verify existing HUD, `.help`, `.join`
and telemetry still work. Open `.help` during a record page and check that the
record resumes afterwards without overlapping it. Runtime linking and client
readiness require Plutonium.
Try `.leaders` and `.LEADERS` after the announcement ends and during a later
page: each should start one private announcement from the first page. Repeat
while help is open, then disconnect or end the game during a replay to check
cleanup. Disable the leaderboard module and verify `.leaders` is absent from help.

## Telemetry authentication (T5 and T6)

Both reporters default to `https://ec.thewrightserver.net/api/telemetry` and send
`Authorization: Bearer <token>` plus `Content-Type: application/json` on every
POST. Authentication is also used for retried/replayed reports. An explicit
`tws_telemetry_url` override must also use HTTPS; remove any old HTTP override.

Before upgrading the API or loading these reporters:

1. Set the Electric Cherry service's **environment variable**
   `EC_TELEMETRY_TOKEN` to a separate shared secret of at least 32 characters.
   A missing service token disables ingestion.
2. Both telemetry scripts embed the matching reporter secret in
   `level.telemetry_token`. No game-server token dvar or private config is needed.
3. Keep `tws_server_id` set to the exact registered server name, then restart
   the map. Both reporters refuse to start with a short embedded token or a
   non-HTTPS URL. Rotating the service token requires editing the token in both
   scripts, deploying them to every game server, and restarting the maps.

The API also accepts `X-EC-Telemetry-Token`; these reporters use Bearer auth.
Cookies, query parameters, and JSON fields cannot authenticate telemetry.
Missing or incorrect request tokens are rejected before payload parsing or
recording. Reporters that cannot set headers need a trusted relay with an HTTPS
reporter-facing endpoint. The plugins used here accept a headers array.

After deployment, verify all four event types reach Electric Cherry. On a test
server, also verify that a short embedded token and an HTTP URL prevent startup,
that a wrong token is rejected without recording an event, and that deploying
the correct token restores ingestion. Plugin transport completion alone does
not prove API acceptance; inspect the API response or ingestion records.

## Validating T6

```powershell
powershell -File t6/scripts/tools/check.ps1 -CompilerPath C:\path\to\gsc-tool.exe
```

Requires PowerShell and `gsc-tool` (T6 PC).