Files
tws-zombies-scripts/README.md
T
josh 463b4bcc51 Add safe bank failure diagnostics for T5 and T6
Address #14 with request correlation, bounded parser reasons, startup version reporting, and rate-limited storage and settlement logs. Preserve accounting and API identity contracts, extend portable tests, and update release versions and rollout guidance.
2026-10-13 10:02:36 -04:00

29 KiB
Raw Blame History

tws-zombies-scripts

Combined working repo for the TWS Plutonium Zombies scripts and native plugins across two games.

Changelog · Current component versions

Layout

The scripts still deploy to the game's existing storage/t5/scripts/sp/ or storage/t6/scripts/zm/ directories; the new source layout does not change GSC namespaces.

Versioning

versions.json records the current source version of each individual script and each native plugin. Every .gsc file has its own version, keyed by its exact filename. Changing one script bumps only that script's version; other scripts increase only when their own source changes.

Component Manifest key Source
Each T5 script Exact filename, e.g. tws_t5_afk.gsc Corresponding file in t5/scripts/
Each T6 script Exact filename, e.g. tws_t6_afk.gsc Corresponding file in t6/scripts/
T5 plugin t5-plugin t5/plugins/tws-t5-gsc-utils/
T6 plugin t6-plugin t6/plugins/tws-t6-gsc-utils/

Every completed script or plugin change includes a version bump for each affected component and an entry in CHANGELOG.md. Each entry identifies each changed script or plugin and its new version, compatibility requirements, and deployment steps. Shared helpers such as tws_t5_common.gsc have their own versions; document required dependency versions when compatibility changes. A shared helper change alone does not bump its unchanged consumers. Changes to shared build/dependency tooling also bump the plugins they affect.

Versions use MAJOR.MINOR.PATCH: increase the patch for fixes and internal changes, the minor for compatible features, and the major for breaking changes to commands, configuration, native APIs, or data contracts. Reset lower fields when increasing the minor or major. Bump once per logical change; documentation-only and test-only changes are exempt. All 16 scripts and both plugins advance independently.

The initial 1.0.0 versions document the existing source baseline. These TWS versions are separate from upstream revisions in each plugin's upstream.json and the existing native loader plugin_version() value. Component versions identify source changes; consult the changelog for compatible script/plugin combinations and receiver requirements. Maintenance rules are in AGENTS.md.

Building plugins

Plugin source is vendored as regular tracked files so local modifications travel with this repository. Each plugin's upstream.json records its upstream revision and exact Git dependency revisions. Dependencies and build outputs are ignored. The dependency loader replaces upstream's submodule setup; nested upstream .github files are retained for reference and do not run as this repo's CI.

Use Windows with Visual Studio 2022 C++ build tools, a Windows SDK, Git, and Python 3.9+. From a Developer PowerShell with msbuild on PATH, at the repository root:

.\tools\build_plugins.ps1 -Game all -Configuration Release

This restores pinned dependencies, generates both projects, and builds x86 DLLs. Upstream's Premake setup also downloads its MySQL client dependency on first use. Outputs are t5/plugins/tws-t5-gsc-utils/build/bin/Release/tws-t5-gsc-utils.dll and t6/plugins/tws-t6-gsc-utils/build/bin/Release/tws-t6-gsc-utils.dll. Remove the old t5-gsc-utils.dll or t6-gsc-utils.dll before installing the corresponding tws- DLL in Plutonium/plugins/; load only one version per game. Restart the server process; restarting only the map cannot load an updated plugin.

Use python tools/fetch_plugin_deps.py all --check to verify dependency revisions without fetching. Use python3 tools/test_plugin_http.py on macOS/Linux with a C++20 compiler and libcurl development files to compile the real HTTP utilities and test local stalled connections, response deadlines, status reporting and recovery. These portable tests do not build the Windows DLLs or exercise Plutonium's VM. Run python3 tools/test_plugin_component.py to compile the completion/option handling against host engine shims with address and undefined-behavior sanitizers.

Both plugins now default to a 5-second connection timeout and 15-second total transfer timeout. Request options can override these with positive millisecond values. Telemetry explicitly sets those limits and logs native HTTP status and timings. See the T5 and T6 plugin notes. Telemetry retries and durable replay require the updated plugins; see the delivery and rollout notes.

Telemetry delivery and rollout (T5 and T6)

Deploy Electric Cherry #144 before these reporters. Replay requires the receiver to associate reports with match and deduplicate seq; the legacy receiver can apply an old end to a newer match. Build and install both updated TWS plugins, restart the server processes, then deploy the corresponding scripts. The new telemetryNewMatch, telemetryLoad, telemetryStore, telemetryRemove and telemetryResponse functions are required even if telemetry is disabled by dvar.

Stat reports now include both score (spendable balance) and score_total (stock cumulative match score) for each player. Deploy Electric Cherry's score_total receiver support before these scripts; older receivers discard the extra field. No additional plugin update is needed for this field. The first stock end_game signal is captured without yielding, before intermission replaces the balance with the total. game_start remains an identity-only roster.

Every captured report includes a match key (random native process token plus an increasing match counter, at most 64 characters) and a positive seq. game_start is sequence 1. Retries and replay send the exact original JSON; sequence numbers are never reassigned. Player snapshots still reflect capture time; no client timestamp has been added and API receipt times retain their existing meaning.

A single sender keeps game_start and game_end at the queue head on transport failure, native timeout, HTTP 429 or 5xx. Delays are 1, 2, 4, 8, 16, then 30 seconds between attempts, capped at 30 seconds without a retry-count limit. Other 4xx (and redirects) are logged and dropped. A 2xx with a JSON boolean "ok": true acknowledges delivery; empty, malformed or negative 2xx replies and missing HTTP diagnostics are unacknowledged and retried for boundaries. Accepted reports log Electric Cherry's telemetry.timing.matchId as ec_match_id when supplied. Heartbeats and round changes remain best effort. Waiting heartbeats are coalesced to the newest snapshot, preserving round reports and any in-flight request. Intentional coalescing can produce gaps in seq.

The native outbox saves start/end snapshots synchronously before sending, and removes them only after acknowledgment or a terminal HTTP rejection. Files live under fs_homepath/tws-telemetry/t5/ or t6/, scoped to the exact server name and endpoint. Keep this directory on persistent writable storage across process or container restarts; it contains player snapshots but no authorization token. Use a single process per server/endpoint and storage directory. Endpoint/server changes use a separate outbox, so restore the original configuration to replay its backlog. Keep tws_server_id stable between maps. On map/process startup, recovered boundaries precede any new game_start. An accepted report whose acknowledgment/removal was lost is safely replayed with the same identity.

Writes replace the committed file only after a complete temporary write. A storage error blocks the affected boundary before sending and logs an error; unsaved in-memory reports cannot survive a restart. Unreadable/corrupt saved data disables telemetry until storage is repaired and the map restarted, preserving the file for recovery. Each outbox is limited to 4,096 reports / 16 MiB, with individual report bodies below 20 KiB; reaching a limit fails visibly without discarding the saved backlog. Disk removal failures retry cleanup without sending an already resolved report again during the same map. The plugin cannot recover a game_end that was never captured (for example, a process crash during play).

Run node tools/check_telemetry.mjs for both reporters' queue/retry fixtures and python3 tools/test_plugin_telemetry.py for native storage, acknowledgment and process-restart checks with sanitizers. The native test uses each restored JSON dependency; alternatively pass --json-header '/path/{game}-json.hpp' with the single headers from each plugin's pinned revision. These tests use engine shims. Validate Windows DLL linking and both games in Plutonium before rollout: force a timeout/429/503 at start and end, confirm permanent 400/401 failures do not retry, restart a map and then a process with pending boundaries, and confirm their original match IDs complete before the next start. Also verify a 2xx without ok: true stays pending, an outage retains only the latest waiting heartbeat, and a disk failure is visible without advancing past an unsaved boundary.

Parity

T5 and T6 are separate source trees, not a shared library — GSC namespaces, includes, and stock APIs differ enough between the two engines that a shared file is not practical. Feature parity is enforced pairwise: a change landed in t5/ should be considered for a companion change in t6/, adapted to T6 conventions.

Modules currently in each tree:

Module T5 T6
common t5/scripts/tws_t5_common.gsc t6/scripts/tws_t6_common.gsc
hud t5/scripts/tws_t5_hud.gsc t6/scripts/tws_t6_hud.gsc
afk t5/scripts/tws_t5_afk.gsc t6/scripts/tws_t6_afk.gsc
lock t5/scripts/tws_t5_lock.gsc t6/scripts/tws_t6_lock.gsc
telemetry t5/scripts/tws_t5_telemetry.gsc t6/scripts/tws_t6_telemetry.gsc — queued game_start / heartbeat / round_change / game_end matching T5
join t5/scripts/tws_t5_join.gsc t6/scripts/tws_t6_join.gsc
bank tws_t5_bank.gsc tws_t6_bank.gsc — persistent shared GUID bank
leaderboard t5/scripts/tws_t5_leaderboard.gsc t6/scripts/tws_t6_leaderboard.gsc

Player bank (T5 and T6)

Install tws_t5_bank.gsc or tws_t6_bank.gsc alongside the game's common and telemetry scripts (1.0.0 or later). Players can use .b to read their balance, .d <amount|all> to deposit spendable points, and .w <amount|all> to withdraw. Configured bank modules register all three commands in .help. Balances are shared by GUID across games and survive map/server restarts. Transfers do not change score_total; bank-assisted matches still count on existing leaderboards.

Deploy Electric Cherry #146 first, then build/install both native plugins 1.2.0 and restart the server processes. The plugins provide bankLoad, bankStore, bankRemove and bankResponse. Removing the bank script is necessary to omit these linking requirements; disabling it by dvar does not avoid linking.

Electric Cherry generates a unique bank token when it prepares a native server instance and injects it through that instance's private electric_cherry_bank.cfg, with tws_server_id supplied by the launch configuration. Both bank scripts 2.0.0 always use https://ec.thewrightserver.net/api/bank; tws_bank_url is no longer read, even if an old configuration still sets it. The host agent's enrolled controller address is independent and continues to serve agent traffic. Agent 1.9.1+ writes only the private tws_bank_token in the bank configuration. Rebuilds and host moves keep the instance token; destroying the instance revokes it. Reusing a server name gets a new token. There is no EC_BANK_TOKENS map or per-server secret to maintain. Keep secrets outside shared scripts and promoted configs. Set tws_bank_enabled 0 to disable the module; restart maps after script/config changes. Electric Cherry's EC_BANK_MAX_BALANCE defaults to 250000 (range 1–1000000). Each transfer and bank-funded spendable balance is capped at 1000000 points.

Transfers require a living player recorded in the server's current telemetry match. Newcomers may wait until the next 30-second heartbeat. The game validates points and player state again after a prepare reply; failed or uncertain prepares leave game points unchanged and cancel the reservation. After points move, a durable commit decision retries settlement without applying them again. A transfer in progress blocks new bank requests on that server. A GUID can have only one prepared transfer across the fleet.

Preserve tws-bank/t5/ and tws-bank/t6/, relative to the game process working directory, across process/container restarts, with one process per server/endpoint directory. EC host agents 1.9.1+ settle pending journals before deleting runtime storage on removal, rebuild, migration or launch cancellation. Update the agents before enabling the bank. An outage or invalid journal blocks cleanup and preserves the files. Decisions are scoped to server name and URL and contain no token. Recoveries run before new bank requests; failed cleanup/replies retry with the original identity. Corrupt storage blocks the bank; failed writes prevent points changes. Restore storage or credentials to finish pending work, rather than deleting the journal. Holds have no automatic expiry. Already-applied withdrawals are match points and are not refunded when a match ends or its process crashes. As with any live score, the journal cannot make game memory and the remote database atomic across a machine/storage failure.

Bank diagnostics — TWS #14

Bank scripts 2.0.2 require their matching TWS plugins 1.2.0+. Startup logs report the script version, the loaded plugin's TWS version via bankVersion(), the effective endpoint, a validated server ID, credential format validation, and the loaded journal count. Credential validation checks the local format; HTTP 401/403 still identifies a credential rejected by EC. A failed journal load logs pending=unknown and disables the bank; invalid configuration reports that journals were not loaded. Server IDs outside the bounded log-safe token alphabet are shown as invalid/unavailable without changing the configured API identity.

Each HTTP attempt receives a local correlation=bank-<map milliseconds>-<counter> for console diagnostics, including .b. These IDs are scoped to a map/process log, never sent to EC, and never replace durable transfer IDs. Balance requests still omit the API request ID and require an empty response ID. Transfer logs also include their original request ID, linking prepare, journal and settlement.

Outcomes distinguish request creation, transport failure, HTTP rejection, missing diagnostics/response, invalid acknowledgment (with a fixed parser reason), request-ID mismatch, unexpected state, and valid application rejection. Available HTTP status, curl code and cumulative DNS/connect/TLS/total milliseconds are included; absent or malformed fields say unavailable. Only allowlisted response codes reach logs; unfamiliar codes say unknown. No token, authorization header, raw response/request body, transport exception or private configuration contents are logged. Existing point accounting, journal format, API payloads, retry delays and player-facing messages are unchanged.

Settlement lines add the durable decision, attempt count, pending seconds, next retry delay, cleanup failures and eventual resolution. Pending time begins when a new cancel decision is attempted; recovered journals measure from this map's load, so downtime is not included. Counters reset on map/process restart. Repeated identical failures per action are summarized at most every 30 seconds, with a suppressed count. Reason/status/code changes and recovery log immediately. Native storage logs similarly identify load/write/remove failures and recovery. Rate-limited attempts retain their correlation internally; only emitted lines appear in the console. No log history or retry metadata is added to the journal.

Build/install both updated DLLs and restart each full server process, then install the corresponding scripts and restart the map if they were copied after startup. bankVersion() is a new linking requirement even when the bank is disabled by dvar. No EC/agent change or new third-party dependency is required; existing bank API, common/telemetry and endpoint-migration requirements still apply. Verify deployed DLL and GSC SHA-256 hashes against the built release artifacts, and confirm startup reports script=2.0.2 plugin=1.2.0 before interpreting results. An old bank script checksum on Ascension was reported in issue #14; deploying only the DLL does not update GSC. No deployment was performed here.

Run node tools/check_bank.mjs --ec-root /path/to/electric-cherry and python3 tools/test_plugin_bank.py. On both T5 and T6, compile/link and test successful .b/deposit/withdrawal, HTTP 401/403, timeout, malformed JSON/schema, wrong response ID, unwritable/corrupt journals and settlement recovery after a map/process restart. Check first/change/recovery logs, 30-second suppression, HTTP timings, unchanged player messages/points and one remote ledger entry per transfer. Keep journal files intact during failure tests. Windows DLL/GSC builds, installed-artifact verification and these live game checks remain required.

Bank identity validation

Bank scripts 2.0.1 use T5's player getGuid() and T6's persistent player.guid, matching each game's telemetry identity source. Both normalize integer identities to strings and preserve existing strings exactly, including large or zero-prefixed GUIDs. Missing or invalid identities show a private "Bank player identity unavailable" message before any request, journal write or score change. GUIDs must meet EC's existing 1–64 character token contract (A–Z, a–z, 0–9, _, ., :, -). Persisted decisions and their retries retain that original string identity, even after the player disconnects.

No new plugin or EC update is required for this fix. Install the updated bank script in each game's script directory and restart the map. This does not repair an older numeric-GUID journal: preserve any such pending decisions and their request IDs for scoped recovery; never delete them to unblock a rebuild. The endpoint upgrade procedure below still applies to legacy URLs.

Run node tools/check_bank.mjs for both scripts' portable fixtures. To also exercise the actual EC bank service and bank migrations with isolated SQLite:

node tools/check_bank.mjs --ec-root /path/to/electric-cherry
python3 tools/test_plugin_bank.py

The optional EC check requires that checkout's installed typescript and better-sqlite3 dependencies. It uses no EC configuration, credentials, network or production database. It covers numeric rejection, normalized balances and transfers, zero-balance .w all, cancellation, and idempotent commit replay. Native checks require the pinned JSON headers as described above. These tests do not validate game VM linking, HTTP route authentication or deployed artifacts.

After deployment, compare the SHA-256 hashes of both installed bank scripts with the source files from this release, and confirm the map restarted. On T5 and T6, check .b, a deposit, a withdrawal, and .w all with zero bank balance (an insufficient-balance message with no point change). Confirm EC records string GUIDs matching telemetry, one ledger entry per completed transfer, and unchanged score_total. On a test server, interrupt settlement and restart the map/process: the original request ID and decision must settle once without reapplying game points. GSC compilation and these live checks remain required.

Upgrading the bank endpoint

Deploy the Electric Cherry agent 1.9.1+ companion to TWS #13 before installing bank scripts 2.0.0. It accepts journals for the public game endpoint and the legacy bank URL derived from that host's enrolled controller. It settles through the existing authenticated, instance-scoped agent API at the controller; it never follows journal URLs and rejects other endpoint identities. The instance token and server identity must stay the same through the upgrade.

If the old tws_bank_url differs from the public endpoint, rebuild the server through EC with agent 1.9.1+ and the new scripts. The agent stops the old process and settles its original commit/cancel decisions before starting the replacement and deleting old storage. A lost reply replays the same request ID; EC deduplicates it. An outage, corrupt journal or failed acknowledgment retains storage and blocks the replacement. Resolve the failure and retry the rebuild. Keep the enrolled controller address unchanged until legacy decisions drain. Do not merely replace scripts and restart the map: native journals are keyed by server and endpoint, so the new scripts cannot see a legacy endpoint's pending decisions. Do not rename, rewrite or delete those journals to force migration.

If the previous endpoint already was the exact public URL, install the scripts and restart the map; saved decisions replay under the same journal identity. No new plugin build or full process restart is required for that case. Other legacy endpoints are refused by the agent and require recovery with their original configuration before switching. No deployment has been performed here.

Run node tools/check_bank.mjs and python3 tools/test_plugin_bank.py. Before rollout, build/link Windows DLLs, compile GSC, and verify both games: help, balance and all commands, score HUD/persistence, max/overdraw errors, death/respawn/disconnect during HTTP, late telemetry, lost prepare/commit responses, disk failures and map/process restart with pending decisions. For this upgrade also verify stale URL settings are ignored, an empty .b shows zero, .w all reports insufficient balance, and an EC rebuild/removal settles legacy and public journals without duplicate ledger entries or lost points.

Map record announcements

Install tws_t5_leaderboard.gsc for T5 or tws_t6_leaderboard.gsc for T6, along with the updated common module for that game. The new module privately announces the current map's fleet-wide highest recorded round and every tied holder once per connection, including spectator newcomers. Type .leaders to replay the record privately from the latest usable cache. Repeating the command restarts the announcement from its first page. It appears in .help when the leaderboard module is enabled. If no usable record is available, the command asks you to try again shortly. It reads Electric Cherry's public /api/leaderboard endpoint without a token; only completed matches contribute. Ties may come from the same run or separate runs. Names are the API's latest player names, with players identified by GUID.

Both versions default to https://ec.thewrightserver.net/api/leaderboard. T5 discovers the map by engine identifier. T6 selects the exact variant using the active mode/location, so Town Survival, TranZit and Town Grief stay separate. See the game READMEs for plugin requirements and installation paths.

Dvar Default Purpose
tws_leaderboard_enabled enabled 0 disables announcements and requests
tws_leaderboard_url endpoint above Full HTTPS API endpoint, without query parameters
tws_leaderboard_map_key automatic Exact, decoded API map key when automatic selection is unavailable

For example, set tws_leaderboard_map_key "t6:zm_transit:town" explicitly selects Town Survival. Normally this setting is unnecessary. An override must exist in the API's map list and match the current game and engine map. Clear it when changing variants; restart the map after changing these settings.

The announcement starts five seconds after player readiness and waits for the initial record fetch if necessary. It appears in the same HUD area as .help, below the health bar, with a heading such as Record: Round 23 and holder names underneath. Each page shows up to three holder lines for 12 seconds. Opening .help hides the record and pauses its display time until help closes. All holders are included; colour/control codes are removed, and unusually long names are shortened to keep messages readable. HUD elements are removed when the announcement finishes, the player disconnects, or the game ends.

One level worker fetches the map list and pages of three players, stopping when the highest-round tie ends. Successful snapshots refresh every five minutes. Failures retry after 30 seconds and retain the last complete snapshot for up to 15 minutes. An empty selection says “No recorded high round yet”; unavailable, malformed or ambiguous results are logged and skipped when no usable cache exists. Partial page results never replace the cache. The API has no pagination snapshot token: changed totals/record rounds reject the refresh, but concurrent renames can still move entries between pages. Refreshes deduplicate GUIDs.

Requests have a 15-second GSC wait limit. The TWS plugins also enforce native 5-second connection and 15-second transfer limits. The leaderboard waits for native completion before starting another request; player threads never wait directly on HTTP. Responses at the plugins' 20 KiB truncation boundary are rejected. The complete JSON grammar, UTF-8, string escapes and response shape are checked before consuming data. These guards also reject HTML/error bodies. Leaderboard still uses body validation; the new native HTTP status diagnostics are consumed by telemetry.

Run the portable data/structure checks with node tools/check_leaderboard.mjs. They execute the scripts' data helpers with engine-value shims, not a game VM. Live checks: join while playing and spectating; die/respawn (no repeat); reconnect (one new announcement); compare Town/TranZit/Grief; check one holder, ties across pages, no records, and a slow/unavailable/malformed API. Disconnect or end the game mid-announcement and confirm messages stop. Verify existing HUD, .help, .join and telemetry still work. Open .help during a record page and check that the record resumes afterwards without overlapping it. Runtime linking and client readiness require Plutonium. Try .leaders and .LEADERS after the announcement ends and during a later page: each should start one private announcement from the first page. Repeat while help is open, then disconnect or end the game during a replay to check cleanup. Disable the leaderboard module and verify .leaders is absent from help.

Telemetry authentication (T5 and T6)

Both reporters default to https://ec.thewrightserver.net/api/telemetry and send Authorization: Bearer <token> plus Content-Type: application/json on every POST. Authentication is also used for retried/replayed reports. An explicit tws_telemetry_url override must also use HTTPS; remove any old HTTP override.

Before upgrading the API or loading these reporters:

  1. Set the Electric Cherry service's environment variable EC_TELEMETRY_TOKEN to a separate shared secret of at least 32 characters. A missing service token disables ingestion.
  2. Both telemetry scripts embed the matching reporter secret in level.telemetry_token. No game-server token dvar or private config is needed.
  3. Keep tws_server_id set to the exact registered server name, then restart the map. Both reporters refuse to start with a short embedded token or a non-HTTPS URL. Rotating the service token requires editing the token in both scripts, deploying them to every game server, and restarting the maps.

The API also accepts X-EC-Telemetry-Token; these reporters use Bearer auth. Cookies, query parameters, and JSON fields cannot authenticate telemetry. Missing or incorrect request tokens are rejected before payload parsing or recording. Reporters that cannot set headers need a trusted relay with an HTTPS reporter-facing endpoint. The plugins used here accept a headers array.

After deployment, verify all four event types reach Electric Cherry. On a test server, also verify that a short embedded token and an HTTP URL prevent startup, that a wrong token is rejected without recording an event, and that deploying the correct token restores ingestion. Plugin transport completion alone does not prove API acceptance; inspect the API response or ingestion records.

Validating T6

powershell -File t6/scripts/tools/check.ps1 -CompilerPath C:\path\to\gsc-tool.exe

Requires PowerShell and gsc-tool (T6 PC).