josh/tws-zombies-scripts
0
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Preserve and retry telemetry match boundaries across restarts

Browse Source
This commit is contained in:
josh
2026-10-12 12:25:14 -04:00
parent 1121b44857
commit 158916b664
16 changed files with 1603 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
t5/scripts/README.md
+32 -39
View File
@@ -333,6 +333,8 @@ The reporter emits exactly four event types:
| `heartbeat` | Every 30 seconds after `game_start`, until game end | Same fields as `round_change`, with current round and player stats |
| `game_end` | First stock `end_game`, `game_ended`, or `intermission` notification | `server`, `map`, final `round`, full player stats |
All four events also include `match` and `seq` (see delivery below).
Each player in round/heartbeat/end reports has `name`, `guid`, `kills`, `headshots`,
`downs`, `revives`, and `score`. Names and GUIDs remain strings; GUIDs never
pass through numeric conversion. Round and stats are nonnegative integers,
@@ -350,14 +352,14 @@ serializes an empty GSC array as `null`; player objects and strings still
use `jsonSerialize` for escaping.
```json
{"event":"game_start","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","players":[{"name":"Josh","guid":"12345678901234567"}]}
{"event":"round_change","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":420,"headshots":160,"downs":1,"revives":3,"score":32000}]}
{"event":"heartbeat","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":430,"headshots":165,"downs":1,"revives":3,"score":32500}]}
{"event":"game_end","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":18,"players":[{"name":"Josh","guid":"12345678901234567","kills":445,"headshots":170,"downs":2,"revives":3,"score":33500}]}
{"event":"game_start","match":"process-token-1","seq":1,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","players":[{"name":"Josh","guid":"12345678901234567"}]}
{"event":"round_change","match":"process-token-1","seq":2,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":420,"headshots":160,"downs":1,"revives":3,"score":32000}]}
{"event":"heartbeat","match":"process-token-1","seq":3,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":430,"headshots":165,"downs":1,"revives":3,"score":32500}]}
{"event":"game_end","match":"process-token-1","seq":4,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":18,"players":[{"name":"Josh","guid":"12345678901234567","kills":445,"headshots":170,"downs":2,"revives":3,"score":33500}]}
```
Events are captured immediately into an in-memory queue. One HTTP worker
POSTs each snapshot once, in gameplay order, with `Content-Type:
POSTs snapshots in gameplay order, with `Content-Type:
application/json` and `Authorization: Bearer <token>`, waiting for completion
before sending the next. Gameplay and round detection continue while requests
are pending. Repeated joins or
@@ -370,35 +372,33 @@ The first heartbeat is captured 30 seconds after match start, then every
changes and later spawns do not reset the interval or add heartbeat workers.
Heartbeats use the same snapshot builder and ordered sender as round reports;
pending HTTP requests can delay transmission. Game end stops the heartbeat
worker, and already queued heartbeats drain before `game_end`.
worker. Only the newest waiting heartbeat is retained; round reports and the
in-flight request retain their order before `game_end`.
There is no discovery, beta token, session ID, sequence field, custom event,
external helper, or separate beta acknowledgement. Delivery is best effort:
no retries or persistence across map reloads, and no cross-game exactly-once
guarantee. The [TWS plugin build](../plugins/README.md) enforces a 5-second
connection and 15-second total transfer timeout. A timed-out report is logged
and skipped so later reports can proceed. A map reload can discard queued reports.
Each report includes a native-generated `match` key and a positive `seq`, starting
at 1 for `game_start`. Start/end snapshots are written to disk before sending and
replayed before a new match after map/process restarts. Retryable failures retain
the queue head with 1–30 second backoff; other 4xx are terminal. Acceptance requires
a 2xx response containing the JSON boolean `ok: true`, and logs include Electric
Cherry's `telemetry.timing.matchId`. The native 5-second connect / 15-second total
transfer limits still apply. Ordinary round/heartbeat failures are best effort.
Transport failures and non-2xx HTTP responses log to the server console. The TWS
plugin exposes HTTP status and cumulative DNS/TCP/TLS/first-byte/total timings,
plus worker-queue and completion-notification delays. Transport success alone
does not establish API acceptance.
Verify acceptance at Electric Cherry: success is HTTP 200 with `ok: true`
and telemetry containing `receivedAt`; invalid payloads return 400 and
non-JSON content types return 415.
**Deploy Electric Cherry #144 and the updated TWS plugin before these scripts.**
See the shared [delivery and rollout notes](../../README.md#telemetry-delivery-and-rollout-t5-and-t6)
for outbox location, persistence requirements, capacity/error handling and tests.
**Diagnosing a lobby that stops reporting**
Default logging includes initialization, match start/end, each request's
event/round and local ID, queue wait and request duration in milliseconds,
transport failures/recovery, HTTP rejection/status/timings, and the final queue drain. IDs reset on map
load and only appear in logs; payloads are unchanged. Every log includes
transport failures/recovery, HTTP rejection/status/timings, and the final queue drain.
Each log identifies the original match key and payload sequence, including replays. Every log includes
the server, map, and game time in milliseconds.
An independent watchdog prints health every 30 seconds, even before match
start and while HTTP is pending or the game-end queue is draining. It shows
the sender stage, current snapshot capture, pending queue size (excluding
the in-flight request), total queued, completed transports, transport failures,
the sender stage, current snapshot capture, pending queue size (including
the retained in-flight head), total captured, accepted reports, completed transports, transport failures,
HTTP rejections, and
time since the last completion (`-1` means none). Warnings remain enabled
at log level `0`:
@@ -406,7 +406,7 @@ at log level `0`:
- `HTTP stalled`: a request has been pending for at least 30 seconds. The
warning repeats on watchdog ticks with its ID, event, age, and queue size.
- `sender is not consuming`: the oldest queued report has waited at least
30 seconds with no in-flight request.
30 seconds with no in-flight request outside an intentional retry backoff.
- `round watcher has stopped advancing`: no polling progress for over
5 seconds, detected on the next watchdog tick.
- `no snapshot captured`: an active match has gone at least 65 seconds
@@ -420,17 +420,12 @@ indicate a proxy error. Previews can include player identities echoed by
the API. Restore `1` afterward, or use `0` to retain only warnings, errors,
and transport recovery messages.
Missing request handles and reported transport failures are counted and
skipped so the sender can continue with the next event. Empty responses
produce a warning. Arbitrary response bodies are not parsed as JSON,
because malformed/proxy responses must not break the sender. The watchdog
does not launch overlapping requests or retry failed events. Native timeouts
release stalled transfers. A request still pending after 30 seconds warrants
checking the installed plugin version, worker-queue delay and completion delivery.
If diagnostics are absent, the reporter warns that native deadlines are unverified;
install the TWS plugin build and restart the server process.
Native deadlines and completion diagnostics require the [TWS plugin build](../plugins/README.md).
Missing handles, transport errors, empty/invalid acknowledgments and retryable
HTTP responses keep start/end pending. The watchdog never launches overlapping
requests. A request pending beyond 30 seconds warrants checking the installed
plugin version, worker-queue delay and completion delivery. Native deadlines,
diagnostics and the telemetry outbox/acknowledgment helpers require the updated
[TWS plugin build](../plugins/README.md).
The upstream baseline contains
[`http::request`](https://github.com/alicealys/t5-gsc-utils/blob/main/src/component/http.cpp)
and [`jsonSerialize`](https://github.com/alicealys/t5-gsc-utils/blob/main/src/component/json.cpp).
@@ -465,10 +460,8 @@ and [score tracking](https://github.com/plutoniummod/t5-scripts/blob/main/ZM/Com
though transport completes. Also check an empty or HTML response.
7. Point a test server at an endpoint that accepts a connection but does
not reply. With the TWS plugin, verify a native timeout near 15 seconds and
continued queue consumption. With an older plugin, expect recurring
`HTTP stalled` warnings and increasing
pending counts while gameplay continues, including after game over.
Release the delayed reply and verify the queue resumes in order and
a retained start/end retry with bounded backoff while gameplay continues.
Restore the endpoint and verify the queue resumes in order and
the watchdog stops after the final drain. Test a refused connection
separately to check failure counts and the recovery log once restored.