josh/tws-zombies-scripts
0
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Preserve and retry telemetry match boundaries across restarts

Browse Source
This commit is contained in:
josh
2026-10-12 12:25:14 -04:00
parent 1121b44857
commit 158916b664
16 changed files with 1603 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
t5/scripts/README.md
+32 -39
View File
@@ -333,6 +333,8 @@ The reporter emits exactly four event types:
| `heartbeat` | Every 30 seconds after `game_start`, until game end | Same fields as `round_change`, with current round and player stats |
| `game_end` | First stock `end_game`, `game_ended`, or `intermission` notification | `server`, `map`, final `round`, full player stats |
All four events also include `match` and `seq` (see delivery below).
Each player in round/heartbeat/end reports has `name`, `guid`, `kills`, `headshots`,
`downs`, `revives`, and `score`. Names and GUIDs remain strings; GUIDs never
pass through numeric conversion. Round and stats are nonnegative integers,
@@ -350,14 +352,14 @@ serializes an empty GSC array as `null`; player objects and strings still
use `jsonSerialize` for escaping.
```json
{"event":"game_start","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","players":[{"name":"Josh","guid":"12345678901234567"}]}
{"event":"round_change","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":420,"headshots":160,"downs":1,"revives":3,"score":32000}]}
{"event":"heartbeat","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":430,"headshots":165,"downs":1,"revives":3,"score":32500}]}
{"event":"game_end","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":18,"players":[{"name":"Josh","guid":"12345678901234567","kills":445,"headshots":170,"downs":2,"revives":3,"score":33500}]}
{"event":"game_start","match":"process-token-1","seq":1,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","players":[{"name":"Josh","guid":"12345678901234567"}]}
{"event":"round_change","match":"process-token-1","seq":2,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":420,"headshots":160,"downs":1,"revives":3,"score":32000}]}
{"event":"heartbeat","match":"process-token-1","seq":3,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":430,"headshots":165,"downs":1,"revives":3,"score":32500}]}
{"event":"game_end","match":"process-token-1","seq":4,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":18,"players":[{"name":"Josh","guid":"12345678901234567","kills":445,"headshots":170,"downs":2,"revives":3,"score":33500}]}
```
Events are captured immediately into an in-memory queue. One HTTP worker
POSTs each snapshot once, in gameplay order, with `Content-Type:
POSTs snapshots in gameplay order, with `Content-Type:
application/json` and `Authorization: Bearer <token>`, waiting for completion
before sending the next. Gameplay and round detection continue while requests
are pending. Repeated joins or
@@ -370,35 +372,33 @@ The first heartbeat is captured 30 seconds after match start, then every
changes and later spawns do not reset the interval or add heartbeat workers.
Heartbeats use the same snapshot builder and ordered sender as round reports;
pending HTTP requests can delay transmission. Game end stops the heartbeat
worker, and already queued heartbeats drain before `game_end`.
worker. Only the newest waiting heartbeat is retained; round reports and the
in-flight request retain their order before `game_end`.
There is no discovery, beta token, session ID, sequence field, custom event,
external helper, or separate beta acknowledgement. Delivery is best effort:
no retries or persistence across map reloads, and no cross-game exactly-once
guarantee. The [TWS plugin build](../plugins/README.md) enforces a 5-second
connection and 15-second total transfer timeout. A timed-out report is logged
and skipped so later reports can proceed. A map reload can discard queued reports.
Each report includes a native-generated `match` key and a positive `seq`, starting
at 1 for `game_start`. Start/end snapshots are written to disk before sending and
replayed before a new match after map/process restarts. Retryable failures retain
the queue head with 1–30 second backoff; other 4xx are terminal. Acceptance requires
a 2xx response containing the JSON boolean `ok: true`, and logs include Electric
Cherry's `telemetry.timing.matchId`. The native 5-second connect / 15-second total
transfer limits still apply. Ordinary round/heartbeat failures are best effort.
Transport failures and non-2xx HTTP responses log to the server console. The TWS
plugin exposes HTTP status and cumulative DNS/TCP/TLS/first-byte/total timings,
plus worker-queue and completion-notification delays. Transport success alone
does not establish API acceptance.
Verify acceptance at Electric Cherry: success is HTTP 200 with `ok: true`
and telemetry containing `receivedAt`; invalid payloads return 400 and
non-JSON content types return 415.
**Deploy Electric Cherry #144 and the updated TWS plugin before these scripts.**
See the shared [delivery and rollout notes](../../README.md#telemetry-delivery-and-rollout-t5-and-t6)
for outbox location, persistence requirements, capacity/error handling and tests.
**Diagnosing a lobby that stops reporting**
Default logging includes initialization, match start/end, each request's
event/round and local ID, queue wait and request duration in milliseconds,
transport failures/recovery, HTTP rejection/status/timings, and the final queue drain. IDs reset on map
load and only appear in logs; payloads are unchanged. Every log includes
transport failures/recovery, HTTP rejection/status/timings, and the final queue drain.
Each log identifies the original match key and payload sequence, including replays. Every log includes
the server, map, and game time in milliseconds.
An independent watchdog prints health every 30 seconds, even before match
start and while HTTP is pending or the game-end queue is draining. It shows
the sender stage, current snapshot capture, pending queue size (excluding
the in-flight request), total queued, completed transports, transport failures,
the sender stage, current snapshot capture, pending queue size (including
the retained in-flight head), total captured, accepted reports, completed transports, transport failures,
HTTP rejections, and
time since the last completion (`-1` means none). Warnings remain enabled
at log level `0`:
@@ -406,7 +406,7 @@ at log level `0`:
- `HTTP stalled`: a request has been pending for at least 30 seconds. The
warning repeats on watchdog ticks with its ID, event, age, and queue size.
- `sender is not consuming`: the oldest queued report has waited at least
30 seconds with no in-flight request.
30 seconds with no in-flight request outside an intentional retry backoff.
- `round watcher has stopped advancing`: no polling progress for over
5 seconds, detected on the next watchdog tick.
- `no snapshot captured`: an active match has gone at least 65 seconds
@@ -420,17 +420,12 @@ indicate a proxy error. Previews can include player identities echoed by
the API. Restore `1` afterward, or use `0` to retain only warnings, errors,
and transport recovery messages.
Missing request handles and reported transport failures are counted and
skipped so the sender can continue with the next event. Empty responses
produce a warning. Arbitrary response bodies are not parsed as JSON,
because malformed/proxy responses must not break the sender. The watchdog
does not launch overlapping requests or retry failed events. Native timeouts
release stalled transfers. A request still pending after 30 seconds warrants
checking the installed plugin version, worker-queue delay and completion delivery.
If diagnostics are absent, the reporter warns that native deadlines are unverified;
install the TWS plugin build and restart the server process.
Native deadlines and completion diagnostics require the [TWS plugin build](../plugins/README.md).
Missing handles, transport errors, empty/invalid acknowledgments and retryable
HTTP responses keep start/end pending. The watchdog never launches overlapping
requests. A request pending beyond 30 seconds warrants checking the installed
plugin version, worker-queue delay and completion delivery. Native deadlines,
diagnostics and the telemetry outbox/acknowledgment helpers require the updated
[TWS plugin build](../plugins/README.md).
The upstream baseline contains
[`http::request`](https://github.com/alicealys/t5-gsc-utils/blob/main/src/component/http.cpp)
and [`jsonSerialize`](https://github.com/alicealys/t5-gsc-utils/blob/main/src/component/json.cpp).
@@ -465,10 +460,8 @@ and [score tracking](https://github.com/plutoniummod/t5-scripts/blob/main/ZM/Com
though transport completes. Also check an empty or HTML response.
7. Point a test server at an endpoint that accepts a connection but does
not reply. With the TWS plugin, verify a native timeout near 15 seconds and
continued queue consumption. With an older plugin, expect recurring
`HTTP stalled` warnings and increasing
pending counts while gameplay continues, including after game over.
Release the delayed reply and verify the queue resumes in order and
a retained start/end retry with bounded backoff while gameplay continues.
Restore the endpoint and verify the queue resumes in order and
the watchdog stops after the final drain. Test a refused connection
separately to check failure counts and the recovery log once restored.
t5/scripts/tws_t5_telemetry.gsc
+161 -33
View File
@@ -45,15 +45,26 @@ init()
return;
}
// Recover before registering producers, so old boundaries precede this match.
recovered = telemetryLoad(level.telemetry_server, level.telemetry_url);
if (!isDefined(recovered))
{
telemetry_log(0, "ERROR: cannot recover durable outbox; reporting disabled until storage is repaired and the map restarted");
return;
}
level.telemetry_initialized = true;
level.telemetry_started = false;
level.telemetry_ended = false;
level.telemetry_last_round = -1;
level.telemetry_queue = [];
level.telemetry_queue = recovered;
level.telemetry_queue_read = 0;
level.telemetry_queue_write = 0;
level.telemetry_queue_write = recovered.size;
for (i = 0; i < recovered.size; i++)
level.telemetry_queue[i]["captured"] = getTime();
level.telemetry_sequence = 0;
level.telemetry_completed = 0;
level.telemetry_accepted = 0;
level.telemetry_failed = 0;
level.telemetry_http_rejected = 0;
level.telemetry_missing_diagnostics_warned = false;
@@ -65,7 +76,7 @@ init()
level.telemetry_sender_state = "starting";
level.telemetry_inflight = undefined;
telemetry_log(1, "Initialized; waiting for match start. log_level=" + telemetry_log_level());
telemetry_log(1, "Initialized; recovered=" + recovered.size + " waiting for match start. log_level=" + telemetry_log_level());
// Use the shared dispatcher; a second connected listener breaks T5 SP.
scripts\sp\tws_t5_common::register_spawn_handler(::telemetry_game_start);
@@ -83,6 +94,12 @@ telemetry_game_start()
if (level.telemetry_started || level.telemetry_ended)
return;
level.telemetry_match = telemetryNewMatch();
if (!isDefined(level.telemetry_match))
{
telemetry_log(0, "ERROR: cannot allocate match identity");
return;
}
level.telemetry_started = true;
level.telemetry_last_capture = getTime();
telemetry_log(1, "Match start detected");
@@ -146,6 +163,8 @@ telemetry_game_end_watcher()
// Preserve order even if the match ends before the next polling tick.
telemetry_game_start();
if (!level.telemetry_started)
return;
telemetry_report_round();
level.telemetry_ended = true;
level notify("telemetry_finished");
@@ -227,30 +246,78 @@ telemetry_enqueue(event_name)
level.telemetry_capture_event = event_name;
telemetry_log(2, "Capturing event=" + event_name);
// Snapshot and serialize before yielding, including during game over.
level.telemetry_sequence++;
body = "{\"event\":" + jsonSerialize(event_name);
body += ",\"server\":" + jsonSerialize(level.telemetry_server);
body += ",\"map\":" + jsonSerialize(GetDvar("mapname"));
body += ",\"match\":" + jsonSerialize(level.telemetry_match);
body += ",\"seq\":" + level.telemetry_sequence;
if (event_name != "game_start")
body += ",\"round\":" + telemetry_nonnegative_int(level.round_number);
body += ",\"players\":" + telemetry_players_json(event_name != "game_start") + "}";
entry = [];
level.telemetry_sequence++;
entry["match"] = level.telemetry_match;
entry["id"] = level.telemetry_sequence;
entry["event"] = event_name;
entry["round"] = telemetry_nonnegative_int(level.round_number);
entry["captured"] = getTime();
entry["body"] = body;
level.telemetry_queue[level.telemetry_queue_write] = entry;
level.telemetry_queue_write++;
// Persist boundaries before yielding or handing any request to the plugin.
// A failed save stays in memory and blocks sending until storage recovers.
entry["stored"] = false;
telemetry_queue_push(entry);
if (telemetry_critical(entry))
telemetry_persist_pending();
level.telemetry_last_capture = getTime();
level.telemetry_capture_event = "none";
telemetry_log(2, "Queued " + telemetry_event_label(entry) + " pending=" + telemetry_pending());
level notify("telemetry_queued");
}
// A single worker keeps one request in flight. It survives game-over notifies
// to drain the final snapshot; no retries, discovery, or beta-specific calls.
// If storage was unavailable at an earlier capture, save it first. Otherwise a
// newly captured end could be committed ahead of its unsaved start on restart.
telemetry_persist_pending()
{
for (i = level.telemetry_queue_read; i < level.telemetry_queue_write; i++)
{
entry = level.telemetry_queue[i];
if (!telemetry_critical(entry) || entry["stored"])
continue;
entry["stored"] = telemetryStore(level.telemetry_server, level.telemetry_url, entry["body"]);
if (!entry["stored"])
return;
}
}
telemetry_critical(entry)
{
return entry["event"] == "game_start" || entry["event"] == "game_end";
}
// Remove obsolete waiting heartbeats, keeping round reports and the in-flight
// head in order. The new heartbeat keeps its capture-time sequence and snapshot.
telemetry_queue_push(entry)
{
queue = [];
count = 0;
for (i = level.telemetry_queue_read; i < level.telemetry_queue_write; i++)
{
previous = level.telemetry_queue[i];
inflight = i == level.telemetry_queue_read && isDefined(level.telemetry_inflight);
if (entry["event"] == "heartbeat" && previous["event"] == "heartbeat" && !inflight)
continue;
queue[count] = previous;
count++;
}
queue[count] = entry;
level.telemetry_queue = queue;
level.telemetry_queue_read = 0;
level.telemetry_queue_write = count + 1;
}
// One worker retains the head through retries. No later report can overtake an
// unacknowledged boundary. Retry delays grow 1, 2, 4, 8, 16, then at most 30s.
telemetry_sender()
{
for (;;)
@@ -263,7 +330,7 @@ telemetry_sender()
if (level.telemetry_ended)
{
level.telemetry_sender_state = "drained";
telemetry_log(1, "Queue drained after game end; completed=" + level.telemetry_completed + " failed=" + level.telemetry_failed);
telemetry_log(1, "Queue drained after game end; accepted=" + level.telemetry_accepted + " failed=" + level.telemetry_failed);
level notify("telemetry_drained");
return;
}
@@ -274,12 +341,53 @@ telemetry_sender()
}
entry = level.telemetry_queue[level.telemetry_queue_read];
if (!telemetry_process(entry))
{
delay = telemetry_retry_delay(entry);
level.telemetry_sender_state = "retry_backoff";
telemetry_log(0, "Retry pending " + telemetry_event_label(entry) + " delay_s=" + delay);
wait (delay);
continue;
}
level.telemetry_queue[level.telemetry_queue_read] = undefined;
level.telemetry_queue_read++;
telemetry_send(entry);
}
}
telemetry_retry_delay(entry)
{
if (!isDefined(entry["retry_delay"]))
entry["retry_delay"] = 1;
else
entry["retry_delay"] *= 2;
if (entry["retry_delay"] > 30)
entry["retry_delay"] = 30;
return entry["retry_delay"];
}
telemetry_process(entry)
{
critical = telemetry_critical(entry);
if (critical && !entry["stored"])
{
entry["stored"] = telemetryStore(level.telemetry_server, level.telemetry_url, entry["body"]);
if (!entry["stored"])
return false;
}
// If removing an acknowledged/terminal entry fails, retry the disk update
// without POSTing again. A restart before removal safely replays its key/seq.
if (!isDefined(entry["resolved"]))
{
outcome = telemetry_send(entry);
if (critical && outcome == "retry")
return false;
entry["resolved"] = true;
}
if (critical)
return telemetryRemove(level.telemetry_server, level.telemetry_url, entry["match"], entry["id"]);
return true;
}
telemetry_send(entry)
{
level.telemetry_inflight = entry;
@@ -302,10 +410,10 @@ telemetry_send(entry)
request = http::request(level.telemetry_url, options);
if (!isDefined(request))
{
telemetry_transport_failed(label + " request handle is undefined; skipping event");
telemetry_transport_failed(label + " request handle is undefined");
level.telemetry_inflight = undefined;
level.telemetry_sender_state = "idle";
return;
return "retry";
}
level.telemetry_sender_state = "waiting_http";
@@ -319,8 +427,8 @@ telemetry_send(entry)
telemetry_log(0, "WARNING: plugin returned no HTTP diagnostics; native timeouts are unverified. Install the TWS plugin build and restart the server process");
}
// The plugin's success flag reports transport completion, not HTTP status.
// No automatic retry: the receiver may already have accepted the event.
// The plugin's success flag reports transport completion, not API acceptance.
outcome = "retry";
if (!isDefined(success) || !success)
{
if (!isDefined(error))
@@ -330,29 +438,49 @@ telemetry_send(entry)
else
{
level.telemetry_completed++;
if (level.telemetry_failure_streak > 0)
telemetry_log(0, "Transport recovered after " + level.telemetry_failure_streak + " failures; " + label);
level.telemetry_failure_streak = 0;
if (isDefined(diagnostics) && isDefined(diagnostics["http_status"]))
outcome = telemetry_response_outcome(response, diagnostics);
if (outcome == "accepted")
{
if (diagnostics["http_status"] < 200 || diagnostics["http_status"] >= 300)
{
level.telemetry_http_rejected++;
telemetry_log(0, "ERROR: HTTP rejected " + label + " elapsed_ms=" + elapsed + timing + " body=" + telemetry_preview(response));
}
else
telemetry_log(1, "HTTP completed " + label + " elapsed_ms=" + elapsed + timing);
ack = telemetryResponse(response);
level.telemetry_accepted++;
if (level.telemetry_failure_streak > 0)
telemetry_log(0, "API recovered after " + level.telemetry_failure_streak + " failures; " + label);
level.telemetry_failure_streak = 0;
telemetry_log(1, "API accepted " + label + " ec_match_id=" + ack["match_id"] + " elapsed_ms=" + elapsed + timing);
}
else
telemetry_log(1, "Transport completed " + label + " elapsed_ms=" + elapsed + timing + " (API acceptance unverified)");
if (!isDefined(response) || response == "")
telemetry_log(0, "WARNING: empty API response for " + label);
else
telemetry_log(2, "Response " + label + " body=" + telemetry_preview(response));
{
level.telemetry_failure_streak++;
level.telemetry_http_rejected++;
telemetry_log(0, "ERROR: API unacknowledged " + label + " outcome=" + outcome + " elapsed_ms=" + elapsed + timing + " body=" + telemetry_preview(response));
}
telemetry_log(2, "Response " + label + " body=" + telemetry_preview(response));
}
if (outcome != "accepted" && (!telemetry_critical(entry) || outcome == "drop"))
telemetry_log(0, "Dropping unaccepted " + label + " outcome=" + outcome);
level.telemetry_inflight = undefined;
level.telemetry_sender_state = "idle";
return outcome;
}
telemetry_response_outcome(response, diagnostics)
{
if (!isDefined(diagnostics) || !isDefined(diagnostics["http_status"]))
return "retry";
status = diagnostics["http_status"];
if (status == 0 || status == 429 || (status >= 500 && status < 600))
return "retry";
if (status < 200 || status >= 300)
return "drop";
if (!isDefined(response))
return "retry";
// Native nonthrowing JSON parsing requires an actual boolean true, rejects
// truncated/proxy bodies, and preserves large Electric Cherry IDs as strings.
ack = telemetryResponse(response);
if (ack["accepted"])
return "accepted";
return "retry";
}
// The native plugin appends diagnostics while preserving the old done arguments.
@@ -387,7 +515,7 @@ telemetry_pending()
telemetry_event_label(entry)
{
return "id=" + entry["id"] + " event=" + entry["event"] + " round=" + entry["round"];
return "match=" + entry["match"] + " seq=" + entry["id"] + " event=" + entry["event"] + " round=" + entry["round"];
}
// Read live so diagnostics can be increased without restarting the map.
@@ -439,7 +567,7 @@ telemetry_watchdog()
telemetry_log(1, "Health started=" + level.telemetry_started + " ended=" + level.telemetry_ended +
" sender=" + level.telemetry_sender_state + " capturing=" + level.telemetry_capture_event +
" pending=" + pending + " queued_total=" + level.telemetry_sequence +
" completed=" + level.telemetry_completed + " failed=" + level.telemetry_failed + " http_rejected=" + level.telemetry_http_rejected +
" accepted=" + level.telemetry_accepted + " completed=" + level.telemetry_completed + " failed=" + level.telemetry_failed + " http_rejected=" + level.telemetry_http_rejected +
" last_done_ms_ago=" + completion_age);
if (isDefined(level.telemetry_inflight))
@@ -450,7 +578,7 @@ telemetry_watchdog()
" elapsed_ms=" + age + " pending=" + pending + " stage=" + level.telemetry_sender_state +
"; native deadline exceeded; check plugin version, worker queue and completion delivery");
}
else if (pending > 0)
else if (pending > 0 && level.telemetry_sender_state != "retry_backoff")
{
oldest = level.telemetry_queue[level.telemetry_queue_read];
if (now - oldest["captured"] >= 30000)