The telemetry reporters send each report once. A failed or timed-out game_start or game_end is dropped, and reports still waiting in the queue when the map changes are lost, so Electric Cherry records a finished game as interrupted and leaves its stats off the leaderboard.
t6/scripts/tws_t6_telemetry.gsc:339 (T5 :323) logs a failed send and moves on without retrying.
The queue lives in level variables (:57, T5 :52), so a map restart discards anything not yet handed to the plugin. Requests already handed over do finish: both plugins wait for them on shutdown (component/http.cpp:154).
One request is in flight at a time (telemetry_sender, :267), so a game_end captured at game over (:151-159) waits behind any slow request. With the new 15-second deadline it can still miss a map change.
Reports carry no match identity. telemetry_sequence (:251) numbers reports for log lines only; it isn't sent and resets each level.
README.md:51: "Requests are still sent once, in order, with no retries or event timestamp changes."
Example: Electric Cherry match #921 (t6_zombie_buried_1, T6 telemetry v13, 9 October 05:27 UTC). Its last report arrived 10.9 s late, then neither its game_end nor the next game's game_start reached Electric Cherry, and the next match was opened 11.5 s later by a lower-round report. The game counts as interrupted and its five players' stats are excluded from the leaderboard. The container's console log was deleted with the server on 11 October, so the exact send failure can't be checked.
The Electric Cherry half (accepting match and seq and using them to attach reports to the right match) is tracked in josh/electric-cherry#144.
Fix (T5 and T6)
Generate a match key at game start that is unique per server process, for example a random boot token kept in a dvar plus a match counter dvar, and send match and seq on every report, with seq 1 for game_start
Retry game_start and game_end at the head of the queue on transport failure, native timeout, HTTP 429 or 5xx, with bounded backoff, and don't retry other 4xx. This is safe with Electric Cherry today because a report identical to the one it just accepted is ignored
Keep unacknowledged game_start and game_end reports across a map change (dvar) or a process restart (writefile), and resend them before the next game_start. Ship this after josh/electric-cherry#144: until reports carry a key, a resent game_end would be applied to the newer match
Collapse queued heartbeats while sends are failing, so recovery sends only the newest one
Treat a 2xx response with "ok": true as accepted, and log the Electric Cherry match id it returns (telemetry.timing.matchId)
The telemetry reporters send each report once. A failed or timed-out `game_start` or `game_end` is dropped, and reports still waiting in the queue when the map changes are lost, so Electric Cherry records a finished game as interrupted and leaves its stats off the leaderboard.
- `t6/scripts/tws_t6_telemetry.gsc:339` (T5 `:323`) logs a failed send and moves on without retrying.
- The queue lives in level variables (`:57`, T5 `:52`), so a map restart discards anything not yet handed to the plugin. Requests already handed over do finish: both plugins wait for them on shutdown (`component/http.cpp:154`).
- One request is in flight at a time (`telemetry_sender`, `:267`), so a `game_end` captured at game over (`:151-159`) waits behind any slow request. With the new 15-second deadline it can still miss a map change.
- Reports carry no match identity. `telemetry_sequence` (`:251`) numbers reports for log lines only; it isn't sent and resets each level.
- `README.md:51`: "Requests are still sent once, in order, with no retries or event timestamp changes."
Example: Electric Cherry match #921 (`t6_zombie_buried_1`, T6 telemetry v13, 9 October 05:27 UTC). Its last report arrived 10.9 s late, then neither its `game_end` nor the next game's `game_start` reached Electric Cherry, and the next match was opened 11.5 s later by a lower-round report. The game counts as interrupted and its five players' stats are excluded from the leaderboard. The container's console log was deleted with the server on 11 October, so the exact send failure can't be checked.
The Electric Cherry half (accepting `match` and `seq` and using them to attach reports to the right match) is tracked in [josh/electric-cherry#144](https://gitea.thewrightserver.net/josh/electric-cherry/issues/144).
**Fix** (T5 and T6)
- [x] Generate a match key at game start that is unique per server process, for example a random boot token kept in a dvar plus a match counter dvar, and send `match` and `seq` on every report, with `seq` 1 for `game_start`
- [x] Retry `game_start` and `game_end` at the head of the queue on transport failure, native timeout, HTTP 429 or 5xx, with bounded backoff, and don't retry other 4xx. This is safe with Electric Cherry today because a report identical to the one it just accepted is ignored
- [x] Keep unacknowledged `game_start` and `game_end` reports across a map change (dvar) or a process restart (`writefile`), and resend them before the next `game_start`. Ship this after josh/electric-cherry#144: until reports carry a key, a resent `game_end` would be applied to the newer match
- [x] Collapse queued heartbeats while sends are failing, so recovery sends only the newest one
- [x] Treat a 2xx response with `"ok": true` as accepted, and log the Electric Cherry match id it returns (`telemetry.timing.matchId`)
josh
was assigned by goddard2026-10-12 00:40:13 -04:00
Reviewed 3bb0b5178e. All five fix items are implemented; closing this ticket.
This review checks the current implementation against every acceptance criterion, regardless of commit references.
Checklist
Match identity. Both telemetry scripts allocate a native boot-token/counter match key and capture server/match/seq in every payload; game_start is sequence 1.
Boundary retries. telemetry_process retains start/end at the FIFO head for network failures, timeout, 429 and 5xx. Backoff grows from 1 to 30 seconds; other 4xx are terminal.
Durable recovery. Native telemetryStore/Load/Remove persist boundary snapshots to an atomically replaced, server/endpoint-scoped outbox. Initialization restores it ahead of the next game_start. EC #144 is closed.
Heartbeat collapse. telemetry_queue_push retains the newest waiting heartbeat and preserves the in-flight report and boundaries.
Acknowledgment. telemetry_response_outcome accepts 2xx only with a parsed boolean ok:true; the acceptance log includes telemetry.timing.matchId.
Review notes
Implemented by 158916b664, whose commit message does not reference this issue. Verified the surrounding implementation at current main.
node tools/check_telemetry.mjs: 23 checks passed across both games, including replay ordering, exact snapshots, failure/recovery, lost acknowledgments and storage errors.
Native telemetry tests passed for both source trees with the pinned JSON headers, ASan/UBSan, separate-process recovery and component syntax checks. These do not validate Windows DLL linking or the live Plutonium VM.
Intermediate reports still use best-effort delivery and numbered heartbeat coalescing still leaves gaps. Those are the explicitly separate scope of #11, not a reason to leave this boundary-delivery ticket open.
Reviewed 3bb0b5178e40ea6d322093283edb39f160542a07. All five fix items are implemented; closing this ticket.
This review checks the current implementation against every acceptance criterion, regardless of commit references.
**Checklist**
- [x] **Match identity.** Both telemetry scripts allocate a native boot-token/counter match key and capture server/match/seq in every payload; game_start is sequence 1.
- [x] **Boundary retries.** telemetry_process retains start/end at the FIFO head for network failures, timeout, 429 and 5xx. Backoff grows from 1 to 30 seconds; other 4xx are terminal.
- [x] **Durable recovery.** Native telemetryStore/Load/Remove persist boundary snapshots to an atomically replaced, server/endpoint-scoped outbox. Initialization restores it ahead of the next game_start. EC #144 is closed.
- [x] **Heartbeat collapse.** telemetry_queue_push retains the newest waiting heartbeat and preserves the in-flight report and boundaries.
- [x] **Acknowledgment.** telemetry_response_outcome accepts 2xx only with a parsed boolean ok:true; the acceptance log includes telemetry.timing.matchId.
**Review notes**
- Implemented by 158916b664300c24e5952494c16f4b8a0d5e3d03, whose commit message does not reference this issue. Verified the surrounding implementation at current main.
- node tools/check_telemetry.mjs: 23 checks passed across both games, including replay ordering, exact snapshots, failure/recovery, lost acknowledgments and storage errors.
- Native telemetry tests passed for both source trees with the pinned JSON headers, ASan/UBSan, separate-process recovery and component syntax checks. These do not validate Windows DLL linking or the live Plutonium VM.
- Intermediate reports still use best-effort delivery and numbered heartbeat coalescing still leaves gaps. Those are the explicitly separate scope of #11, not a reason to leave this boundary-delivery ticket open.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The telemetry reporters send each report once. A failed or timed-out
game_startorgame_endis dropped, and reports still waiting in the queue when the map changes are lost, so Electric Cherry records a finished game as interrupted and leaves its stats off the leaderboard.t6/scripts/tws_t6_telemetry.gsc:339(T5:323) logs a failed send and moves on without retrying.:57, T5:52), so a map restart discards anything not yet handed to the plugin. Requests already handed over do finish: both plugins wait for them on shutdown (component/http.cpp:154).telemetry_sender,:267), so agame_endcaptured at game over (:151-159) waits behind any slow request. With the new 15-second deadline it can still miss a map change.telemetry_sequence(:251) numbers reports for log lines only; it isn't sent and resets each level.README.md:51: "Requests are still sent once, in order, with no retries or event timestamp changes."Example: Electric Cherry match #921 (
t6_zombie_buried_1, T6 telemetry v13, 9 October 05:27 UTC). Its last report arrived 10.9 s late, then neither itsgame_endnor the next game'sgame_startreached Electric Cherry, and the next match was opened 11.5 s later by a lower-round report. The game counts as interrupted and its five players' stats are excluded from the leaderboard. The container's console log was deleted with the server on 11 October, so the exact send failure can't be checked.The Electric Cherry half (accepting
matchandseqand using them to attach reports to the right match) is tracked in josh/electric-cherry#144.Fix (T5 and T6)
matchandseqon every report, withseq1 forgame_startgame_startandgame_endat the head of the queue on transport failure, native timeout, HTTP 429 or 5xx, with bounded backoff, and don't retry other 4xx. This is safe with Electric Cherry today because a report identical to the one it just accepted is ignoredgame_startandgame_endreports across a map change (dvar) or a process restart (writefile), and resend them before the nextgame_start. Ship this after josh/electric-cherry#144: until reports carry a key, a resentgame_endwould be applied to the newer match"ok": trueas accepted, and log the Electric Cherry match id it returns (telemetry.timing.matchId)Reviewed
3bb0b5178e. All five fix items are implemented; closing this ticket.This review checks the current implementation against every acceptance criterion, regardless of commit references.
Checklist
Review notes
158916b664, whose commit message does not reference this issue. Verified the surrounding implementation at current main.