Preserve and retry telemetry match boundaries across restarts

README.md

@@ -48,7 +48,69 @@ Both plugins now default to a 5-second connection timeout and 15-second total
 transfer timeout. Request options can override these with positive millisecond
 values. Telemetry explicitly sets those limits and logs native HTTP status and
 timings. See the [T5](t5/plugins/README.md) and [T6](t6/plugins/README.md) plugin notes.
-Requests are still sent once, in order, with no retries or event timestamp changes.
+Telemetry retries and durable replay require the updated plugins; see the
+[delivery and rollout notes](#telemetry-delivery-and-rollout-t5-and-t6).
+
+## Telemetry delivery and rollout (T5 and T6)
+
+**Deploy Electric Cherry [#144](https://gitea.thewrightserver.net/josh/electric-cherry/issues/144)
+before these reporters.** Replay requires the receiver to associate reports with
+`match` and deduplicate `seq`; the legacy receiver can apply an old end to a newer
+match. Build and install both updated TWS plugins, restart the server processes,
+then deploy the corresponding scripts. The new `telemetryNewMatch`,
+`telemetryLoad`, `telemetryStore`, `telemetryRemove` and `telemetryResponse`
+functions are required even if telemetry is disabled by dvar.
+
+Every captured report includes a `match` key (random native process token plus
+an increasing match counter, at most 64 characters) and a positive `seq`.
+`game_start` is sequence 1. Retries and replay send the exact original JSON;
+sequence numbers are never reassigned. Player snapshots still reflect capture
+time; no client timestamp has been added and API receipt times retain their
+existing meaning.
+
+A single sender keeps `game_start` and `game_end` at the queue head on transport
+failure, native timeout, HTTP 429 or 5xx. Delays are 1, 2, 4, 8, 16, then 30 seconds
+between attempts, capped at 30 seconds without a retry-count limit. Other 4xx
+(and redirects) are logged and dropped. A 2xx with a JSON boolean `"ok": true`
+acknowledges delivery; empty, malformed or negative 2xx replies and missing HTTP
+diagnostics are unacknowledged and retried for boundaries. Accepted reports log
+Electric Cherry's `telemetry.timing.matchId` as `ec_match_id` when supplied.
+Heartbeats and round changes remain best effort. Waiting heartbeats are coalesced
+to the newest snapshot, preserving round reports and any in-flight request.
+Intentional coalescing can produce gaps in `seq`.
+
+The native outbox saves start/end snapshots synchronously before sending, and
+removes them only after acknowledgment or a terminal HTTP rejection. Files live
+under `fs_homepath/tws-telemetry/t5/` or `t6/`, scoped to the exact server name and
+endpoint. Keep this directory on persistent writable storage across process or
+container restarts; it contains player snapshots but no authorization token.
+Use a single process per server/endpoint and storage directory. Endpoint/server
+changes use a separate outbox, so restore the original configuration to replay
+its backlog. Keep `tws_server_id` stable between maps. On map/process startup,
+recovered boundaries precede any new `game_start`. An accepted report whose
+acknowledgment/removal was lost is safely replayed with the same identity.
+
+Writes replace the committed file only after a complete temporary write. A
+storage error blocks the affected boundary before sending and logs an error;
+unsaved in-memory reports cannot survive a restart. Unreadable/corrupt saved data
+disables telemetry until storage is repaired and the map restarted, preserving
+the file for recovery. Each outbox is limited to 4,096 reports / 16 MiB, with
+individual report bodies below 20 KiB; reaching a limit fails visibly without
+discarding the saved backlog. Disk removal failures retry cleanup without sending
+an already resolved report again during the same map. The plugin cannot recover
+a `game_end` that was never captured (for example, a process crash during play).
+
+Run `node tools/check_telemetry.mjs` for both reporters' queue/retry fixtures and
+`python3 tools/test_plugin_telemetry.py` for native storage, acknowledgment and
+process-restart checks with sanitizers. The native test uses each restored JSON
+dependency; alternatively pass `--json-header '/path/{game}-json.hpp'` with the
+single headers from each plugin's pinned revision. These tests use engine shims.
+Validate Windows DLL linking and both games in Plutonium before rollout: force a
+timeout/429/503 at start and end, confirm permanent 400/401 failures do not retry,
+restart a map and then a process with pending boundaries, and confirm their
+original match IDs complete before the next start. Also verify a 2xx without
+`ok: true` stays pending, an outage retains only the latest waiting heartbeat,
+and a disk failure is visible without advancing past an unsaved boundary.
 
 ## Parity
 
@@ -141,7 +203,7 @@ cleanup. Disable the leaderboard module and verify `.leaders` is absent from hel
 
 Both reporters default to `https://ec.thewrightserver.net/api/telemetry` and send
 `Authorization: Bearer <token>` plus `Content-Type: application/json` on every
-POST. Existing event fields and queue behavior are unchanged. An explicit
+POST. Authentication is also used for retried/replayed reports. An explicit
 `tws_telemetry_url` override must also use HTTPS; remove any old HTTP override.
 
 Before upgrading the API or loading these reporters: