Preserve and retry telemetry match boundaries across restarts

t6/plugins/README.md

@@ -19,7 +19,7 @@ These are libcurl transfer deadlines, starting when a worker begins the request.
 Time waiting for a worker or the game thread to deliver completion is measured
 separately. A native timeout releases the request and emits `done` with
 `success = false`, a transport error, and diagnostics. Each request is sent once.
-The telemetry sender then logs the failed event and advances its ordered queue.
+The telemetry sender retries start/end reports at the head of its ordered queue.
 A timeout does not prove the API did not already accept the event.
 
 The existing completion values remain in the same positions. An optional fourth
@@ -31,8 +31,8 @@ request waittill("done", response, success, error, diagnostics);
 
 `success` continues to mean transport success, including completed HTTP 4xx/5xx
 responses. Existing scripts that read only the first three values still work.
-An updated reporter on an old plugin logs that diagnostics and native deadlines
-are unverified; deploy the matching DLL to enable the fix.
+Current reporters also require the native telemetry helpers below; deploy the
+matching DLL and restart the process before loading the updated scripts.
 
 | Field | Meaning |
 | --- | --- |
@@ -55,20 +55,45 @@ included in diagnostics. Responses retain the existing 20 KiB truncation behavio
 Telemetry logs the status and timings for each completion at log level 1.
 Non-2xx responses log at level 0 with a bounded response preview. Health counters
 keep transport `completed`/`failed` and track `http_rejected` separately; an HTTP
-rejection is a completed transport. Producers, queue ordering and payloads are
-unchanged. Heartbeats are not coalesced, failed reports are not retried, and API
-receipt timestamps remain the existing behavior.
+rejection is a completed transport. `accepted` counts only 2xx replies with the
+JSON boolean `ok: true`. Report identities, retry behavior, heartbeat coalescing
+and persistence are described in the shared
+[delivery and rollout notes](../../README.md#telemetry-delivery-and-rollout-t5-and-t6).
+
+## Durable telemetry helpers
+
+The paired `component/telemetry.cpp` and `utils/telemetry.*` provide:
+
+- `telemetryNewMatch()`: a random process token plus a process-local match counter.
+- `telemetryLoad(server, endpoint)`: ordered saved boundary entries, or undefined
+  on an unreadable/corrupt outbox. An empty outbox returns an empty GSC array.
+- `telemetryStore(server, endpoint, body)`: idempotently commit an exact boundary
+  snapshot; false on storage, validation or capacity failure.
+- `telemetryRemove(server, endpoint, match, seq)`: atomically remove a resolved
+  entry; false on failure. The old committed file remains available on failure.
+- `telemetryResponse(body)`: parse bounded JSON without throwing on malformed
+  responses, returning `accepted` (strict boolean `ok`) and `match_id` (a string).
+  The GSC reporter separately checks the HTTP status before accepting it.
+
+The native helpers avoid the generic JSON-to-GSC converter's boolean handling.
+Outbox files are scoped by game, server and endpoint under `fs_homepath/tws-telemetry`.
+They contain snapshots, never authentication headers. One writer per scope is
+required. Writes replace the old file with `MoveFileExW`; stale `.tmp` files are
+ignored during recovery. Native memory supplies identity only; pending reports
+come from disk after every script or process restart.
 
 ## Validation and rollout
 
 Run `python3 tools/test_plugin_http.py` from the repository root for portable
-native HTTP tests. Build the x86 DLL on Windows, then on a test game server check:
+native HTTP tests, and `python3 tools/test_plugin_telemetry.py` for outbox and
+acknowledgment tests (requires the pinned JSON headers). Deploy Electric Cherry
+#144 first as described in the shared rollout notes. Build the x86 DLL on Windows, then on a test game server check:
 
 1. Updated telemetry reports HTTP status and all timing fields on normal requests.
 2. An endpoint that accepts a connection but does not reply produces a native
-   timeout near 15 seconds, followed by the next queued request.
-3. HTTP 401/429/500 responses increment `http_rejected` without blocking the queue;
-   transport errors increment `failed` instead.
+   timeout near 15 seconds; a start/end is retried before the next queued report.
+3. HTTP 401 is terminal; 429/500 retain start/end at the queue head and retry.
+   Non-2xx replies increment `http_rejected`; transport errors increment `failed`.
 4. Leaderboard and older three-value HTTP consumers still receive completion.
 5. After a full server-process restart, verify all four telemetry event types in
    Electric Cherry. A successful transport alone is not API acceptance.