Preserve and retry telemetry match boundaries across restarts

t5/plugins/README.md

@@ -19,7 +19,7 @@ These are libcurl transfer deadlines, starting when a worker begins the request.
 Time waiting for a worker or the game thread to deliver completion is measured
 separately. A native timeout releases the request and emits `done` with
 `success = false`, a transport error, and diagnostics. Each request is sent once.
-The telemetry sender then logs the failed event and advances its ordered queue.
+The telemetry sender retries start/end reports at the head of its ordered queue.
 A timeout does not prove the API did not already accept the event.
 
 The existing completion values remain in the same positions. An optional fourth
@@ -31,8 +31,8 @@ request waittill("done", response, success, error, diagnostics);
 
 `success` continues to mean transport success, including completed HTTP 4xx/5xx
 responses. Existing scripts that read only the first three values still work.
-An updated reporter on an old plugin logs that diagnostics and native deadlines
-are unverified; deploy the matching DLL to enable the fix.
+Current reporters also require the native telemetry helpers below; deploy the
+matching DLL and restart the process before loading the updated scripts.
 
 | Field | Meaning |
 | --- | --- |
@@ -55,20 +55,45 @@ included in diagnostics. Responses retain the existing 20 KiB truncation behavio
 Telemetry logs the status and timings for each completion at log level 1.
 Non-2xx responses log at level 0 with a bounded response preview. Health counters
 keep transport `completed`/`failed` and track `http_rejected` separately; an HTTP
-rejection is a completed transport. Producers, queue ordering and payloads are
-unchanged. Heartbeats are not coalesced, failed reports are not retried, and API
-receipt timestamps remain the existing behavior.
+rejection is a completed transport. `accepted` counts only 2xx replies with the
+JSON boolean `ok: true`. Report identities, retry behavior, heartbeat coalescing
+and persistence are described in the shared
+[delivery and rollout notes](../../README.md#telemetry-delivery-and-rollout-t5-and-t6).
+
+## Durable telemetry helpers
+
+The paired `component/telemetry.cpp` and `utils/telemetry.*` provide:
+
+- `telemetryNewMatch()`: a random process token plus a process-local match counter.
+- `telemetryLoad(server, endpoint)`: ordered saved boundary entries, or undefined
+  on an unreadable/corrupt outbox. An empty outbox returns an empty GSC array.
+- `telemetryStore(server, endpoint, body)`: idempotently commit an exact boundary
+  snapshot; false on storage, validation or capacity failure.
+- `telemetryRemove(server, endpoint, match, seq)`: atomically remove a resolved
+  entry; false on failure. The old committed file remains available on failure.
+- `telemetryResponse(body)`: parse bounded JSON without throwing on malformed
+  responses, returning `accepted` (strict boolean `ok`) and `match_id` (a string).
+  The GSC reporter separately checks the HTTP status before accepting it.
+
+The native helpers avoid the generic JSON-to-GSC converter's boolean handling.
+Outbox files are scoped by game, server and endpoint under `fs_homepath/tws-telemetry`.
+They contain snapshots, never authentication headers. One writer per scope is
+required. Writes replace the old file with `MoveFileExW`; stale `.tmp` files are
+ignored during recovery. Native memory supplies identity only; pending reports
+come from disk after every script or process restart.
 
 ## Validation and rollout
 
 Run `python3 tools/test_plugin_http.py` from the repository root for portable
-native HTTP tests. Build the x86 DLL on Windows, then on a test game server check:
+native HTTP tests, and `python3 tools/test_plugin_telemetry.py` for outbox and
+acknowledgment tests (requires the pinned JSON headers). Deploy Electric Cherry
+#144 first as described in the shared rollout notes. Build the x86 DLL on Windows, then on a test game server check:
 
 1. Updated telemetry reports HTTP status and all timing fields on normal requests.
 2. An endpoint that accepts a connection but does not reply produces a native
-   timeout near 15 seconds, followed by the next queued request.
-3. HTTP 401/429/500 responses increment `http_rejected` without blocking the queue;
-   transport errors increment `failed` instead.
+   timeout near 15 seconds; a start/end is retried before the next queued report.
+3. HTTP 401 is terminal; 429/500 retain start/end at the queue head and retry.
+   Non-2xx replies increment `http_rejected`; transport errors increment `failed`.
 4. Leaderboard and older three-value HTTP consumers still receive completion.
 5. After a full server-process restart, verify all four telemetry event types in
    Electric Cherry. A successful transport alone is not API acceptance.

t5/plugins/tws-t5-gsc-utils/src/component/telemetry.cpp

@@ -0,0 +1,75 @@
+#include <stdinc.hpp>
+#include "loader/component_loader.hpp"
+#include "gsc.hpp"
+#include "game/scripting/array.hpp"
+#include <utils/telemetry.hpp>
+
+namespace telemetry
+{
+    namespace
+    {
+        utils::telemetry::outbox spool(const std::string& server, const std::string& endpoint)
+        {
+            return {"tws-telemetry/t5", server, endpoint};
+        }
+
+        void storage_error()
+        {
+            // Do not echo parse exceptions: their text can contain player data.
+            printf("^1[tws_telemetry] Outbox operation failed; check tws-telemetry/t5 files, disk space and permissions. Existing reports retained.\n");
+        }
+    }
+
+    class component final : public component_interface
+    {
+    public:
+        void on_startup([[maybe_unused]] plugin::plugin* plugin) override
+        {
+            gsc::function::add("telemetrynewmatch", []() -> scripting::script_value
+            {
+                try { return utils::telemetry::new_match_key(); }
+                catch (...) { return {}; }
+            });
+            gsc::function::add("telemetryload", [](const std::string& server, const std::string& endpoint) -> scripting::script_value
+            {
+                try
+                {
+                    scripting::array entries;
+                    for (const auto& value : spool(server, endpoint).load())
+                    {
+                        scripting::array entry;
+                        entry["match"] = value.match;
+                        entry["id"] = value.seq;
+                        entry["event"] = value.event;
+                        entry["round"] = value.round;
+                        entry["body"] = value.body;
+                        entry["stored"] = true;
+                        entries.emplace_back(entry);
+                    }
+                    return entries;
+                }
+                catch (...) { storage_error(); return {}; }
+            });
+            gsc::function::add("telemetrystore", [](const std::string& server, const std::string& endpoint, const std::string& body)
+            {
+                try { spool(server, endpoint).store(body); return true; }
+                catch (...) { storage_error(); return false; }
+            });
+            gsc::function::add("telemetryremove", [](const std::string& server, const std::string& endpoint, const std::string& match, const int seq)
+            {
+                try { spool(server, endpoint).erase(match, seq); return true; }
+                catch (...) { storage_error(); return false; }
+            });
+            gsc::function::add("telemetryresponse", [](const std::string& response)
+            {
+                const auto ack = utils::telemetry::parse_acknowledgement(response);
+                scripting::array result;
+                result["accepted"] = ack.accepted;
+                result["match_id"] = ack.match_id;
+                return result;
+            });
+        }
+    };
+}
+
+REGISTER_COMPONENT(telemetry::component)

t5/plugins/tws-t5-gsc-utils/src/utils/telemetry.cpp

@@ -0,0 +1,199 @@
+#include <stdinc.hpp>
+#include "telemetry.hpp"
+
+#include <atomic>
+#include <cstdint>
+#include <iomanip>
+#include <limits>
+#include <random>
+#include <set>
+
+namespace utils::telemetry
+{
+    namespace
+    {
+        constexpr size_t max_body_size = 0x5000;
+        constexpr size_t max_file_size = 16 * 1024 * 1024;
+        constexpr size_t max_reports = 4096;
+
+        // Stable, compact filename for arbitrary server names and endpoints.
+        // The envelope also checks both strings, so a collision fails closed.
+        std::string scope_name(const std::string& scope)
+        {
+            uint64_t hash = 14695981039346656037ull;
+            for (const unsigned char byte : scope)
+                hash = (hash ^ byte) * 1099511628211ull;
+            std::ostringstream result;
+            result << std::hex << std::setfill('0') << std::setw(16) << hash;
+            return result.str();
+        }
+
+        std::string boot_token()
+        {
+            std::random_device random;
+            std::ostringstream result;
+            result << std::hex << std::setfill('0');
+            for (int i = 0; i < 4; ++i)
+                result << std::setw(8) << static_cast<uint32_t>(random());
+            return result.str();
+        }
+
+        int positive_int(const nlohmann::json& value)
+        {
+            if (!value.is_number_integer() || value < 1 || value > std::numeric_limits<int>::max())
+                throw std::runtime_error("invalid telemetry sequence");
+            return value.get<int>();
+        }
+    }
+
+    std::string new_match_key()
+    {
+        static const auto boot = boot_token();
+        static std::atomic<uint64_t> counter{0};
+        return boot + "-" + std::to_string(++counter);
+    }
+
+    acknowledgement parse_acknowledgement(const std::string& response)
+    {
+        acknowledgement result;
+        // HTTP responses at this boundary have potentially been truncated.
+        if (response.empty() || response.size() >= max_body_size)
+            return result;
+        const auto body = nlohmann::json::parse(response, nullptr, false);
+        if (!body.is_object() || !body.contains("ok") || !body["ok"].is_boolean() || !body["ok"].get<bool>())
+            return result;
+        result.accepted = true;
+        const auto telemetry = body.find("telemetry");
+        if (telemetry != body.end() && telemetry->is_object())
+        {
+            const auto timing = telemetry->find("timing");
+            if (timing != telemetry->end() && timing->is_object())
+            {
+                const auto id = timing->find("matchId");
+                if (id != timing->end() && id->is_number_integer() && *id > 0)
+                    result.match_id = id->dump();
+            }
+        }
+        return result;
+    }
+
+    outbox::outbox(std::filesystem::path root, std::string server, std::string endpoint)
+        : server_(std::move(server)), endpoint_(std::move(endpoint))
+    {
+        file_ = root / (scope_name(server_ + '\0' + endpoint_) + ".json");
+    }
+
+    report outbox::decode(const std::string& body) const
+    {
+        if (body.empty() || body.size() >= max_body_size)
+            throw std::runtime_error("invalid telemetry report size");
+        const auto data = nlohmann::json::parse(body);
+        if (!data.is_object() || data.at("server") != server_ ||
+            !data.at("map").is_string() || !data.at("players").is_array())
+            throw std::runtime_error("invalid telemetry report envelope");
+        report value;
+        value.match = data.at("match").get<std::string>();
+        value.seq = positive_int(data.at("seq"));
+        value.event = data.at("event").get<std::string>();
+        if (value.match.empty() || value.match.size() > 64 ||
+            (value.event != "game_start" && value.event != "game_end") ||
+            (value.event == "game_start" && value.seq != 1))
+            throw std::runtime_error("invalid durable telemetry identity/event");
+        value.round = 0;
+        if (value.event == "game_end")
+        {
+            const auto& round = data.at("round");
+            if (!round.is_number_integer() || round < 0 || round > std::numeric_limits<int>::max())
+                throw std::runtime_error("invalid telemetry round");
+            value.round = round.get<int>();
+        }
+        value.body = body;
+        return value;
+    }
+
+    std::vector<report> outbox::load() const
+    {
+        if (!std::filesystem::exists(file_))
+            return {};
+        if (std::filesystem::file_size(file_) > max_file_size)
+            throw std::runtime_error("telemetry outbox exceeds size limit");
+        std::ifstream stream(file_, std::ios::binary);
+        if (!stream)
+            throw std::runtime_error("cannot open telemetry outbox");
+        const auto data = nlohmann::json::parse(stream);
+        if (stream.bad() || !data.is_object() || data.at("version") != 1 ||
+            data.at("server") != server_ || data.at("endpoint") != endpoint_ ||
+            !data.at("reports").is_array() || data.at("reports").size() > max_reports)
+            throw std::runtime_error("invalid telemetry outbox; preserve file for recovery");
+        std::vector<report> reports;
+        std::set<std::pair<std::string, int>> identities;
+        for (const auto& body : data.at("reports"))
+        {
+            auto value = decode(body.get<std::string>());
+            if (!identities.emplace(value.match, value.seq).second)
+                throw std::runtime_error("duplicate identity in telemetry outbox");
+            reports.push_back(std::move(value));
+        }
+        return reports;
+    }
+
+    void outbox::save(const std::vector<report>& reports) const
+    {
+        if (reports.size() > max_reports)
+            throw std::runtime_error("telemetry outbox is full; delivery blocked");
+        nlohmann::json data = {{"version", 1}, {"server", server_}, {"endpoint", endpoint_},
+            {"reports", nlohmann::json::array()}};
+        for (const auto& value : reports)
+            data["reports"].push_back(value.body);
+        const auto encoded = data.dump();
+        if (encoded.size() > max_file_size)
+            throw std::runtime_error("telemetry outbox is full; delivery blocked");
+        std::filesystem::create_directories(file_.parent_path());
+        auto temporary = file_;
+        temporary += ".tmp";
+        std::ofstream stream(temporary, std::ios::binary | std::ios::trunc);
+        stream.write(encoded.data(), static_cast<std::streamsize>(encoded.size()));
+        stream.flush();
+        if (!stream)
+            throw std::runtime_error("cannot write telemetry outbox; old file retained");
+        stream.close();
+        if (stream.fail())
+            throw std::runtime_error("cannot close telemetry outbox; old file retained");
+#ifdef _WIN32
+        if (!MoveFileExW(temporary.c_str(), file_.c_str(), MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH))
+            throw std::runtime_error("cannot replace telemetry outbox; old file retained");
+#else
+        // Used by the portable tests; the production plugin targets Windows.
+        std::filesystem::rename(temporary, file_);
+#endif
+    }
+
+    void outbox::store(const std::string& body) const
+    {
+        const auto incoming = decode(body);
+        auto reports = load();
+        for (const auto& value : reports)
+        {
+            if (value.match != incoming.match || value.seq != incoming.seq)
+                continue;
+            if (value.body != body)
+                throw std::runtime_error("telemetry identity reused for a different snapshot");
+            return;
+        }
+        reports.push_back(incoming);
+        save(reports);
+    }
+
+    void outbox::erase(const std::string& match, const int seq) const
+    {
+        auto reports = load();
+        const auto end = std::remove_if(reports.begin(), reports.end(), [&](const report& value)
+        {
+            return value.match == match && value.seq == seq;
+        });
+        if (end == reports.end())
+            return;
+        reports.erase(end, reports.end());
+        save(reports);
+    }
+}

t5/plugins/tws-t5-gsc-utils/src/utils/telemetry.hpp

@@ -0,0 +1,45 @@
+#pragma once
+
+#include <filesystem>
+#include <string>
+#include <vector>
+
+namespace utils::telemetry
+{
+    struct report
+    {
+        std::string match;
+        int seq;
+        std::string event;
+        int round;
+        std::string body;
+    };
+
+    struct acknowledgement
+    {
+        bool accepted = false;
+        std::string match_id;
+    };
+
+    // Native state survives script/map shutdown; a new process gets a new token.
+    std::string new_match_key();
+    acknowledgement parse_acknowledgement(const std::string& response);
+
+    // One game-thread writer per server/endpoint. Commit a replacement before
+    // discarding the previous file; a failed write leaves it available for replay.
+    class outbox
+    {
+    public:
+        outbox(std::filesystem::path root, std::string server, std::string endpoint);
+        std::vector<report> load() const;
+        void store(const std::string& body) const;
+        void erase(const std::string& match, int seq) const;
+
+    private:
+        std::filesystem::path file_;
+        std::string server_;
+        std::string endpoint_;
+        report decode(const std::string& body) const;
+        void save(const std::vector<report>& reports) const;
+    };
+}

t5/scripts/README.md

@@ -333,6 +333,8 @@ The reporter emits exactly four event types:
 | `heartbeat` | Every 30 seconds after `game_start`, until game end | Same fields as `round_change`, with current round and player stats |
 | `game_end` | First stock `end_game`, `game_ended`, or `intermission` notification | `server`, `map`, final `round`, full player stats |
 
+All four events also include `match` and `seq` (see delivery below).
+
 Each player in round/heartbeat/end reports has `name`, `guid`, `kills`, `headshots`,
 `downs`, `revives`, and `score`. Names and GUIDs remain strings; GUIDs never
 pass through numeric conversion. Round and stats are nonnegative integers,
@@ -350,14 +352,14 @@ serializes an empty GSC array as `null`; player objects and strings still
 use `jsonSerialize` for escaping.
 
 ```json
-{"event":"game_start","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","players":[{"name":"Josh","guid":"12345678901234567"}]}
-{"event":"round_change","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":420,"headshots":160,"downs":1,"revives":3,"score":32000}]}
-{"event":"heartbeat","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":430,"headshots":165,"downs":1,"revives":3,"score":32500}]}
-{"event":"game_end","server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":18,"players":[{"name":"Josh","guid":"12345678901234567","kills":445,"headshots":170,"downs":2,"revives":3,"score":33500}]}
+{"event":"game_start","match":"process-token-1","seq":1,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","players":[{"name":"Josh","guid":"12345678901234567"}]}
+{"event":"round_change","match":"process-token-1","seq":2,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":420,"headshots":160,"downs":1,"revives":3,"score":32000}]}
+{"event":"heartbeat","match":"process-token-1","seq":3,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":17,"players":[{"name":"Josh","guid":"12345678901234567","kills":430,"headshots":165,"downs":1,"revives":3,"score":32500}]}
+{"event":"game_end","match":"process-token-1","seq":4,"server":"t5_zombie_prototype_1","map":"zombie_cod5_prototype","round":18,"players":[{"name":"Josh","guid":"12345678901234567","kills":445,"headshots":170,"downs":2,"revives":3,"score":33500}]}
 ```
 
 Events are captured immediately into an in-memory queue. One HTTP worker
-POSTs each snapshot once, in gameplay order, with `Content-Type:
+POSTs snapshots in gameplay order, with `Content-Type:
 application/json` and `Authorization: Bearer <token>`, waiting for completion
 before sending the next. Gameplay and round detection continue while requests
 are pending. Repeated joins or
@@ -370,35 +372,33 @@ The first heartbeat is captured 30 seconds after match start, then every
 changes and later spawns do not reset the interval or add heartbeat workers.
 Heartbeats use the same snapshot builder and ordered sender as round reports;
 pending HTTP requests can delay transmission. Game end stops the heartbeat
-worker, and already queued heartbeats drain before `game_end`.
+worker. Only the newest waiting heartbeat is retained; round reports and the
+in-flight request retain their order before `game_end`.
 
-There is no discovery, beta token, session ID, sequence field, custom event,
-external helper, or separate beta acknowledgement. Delivery is best effort:
-no retries or persistence across map reloads, and no cross-game exactly-once
-guarantee. The [TWS plugin build](../plugins/README.md) enforces a 5-second
-connection and 15-second total transfer timeout. A timed-out report is logged
-and skipped so later reports can proceed. A map reload can discard queued reports.
+Each report includes a native-generated `match` key and a positive `seq`, starting
+at 1 for `game_start`. Start/end snapshots are written to disk before sending and
+replayed before a new match after map/process restarts. Retryable failures retain
+the queue head with 1–30 second backoff; other 4xx are terminal. Acceptance requires
+a 2xx response containing the JSON boolean `ok: true`, and logs include Electric
+Cherry's `telemetry.timing.matchId`. The native 5-second connect / 15-second total
+transfer limits still apply. Ordinary round/heartbeat failures are best effort.
 
-Transport failures and non-2xx HTTP responses log to the server console. The TWS
-plugin exposes HTTP status and cumulative DNS/TCP/TLS/first-byte/total timings,
-plus worker-queue and completion-notification delays. Transport success alone
-does not establish API acceptance.
-Verify acceptance at Electric Cherry: success is HTTP 200 with `ok: true`
-and telemetry containing `receivedAt`; invalid payloads return 400 and
-non-JSON content types return 415.
+**Deploy Electric Cherry #144 and the updated TWS plugin before these scripts.**
+See the shared [delivery and rollout notes](../../README.md#telemetry-delivery-and-rollout-t5-and-t6)
+for outbox location, persistence requirements, capacity/error handling and tests.
 
 **Diagnosing a lobby that stops reporting**
 
 Default logging includes initialization, match start/end, each request's
 event/round and local ID, queue wait and request duration in milliseconds,
-transport failures/recovery, HTTP rejection/status/timings, and the final queue drain. IDs reset on map
-load and only appear in logs; payloads are unchanged. Every log includes
+transport failures/recovery, HTTP rejection/status/timings, and the final queue drain.
+Each log identifies the original match key and payload sequence, including replays. Every log includes
 the server, map, and game time in milliseconds.
 
 An independent watchdog prints health every 30 seconds, even before match
 start and while HTTP is pending or the game-end queue is draining. It shows
-the sender stage, current snapshot capture, pending queue size (excluding
-the in-flight request), total queued, completed transports, transport failures,
+the sender stage, current snapshot capture, pending queue size (including
+the retained in-flight head), total captured, accepted reports, completed transports, transport failures,
 HTTP rejections, and
 time since the last completion (`-1` means none). Warnings remain enabled
 at log level `0`:
@@ -406,7 +406,7 @@ at log level `0`:
 - `HTTP stalled`: a request has been pending for at least 30 seconds. The
   warning repeats on watchdog ticks with its ID, event, age, and queue size.
 - `sender is not consuming`: the oldest queued report has waited at least
-  30 seconds with no in-flight request.
+  30 seconds with no in-flight request outside an intentional retry backoff.
 - `round watcher has stopped advancing`: no polling progress for over
   5 seconds, detected on the next watchdog tick.
 - `no snapshot captured`: an active match has gone at least 65 seconds
@@ -420,17 +420,12 @@ indicate a proxy error. Previews can include player identities echoed by
 the API. Restore `1` afterward, or use `0` to retain only warnings, errors,
 and transport recovery messages.
 
-Missing request handles and reported transport failures are counted and
-skipped so the sender can continue with the next event. Empty responses
-produce a warning. Arbitrary response bodies are not parsed as JSON,
-because malformed/proxy responses must not break the sender. The watchdog
-does not launch overlapping requests or retry failed events. Native timeouts
-release stalled transfers. A request still pending after 30 seconds warrants
-checking the installed plugin version, worker-queue delay and completion delivery.
-If diagnostics are absent, the reporter warns that native deadlines are unverified;
-install the TWS plugin build and restart the server process.
-
-Native deadlines and completion diagnostics require the [TWS plugin build](../plugins/README.md).
+Missing handles, transport errors, empty/invalid acknowledgments and retryable
+HTTP responses keep start/end pending. The watchdog never launches overlapping
+requests. A request pending beyond 30 seconds warrants checking the installed
+plugin version, worker-queue delay and completion delivery. Native deadlines,
+diagnostics and the telemetry outbox/acknowledgment helpers require the updated
+[TWS plugin build](../plugins/README.md).
 The upstream baseline contains
 [`http::request`](https://github.com/alicealys/t5-gsc-utils/blob/main/src/component/http.cpp)
 and [`jsonSerialize`](https://github.com/alicealys/t5-gsc-utils/blob/main/src/component/json.cpp).
@@ -465,10 +460,8 @@ and [score tracking](https://github.com/plutoniummod/t5-scripts/blob/main/ZM/Com
    though transport completes. Also check an empty or HTML response.
 7. Point a test server at an endpoint that accepts a connection but does
    not reply. With the TWS plugin, verify a native timeout near 15 seconds and
-   continued queue consumption. With an older plugin, expect recurring
-   `HTTP stalled` warnings and increasing
-   pending counts while gameplay continues, including after game over.
-   Release the delayed reply and verify the queue resumes in order and
+   a retained start/end retry with bounded backoff while gameplay continues.
+   Restore the endpoint and verify the queue resumes in order and
    the watchdog stops after the final drain. Test a refused connection
    separately to check failure counts and the recovery log once restored.
 

t5/scripts/tws_t5_telemetry.gsc

@@ -45,15 +45,26 @@ init()
         return;
     }
 
+    // Recover before registering producers, so old boundaries precede this match.
+    recovered = telemetryLoad(level.telemetry_server, level.telemetry_url);
+    if (!isDefined(recovered))
+    {
+        telemetry_log(0, "ERROR: cannot recover durable outbox; reporting disabled until storage is repaired and the map restarted");
+        return;
+    }
+
     level.telemetry_initialized = true;
     level.telemetry_started = false;
     level.telemetry_ended = false;
     level.telemetry_last_round = -1;
-    level.telemetry_queue = [];
+    level.telemetry_queue = recovered;
     level.telemetry_queue_read = 0;
-    level.telemetry_queue_write = 0;
+    level.telemetry_queue_write = recovered.size;
+    for (i = 0; i < recovered.size; i++)
+        level.telemetry_queue[i]["captured"] = getTime();
     level.telemetry_sequence = 0;
     level.telemetry_completed = 0;
+    level.telemetry_accepted = 0;
     level.telemetry_failed = 0;
     level.telemetry_http_rejected = 0;
     level.telemetry_missing_diagnostics_warned = false;
@@ -65,7 +76,7 @@ init()
     level.telemetry_sender_state = "starting";
     level.telemetry_inflight = undefined;
 
-    telemetry_log(1, "Initialized; waiting for match start. log_level=" + telemetry_log_level());
+    telemetry_log(1, "Initialized; recovered=" + recovered.size + " waiting for match start. log_level=" + telemetry_log_level());
 
     // Use the shared dispatcher; a second connected listener breaks T5 SP.
     scripts\sp\tws_t5_common::register_spawn_handler(::telemetry_game_start);
@@ -83,6 +94,12 @@ telemetry_game_start()
     if (level.telemetry_started || level.telemetry_ended)
         return;
 
+    level.telemetry_match = telemetryNewMatch();
+    if (!isDefined(level.telemetry_match))
+    {
+        telemetry_log(0, "ERROR: cannot allocate match identity");
+        return;
+    }
     level.telemetry_started = true;
     level.telemetry_last_capture = getTime();
     telemetry_log(1, "Match start detected");
@@ -146,6 +163,8 @@ telemetry_game_end_watcher()
 
     // Preserve order even if the match ends before the next polling tick.
     telemetry_game_start();
+    if (!level.telemetry_started)
+        return;
     telemetry_report_round();
     level.telemetry_ended = true;
     level notify("telemetry_finished");
@@ -227,30 +246,78 @@ telemetry_enqueue(event_name)
     level.telemetry_capture_event = event_name;
     telemetry_log(2, "Capturing event=" + event_name);
     // Snapshot and serialize before yielding, including during game over.
+    level.telemetry_sequence++;
     body = "{\"event\":" + jsonSerialize(event_name);
     body += ",\"server\":" + jsonSerialize(level.telemetry_server);
     body += ",\"map\":" + jsonSerialize(GetDvar("mapname"));
+    body += ",\"match\":" + jsonSerialize(level.telemetry_match);
+    body += ",\"seq\":" + level.telemetry_sequence;
     if (event_name != "game_start")
         body += ",\"round\":" + telemetry_nonnegative_int(level.round_number);
     body += ",\"players\":" + telemetry_players_json(event_name != "game_start") + "}";
 
     entry = [];
-    level.telemetry_sequence++;
+    entry["match"] = level.telemetry_match;
     entry["id"] = level.telemetry_sequence;
     entry["event"] = event_name;
     entry["round"] = telemetry_nonnegative_int(level.round_number);
     entry["captured"] = getTime();
     entry["body"] = body;
-    level.telemetry_queue[level.telemetry_queue_write] = entry;
-    level.telemetry_queue_write++;
+    // Persist boundaries before yielding or handing any request to the plugin.
+    // A failed save stays in memory and blocks sending until storage recovers.
+    entry["stored"] = false;
+    telemetry_queue_push(entry);
+    if (telemetry_critical(entry))
+        telemetry_persist_pending();
     level.telemetry_last_capture = getTime();
     level.telemetry_capture_event = "none";
     telemetry_log(2, "Queued " + telemetry_event_label(entry) + " pending=" + telemetry_pending());
     level notify("telemetry_queued");
 }
 
-// A single worker keeps one request in flight. It survives game-over notifies
-// to drain the final snapshot; no retries, discovery, or beta-specific calls.
+// If storage was unavailable at an earlier capture, save it first. Otherwise a
+// newly captured end could be committed ahead of its unsaved start on restart.
+telemetry_persist_pending()
+{
+    for (i = level.telemetry_queue_read; i < level.telemetry_queue_write; i++)
+    {
+        entry = level.telemetry_queue[i];
+        if (!telemetry_critical(entry) || entry["stored"])
+            continue;
+        entry["stored"] = telemetryStore(level.telemetry_server, level.telemetry_url, entry["body"]);
+        if (!entry["stored"])
+            return;
+    }
+}
+
+telemetry_critical(entry)
+{
+    return entry["event"] == "game_start" || entry["event"] == "game_end";
+}
+
+// Remove obsolete waiting heartbeats, keeping round reports and the in-flight
+// head in order. The new heartbeat keeps its capture-time sequence and snapshot.
+telemetry_queue_push(entry)
+{
+    queue = [];
+    count = 0;
+    for (i = level.telemetry_queue_read; i < level.telemetry_queue_write; i++)
+    {
+        previous = level.telemetry_queue[i];
+        inflight = i == level.telemetry_queue_read && isDefined(level.telemetry_inflight);
+        if (entry["event"] == "heartbeat" && previous["event"] == "heartbeat" && !inflight)
+            continue;
+        queue[count] = previous;
+        count++;
+    }
+    queue[count] = entry;
+    level.telemetry_queue = queue;
+    level.telemetry_queue_read = 0;
+    level.telemetry_queue_write = count + 1;
+}
+
+// One worker retains the head through retries. No later report can overtake an
+// unacknowledged boundary. Retry delays grow 1, 2, 4, 8, 16, then at most 30s.
 telemetry_sender()
 {
     for (;;)
@@ -263,7 +330,7 @@ telemetry_sender()
             if (level.telemetry_ended)
             {
                 level.telemetry_sender_state = "drained";
-                telemetry_log(1, "Queue drained after game end; completed=" + level.telemetry_completed + " failed=" + level.telemetry_failed);
+                telemetry_log(1, "Queue drained after game end; accepted=" + level.telemetry_accepted + " failed=" + level.telemetry_failed);
                 level notify("telemetry_drained");
                 return;
             }
@@ -274,12 +341,53 @@ telemetry_sender()
         }
 
         entry = level.telemetry_queue[level.telemetry_queue_read];
+        if (!telemetry_process(entry))
+        {
+            delay = telemetry_retry_delay(entry);
+            level.telemetry_sender_state = "retry_backoff";
+            telemetry_log(0, "Retry pending " + telemetry_event_label(entry) + " delay_s=" + delay);
+            wait (delay);
+            continue;
+        }
         level.telemetry_queue[level.telemetry_queue_read] = undefined;
         level.telemetry_queue_read++;
-        telemetry_send(entry);
     }
 }
 
+telemetry_retry_delay(entry)
+{
+    if (!isDefined(entry["retry_delay"]))
+        entry["retry_delay"] = 1;
+    else
+        entry["retry_delay"] *= 2;
+    if (entry["retry_delay"] > 30)
+        entry["retry_delay"] = 30;
+    return entry["retry_delay"];
+}
+
+telemetry_process(entry)
+{
+    critical = telemetry_critical(entry);
+    if (critical && !entry["stored"])
+    {
+        entry["stored"] = telemetryStore(level.telemetry_server, level.telemetry_url, entry["body"]);
+        if (!entry["stored"])
+            return false;
+    }
+    // If removing an acknowledged/terminal entry fails, retry the disk update
+    // without POSTing again. A restart before removal safely replays its key/seq.
+    if (!isDefined(entry["resolved"]))
+    {
+        outcome = telemetry_send(entry);
+        if (critical && outcome == "retry")
+            return false;
+        entry["resolved"] = true;
+    }
+    if (critical)
+        return telemetryRemove(level.telemetry_server, level.telemetry_url, entry["match"], entry["id"]);
+    return true;
+}
+
 telemetry_send(entry)
 {
     level.telemetry_inflight = entry;
@@ -302,10 +410,10 @@ telemetry_send(entry)
     request = http::request(level.telemetry_url, options);
     if (!isDefined(request))
     {
-        telemetry_transport_failed(label + " request handle is undefined; skipping event");
+        telemetry_transport_failed(label + " request handle is undefined");
         level.telemetry_inflight = undefined;
         level.telemetry_sender_state = "idle";
-        return;
+        return "retry";
     }
 
     level.telemetry_sender_state = "waiting_http";
@@ -319,8 +427,8 @@ telemetry_send(entry)
         telemetry_log(0, "WARNING: plugin returned no HTTP diagnostics; native timeouts are unverified. Install the TWS plugin build and restart the server process");
     }
 
-    // The plugin's success flag reports transport completion, not HTTP status.
-    // No automatic retry: the receiver may already have accepted the event.
+    // The plugin's success flag reports transport completion, not API acceptance.
+    outcome = "retry";
     if (!isDefined(success) || !success)
     {
         if (!isDefined(error))
@@ -330,29 +438,49 @@ telemetry_send(entry)
     else
     {
         level.telemetry_completed++;
-        if (level.telemetry_failure_streak > 0)
-            telemetry_log(0, "Transport recovered after " + level.telemetry_failure_streak + " failures; " + label);
-        level.telemetry_failure_streak = 0;
-        if (isDefined(diagnostics) && isDefined(diagnostics["http_status"]))
+        outcome = telemetry_response_outcome(response, diagnostics);
+        if (outcome == "accepted")
         {
-            if (diagnostics["http_status"] < 200 || diagnostics["http_status"] >= 300)
-            {
-                level.telemetry_http_rejected++;
-                telemetry_log(0, "ERROR: HTTP rejected " + label + " elapsed_ms=" + elapsed + timing + " body=" + telemetry_preview(response));
-            }
-            else
-                telemetry_log(1, "HTTP completed " + label + " elapsed_ms=" + elapsed + timing);
+            ack = telemetryResponse(response);
+            level.telemetry_accepted++;
+            if (level.telemetry_failure_streak > 0)
+                telemetry_log(0, "API recovered after " + level.telemetry_failure_streak + " failures; " + label);
+            level.telemetry_failure_streak = 0;
+            telemetry_log(1, "API accepted " + label + " ec_match_id=" + ack["match_id"] + " elapsed_ms=" + elapsed + timing);
         }
         else
-            telemetry_log(1, "Transport completed " + label + " elapsed_ms=" + elapsed + timing + " (API acceptance unverified)");
-        if (!isDefined(response) || response == "")
-            telemetry_log(0, "WARNING: empty API response for " + label);
-        else
-            telemetry_log(2, "Response " + label + " body=" + telemetry_preview(response));
+        {
+            level.telemetry_failure_streak++;
+            level.telemetry_http_rejected++;
+            telemetry_log(0, "ERROR: API unacknowledged " + label + " outcome=" + outcome + " elapsed_ms=" + elapsed + timing + " body=" + telemetry_preview(response));
+        }
+        telemetry_log(2, "Response " + label + " body=" + telemetry_preview(response));
     }
 
+    if (outcome != "accepted" && (!telemetry_critical(entry) || outcome == "drop"))
+        telemetry_log(0, "Dropping unaccepted " + label + " outcome=" + outcome);
     level.telemetry_inflight = undefined;
     level.telemetry_sender_state = "idle";
+    return outcome;
+}
+
+telemetry_response_outcome(response, diagnostics)
+{
+    if (!isDefined(diagnostics) || !isDefined(diagnostics["http_status"]))
+        return "retry";
+    status = diagnostics["http_status"];
+    if (status == 0 || status == 429 || (status >= 500 && status < 600))
+        return "retry";
+    if (status < 200 || status >= 300)
+        return "drop";
+    if (!isDefined(response))
+        return "retry";
+    // Native nonthrowing JSON parsing requires an actual boolean true, rejects
+    // truncated/proxy bodies, and preserves large Electric Cherry IDs as strings.
+    ack = telemetryResponse(response);
+    if (ack["accepted"])
+        return "accepted";
+    return "retry";
 }
 
 // The native plugin appends diagnostics while preserving the old done arguments.
@@ -387,7 +515,7 @@ telemetry_pending()
 
 telemetry_event_label(entry)
 {
-    return "id=" + entry["id"] + " event=" + entry["event"] + " round=" + entry["round"];
+    return "match=" + entry["match"] + " seq=" + entry["id"] + " event=" + entry["event"] + " round=" + entry["round"];
 }
 
 // Read live so diagnostics can be increased without restarting the map.
@@ -439,7 +567,7 @@ telemetry_watchdog()
         telemetry_log(1, "Health started=" + level.telemetry_started + " ended=" + level.telemetry_ended +
             " sender=" + level.telemetry_sender_state + " capturing=" + level.telemetry_capture_event +
             " pending=" + pending + " queued_total=" + level.telemetry_sequence +
-            " completed=" + level.telemetry_completed + " failed=" + level.telemetry_failed + " http_rejected=" + level.telemetry_http_rejected +
+            " accepted=" + level.telemetry_accepted + " completed=" + level.telemetry_completed + " failed=" + level.telemetry_failed + " http_rejected=" + level.telemetry_http_rejected +
             " last_done_ms_ago=" + completion_age);
 
         if (isDefined(level.telemetry_inflight))
@@ -450,7 +578,7 @@ telemetry_watchdog()
                     " elapsed_ms=" + age + " pending=" + pending + " stage=" + level.telemetry_sender_state +
                     "; native deadline exceeded; check plugin version, worker queue and completion delivery");
         }
-        else if (pending > 0)
+        else if (pending > 0 && level.telemetry_sender_state != "retry_backoff")
         {
             oldest = level.telemetry_queue[level.telemetry_queue_read];
             if (now - oldest["captured"] >= 30000)