Preserve and retry telemetry match boundaries across restarts

t6/plugins/README.md

@@ -19,7 +19,7 @@ These are libcurl transfer deadlines, starting when a worker begins the request.
 Time waiting for a worker or the game thread to deliver completion is measured
 separately. A native timeout releases the request and emits `done` with
 `success = false`, a transport error, and diagnostics. Each request is sent once.
-The telemetry sender then logs the failed event and advances its ordered queue.
+The telemetry sender retries start/end reports at the head of its ordered queue.
 A timeout does not prove the API did not already accept the event.
 
 The existing completion values remain in the same positions. An optional fourth
@@ -31,8 +31,8 @@ request waittill("done", response, success, error, diagnostics);
 
 `success` continues to mean transport success, including completed HTTP 4xx/5xx
 responses. Existing scripts that read only the first three values still work.
-An updated reporter on an old plugin logs that diagnostics and native deadlines
-are unverified; deploy the matching DLL to enable the fix.
+Current reporters also require the native telemetry helpers below; deploy the
+matching DLL and restart the process before loading the updated scripts.
 
 | Field | Meaning |
 | --- | --- |
@@ -55,20 +55,45 @@ included in diagnostics. Responses retain the existing 20 KiB truncation behavio
 Telemetry logs the status and timings for each completion at log level 1.
 Non-2xx responses log at level 0 with a bounded response preview. Health counters
 keep transport `completed`/`failed` and track `http_rejected` separately; an HTTP
-rejection is a completed transport. Producers, queue ordering and payloads are
-unchanged. Heartbeats are not coalesced, failed reports are not retried, and API
-receipt timestamps remain the existing behavior.
+rejection is a completed transport. `accepted` counts only 2xx replies with the
+JSON boolean `ok: true`. Report identities, retry behavior, heartbeat coalescing
+and persistence are described in the shared
+[delivery and rollout notes](../../README.md#telemetry-delivery-and-rollout-t5-and-t6).
+
+## Durable telemetry helpers
+
+The paired `component/telemetry.cpp` and `utils/telemetry.*` provide:
+
+- `telemetryNewMatch()`: a random process token plus a process-local match counter.
+- `telemetryLoad(server, endpoint)`: ordered saved boundary entries, or undefined
+  on an unreadable/corrupt outbox. An empty outbox returns an empty GSC array.
+- `telemetryStore(server, endpoint, body)`: idempotently commit an exact boundary
+  snapshot; false on storage, validation or capacity failure.
+- `telemetryRemove(server, endpoint, match, seq)`: atomically remove a resolved
+  entry; false on failure. The old committed file remains available on failure.
+- `telemetryResponse(body)`: parse bounded JSON without throwing on malformed
+  responses, returning `accepted` (strict boolean `ok`) and `match_id` (a string).
+  The GSC reporter separately checks the HTTP status before accepting it.
+
+The native helpers avoid the generic JSON-to-GSC converter's boolean handling.
+Outbox files are scoped by game, server and endpoint under `fs_homepath/tws-telemetry`.
+They contain snapshots, never authentication headers. One writer per scope is
+required. Writes replace the old file with `MoveFileExW`; stale `.tmp` files are
+ignored during recovery. Native memory supplies identity only; pending reports
+come from disk after every script or process restart.
 
 ## Validation and rollout
 
 Run `python3 tools/test_plugin_http.py` from the repository root for portable
-native HTTP tests. Build the x86 DLL on Windows, then on a test game server check:
+native HTTP tests, and `python3 tools/test_plugin_telemetry.py` for outbox and
+acknowledgment tests (requires the pinned JSON headers). Deploy Electric Cherry
+#144 first as described in the shared rollout notes. Build the x86 DLL on Windows, then on a test game server check:
 
 1. Updated telemetry reports HTTP status and all timing fields on normal requests.
 2. An endpoint that accepts a connection but does not reply produces a native
-   timeout near 15 seconds, followed by the next queued request.
-3. HTTP 401/429/500 responses increment `http_rejected` without blocking the queue;
-   transport errors increment `failed` instead.
+   timeout near 15 seconds; a start/end is retried before the next queued report.
+3. HTTP 401 is terminal; 429/500 retain start/end at the queue head and retry.
+   Non-2xx replies increment `http_rejected`; transport errors increment `failed`.
 4. Leaderboard and older three-value HTTP consumers still receive completion.
 5. After a full server-process restart, verify all four telemetry event types in
    Electric Cherry. A successful transport alone is not API acceptance.

t6/plugins/tws-t6-gsc-utils/src/component/telemetry.cpp

@@ -0,0 +1,75 @@
+#include <stdinc.hpp>
+#include "loader/component_loader.hpp"
+#include "gsc.hpp"
+#include "game/scripting/array.hpp"
+#include <utils/telemetry.hpp>
+
+namespace telemetry
+{
+    namespace
+    {
+        utils::telemetry::outbox spool(const std::string& server, const std::string& endpoint)
+        {
+            return {"tws-telemetry/t6", server, endpoint};
+        }
+
+        void storage_error()
+        {
+            // Do not echo parse exceptions: their text can contain player data.
+            printf("^1[tws_telemetry] Outbox operation failed; check tws-telemetry/t6 files, disk space and permissions. Existing reports retained.\n");
+        }
+    }
+
+    class component final : public component_interface
+    {
+    public:
+        void on_startup([[maybe_unused]] plugin::plugin* plugin) override
+        {
+            gsc::function::add("telemetrynewmatch", []() -> scripting::script_value
+            {
+                try { return utils::telemetry::new_match_key(); }
+                catch (...) { return {}; }
+            });
+            gsc::function::add("telemetryload", [](const std::string& server, const std::string& endpoint) -> scripting::script_value
+            {
+                try
+                {
+                    scripting::array entries;
+                    for (const auto& value : spool(server, endpoint).load())
+                    {
+                        scripting::array entry;
+                        entry["match"] = value.match;
+                        entry["id"] = value.seq;
+                        entry["event"] = value.event;
+                        entry["round"] = value.round;
+                        entry["body"] = value.body;
+                        entry["stored"] = true;
+                        entries.emplace_back(entry);
+                    }
+                    return entries;
+                }
+                catch (...) { storage_error(); return {}; }
+            });
+            gsc::function::add("telemetrystore", [](const std::string& server, const std::string& endpoint, const std::string& body)
+            {
+                try { spool(server, endpoint).store(body); return true; }
+                catch (...) { storage_error(); return false; }
+            });
+            gsc::function::add("telemetryremove", [](const std::string& server, const std::string& endpoint, const std::string& match, const int seq)
+            {
+                try { spool(server, endpoint).erase(match, seq); return true; }
+                catch (...) { storage_error(); return false; }
+            });
+            gsc::function::add("telemetryresponse", [](const std::string& response)
+            {
+                const auto ack = utils::telemetry::parse_acknowledgement(response);
+                scripting::array result;
+                result["accepted"] = ack.accepted;
+                result["match_id"] = ack.match_id;
+                return result;
+            });
+        }
+    };
+}
+
+REGISTER_COMPONENT(telemetry::component)

t6/plugins/tws-t6-gsc-utils/src/utils/telemetry.cpp

@@ -0,0 +1,199 @@
+#include <stdinc.hpp>
+#include "telemetry.hpp"
+
+#include <atomic>
+#include <cstdint>
+#include <iomanip>
+#include <limits>
+#include <random>
+#include <set>
+
+namespace utils::telemetry
+{
+    namespace
+    {
+        constexpr size_t max_body_size = 0x5000;
+        constexpr size_t max_file_size = 16 * 1024 * 1024;
+        constexpr size_t max_reports = 4096;
+
+        // Stable, compact filename for arbitrary server names and endpoints.
+        // The envelope also checks both strings, so a collision fails closed.
+        std::string scope_name(const std::string& scope)
+        {
+            uint64_t hash = 14695981039346656037ull;
+            for (const unsigned char byte : scope)
+                hash = (hash ^ byte) * 1099511628211ull;
+            std::ostringstream result;
+            result << std::hex << std::setfill('0') << std::setw(16) << hash;
+            return result.str();
+        }
+
+        std::string boot_token()
+        {
+            std::random_device random;
+            std::ostringstream result;
+            result << std::hex << std::setfill('0');
+            for (int i = 0; i < 4; ++i)
+                result << std::setw(8) << static_cast<uint32_t>(random());
+            return result.str();
+        }
+
+        int positive_int(const nlohmann::json& value)
+        {
+            if (!value.is_number_integer() || value < 1 || value > std::numeric_limits<int>::max())
+                throw std::runtime_error("invalid telemetry sequence");
+            return value.get<int>();
+        }
+    }
+
+    std::string new_match_key()
+    {
+        static const auto boot = boot_token();
+        static std::atomic<uint64_t> counter{0};
+        return boot + "-" + std::to_string(++counter);
+    }
+
+    acknowledgement parse_acknowledgement(const std::string& response)
+    {
+        acknowledgement result;
+        // HTTP responses at this boundary have potentially been truncated.
+        if (response.empty() || response.size() >= max_body_size)
+            return result;
+        const auto body = nlohmann::json::parse(response, nullptr, false);
+        if (!body.is_object() || !body.contains("ok") || !body["ok"].is_boolean() || !body["ok"].get<bool>())
+            return result;
+        result.accepted = true;
+        const auto telemetry = body.find("telemetry");
+        if (telemetry != body.end() && telemetry->is_object())
+        {
+            const auto timing = telemetry->find("timing");
+            if (timing != telemetry->end() && timing->is_object())
+            {
+                const auto id = timing->find("matchId");
+                if (id != timing->end() && id->is_number_integer() && *id > 0)
+                    result.match_id = id->dump();
+            }
+        }
+        return result;
+    }
+
+    outbox::outbox(std::filesystem::path root, std::string server, std::string endpoint)
+        : server_(std::move(server)), endpoint_(std::move(endpoint))
+    {
+        file_ = root / (scope_name(server_ + '\0' + endpoint_) + ".json");
+    }
+
+    report outbox::decode(const std::string& body) const
+    {
+        if (body.empty() || body.size() >= max_body_size)
+            throw std::runtime_error("invalid telemetry report size");
+        const auto data = nlohmann::json::parse(body);
+        if (!data.is_object() || data.at("server") != server_ ||
+            !data.at("map").is_string() || !data.at("players").is_array())
+            throw std::runtime_error("invalid telemetry report envelope");
+        report value;
+        value.match = data.at("match").get<std::string>();
+        value.seq = positive_int(data.at("seq"));
+        value.event = data.at("event").get<std::string>();
+        if (value.match.empty() || value.match.size() > 64 ||
+            (value.event != "game_start" && value.event != "game_end") ||
+            (value.event == "game_start" && value.seq != 1))
+            throw std::runtime_error("invalid durable telemetry identity/event");
+        value.round = 0;
+        if (value.event == "game_end")
+        {
+            const auto& round = data.at("round");
+            if (!round.is_number_integer() || round < 0 || round > std::numeric_limits<int>::max())
+                throw std::runtime_error("invalid telemetry round");
+            value.round = round.get<int>();
+        }
+        value.body = body;
+        return value;
+    }
+
+    std::vector<report> outbox::load() const
+    {
+        if (!std::filesystem::exists(file_))
+            return {};
+        if (std::filesystem::file_size(file_) > max_file_size)
+            throw std::runtime_error("telemetry outbox exceeds size limit");
+        std::ifstream stream(file_, std::ios::binary);
+        if (!stream)
+            throw std::runtime_error("cannot open telemetry outbox");
+        const auto data = nlohmann::json::parse(stream);
+        if (stream.bad() || !data.is_object() || data.at("version") != 1 ||
+            data.at("server") != server_ || data.at("endpoint") != endpoint_ ||
+            !data.at("reports").is_array() || data.at("reports").size() > max_reports)
+            throw std::runtime_error("invalid telemetry outbox; preserve file for recovery");
+        std::vector<report> reports;
+        std::set<std::pair<std::string, int>> identities;
+        for (const auto& body : data.at("reports"))
+        {
+            auto value = decode(body.get<std::string>());
+            if (!identities.emplace(value.match, value.seq).second)
+                throw std::runtime_error("duplicate identity in telemetry outbox");
+            reports.push_back(std::move(value));
+        }
+        return reports;
+    }
+
+    void outbox::save(const std::vector<report>& reports) const
+    {
+        if (reports.size() > max_reports)
+            throw std::runtime_error("telemetry outbox is full; delivery blocked");
+        nlohmann::json data = {{"version", 1}, {"server", server_}, {"endpoint", endpoint_},
+            {"reports", nlohmann::json::array()}};
+        for (const auto& value : reports)
+            data["reports"].push_back(value.body);
+        const auto encoded = data.dump();
+        if (encoded.size() > max_file_size)
+            throw std::runtime_error("telemetry outbox is full; delivery blocked");
+        std::filesystem::create_directories(file_.parent_path());
+        auto temporary = file_;
+        temporary += ".tmp";
+        std::ofstream stream(temporary, std::ios::binary | std::ios::trunc);
+        stream.write(encoded.data(), static_cast<std::streamsize>(encoded.size()));
+        stream.flush();
+        if (!stream)
+            throw std::runtime_error("cannot write telemetry outbox; old file retained");
+        stream.close();
+        if (stream.fail())
+            throw std::runtime_error("cannot close telemetry outbox; old file retained");
+#ifdef _WIN32
+        if (!MoveFileExW(temporary.c_str(), file_.c_str(), MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH))
+            throw std::runtime_error("cannot replace telemetry outbox; old file retained");
+#else
+        // Used by the portable tests; the production plugin targets Windows.
+        std::filesystem::rename(temporary, file_);
+#endif
+    }
+
+    void outbox::store(const std::string& body) const
+    {
+        const auto incoming = decode(body);
+        auto reports = load();
+        for (const auto& value : reports)
+        {
+            if (value.match != incoming.match || value.seq != incoming.seq)
+                continue;
+            if (value.body != body)
+                throw std::runtime_error("telemetry identity reused for a different snapshot");
+            return;
+        }
+        reports.push_back(incoming);
+        save(reports);
+    }
+
+    void outbox::erase(const std::string& match, const int seq) const
+    {
+        auto reports = load();
+        const auto end = std::remove_if(reports.begin(), reports.end(), [&](const report& value)
+        {
+            return value.match == match && value.seq == seq;
+        });
+        if (end == reports.end())
+            return;
+        reports.erase(end, reports.end());
+        save(reports);
+    }
+}

t6/plugins/tws-t6-gsc-utils/src/utils/telemetry.hpp

@@ -0,0 +1,45 @@
+#pragma once
+
+#include <filesystem>
+#include <string>
+#include <vector>
+
+namespace utils::telemetry
+{
+    struct report
+    {
+        std::string match;
+        int seq;
+        std::string event;
+        int round;
+        std::string body;
+    };
+
+    struct acknowledgement
+    {
+        bool accepted = false;
+        std::string match_id;
+    };
+
+    // Native state survives script/map shutdown; a new process gets a new token.
+    std::string new_match_key();
+    acknowledgement parse_acknowledgement(const std::string& response);
+
+    // One game-thread writer per server/endpoint. Commit a replacement before
+    // discarding the previous file; a failed write leaves it available for replay.
+    class outbox
+    {
+    public:
+        outbox(std::filesystem::path root, std::string server, std::string endpoint);
+        std::vector<report> load() const;
+        void store(const std::string& body) const;
+        void erase(const std::string& match, int seq) const;
+
+    private:
+        std::filesystem::path file_;
+        std::string server_;
+        std::string endpoint_;
+        report decode(const std::string& body) const;
+        void save(const std::vector<report>& reports) const;
+    };
+}

t6/scripts/README.md

@@ -129,7 +129,7 @@ It sends four event types over the life of a match, mirroring the T5 script:
 | `round_change` | Every new positive round | new round |
 | `game_end` | `end_game` / `game_ended` / `intermission` | final round |
 
-Each event payload includes `event`, `server`, `map`, `round` (except `game_start`)
+Each event payload includes `event`, `server`, `map`, `match`, `seq`, `round` (except `game_start`)
 and a `players` array with `name`, `guid`, `kills`, `headshots`, `downs`, `revives`
 and `score` per player. `guid` reads Plutonium's `player.guid` engine property
 (T5 uses the `player getGuid()` method — T6 has no such method, but the field is
@@ -137,17 +137,21 @@ equivalent for identity purposes). `kills`/`headshots`/`downs`/`revives` read
 from T6's stock match counters in `player.pers[]`; missing or negative values
 are reported as `0`. `score` is the player's current spendable `player.score`.
 
-Delivery is asynchronous with a sequenced FIFO queue and one in-flight request.
-The queue survives the game-end signal so the final snapshot drains. There is no
-automatic retry: the receiver may already have accepted the event. Transport
-success reports the HTTP call finished, not that the API accepted it. The TWS
-plugin enforces 5-second connection and 15-second transfer deadlines, then
-releases a timed-out request so the sender can advance. Telemetry logs HTTP
-status and native timing diagnostics; non-2xx responses increment `http_rejected`
-separately from transport failures. Older plugins trigger a warning that native
-deadlines are unverified. Increase
-`tws_telemetry_log_level` to `2` for per-event capture/queue/response logging;
-set to `0` to silence non-error output.
+Delivery uses one in-flight request and a FIFO queue. Every match receives a
+native-generated key; `seq` starts at 1 for `game_start` and increases at capture.
+Start/end snapshots are saved to disk before sending and replayed before the next
+start after map/process restarts. Transport failures, native timeouts, HTTP 429
+and 5xx retain the head with 1–30 second backoff. Other 4xx are terminal. Round
+and heartbeat failures remain best effort; waiting heartbeats collapse to the
+newest snapshot. Acceptance requires 2xx plus the JSON boolean `ok: true`; logs
+include `telemetry.timing.matchId` as `ec_match_id` when available.
+
+The TWS plugin still enforces 5-second connection and 15-second transfer deadlines.
+Increase `tws_telemetry_log_level` to `2` for capture/queue/response logging; set to
+`0` for errors and recovery. **Deploy Electric Cherry #144 and the updated native
+plugin before these scripts.** See the shared
+[delivery and rollout notes](../../README.md#telemetry-delivery-and-rollout-t5-and-t6)
+for outbox storage, error handling, limits and live validation.
 
 ## `.join` for newcomers
 

t6/scripts/tws_t6_telemetry.gsc

@@ -50,15 +50,26 @@ init()
         return;
     }
 
+    // Recover before registering producers, so old boundaries precede this match.
+    recovered = telemetryLoad(level.telemetry_server, level.telemetry_url);
+    if (!isDefined(recovered))
+    {
+        telemetry_log(0, "ERROR: cannot recover durable outbox; reporting disabled until storage is repaired and the map restarted");
+        return;
+    }
+
     level.telemetry_initialized = true;
     level.telemetry_started = false;
     level.telemetry_ended = false;
     level.telemetry_last_round = -1;
-    level.telemetry_queue = [];
+    level.telemetry_queue = recovered;
     level.telemetry_queue_read = 0;
-    level.telemetry_queue_write = 0;
+    level.telemetry_queue_write = recovered.size;
+    for (i = 0; i < recovered.size; i++)
+        level.telemetry_queue[i]["captured"] = getTime();
     level.telemetry_sequence = 0;
     level.telemetry_completed = 0;
+    level.telemetry_accepted = 0;
     level.telemetry_failed = 0;
     level.telemetry_http_rejected = 0;
     level.telemetry_missing_diagnostics_warned = false;
@@ -70,7 +81,7 @@ init()
     level.telemetry_sender_state = "starting";
     level.telemetry_inflight = undefined;
 
-    telemetry_log(1, "Initialized; waiting for match start. log_level=" + telemetry_log_level());
+    telemetry_log(1, "Initialized; recovered=" + recovered.size + " waiting for match start. log_level=" + telemetry_log_level());
 
     // Route through the shared dispatcher; a duplicate connected listener
     // would fight the T6 common spawn dispatcher.
@@ -89,6 +100,12 @@ telemetry_game_start()
     if (level.telemetry_started || level.telemetry_ended)
         return;
 
+    level.telemetry_match = telemetryNewMatch();
+    if (!isDefined(level.telemetry_match))
+    {
+        telemetry_log(0, "ERROR: cannot allocate match identity");
+        return;
+    }
     level.telemetry_started = true;
     level.telemetry_last_capture = getTime();
     telemetry_log(1, "Match start detected");
@@ -153,6 +170,8 @@ telemetry_game_end_watcher()
 
     // Preserve order even if the match ends before the next polling tick.
     telemetry_game_start();
+    if (!level.telemetry_started)
+        return;
     telemetry_report_round();
     level.telemetry_ended = true;
     level notify("telemetry_finished");
@@ -240,30 +259,78 @@ telemetry_enqueue(event_name)
     level.telemetry_capture_event = event_name;
     telemetry_log(2, "Capturing event=" + event_name);
     // Snapshot and serialize before yielding, including during game over.
+    level.telemetry_sequence++;
     body = "{\"event\":" + jsonSerialize(event_name);
     body += ",\"server\":" + jsonSerialize(level.telemetry_server);
     body += ",\"map\":" + jsonSerialize(GetDvar("mapname"));
+    body += ",\"match\":" + jsonSerialize(level.telemetry_match);
+    body += ",\"seq\":" + level.telemetry_sequence;
     if (event_name != "game_start")
         body += ",\"round\":" + telemetry_nonnegative_int(level.round_number);
     body += ",\"players\":" + telemetry_players_json(event_name != "game_start") + "}";
 
     entry = [];
-    level.telemetry_sequence++;
+    entry["match"] = level.telemetry_match;
     entry["id"] = level.telemetry_sequence;
     entry["event"] = event_name;
     entry["round"] = telemetry_nonnegative_int(level.round_number);
     entry["captured"] = getTime();
     entry["body"] = body;
-    level.telemetry_queue[level.telemetry_queue_write] = entry;
-    level.telemetry_queue_write++;
+    // Persist boundaries before yielding or handing any request to the plugin.
+    // A failed save stays in memory and blocks sending until storage recovers.
+    entry["stored"] = false;
+    telemetry_queue_push(entry);
+    if (telemetry_critical(entry))
+        telemetry_persist_pending();
     level.telemetry_last_capture = getTime();
     level.telemetry_capture_event = "none";
     telemetry_log(2, "Queued " + telemetry_event_label(entry) + " pending=" + telemetry_pending());
     level notify("telemetry_queued");
 }
 
-// A single worker keeps one request in flight. It survives game-over notifies
-// to drain the final snapshot; no retries, discovery, or beta-specific calls.
+// If storage was unavailable at an earlier capture, save it first. Otherwise a
+// newly captured end could be committed ahead of its unsaved start on restart.
+telemetry_persist_pending()
+{
+    for (i = level.telemetry_queue_read; i < level.telemetry_queue_write; i++)
+    {
+        entry = level.telemetry_queue[i];
+        if (!telemetry_critical(entry) || entry["stored"])
+            continue;
+        entry["stored"] = telemetryStore(level.telemetry_server, level.telemetry_url, entry["body"]);
+        if (!entry["stored"])
+            return;
+    }
+}
+
+telemetry_critical(entry)
+{
+    return entry["event"] == "game_start" || entry["event"] == "game_end";
+}
+
+// Remove obsolete waiting heartbeats, keeping round reports and the in-flight
+// head in order. The new heartbeat keeps its capture-time sequence and snapshot.
+telemetry_queue_push(entry)
+{
+    queue = [];
+    count = 0;
+    for (i = level.telemetry_queue_read; i < level.telemetry_queue_write; i++)
+    {
+        previous = level.telemetry_queue[i];
+        inflight = i == level.telemetry_queue_read && isDefined(level.telemetry_inflight);
+        if (entry["event"] == "heartbeat" && previous["event"] == "heartbeat" && !inflight)
+            continue;
+        queue[count] = previous;
+        count++;
+    }
+    queue[count] = entry;
+    level.telemetry_queue = queue;
+    level.telemetry_queue_read = 0;
+    level.telemetry_queue_write = count + 1;
+}
+
+// One worker retains the head through retries. No later report can overtake an
+// unacknowledged boundary. Retry delays grow 1, 2, 4, 8, 16, then at most 30s.
 telemetry_sender()
 {
     for (;;)
@@ -276,7 +343,7 @@ telemetry_sender()
             if (level.telemetry_ended)
             {
                 level.telemetry_sender_state = "drained";
-                telemetry_log(1, "Queue drained after game end; completed=" + level.telemetry_completed + " failed=" + level.telemetry_failed);
+                telemetry_log(1, "Queue drained after game end; accepted=" + level.telemetry_accepted + " failed=" + level.telemetry_failed);
                 level notify("telemetry_drained");
                 return;
             }
@@ -287,15 +354,53 @@ telemetry_sender()
         }
 
         entry = level.telemetry_queue[level.telemetry_queue_read];
+        if (!telemetry_process(entry))
+        {
+            delay = telemetry_retry_delay(entry);
+            level.telemetry_sender_state = "retry_backoff";
+            telemetry_log(0, "Retry pending " + telemetry_event_label(entry) + " delay_s=" + delay);
+            wait (delay);
+            continue;
+        }
         level.telemetry_queue[level.telemetry_queue_read] = undefined;
         level.telemetry_queue_read++;
-        telemetry_send(entry);
     }
 }
 
-// v3.1.1 global alias: httpPost(url, options) with options["method"|"headers"|"body"].
-// The plugin exposes no cancellation API and no HTTP status code; treat the
-// success flag as transport completion only and never overlap requests.
+telemetry_retry_delay(entry)
+{
+    if (!isDefined(entry["retry_delay"]))
+        entry["retry_delay"] = 1;
+    else
+        entry["retry_delay"] *= 2;
+    if (entry["retry_delay"] > 30)
+        entry["retry_delay"] = 30;
+    return entry["retry_delay"];
+}
+
+telemetry_process(entry)
+{
+    critical = telemetry_critical(entry);
+    if (critical && !entry["stored"])
+    {
+        entry["stored"] = telemetryStore(level.telemetry_server, level.telemetry_url, entry["body"]);
+        if (!entry["stored"])
+            return false;
+    }
+    // If removing an acknowledged/terminal entry fails, retry the disk update
+    // without POSTing again. A restart before removal safely replays its key/seq.
+    if (!isDefined(entry["resolved"]))
+    {
+        outcome = telemetry_send(entry);
+        if (critical && outcome == "retry")
+            return false;
+        entry["resolved"] = true;
+    }
+    if (critical)
+        return telemetryRemove(level.telemetry_server, level.telemetry_url, entry["match"], entry["id"]);
+    return true;
+}
+
 telemetry_send(entry)
 {
     level.telemetry_inflight = entry;
@@ -318,10 +423,10 @@ telemetry_send(entry)
     request = httpPost(level.telemetry_url, options);
     if (!isDefined(request))
     {
-        telemetry_transport_failed(label + " request handle is undefined; skipping event");
+        telemetry_transport_failed(label + " request handle is undefined");
         level.telemetry_inflight = undefined;
         level.telemetry_sender_state = "idle";
-        return;
+        return "retry";
     }
 
     level.telemetry_sender_state = "waiting_http";
@@ -335,8 +440,8 @@ telemetry_send(entry)
         telemetry_log(0, "WARNING: plugin returned no HTTP diagnostics; native timeouts are unverified. Install the TWS plugin build and restart the server process");
     }
 
-    // The plugin's success flag reports transport completion, not HTTP status.
-    // No automatic retry: the receiver may already have accepted the event.
+    // The plugin's success flag reports transport completion, not API acceptance.
+    outcome = "retry";
     if (!isDefined(success) || !success)
     {
         if (!isDefined(error))
@@ -346,29 +451,49 @@ telemetry_send(entry)
     else
     {
         level.telemetry_completed++;
-        if (level.telemetry_failure_streak > 0)
-            telemetry_log(0, "Transport recovered after " + level.telemetry_failure_streak + " failures; " + label);
-        level.telemetry_failure_streak = 0;
-        if (isDefined(diagnostics) && isDefined(diagnostics["http_status"]))
+        outcome = telemetry_response_outcome(response, diagnostics);
+        if (outcome == "accepted")
         {
-            if (diagnostics["http_status"] < 200 || diagnostics["http_status"] >= 300)
-            {
-                level.telemetry_http_rejected++;
-                telemetry_log(0, "ERROR: HTTP rejected " + label + " elapsed_ms=" + elapsed + timing + " body=" + telemetry_preview(response));
-            }
-            else
-                telemetry_log(1, "HTTP completed " + label + " elapsed_ms=" + elapsed + timing);
+            ack = telemetryResponse(response);
+            level.telemetry_accepted++;
+            if (level.telemetry_failure_streak > 0)
+                telemetry_log(0, "API recovered after " + level.telemetry_failure_streak + " failures; " + label);
+            level.telemetry_failure_streak = 0;
+            telemetry_log(1, "API accepted " + label + " ec_match_id=" + ack["match_id"] + " elapsed_ms=" + elapsed + timing);
         }
         else
-            telemetry_log(1, "Transport completed " + label + " elapsed_ms=" + elapsed + timing + " (API acceptance unverified)");
-        if (!isDefined(response) || response == "")
-            telemetry_log(0, "WARNING: empty API response for " + label);
-        else
-            telemetry_log(2, "Response " + label + " body=" + telemetry_preview(response));
+        {
+            level.telemetry_failure_streak++;
+            level.telemetry_http_rejected++;
+            telemetry_log(0, "ERROR: API unacknowledged " + label + " outcome=" + outcome + " elapsed_ms=" + elapsed + timing + " body=" + telemetry_preview(response));
+        }
+        telemetry_log(2, "Response " + label + " body=" + telemetry_preview(response));
     }
 
+    if (outcome != "accepted" && (!telemetry_critical(entry) || outcome == "drop"))
+        telemetry_log(0, "Dropping unaccepted " + label + " outcome=" + outcome);
     level.telemetry_inflight = undefined;
     level.telemetry_sender_state = "idle";
+    return outcome;
+}
+
+telemetry_response_outcome(response, diagnostics)
+{
+    if (!isDefined(diagnostics) || !isDefined(diagnostics["http_status"]))
+        return "retry";
+    status = diagnostics["http_status"];
+    if (status == 0 || status == 429 || (status >= 500 && status < 600))
+        return "retry";
+    if (status < 200 || status >= 300)
+        return "drop";
+    if (!isDefined(response))
+        return "retry";
+    // Native nonthrowing JSON parsing requires an actual boolean true, rejects
+    // truncated/proxy bodies, and preserves large Electric Cherry IDs as strings.
+    ack = telemetryResponse(response);
+    if (ack["accepted"])
+        return "accepted";
+    return "retry";
 }
 
 // The native plugin appends diagnostics while preserving the old done arguments.
@@ -403,7 +528,7 @@ telemetry_pending()
 
 telemetry_event_label(entry)
 {
-    return "id=" + entry["id"] + " event=" + entry["event"] + " round=" + entry["round"];
+    return "match=" + entry["match"] + " seq=" + entry["id"] + " event=" + entry["event"] + " round=" + entry["round"];
 }
 
 // Read live so diagnostics can be increased without restarting the map.
@@ -455,7 +580,7 @@ telemetry_watchdog()
         telemetry_log(1, "Health started=" + level.telemetry_started + " ended=" + level.telemetry_ended +
             " sender=" + level.telemetry_sender_state + " capturing=" + level.telemetry_capture_event +
             " pending=" + pending + " queued_total=" + level.telemetry_sequence +
-            " completed=" + level.telemetry_completed + " failed=" + level.telemetry_failed + " http_rejected=" + level.telemetry_http_rejected +
+            " accepted=" + level.telemetry_accepted + " completed=" + level.telemetry_completed + " failed=" + level.telemetry_failed + " http_rejected=" + level.telemetry_http_rejected +
             " last_done_ms_ago=" + completion_age);
 
         if (isDefined(level.telemetry_inflight))
@@ -466,7 +591,7 @@ telemetry_watchdog()
                     " elapsed_ms=" + age + " pending=" + pending + " stage=" + level.telemetry_sender_state +
                     "; native deadline exceeded; check plugin version, worker queue and completion delivery");
         }
-        else if (pending > 0)
+        else if (pending > 0 && level.telemetry_sender_state != "retry_backoff")
         {
             oldest = level.telemetry_queue[level.telemetry_queue_read];
             if (now - oldest["captured"] >= 30000)