Preserve and retry telemetry match boundaries across restarts

t6/plugins/README.md

@@ -19,7 +19,7 @@ These are libcurl transfer deadlines, starting when a worker begins the request.
 Time waiting for a worker or the game thread to deliver completion is measured
 separately. A native timeout releases the request and emits `done` with
 `success = false`, a transport error, and diagnostics. Each request is sent once.
-The telemetry sender then logs the failed event and advances its ordered queue.
+The telemetry sender retries start/end reports at the head of its ordered queue.
 A timeout does not prove the API did not already accept the event.
 
 The existing completion values remain in the same positions. An optional fourth
@@ -31,8 +31,8 @@ request waittill("done", response, success, error, diagnostics);
 
 `success` continues to mean transport success, including completed HTTP 4xx/5xx
 responses. Existing scripts that read only the first three values still work.
-An updated reporter on an old plugin logs that diagnostics and native deadlines
-are unverified; deploy the matching DLL to enable the fix.
+Current reporters also require the native telemetry helpers below; deploy the
+matching DLL and restart the process before loading the updated scripts.
 
 | Field | Meaning |
 | --- | --- |
@@ -55,20 +55,45 @@ included in diagnostics. Responses retain the existing 20 KiB truncation behavio
 Telemetry logs the status and timings for each completion at log level 1.
 Non-2xx responses log at level 0 with a bounded response preview. Health counters
 keep transport `completed`/`failed` and track `http_rejected` separately; an HTTP
-rejection is a completed transport. Producers, queue ordering and payloads are
-unchanged. Heartbeats are not coalesced, failed reports are not retried, and API
-receipt timestamps remain the existing behavior.
+rejection is a completed transport. `accepted` counts only 2xx replies with the
+JSON boolean `ok: true`. Report identities, retry behavior, heartbeat coalescing
+and persistence are described in the shared
+[delivery and rollout notes](../../README.md#telemetry-delivery-and-rollout-t5-and-t6).
+
+## Durable telemetry helpers
+
+The paired `component/telemetry.cpp` and `utils/telemetry.*` provide:
+
+- `telemetryNewMatch()`: a random process token plus a process-local match counter.
+- `telemetryLoad(server, endpoint)`: ordered saved boundary entries, or undefined
+  on an unreadable/corrupt outbox. An empty outbox returns an empty GSC array.
+- `telemetryStore(server, endpoint, body)`: idempotently commit an exact boundary
+  snapshot; false on storage, validation or capacity failure.
+- `telemetryRemove(server, endpoint, match, seq)`: atomically remove a resolved
+  entry; false on failure. The old committed file remains available on failure.
+- `telemetryResponse(body)`: parse bounded JSON without throwing on malformed
+  responses, returning `accepted` (strict boolean `ok`) and `match_id` (a string).
+  The GSC reporter separately checks the HTTP status before accepting it.
+
+The native helpers avoid the generic JSON-to-GSC converter's boolean handling.
+Outbox files are scoped by game, server and endpoint under `fs_homepath/tws-telemetry`.
+They contain snapshots, never authentication headers. One writer per scope is
+required. Writes replace the old file with `MoveFileExW`; stale `.tmp` files are
+ignored during recovery. Native memory supplies identity only; pending reports
+come from disk after every script or process restart.
 
 ## Validation and rollout
 
 Run `python3 tools/test_plugin_http.py` from the repository root for portable
-native HTTP tests. Build the x86 DLL on Windows, then on a test game server check:
+native HTTP tests, and `python3 tools/test_plugin_telemetry.py` for outbox and
+acknowledgment tests (requires the pinned JSON headers). Deploy Electric Cherry
+#144 first as described in the shared rollout notes. Build the x86 DLL on Windows, then on a test game server check:
 
 1. Updated telemetry reports HTTP status and all timing fields on normal requests.
 2. An endpoint that accepts a connection but does not reply produces a native
-   timeout near 15 seconds, followed by the next queued request.
-3. HTTP 401/429/500 responses increment `http_rejected` without blocking the queue;
-   transport errors increment `failed` instead.
+   timeout near 15 seconds; a start/end is retried before the next queued report.
+3. HTTP 401 is terminal; 429/500 retain start/end at the queue head and retry.
+   Non-2xx replies increment `http_rejected`; transport errors increment `failed`.
 4. Leaderboard and older three-value HTTP consumers still receive completion.
 5. After a full server-process restart, verify all four telemetry event types in
    Electric Cherry. A successful transport alone is not API acceptance.

t6/plugins/tws-t6-gsc-utils/src/component/telemetry.cpp

@@ -0,0 +1,75 @@
+#include <stdinc.hpp>
+#include "loader/component_loader.hpp"
+#include "gsc.hpp"
+#include "game/scripting/array.hpp"
+#include <utils/telemetry.hpp>
+
+namespace telemetry
+{
+    namespace
+    {
+        utils::telemetry::outbox spool(const std::string& server, const std::string& endpoint)
+        {
+            return {"tws-telemetry/t6", server, endpoint};
+        }
+
+        void storage_error()
+        {
+            // Do not echo parse exceptions: their text can contain player data.
+            printf("^1[tws_telemetry] Outbox operation failed; check tws-telemetry/t6 files, disk space and permissions. Existing reports retained.\n");
+        }
+    }
+
+    class component final : public component_interface
+    {
+    public:
+        void on_startup([[maybe_unused]] plugin::plugin* plugin) override
+        {
+            gsc::function::add("telemetrynewmatch", []() -> scripting::script_value
+            {
+                try { return utils::telemetry::new_match_key(); }
+                catch (...) { return {}; }
+            });
+            gsc::function::add("telemetryload", [](const std::string& server, const std::string& endpoint) -> scripting::script_value
+            {
+                try
+                {
+                    scripting::array entries;
+                    for (const auto& value : spool(server, endpoint).load())
+                    {
+                        scripting::array entry;
+                        entry["match"] = value.match;
+                        entry["id"] = value.seq;
+                        entry["event"] = value.event;
+                        entry["round"] = value.round;
+                        entry["body"] = value.body;
+                        entry["stored"] = true;
+                        entries.emplace_back(entry);
+                    }
+                    return entries;
+                }
+                catch (...) { storage_error(); return {}; }
+            });
+            gsc::function::add("telemetrystore", [](const std::string& server, const std::string& endpoint, const std::string& body)
+            {
+                try { spool(server, endpoint).store(body); return true; }
+                catch (...) { storage_error(); return false; }
+            });
+            gsc::function::add("telemetryremove", [](const std::string& server, const std::string& endpoint, const std::string& match, const int seq)
+            {
+                try { spool(server, endpoint).erase(match, seq); return true; }
+                catch (...) { storage_error(); return false; }
+            });
+            gsc::function::add("telemetryresponse", [](const std::string& response)
+            {
+                const auto ack = utils::telemetry::parse_acknowledgement(response);
+                scripting::array result;
+                result["accepted"] = ack.accepted;
+                result["match_id"] = ack.match_id;
+                return result;
+            });
+        }
+    };
+}
+
+REGISTER_COMPONENT(telemetry::component)

t6/plugins/tws-t6-gsc-utils/src/utils/telemetry.cpp

@@ -0,0 +1,199 @@
+#include <stdinc.hpp>
+#include "telemetry.hpp"
+
+#include <atomic>
+#include <cstdint>
+#include <iomanip>
+#include <limits>
+#include <random>
+#include <set>
+
+namespace utils::telemetry
+{
+    namespace
+    {
+        constexpr size_t max_body_size = 0x5000;
+        constexpr size_t max_file_size = 16 * 1024 * 1024;
+        constexpr size_t max_reports = 4096;
+
+        // Stable, compact filename for arbitrary server names and endpoints.
+        // The envelope also checks both strings, so a collision fails closed.
+        std::string scope_name(const std::string& scope)
+        {
+            uint64_t hash = 14695981039346656037ull;
+            for (const unsigned char byte : scope)
+                hash = (hash ^ byte) * 1099511628211ull;
+            std::ostringstream result;
+            result << std::hex << std::setfill('0') << std::setw(16) << hash;
+            return result.str();
+        }
+
+        std::string boot_token()
+        {
+            std::random_device random;
+            std::ostringstream result;
+            result << std::hex << std::setfill('0');
+            for (int i = 0; i < 4; ++i)
+                result << std::setw(8) << static_cast<uint32_t>(random());
+            return result.str();
+        }
+
+        int positive_int(const nlohmann::json& value)
+        {
+            if (!value.is_number_integer() || value < 1 || value > std::numeric_limits<int>::max())
+                throw std::runtime_error("invalid telemetry sequence");
+            return value.get<int>();
+        }
+    }
+
+    std::string new_match_key()
+    {
+        static const auto boot = boot_token();
+        static std::atomic<uint64_t> counter{0};
+        return boot + "-" + std::to_string(++counter);
+    }
+
+    acknowledgement parse_acknowledgement(const std::string& response)
+    {
+        acknowledgement result;
+        // HTTP responses at this boundary have potentially been truncated.
+        if (response.empty() || response.size() >= max_body_size)
+            return result;
+        const auto body = nlohmann::json::parse(response, nullptr, false);
+        if (!body.is_object() || !body.contains("ok") || !body["ok"].is_boolean() || !body["ok"].get<bool>())
+            return result;
+        result.accepted = true;
+        const auto telemetry = body.find("telemetry");
+        if (telemetry != body.end() && telemetry->is_object())
+        {
+            const auto timing = telemetry->find("timing");
+            if (timing != telemetry->end() && timing->is_object())
+            {
+                const auto id = timing->find("matchId");
+                if (id != timing->end() && id->is_number_integer() && *id > 0)
+                    result.match_id = id->dump();
+            }
+        }
+        return result;
+    }
+
+    outbox::outbox(std::filesystem::path root, std::string server, std::string endpoint)
+        : server_(std::move(server)), endpoint_(std::move(endpoint))
+    {
+        file_ = root / (scope_name(server_ + '\0' + endpoint_) + ".json");
+    }
+
+    report outbox::decode(const std::string& body) const
+    {
+        if (body.empty() || body.size() >= max_body_size)
+            throw std::runtime_error("invalid telemetry report size");
+        const auto data = nlohmann::json::parse(body);
+        if (!data.is_object() || data.at("server") != server_ ||
+            !data.at("map").is_string() || !data.at("players").is_array())
+            throw std::runtime_error("invalid telemetry report envelope");
+        report value;
+        value.match = data.at("match").get<std::string>();
+        value.seq = positive_int(data.at("seq"));
+        value.event = data.at("event").get<std::string>();
+        if (value.match.empty() || value.match.size() > 64 ||
+            (value.event != "game_start" && value.event != "game_end") ||
+            (value.event == "game_start" && value.seq != 1))
+            throw std::runtime_error("invalid durable telemetry identity/event");
+        value.round = 0;
+        if (value.event == "game_end")
+        {
+            const auto& round = data.at("round");
+            if (!round.is_number_integer() || round < 0 || round > std::numeric_limits<int>::max())
+                throw std::runtime_error("invalid telemetry round");
+            value.round = round.get<int>();
+        }
+        value.body = body;
+        return value;
+    }
+
+    std::vector<report> outbox::load() const
+    {
+        if (!std::filesystem::exists(file_))
+            return {};
+        if (std::filesystem::file_size(file_) > max_file_size)
+            throw std::runtime_error("telemetry outbox exceeds size limit");
+        std::ifstream stream(file_, std::ios::binary);
+        if (!stream)
+            throw std::runtime_error("cannot open telemetry outbox");
+        const auto data = nlohmann::json::parse(stream);
+        if (stream.bad() || !data.is_object() || data.at("version") != 1 ||
+            data.at("server") != server_ || data.at("endpoint") != endpoint_ ||
+            !data.at("reports").is_array() || data.at("reports").size() > max_reports)
+            throw std::runtime_error("invalid telemetry outbox; preserve file for recovery");
+        std::vector<report> reports;
+        std::set<std::pair<std::string, int>> identities;
+        for (const auto& body : data.at("reports"))
+        {
+            auto value = decode(body.get<std::string>());
+            if (!identities.emplace(value.match, value.seq).second)
+                throw std::runtime_error("duplicate identity in telemetry outbox");
+            reports.push_back(std::move(value));
+        }
+        return reports;
+    }
+
+    void outbox::save(const std::vector<report>& reports) const
+    {
+        if (reports.size() > max_reports)
+            throw std::runtime_error("telemetry outbox is full; delivery blocked");
+        nlohmann::json data = {{"version", 1}, {"server", server_}, {"endpoint", endpoint_},
+            {"reports", nlohmann::json::array()}};
+        for (const auto& value : reports)
+            data["reports"].push_back(value.body);
+        const auto encoded = data.dump();
+        if (encoded.size() > max_file_size)
+            throw std::runtime_error("telemetry outbox is full; delivery blocked");
+        std::filesystem::create_directories(file_.parent_path());
+        auto temporary = file_;
+        temporary += ".tmp";
+        std::ofstream stream(temporary, std::ios::binary | std::ios::trunc);
+        stream.write(encoded.data(), static_cast<std::streamsize>(encoded.size()));
+        stream.flush();
+        if (!stream)
+            throw std::runtime_error("cannot write telemetry outbox; old file retained");
+        stream.close();
+        if (stream.fail())
+            throw std::runtime_error("cannot close telemetry outbox; old file retained");
+#ifdef _WIN32
+        if (!MoveFileExW(temporary.c_str(), file_.c_str(), MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH))
+            throw std::runtime_error("cannot replace telemetry outbox; old file retained");
+#else
+        // Used by the portable tests; the production plugin targets Windows.
+        std::filesystem::rename(temporary, file_);
+#endif
+    }
+
+    void outbox::store(const std::string& body) const
+    {
+        const auto incoming = decode(body);
+        auto reports = load();
+        for (const auto& value : reports)
+        {
+            if (value.match != incoming.match || value.seq != incoming.seq)
+                continue;
+            if (value.body != body)
+                throw std::runtime_error("telemetry identity reused for a different snapshot");
+            return;
+        }
+        reports.push_back(incoming);
+        save(reports);
+    }
+
+    void outbox::erase(const std::string& match, const int seq) const
+    {
+        auto reports = load();
+        const auto end = std::remove_if(reports.begin(), reports.end(), [&](const report& value)
+        {
+            return value.match == match && value.seq == seq;
+        });
+        if (end == reports.end())
+            return;
+        reports.erase(end, reports.end());
+        save(reports);
+    }
+}

t6/plugins/tws-t6-gsc-utils/src/utils/telemetry.hpp

@@ -0,0 +1,45 @@
+#pragma once
+
+#include <filesystem>
+#include <string>
+#include <vector>
+
+namespace utils::telemetry
+{
+    struct report
+    {
+        std::string match;
+        int seq;
+        std::string event;
+        int round;
+        std::string body;
+    };
+
+    struct acknowledgement
+    {
+        bool accepted = false;
+        std::string match_id;
+    };
+
+    // Native state survives script/map shutdown; a new process gets a new token.
+    std::string new_match_key();
+    acknowledgement parse_acknowledgement(const std::string& response);
+
+    // One game-thread writer per server/endpoint. Commit a replacement before
+    // discarding the previous file; a failed write leaves it available for replay.
+    class outbox
+    {
+    public:
+        outbox(std::filesystem::path root, std::string server, std::string endpoint);
+        std::vector<report> load() const;
+        void store(const std::string& body) const;
+        void erase(const std::string& match, int seq) const;
+
+    private:
+        std::filesystem::path file_;
+        std::string server_;
+        std::string endpoint_;
+        report decode(const std::string& body) const;
+        void save(const std::vector<report>& reports) const;
+    };
+}