# Beta test plans New betas use a saved plan rather than the two legacy wait flags. Existing betas and API callers without `testPlan` retain their original workflow. Schema additions are additive and run at startup; no active beta needs to be closed to install this feature. ## Requirements Use **Add requirement** in Betas. Requirements on a qualifying server are ANDed: | Requirement | Evidence | | --- | --- | | Reach round X | Session telemetry reports round >= X with the required player count | | Complete qualifying matches | Normal completed matches, optionally qualified by round, players and duration | | Populated minutes | Fresh RCON samples at both ends of a populated interval | | Player-hours | Populated minutes × the lower sampled player count / 60 | | Reach player count | Fresh healthy RCON sample | | Sustained population | Consecutive fresh healthy samples above the player threshold | | Continuous healthy minutes | Consecutive fresh container/RCON observations | | Maximum health outages | Transitions from observed healthy to unhealthy | | Maximum unexpected restarts | Distinct reported process boots within the same test run | | Maximum script errors | New console lines matching script errors/exceptions, unhandled exceptions or fatal errors | | Successful recovery cycles | Repeat the other requirements after each additional empty-server restart | | Feature event count | Named, session-scoped game events | | Tester checklist | Tester name and observation, entered on the beta detail page | There is one round requirement: **reach round X**. For repeated high-round runs, use a qualifying-match count with a minimum round. The minimum player count must be present at the qualifying round observation in the same match. A match with many players and a different solo high-round match cannot combine to pass. Unqualified match counts use incremental game logs (`InitGame`, a player join, then `ExitLevel: executed`). Round-qualified or multi-player-qualified matches use session telemetry. Each requirement uses one source, preventing duplicate counts. Match duration is measured from the observed join/session start. Log polling can undercount duration; it does not infer unobserved elapsed time. Round, feature and unexpected-restart checks require the reporting protocol below. Legacy round reports still update Fleet but cannot prove beta progress. The game scripts themselves are uploaded artifacts, not source maintained in this repository; update their reporter to emit the protocol before selecting these requirements in production. ## Coverage, rollout and outcomes Coverage can require all selected servers, a quorum, one per selected map, one per selected game, or one per selected game/map combination. Coverage metadata is pinned when the beta opens. Register game/map metadata first. Set wave size to 1 for a canary, a larger number for batches, or 0 for all selected servers. Later waves reserve their artifact targets but receive no files until every server in the prior waves has passed. All waves must start testing before the final coverage rule can pass. Error-budget violations on any tested server block promotion, including with quorum coverage. The test window starts when the beta opens, including time waiting for empty servers. Missing evidence at the deadline produces **inconclusive**, not a failure or an automatic rollback. Health and log monitoring continue after a pass until promotion; a deteriorating result revokes readiness. For failures, select hold as failed, pause for investigation, or automatic rollback. Rollback restores only files touched by the beta. It can queue rolling restarts when servers empty, or leave restarts manual. Manual rollback uses the same selected restart policy. No beta auto-promotes. Pause holds evidence evaluation and rollout. Resume retains evidence and the original deadline. A fresh attempt retires old evidence and restarts deployed servers when empty; it does not extend the deadline. Expired or incomplete-file deployments should be rolled back and reopened. Recovery cycles similarly require fresh milestones/checklists, while carrying error budgets forward. Time accounting excludes intervals spanning monitoring gaps longer than 60 seconds. A missing first sample earns no time. A log that cannot be read cannot pass a clean-log requirement. Log backlog above 2 MB makes the plan inconclusive instead of silently dropping evidence. Partial lines, rotations, byte cursors, attempts and accepted events persist in SQLite. The detail page shows the most recent 200 evidence entries; older entries remain in the database. ## Game reporter protocol This section is a standalone integration guide for the T6 telemetry script. It describes the **currently implemented API**, including its limitations. Examples use `t6_zombie_town_1`; replace that with the exact, case-sensitive Electric Cherry server registry name. The API base URL is the Electric Cherry application, not Komodo or the game server's RCON address. ### 1. Two reporting modes: Fleet display and beta evidence The reporter must work whether or not anybody is beta testing: | Situation | Reporter behavior | | --- | --- | | No eligible beta run for this server | Keep sending ordinary `round_change` reports for Fleet. Do not send beta-only events. | | One eligible run discovered at process startup | Send the beta event stream with that run's `runId`. Scoped `round_change` reports also update Fleet. | | Several eligible runs discovered at startup | Send a copy of each beta event to each run, with separate acknowledgement/queue state per run. | | Beta opened while an existing game process is running | Continue its existing reporting mode. The beta waits for the server to empty and restarts it; the new process discovers the new run. | | Beta has passed and is awaiting promotion | Continue reporting. A passed test still accepts evidence until promotion or another outcome stops it. | | Beta paused, failed, expired, promoted, rolled back, or run superseded | Stop crediting events to that inactive run; continue ordinary round reporting. See the suspension rules below. | **No active beta is normal, not an error.** Fleet telemetry does not depend on a beta, a run token, or a successful beta discovery request. ### 2. Startup discovery and an empty result On **actual game-process startup**, request: ```http GET /api/telemetry?server=t6_zombie_town_1 ``` For a registered server without an eligible run, this returns **HTTP 200**: ```json { "runs": [] } ``` This can mean there is no beta, the server is not a beta target, its wave has not started, the beta is waiting to restart it, or the test is not accepting evidence. Legacy betas without a test plan are not returned either. The response is a list of eligible **runs**, not a list of every beta in the application. With an eligible run, the response is **HTTP 200**: ```json { "runs": [ { "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", "betaId": 42 } ] } ``` The response has `Cache-Control: no-store`. Do not assume it contains exactly one entry, and do not choose only the first entry. `betaId` is informational; POST requests use `runId`, not `betaId`, to identify the test attempt. A missing or unknown `server` query parameter returns **404**, not an empty success. Treat that as a configuration issue. The GET handler currently only checks registry existence; POST additionally rejects a server marked destroyed. Do not interpret successful discovery as permission to report for another server. Cache the startup run list for this process lifetime. Electric Cherry creates a new run before requesting each beta-controlled restart. That restart is the boundary that makes gameplay belong to the newly deployed build. **Do not poll for and adopt new run IDs in an old process, at a map reload, or mid-match.** Never move queued events to a newer run ID. If discovery fails, use bounded retries with backoff while still initializing, before gameplay starts. If startup discovery cannot finish, run in Fleet-only mode for that process and log the problem. Do not attach later gameplay to a run discovered after that startup boundary. A new process/retry is needed for a new safe beta attachment; the current API has no late-attachment handshake. ### 3. Reporting without a beta Send this on each round change, even when `runs` is empty: ```http POST /api/telemetry Content-Type: application/json ``` ```json { "event": "round_change", "server": "t6_zombie_town_1", "round": 17, "players": 4 } ``` Example success response: ```json { "ok": true, "telemetry": { "server": "t6_zombie_town_1", "round": 17, "players": 4, "receivedAt": "2026-09-18T12:00:00.000Z" } } ``` Omit `runId` entirely in this mode. `"runId": null` and `"runId": ""` are invalid; they are not equivalent to omitting the field. The response has no `test` property when `runId` was omitted. Without `runId`, only `round_change` is supported. Do not send unscoped `boot`, `session_start`, `feature`, or `match_end` requests: they return 400. Do not buffer feature occurrences from normal operation to submit when a future beta opens. Electric Cherry stores only the latest Fleet round report per server and stamps its receipt time. Lower rounds are valid when a new match begins. These reports do not change RCON player counts, restart safety decisions, or beta progress. ### 4. Identifier lifetimes and the T6 process boundary | Field | Who creates it | Lifetime and rules | | --- | --- | --- | | `server` | Configuration | Exact registered name, unchanged across matches and process restarts. | | `runId` | Electric Cherry | One server attempt/recovery phase. Use the value discovered at process startup. Treat it as opaque. | | `bootId` | Reporter | Unique for each actual game-process lifetime. Keep it across matches, map reloads, and reporter reinitialization within that process. | | `sessionId` | Reporter | Unique for each actual played match. Generate a new value for the next match, even if the players and map are unchanged. | | `eventId` | Reporter | Unique for each logical event within a run, including across sessions/boots. A delivery retry reuses the exact same ID. | | `sequence` | Reporter | Start at 0 for `session_start`; increment for every subsequent event in that session, across all event types. Reset only for a new session. | Generate collision-resistant IDs, such as UUIDs. A boot ID plus a session ID and counter is also suitable for event IDs if the combined string fits the length limit. Example IDs below are illustrative; never hardcode them in the script. The same event ID can be used for copies sent to different runs because deduplication is scoped to `(runId, eventId)`. **A GSC script's initialization is not necessarily a game-process startup.** The T6 implementation must establish where its state survives match/map changes. If script state resets on each map load, preserve process identity, the startup run list, and delivery state in a process-lifetime facility or a helper that can reliably identify the game process. A helper's own restart must not invent a new game boot. Do not assume map-local globals provide process-lifetime storage. If that distinction is unavailable, leave beta reporting disabled rather than reporting a fabricated boot; ordinary Fleet round reports still work. An unexpected game-process restart may discover the **same** run ID. Send a new boot ID: the backend counts additional distinct boots in a run as unexpected restarts. A planned recovery restart receives a **new** run ID and starts a new evidence phase. Restarting only the Electric Cherry web app changes neither the game boot nor the reporter's cached IDs. ### 5. Common request fields All requests use JSON objects and `Content-Type: application/json`. Every beta-scoped event includes these fields unless the table says otherwise: | Field | Type | Constraints | | --- | --- | --- | | `event` | string | Exactly `boot`, `session_start`, `round_change`, `feature`, or `match_end`. | | `server` | string | Registered server name, 1-255 characters, case-sensitive. | | `runId` | string | Required for beta evidence; 1-128 characters from `A-Z a-z 0-9 _ . : -`. | | `eventId` | string | Same identifier character/length limits. | | `bootId` | string | Same identifier character/length limits. | | `sessionId` | string | Same identifier limits; required except for `boot`. Omit for `boot`. | | `sequence` | number | Nonnegative safe integer. Use 0 for `boot` and `session_start`. | | `round` | number | Nonnegative safe integer; the actual player-facing round. Use 0 for `boot`. | | `players` | number | Nonnegative safe integer; actual participating player count at the event. Use 0 for `boot`. | | `name` | string | Required only for `feature`; 1-120 characters from the identifier character set. | Numbers must be JSON numbers, not strings, fractions, `null`, or negative values. The maximum safe integer is 9007199254740991. Sequence numbers need to increase; the backend does not require them to be contiguous, but the sender should emit a single ordered stream. Do not use a separate counter for each event type. The examples below show excerpts from one process and match. Earlier round reports are omitted, hence the sequence jumps to 25. The real script should send each round change and increment its counter for every event it produces. ### 6. Full event examples #### Process boot Send once per discovered run at real game-process startup, before any session events. A retry is the same request, not another boot with another ID. ```json { "event": "boot", "server": "t6_zombie_town_1", "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", "eventId": "boot-a81f:boot", "bootId": "boot-a81f", "sequence": 0, "round": 0, "players": 0 } ``` #### Start of a played match Send when actual gameplay begins with players, not merely when an empty map loads. Retain this `sessionId` until that match ends or is abandoned. ```json { "event": "session_start", "server": "t6_zombie_town_1", "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", "eventId": "boot-a81f:match-b92c:0", "bootId": "boot-a81f", "sessionId": "match-b92c", "sequence": 0, "round": 1, "players": 4 } ``` After this is acknowledged, report the initial round with `round_change` too. The `round` on `session_start` does not itself satisfy a round requirement. #### Reaching a round Send on entry to each round, with the player count sampled at that event. There is no separate "round completed" event. ```json { "event": "round_change", "server": "t6_zombie_town_1", "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", "eventId": "boot-a81f:match-b92c:25", "bootId": "boot-a81f", "sessionId": "match-b92c", "sequence": 25, "round": 25, "players": 4 } ``` This can satisfy a round-25 check requiring four players. It also records the qualifying round for a later completed-match check. A `match_end` reporting round 25 does not replace a missing `round_change` event for that round. #### Feature: a successful perk purchase The top-level event is always `"feature"`. The custom feature identifier goes in `name`; it is not a new top-level event type. ```json { "event": "feature", "server": "t6_zombie_town_1", "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", "eventId": "boot-a81f:match-b92c:26", "bootId": "boot-a81f", "sessionId": "match-b92c", "sequence": 26, "round": 25, "players": 4, "name": "perk.purchased" } ``` Emit when the purchase actually succeeds, not when a player opens a menu or attempts an unaffordable purchase. One accepted event adds **one** to each matching feature requirement whose minimum round/player qualifiers are met. This event does not require waiting for the match to end. For example, configure **Feature event count** with event name `perk.purchased`, target 10, minimum round 20, and minimum players 4. The example above contributes one occurrence. A purchase at round 19 or with only three players can still be accepted into the event stream, but contributes zero to that requirement. #### Feature: successful Pack-a-Punch completion ```json { "event": "feature", "server": "t6_zombie_town_1", "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", "eventId": "boot-a81f:match-b92c:27", "bootId": "boot-a81f", "sessionId": "match-b92c", "sequence": 27, "round": 25, "players": 4, "name": "weapon.pack_a_punch.completed" } ``` #### Feature: successful execution of a custom mechanic ```json { "event": "feature", "server": "t6_zombie_town_1", "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", "eventId": "boot-a81f:match-b92c:28", "bootId": "boot-a81f", "sessionId": "match-b92c", "sequence": 28, "round": 25, "players": 4, "name": "mechanic.teleporter.completed" } ``` These names are **suggested conventions**, not preinstalled T6 hooks. The script must instrument the corresponding successful actions. Only emit a mechanic's event on maps/builds where that mechanic actually exists. Names are matched exactly and case-sensitively against the UI's **Reported event name**. A requirement named `perk.purchased` with target 10 counts ten accepted occurrences of that exact name; `perk.*` is not a wildcard. For a specific perk, use a distinct name such as `perk.juggernog.purchased`. The current API does not filter on a separate `perk`, `weapon`, or `playerId` field. You may deliberately emit both a generic and a specific marker for one action, but they must be separate events with separate event IDs and sequences; each requirement still counts only its exact matching name. There is no bulk-count field. `"count": 10`, `value`, `metadata`, `timestamp`, `duration`, and other extra properties are not consumed or preserved as beta signal fields. Send one event for each real occurrence. Keep any additional diagnostics in script logs until the API explicitly supports them. #### Normal match completion ```json { "event": "match_end", "server": "t6_zombie_town_1", "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", "eventId": "boot-a81f:match-b92c:29", "bootId": "boot-a81f", "sessionId": "match-b92c", "sequence": 29, "round": 25, "players": 4 } ``` Send once when a played match ends normally, including a normal game-over. Capture the actual terminal player count; the example's four players are not a value to fabricate if players have already left. The backend retains earlier round/player observations for qualification. Do **not** send `match_end` for a crash, forced process shutdown, arbitrary map rotation, or an idle map cycle. A generic script teardown callback alone is not proof of normal match completion. There is no `session_abort` event at present; abandon that session and use a new ID when actual gameplay next begins. Never replay events for an abandoned session into the new one. Accepted `match_end` closes the session. Later events for that session are not accepted. Round-qualified or multi-player-qualified match requirements use this stream; unqualified completed-match requirements use game logs instead, so the same match is not counted twice for one requirement. ### 7. Responses: transport success is not evidence acceptance An accepted `boot`, `session_start`, `feature`, or `match_end` returns HTTP 200: ```json { "ok": true, "test": { "accepted": true } } ``` An accepted scoped `round_change` additionally updates the Fleet display: ```json { "ok": true, "telemetry": { "server": "t6_zombie_town_1", "round": 25, "players": 4, "receivedAt": "2026-09-18T12:25:00.000Z" }, "test": { "accepted": true } } ``` Acceptance means the event was recorded in the run's stream. It does not mean the whole test passed, or even that a feature/round event met a configured requirement's qualifiers. A retry of an already accepted event returns HTTP 200: ```json { "ok": true, "test": { "accepted": false, "reason": "Duplicate event" } } ``` Treat that exact duplicate response as delivery already acknowledged. Do not generate a new ID to force the same action to count twice. An inactive run returns HTTP 200: ```json { "ok": true, "test": { "accepted": false, "reason": "Run is inactive or superseded; obtain the current run at process startup" } } ``` Invalid ordering or session identity also returns HTTP 200: ```json { "ok": true, "test": { "accepted": false, "reason": "Duplicate, out-of-order, closed, or unknown session/boot" } } ``` That last response is **not** an acknowledgement: inspect the boot/session/ sequence handling. The API currently provides explanatory strings, not distinct machine-readable error codes for every rejection. Do not treat every `accepted: false` as a harmless duplicate or assume every inactive run is closed permanently; paused runs use the same inactive response. **Fleet updates are independent of beta acceptance.** A syntactically valid scoped `round_change` still writes the latest Fleet round even when `test.accepted` is false, including a duplicate or inactive-run rejection. Its response contains both `telemetry` and `test`. Do not replay an old round backlog after newer reports: it can move the Fleet display backwards. There is no client-timestamp freshness check. | HTTP result | Meaning and handling | | --- | --- | | 200, no `test` | Ordinary Fleet report accepted. | | 200, `test.accepted: true` | Beta event accepted; advance that run's queue. | | 200, exact `Duplicate event` reason | Already accepted earlier; advance that run's queue. | | 200, inactive-run reason | Suspend beta delivery for that run; continue Fleet reporting. | | 200, session/order rejection | Log an integration error and stop crediting that broken session. Do not invent a replacement session for an in-progress match. | | 400 | Invalid JSON, fields, or event type; fix the payload instead of retrying unchanged forever. | | 404 on POST | Server is unregistered or destroyed; check configuration/lifecycle. | | 415 | Missing or incorrect JSON content type. | | Network error, timeout, or 5xx | Delivery is uncertain; retry the same request with backoff within a bounded queue. | ### 8. Ordering, retries, pauses, and multiple runs **Recommended sender behavior** (client implementation guidance, not extra API features): 1. Keep delivery asynchronous so a web-service outage never blocks the game. 2. Send `boot` first for each run and wait for acceptance or an acknowledged duplicate before delivering its `session_start`. 3. Serialize the remaining events per run. Do not allow a later event to overtake an uncertain request. Use the same session counter for rounds, features, and match completion. 4. On timeout, retry with exactly the same event ID, sequence, and payload. The first request may have committed even if its response was lost. 5. Bound queue size, retry count, and event age. For example, back off through 1, 2, 5, and 10 seconds rather than retrying every frame. These intervals are a suggested client policy, not server-enforced values. 6. If a long outage makes the queue stale, abandon beta evidence for that session and keep the current Fleet round visible. Do not replay historical feature or completion events hours later. Resume beta evidence at a later real match only if the same startup-pinned run is still eligible and boot delivery is established. 7. Keep acknowledgement state separately for every discovered run. Failure on one run does not change another run's IDs or acknowledged sequence. Reuse the logical event ID across copies, changing only `runId`. The server stamps **receipt time**, not game event time. In particular, match duration is measured between accepted `session_start` and `match_end` receipts. Delayed replay can distort duration and no client-provided timestamp fixes it. Keep beta events near real time; do not use them as a historical-import API. While scoped round delivery is healthy, no extra unscoped copy is necessary. If beta delivery is suspended or lagging, ordinary round reports can keep Fleet current. Retire the affected session queue rather than later sending its old rounds behind newer unscoped reports. A simple implementation may serialize delivery across all run copies to avoid cross-run round reordering. For paused/inactive runs, the simplest safe policy is Fleet-only reporting until the next actual process startup. If automatic resume within the same process is desired, the reporter may recheck **only IDs already pinned at startup**: - Intersect discovery results with the original startup list. Never adopt an unfamiliar run ID returned by a later GET. - When a cached run becomes eligible again, resume at the **next real match**, using the existing boot ID and a new session ID. Establish boot delivery first; reusing the original boot request/event ID is safe if its acknowledgement was lost. - Discard events generated during suspension. Do not manufacture a new session start in the middle of an existing match or replay paused gameplay. - If startup returned `runs: []`, there are no pinned IDs to resume. Stay Fleet-only until a real process restart. Similarly, a process started while a test was paused may require an operator-requested fresh attempt after resume. There is no polling interval or resume notification supplied by this API. The reporter owns its retry/suspension policy. The rule about startup pinning must remain true regardless of that policy. ### 9. T6 implementation handoff checklist - Configure the Electric Cherry URL and exact server registry name. - Preserve ordinary round reports when no beta is active or the service is temporarily unreachable; never block gameplay waiting for HTTP. - Identify actual game-process boots separately from GSC initialization and map/match resets. Cache discovery results for the game process lifetime. - Instrument real session start, round entry, normal game-over, and successful feature actions. Confirm each chosen T6 callback's semantics; do not infer normal completion from generic teardown. - Use one sequence counter per session, unique event IDs, stable retry payloads, and an ordered bounded queue with separate acknowledgements per run. - Decide and document the feature names the script emits so beta creators can enter those exact names. Extra JSON metadata cannot be used as a qualifier. - Keep beta-only events scoped to startup-pinned runs. Do not backfill pre-beta, paused, abandoned, or previous-process gameplay into a new run. - Handle `ok` and `test.accepted` separately. Log rejections with enough context to diagnose the run, boot, session, event ID, and sequence. Suggested verification cases for the telemetry-script session: | Case | Expected result | | --- | --- | | Startup returns an empty run list | Round reports continue; no beta-only POSTs. | | Beta opens during an ordinary match | No new token adopted; beta reporting begins after the beta-controlled process restart. | | Normal new match, same process | Same boot/run IDs; new session ID and sequence starting at 0. | | Actual unexpected process restart | New boot ID; rediscover runs; no old queued session events replayed. | | Retry after a lost response | Same event ID and payload; one evidence entry and a duplicate acknowledgement. | | Round 25 with four players, then normal match end | Round evidence can pass immediately; a qualifying-match count advances only at normal completion. | | Ten successful `perk.purchased` events | Ten uniquely identified feature events, counted only when each event meets the configured round/player qualifiers. | | Crash or idle map reset | No fabricated `match_end`. | | Paused, closed, or superseded run | Beta delivery suspended; Fleet remains current; never switch old queued events to a new token. | | Two startup run IDs | Each receives the event once, with independent acknowledgements. | | Reporter script reload, same game process | No new boot identity or fresh adoption of run IDs. | The endpoint retains the application's existing trusted-network model. A `runId` correlates evidence; it is **not** an authentication secret or proof of loaded artifact bytes. Correct process/session attribution depends on the reporter following the lifecycle rules above. ## API and storage `POST /api/betas` accepts `testPlan`, validated by `parseBetaPlan` in `src/lib/beta-plan.ts`. New plans always verify restart recovery before testing. `GET /api/betas/:id` includes the plan, outcome, per-server requirement progress and recent evidence under `test`. `POST /api/betas/:id/test` accepts actions `pause`, `resume`, `retry`, or `sign-off` (with `server`, `requirementId`, and `note`). `GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are copied into a beta; later preset edits never change an active test. `beta_test_plans`, `beta_test_coverage`, `beta_test_runs`, `beta_test_evidence`, and `beta_test_presets` hold configuration and results. `beta_file_changes` records which files were touched, including partially completed deployments. Legacy beta lifecycle states remain compatible with artifact holds and history; test outcomes are stored separately. Holds last until promotion or rollback.