electric-cherry/docs/guides/beta-test-plans.md

# Beta test plans

Create a plan in Betas, choose how many instances per game must pass, and deploy. The server's game script
reports gameplay to Electric Cherry. The same reports update Fleet and any active
beta on that server.

## Files and automatic server selection

A beta can bundle files for different games, including T5 and T6 together. Each
file installs only on participants matching its game or map scope; an unscoped
file applies to all participants. Every file needs at least one matching test
participant. For example, a T5 script and a T6 script can share one beta without
either script being installed on the other game's servers.

Setup defaults to automatic selection. Enter one **Instances per game** number,
which defaults to 1 and applies to every game represented by the selected files
and their matching servers. Every automatically selected instance must pass
every requirement in the plan.

When you choose **Start beta**, Electric Cherry chooses compatible servers with
the fewest current players, while ensuring that every file is tested. Counts
are exact: with T5 and T6 files selected, entering 2 selects two T5 and two T6
participants, and all four must pass. Server choices are made at launch, so
review shows the count for each game rather than a provisional list of server
names.

Automatic selection requires fresh player observations and available running
servers. Servers held by another active beta or with pending deployment work
are excluded, as are instances on unavailable, draining, or disabled hosts.
Unknown or stale player counts are not treated as empty. If the
requested counts cannot be met, including coverage of every selected map-scoped
file, the beta does not start. Deployment and restarts still wait for confirmed
empty lobbies and restart capacity.

Open **Advanced options → Choose specific servers** for manual selection. The
picker includes servers matching any file in the bundle; each file must have a
matching selected server. Manual selection retains the configurable coverage
rules below. Drafts preserve the selection mode and shared instance count, and
older drafts with explicit server choices resume in manual mode.

## Game reporter protocol

Set `EC_TELEMETRY_TOKEN` on the Electric Cherry controller to a dedicated random
secret of at least 32 characters (for example, generate 64 hexadecimal characters
with `openssl rand -hex 32`). Configure the same secret in each game reporter or
its trusted local relay. Use a production HTTPS URL or a reverse proxy that
terminates TLS; browser login cookies do not authorize game telemetry.

Send JSON to `POST /api/telemetry` with `Content-Type: application/json` and either
`Authorization: Bearer <EC_TELEMETRY_TOKEN>` or
`X-EC-Telemetry-Token: <EC_TELEMETRY_TOKEN>`. If both headers are present, they must
contain the same secret. Keep the secret out of URLs, payloads and browser code,
and use a different secret from host-agent credentials or account passwords.
Request bodies are limited to 1 MiB (1,048,576 bytes), including requests without
`Content-Length`. The limit applies to the UTF-8 JSON body before parsing.
There are four events. Send them whether or not a beta is active.

### Event fields

| Event          | Fields                                                                                                                       |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `game_start`   | `server`, `map`, `players` with each player's `name` and `guid`                                                              |
| `round_change` | `server`, `map`, `round`, `players` with each player's `name`, `guid`, `kills`, `headshots`, `downs`, `revives`, and `score` |
| `heartbeat`    | Same fields as `round_change`; sent periodically during a round to refresh the current round and player stats                |
| `game_end`     | `server`, `map`, `round`, `players` with each player's `name`, `guid`, `kills`, `headshots`, `downs`, `revives`, and `score` |

`server` is the exact registered Electric Cherry server name. `map` is the
current map. `players` is the full player array, including an empty array when
there are no players. `server`, `map`, and each player's `name` and `guid` must be
nonempty strings of at most 255 UTF-16 code units (JavaScript string length).
Values over the limit reject the entire report; accepted values are preserved
without trimming or truncation. Keep GUIDs as strings to preserve their precision
and leading zeros. Round and stat values are nonnegative JSON integers.
Each GUID must appear only once within a report's player array.

Every event also accepts these optional top-level fields:

| Field   | Meaning                                                                                             |
| ------- | --------------------------------------------------------------------------------------------------- |
| `match` | Reporter-generated game identity: a nonblank string of at most 64 UTF-16 code units.                |
| `seq`   | Positive JSON safe integer, at most 9,007,199,254,740,991, identifying this report within the game. |

Send both fields for reliable match boundaries, ordering, and retry handling.
Choose a fresh `match` value for every game, including restarts on the same map,
and keep it unchanged throughout that game. Match keys are scoped to the exact
`server` name. Start `seq` at 1 and increment it once per event, including
heartbeats. Retry a report with its original `match`, `seq`, and payload; do not
allocate a new sequence number for a retry. Either field may be omitted for
compatibility, but sequence-based deduplication requires both fields. Existing
reporters without these fields remain valid.

Every event also accepts an optional per-player `ping`: the game server's current
latency measurement in milliseconds, as a nonnegative JSON integer (for example,
`"ping": 42`). Send `null` or omit the field when unavailable; do not substitute
zero for a missing measurement. Older reporters without this field remain valid.
Fleet prefers telemetry ping and fills missing readings from fresh RCON status,
matching players only by GUID. For T6, RCON's
hexadecimal GUIDs are compared with telemetry's decimal GUIDs using exact integer
conversion; stored GUIDs remain unchanged. T5 GUIDs are compared as reported.
Occupied lobbies continue RCON polling every 30 seconds.
RCON supplements only ping; telemetry still supplies roster membership and scores.
RCON pings expire after 90 seconds and must match the current map and server
lifecycle. Missing or ambiguous matches display `—`. Telemetry ping is retained
and streamed with the latest report; it does not contribute to leaderboard statistics.

### Game start

Send when the game starts. Round and stats are not required.

```json
{
    "event": "game_start",
    "server": "t6_zombie_town_1",
    "match": "town-20261012-001",
    "seq": 1,
    "map": "zm_transit",
    "players": [
        { "name": "Josh", "guid": "12345678901234567" },
        { "name": "Alex", "guid": "23456789012345678" }
    ]
}
```

### Round change

Send the new round and each player's current stats.

```json
{
    "event": "round_change",
    "server": "t6_zombie_town_1",
    "match": "town-20261012-001",
    "seq": 2,
    "map": "zm_transit",
    "round": 25,
    "players": [
        {
            "name": "Josh",
            "guid": "12345678901234567",
            "kills": 420,
            "headshots": 160,
            "downs": 1,
            "revives": 3,
            "score": 32000
        },
        {
            "name": "Alex",
            "guid": "23456789012345678",
            "kills": 380,
            "headshots": 140,
            "downs": 3,
            "revives": 1,
            "score": 28000
        }
    ]
}
```

### Game end

Send the final round and each player's final stats when the game ends.

```json
{
    "event": "game_end",
    "server": "t6_zombie_town_1",
    "match": "town-20261012-001",
    "seq": 3,
    "map": "zm_transit",
    "round": 26,
    "players": [
        {
            "name": "Josh",
            "guid": "12345678901234567",
            "kills": 445,
            "headshots": 170,
            "downs": 2,
            "revives": 3,
            "score": 33500
        },
        {
            "name": "Alex",
            "guid": "23456789012345678",
            "kills": 401,
            "headshots": 146,
            "downs": 4,
            "revives": 1,
            "score": 29750
        }
    ]
}
```

### Responses and delivery

Success returns HTTP 200 with `{ "ok": true, "telemetry": { ... } }`.
The returned telemetry contains the event and an Electric Cherry `receivedAt`
timestamp. A missing/invalid machine token returns 401; an unset or invalid
controller `EC_TELEMETRY_TOKEN` returns 503. Authentication happens before payload
validation. Authenticated bodies over 1 MiB return 413 before JSON parsing.
Invalid JSON or fields return 400, an unknown/destroyed server 404, and a non-JSON
content type 415.

Send events in gameplay order directly to Electric Cherry. There is no discovery
request, beta token, custom feature event, external helper, or separate beta
acknowledgement. Basic payload checks happen in the receiver.

With `match` and `seq`, repeated sequences are ignored even after an application
restart. A lower sequence that arrives late can fill a missing-report gap without
rolling back the current round, roster, or counters. A delayed `game_start` can
record the starting roster without resetting the game's observed progression.
A known older match key cannot replace a newer game's live state. When an unseen
key starts a new game before the previous keyed game's `game_end` arrives,
Matches labels the previous game **End report missing**. A later `game_end`
completes its own match without changing the newer game. Missing sequence counts
are diagnostic: they do not invent absent rounds, rosters, or an ending.
Duplicate sequences, lower-sequence reports, and reports for known older keys do
not advance beta requirements or replace the current scoreboard. A successful
response acknowledges the submitted report; it does not mean that report became
the current game state.

Preserve delivery order across games as well: keep queued retries ahead of
newer games' reports. An unseen key is always treated as the next game, even if
its first report was delayed from an older game. `seq` cannot resolve this because
it orders reports only within a game. A persistent per-server game counter or
equivalent ordering metadata would require a protocol extension. See the
[first-report ordering limitation](leaderboard.md#persistence-and-match-boundaries)
for its effect on match history and the live scoreboard.

Electric Cherry saves the latest accepted report, including player stats, and
streams it to connected clients. A new start clears the previous round display;
an end closes the match. Without a match key, legacy receipt-order and
map/round heuristics remain in effect. Those reports cannot reliably distinguish
delayed events from earlier games; send them once in order.

Match timing is independent of player count: an empty `game_start` still records
the start, and empty rosters or RCON polls do not erase it. Fleet can display a
fresh reported round even when the start was missed; match duration stays unknown
in that case. When match history identifies a new match from subsequent gameplay,
Fleet clears the previous match's timing rather than carrying it into the new one.

Reports also feed the persistent [Leaderboard](leaderboard.md), independently
of beta activity. That page documents cumulative stat accounting, round credit,
and match identity and ordering behavior.

### Reporter deployment

The game scripts are uploaded artifacts, not source maintained in this
repository. Their reporter should emit the four payloads above and add the
authentication header. Existing anonymous reporters must be updated before
telemetry is accepted; if a reporter cannot set HTTP headers, configure a trusted
host-local relay to add the header. This repository does not bundle that relay or
assume a particular GSC HTTP implementation. Rotating the shared secret requires
updating the controller environment and all reporters/relays together.

### Telemetry diagnostics

Open a server's **Diagnostics → Telemetry API requests** to inspect incoming
`POST /api/telemetry` calls whose top-level `server` exactly matches its registered
name. Each entry includes the Electric Cherry receipt time, HTTP result, error
reason, event, map, round, player count, content type, and submitted JSON. Rejected
reports are included, even when the content type is wrong but the body is readable
JSON after authentication. Unauthorized requests are rejected before their bodies
are read and are not retained in per-server diagnostics. Oversized bodies are
rejected before parsing and are not retained either. Authentication headers
and their secrets are never stored. Processing failures return 500 and record the internal error when diagnostic
storage is available; failed database transactions do not update the scoreboard.

The panel refreshes every five seconds while visible and has a manual refresh
button. History is stored in SQLite starting with this version, with up to 200
requests per server from the last seven days and a fleet-wide cap of 10,000
requests. Old entries are pruned when new requests arrive, and expired entries
are excluded from reads. Payload previews are limited to 16,384 characters and
marked when truncated. Request headers other than content type are not stored.
`GET /api/servers/{name}/telemetry-requests` returns the same recent history,
newest first, with caching disabled.

- **No entries:** no attributable requests are retained for this exact name.
  Check the reporter's destination and server field. Calls that never reach
  Electric Cherry, malformed JSON, and bodies without a usable server name
  cannot appear in this server's log. Earlier requests are not backfilled.
  Missing/invalid telemetry credentials and oversized bodies also produce no
  retained request; inspect the reporter's HTTP result for 401, 413, or an
  unconfigured-controller 503.
- **400 / 415:** inspect the error and payload. One invalid player's stats or a
  duplicate GUID rejects the entire report. These requests do not update the
  scoreboard.
- **404:** the reported server was not registered or had been destroyed at the
  time of the request.
- **200:** Electric Cherry processed the report. An empty `players` array
  produces an empty scoreboard, including when sent with `game_end`. If a
  populated accepted report is missing from the scoreboard, investigate the
  live update/browser path.
- **500:** Electric Cherry failed to process the request. Inspect the recorded
  error and the application log.

RCON's online status and player count are separate from gameplay telemetry, so
an occupied server can still have no scoreboard report.

## Requirements

Requirements on a qualifying server are ANDed:

| Requirement                 | Evidence                                                                                         |
| --------------------------- | ------------------------------------------------------------------------------------------------ |
| Reach round X               | A round change, heartbeat, or game end reports round >= X with the required player count         |
| Complete qualifying matches | Game end at round 2 or later, optionally qualified by a higher round, player count, and duration |
| Populated minutes           | Time between consecutive game reports with enough players at both ends                           |
| Player-hours                | Populated minutes multiplied by the lower reported player count, divided by 60                   |
| Reach player count          | Player array length in a game report                                                             |
| Sustained population        | Consecutive report intervals above the player threshold within one game                          |
| Continuous healthy minutes  | Consecutive fresh container/RCON observations                                                    |
| Maximum health outages      | Transitions from observed healthy to unhealthy                                                   |
| Successful recovery cycles  | Repeat the other requirements after each additional empty-server restart                         |
| Tester checklist            | Tester name and observation entered on the beta detail page                                      |

All gameplay requirements use the script reports. Match counts do not read game
logs. The reported round and required player count must occur together in the
same game; observations from different games cannot combine to qualify a match.
The final game-end report can supply that qualifying round. Consecutive duplicate
game ends do not add another completion.

Match duration requires a game start and uses receipt times. Population time
advances only when another report arrives in the same game; it uses the lower
player count at the interval's endpoints. There is no heartbeat requirement or
short interval limit between rounds. These are estimates between reported
snapshots, not continuous measurements of player presence.

Health checks and safe restart gates use the existing container/RCON monitoring.
Health intervals spanning more than 60 seconds earn no healthy time.

## Coverage, waves and outcomes

Automatic selection requires all requested instances to pass. With manual server
selection, coverage can require all selected servers, a quorum, one per selected map,
one per selected game, or one per selected game/map combination. Coverage
metadata is pinned when the beta opens. Every bundled file also needs at least
one passing server that received it; a quorum cannot skip testing a game's file.

Set wave size to 1 for a canary, a larger number for batches, or 0 for all
selected servers. Later waves receive files after the previous waves pass.
Health-outage budget violations on any tested server block promotion.

The test window starts when the beta opens or a revision is submitted, including
time waiting for empty servers. Missing evidence at the deadline produces **inconclusive**.
Health monitoring continues after a pass until promotion.

### Failure and rollback

For failures, select hold as failed, pause for investigation, or automatic
rollback. Rollback queues restoration only for files touched by the beta.
Restoration and restart both wait for an empty server and restart capacity.
The manual option holds those changes in a rollback campaign until you select
**Resume when empty**. Promotion is always manual.

### Promotion

Review the promotion preview before confirming. API clients fetch it with
`GET /api/betas/:id/promote` and send its `previewToken` to the same route with
`POST`. If the reviewed scope or configuration has changed, the request returns
409 and requires a fresh preview.

Confirming promotion creates a campaign without reading the existing files on
additional servers. Each server waits for an empty lobby and restart capacity,
then captures rollback snapshots immediately before applying its queued files
and restarting. Campaign history records snapshot capture, and file comparisons
become available once those snapshots are confirmed. Tested servers retain their
original pre-beta snapshots; their composed configuration is still checked at
promotion to determine whether another activation is needed.

Withdrawing changes that have not activated needs no snapshot and leaves those
servers' files untouched. If activation was submitted but its snapshot has not
been confirmed, reconcile the provisioning operation before rolling back.

### Pause, resume and retry

Pause holds evaluation and wave advancement. Resume retains completed progress and the
current revision's deadline, but starts new timing intervals. A fresh attempt resets
progress and restarts deployed servers when empty. Recovery cycles repeat
gameplay milestones and checklist sign-offs while carrying health outages forward.

Plans, progress, and evidence persist in SQLite. The detail page shows the most
recent 200 evidence entries. On upgrading from the old reporter protocol,
active progress resets. Unsupported checks in existing plans and saved presets
become manual review items; they cannot silently pass.

## Updating an active beta

Choose **Update beta** on the beta detail page, upload replacements for existing
bundle files, and review the changes and selected servers before choosing
**Deploy revision**. Text files show a diff when both files are small enough;
binary and large files show their checksums. Files must actually change.

The beta keeps its ID, servers, scope and test policy. Each revision creates new
file versions, clears current test progress and checklist sign-offs, repeats the
configured waves, and starts a fresh testing window at submission. Files install
and servers restart through the normal empty-server and capacity gates. Promotion
requires the new revision to pass. Pause and retry retain the current revision's
deadline; they do not create a revision.

Revision history retains downloadable bundle files and previous test results,
including the last 200 evidence entries shown at replacement. All underlying
attempts and evidence remain in SQLite, separated by revision. Rejecting the beta
restores the original pre-beta files, even after multiple revisions or while a
new revision is waiting to deploy. Files originally absent are removed.

Updates are available until promotion or rollback, including paused, failed and
expired tests. An activation already in progress must settle first; interrupted
file writes and deleted participants must be resolved before an update. Pending,
unsubmitted restarts are superseded. Adding/removing files, changing targets or
changing the policy still requires a new beta.

## API and storage

`POST /api/betas` accepts `scripts`, `testPlan`, and either
`instancesPerGame` (for example, `{ "t5": 2, "t6": 2 }`) for automatic selection
or `servers` with explicit registered server names for manual selection. Submit
only positive integer counts in `instancesPerGame`. The setup UI sends the same
shared count for every applicable game; API clients can still supply different
counts per game. Automatic selection sets the plan's coverage `scope` to `all`.
Do not send both selection forms. File scopes determine which bundle files each
participant receives in both modes.
`GET /api/betas/:id` includes the plan, outcome, per-server progress, and
recent evidence under `test`.

`POST /api/betas/:id/test` accepts `pause`, `resume`, `retry`, or
`sign-off` (with `server`, `requirementId`, and `note`).
`GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are
copied into a beta; later edits do not change an active plan.

`POST /api/betas/:id/revisions` accepts multipart `expectedRevision`, optional
`notes`, and replacement uploads named `file:<scriptId>`. A stale revision is
rejected. `GET /api/betas/:id` includes the current `revision` and `revisions`
history; `GET /api/betas/:id/revisions/:revision/files/:scriptId` downloads an
archived file (or the original pinned file before its first update).

Internal progress records support attempts and recovery cycles. The reporter
does not know about or supply these records. Legacy betas without a test plan
also use `game_end` at round 2 or later for their match-completion gate.