electric-cherry/docs/beta-test-plans.md

# Beta test plans

New betas use a saved plan rather than the two legacy wait flags. Existing
betas and API callers without `testPlan` retain their original workflow.
Schema additions are additive and run at startup; no active beta needs to be
closed to install this feature.

## Requirements

Use **Add requirement** in Betas. Requirements on a qualifying server are ANDed:

| Requirement | Evidence |
| --- | --- |
| Reach round X | Session telemetry reports round >= X with the required player count |
| Complete qualifying matches | Normal completed matches, optionally qualified by round, players and duration |
| Populated minutes | Fresh RCON samples at both ends of a populated interval |
| Player-hours | Populated minutes × the lower sampled player count / 60 |
| Reach player count | Fresh healthy RCON sample |
| Sustained population | Consecutive fresh healthy samples above the player threshold |
| Continuous healthy minutes | Consecutive fresh container/RCON observations |
| Maximum health outages | Transitions from observed healthy to unhealthy |
| Maximum unexpected restarts | Distinct reported process boots within the same test run |
| Maximum script errors | New console lines matching script errors/exceptions, unhandled exceptions or fatal errors |
| Successful recovery cycles | Repeat the other requirements after each additional empty-server restart |
| Feature event count | Named, session-scoped game events |
| Tester checklist | Tester name and observation, entered on the beta detail page |

There is one round requirement: **reach round X**. For repeated high-round runs,
use a qualifying-match count with a minimum round. The minimum player count must
be present at the qualifying round observation in the same match. A match with
many players and a different solo high-round match cannot combine to pass.

Unqualified match counts use incremental game logs (`InitGame`, a player join,
then `ExitLevel: executed`). Round-qualified or multi-player-qualified matches
use session telemetry. Each requirement uses one source, preventing duplicate
counts. Match duration is measured from the observed join/session start. Log
polling can undercount duration; it does not infer unobserved elapsed time.

Round, feature and unexpected-restart checks require the reporting protocol
below. Legacy round reports still update Fleet but cannot prove beta progress.
The game scripts themselves are uploaded artifacts, not source maintained in
this repository; update their reporter to emit the protocol before selecting
these requirements in production.

## Coverage, rollout and outcomes

Coverage can require all selected servers, a quorum, one per selected map,
one per selected game, or one per selected game/map combination. Coverage
metadata is pinned when the beta opens. Register game/map metadata first.

Set wave size to 1 for a canary, a larger number for batches, or 0 for all
selected servers. Later waves reserve their artifact targets but receive no
files until every server in the prior waves has passed. All waves must start
testing before the final coverage rule can pass. Error-budget violations on
any tested server block promotion, including with quorum coverage.

The test window starts when the beta opens, including time waiting for empty
servers. Missing evidence at the deadline produces **inconclusive**, not a
failure or an automatic rollback. Health and log monitoring continue after a
pass until promotion; a deteriorating result revokes readiness.

For failures, select hold as failed, pause for investigation, or automatic
rollback. Rollback restores only files touched by the beta. It can queue rolling
restarts when servers empty, or leave restarts manual. Manual rollback uses the
same selected restart policy. No beta auto-promotes.

Pause holds evidence evaluation and rollout. Resume retains evidence and the
original deadline. A fresh attempt retires old evidence and restarts deployed
servers when empty; it does not extend the deadline. Expired or incomplete-file
deployments should be rolled back and reopened. Recovery cycles similarly
require fresh milestones/checklists, while carrying error budgets forward.

Time accounting excludes intervals spanning monitoring gaps longer than 60
seconds. A missing first sample earns no time. A log that cannot be read cannot
pass a clean-log requirement. Log backlog above 2 MB makes the plan inconclusive
instead of silently dropping evidence. Partial lines, rotations, byte cursors,
attempts and accepted events persist in SQLite. The detail page shows the most
recent 200 evidence entries; older entries remain in the database.

## Game reporter protocol

On **game-process startup**, fetch:

```http
GET /api/telemetry?server=t5_zombie_theater_1
```

The response contains `runs: [{ runId, betaId }]`. Cache that list for the process
lifetime, generate a unique `bootId`, and send each event once to every returned
run. A run ID is an evidence correlation token, not an authentication credential.
Do not refresh it during a match or relabel queued events with a newer run ID.
Each beta-controlled restart creates a new run ID before requesting the restart.
An unexpected process restart fetches the current run ID and uses a new boot ID.

POST JSON to `/api/telemetry`:

```json
{
  "event": "round_change",
  "server": "t5_zombie_theater_1",
  "runId": "the-run-id-from-startup",
  "eventId": "boot-uuid:session-uuid:12",
  "bootId": "boot-uuid",
  "sessionId": "session-uuid",
  "sequence": 12,
  "round": 25,
  "players": 4
}
```

Send events in order:

1. `boot`: once per process, before all session events. `sessionId` can be
   omitted. Send `sequence`, `round` and `players` as 0.
2. `session_start`: a new unique session ID, sequence 0, current players/round.
   Send when actual gameplay begins, not when an idle map loads.
3. `round_change`: strictly increasing sequence within that session, reported
   round and player count. Emit on entry to each round.
4. `feature`: same sequence ordering, current players/round, plus `name` such
   as `perk.purchased`. Report one event per occurrence.
5. `match_end`: normal completion only, next sequence, final players/round.
   Crashes, forced shutdowns and idle map rotations must not emit this event.

IDs accept letters, numbers, `_`, `.`, `:`, and `-` (up to 128 characters).
Feature names are limited to 120 characters. Numbers must be nonnegative safe
integers. Retries reuse the same event ID and payload. Send sequentially: an
older sequence arriving late is rejected, not retroactively credited.

Responses include `test: { accepted, reason? }`. Duplicates return HTTP 200 with
`accepted: false`; malformed events return 400. Events for retired runs, unknown
boots/sessions, already-ended sessions, or paused/closed tests are not credited.
The existing unauthenticated/trusted-network deployment model is unchanged.

Legacy payloads without `runId` continue to support `round_change` for Fleet
display. They do not alter RCON restart safety data or test evidence.

## API and storage

`POST /api/betas` accepts `testPlan`, validated by `parseBetaPlan` in
`src/lib/beta-plan.ts`. New plans always verify restart recovery before testing.
`GET /api/betas/:id` includes the plan, outcome, per-server requirement progress
and recent evidence under `test`.

`POST /api/betas/:id/test` accepts actions `pause`, `resume`, `retry`, or
`sign-off` (with `server`, `requirementId`, and `note`).
`GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are copied
into a beta; later preset edits never change an active test.

`beta_test_plans`, `beta_test_coverage`, `beta_test_runs`, `beta_test_evidence`,
and `beta_test_presets` hold configuration and results. `beta_file_changes`
records which files were touched, including partially completed deployments.
Legacy beta lifecycle states remain compatible with artifact holds and history;
test outcomes are stored separately. Holds last until promotion or rollback.