electric-cherry/docs/beta-test-plans.md

Beta test plans

New betas use a saved plan rather than the two legacy wait flags. Existing betas and API callers without testPlan retain their original workflow. Schema additions are additive and run at startup; no active beta needs to be closed to install this feature.

Requirements

Use Add requirement in Betas. Requirements on a qualifying server are ANDed:

Requirement	Evidence
Reach round X	Session telemetry reports round >= X with the required player count
Complete qualifying matches	Normal completed matches, optionally qualified by round, players and duration
Populated minutes	Fresh RCON samples at both ends of a populated interval
Player-hours	Populated minutes × the lower sampled player count / 60
Reach player count	Fresh healthy RCON sample
Sustained population	Consecutive fresh healthy samples above the player threshold
Continuous healthy minutes	Consecutive fresh container/RCON observations
Maximum health outages	Transitions from observed healthy to unhealthy
Maximum unexpected restarts	Distinct reported process boots within the same test run
Maximum script errors	New console lines matching script errors/exceptions, unhandled exceptions or fatal errors
Successful recovery cycles	Repeat the other requirements after each additional empty-server restart
Feature event count	Named, session-scoped game events
Tester checklist	Tester name and observation, entered on the beta detail page

There is one round requirement: reach round X. For repeated high-round runs, use a qualifying-match count with a minimum round. The minimum player count must be present at the qualifying round observation in the same match. A match with many players and a different solo high-round match cannot combine to pass.

Unqualified match counts use incremental game logs (InitGame, a player join, then ExitLevel: executed). Round-qualified or multi-player-qualified matches use session telemetry. Each requirement uses one source, preventing duplicate counts. Match duration is measured from the observed join/session start. Log polling can undercount duration; it does not infer unobserved elapsed time.

Round, feature and unexpected-restart checks require the reporting protocol below. Legacy round reports still update Fleet but cannot prove beta progress. The game scripts themselves are uploaded artifacts, not source maintained in this repository; update their reporter to emit the protocol before selecting these requirements in production.

Coverage, rollout and outcomes

Coverage can require all selected servers, a quorum, one per selected map, one per selected game, or one per selected game/map combination. Coverage metadata is pinned when the beta opens. Register game/map metadata first.

Set wave size to 1 for a canary, a larger number for batches, or 0 for all selected servers. Later waves reserve their artifact targets but receive no files until every server in the prior waves has passed. All waves must start testing before the final coverage rule can pass. Error-budget violations on any tested server block promotion, including with quorum coverage.

The test window starts when the beta opens, including time waiting for empty servers. Missing evidence at the deadline produces inconclusive, not a failure or an automatic rollback. Health and log monitoring continue after a pass until promotion; a deteriorating result revokes readiness.

For failures, select hold as failed, pause for investigation, or automatic rollback. Rollback restores only files touched by the beta. It can queue rolling restarts when servers empty, or leave restarts manual. Manual rollback uses the same selected restart policy. No beta auto-promotes.

Pause holds evidence evaluation and rollout. Resume retains evidence and the original deadline. A fresh attempt retires old evidence and restarts deployed servers when empty; it does not extend the deadline. Expired or incomplete-file deployments should be rolled back and reopened. Recovery cycles similarly require fresh milestones/checklists, while carrying error budgets forward.

Time accounting excludes intervals spanning monitoring gaps longer than 60 seconds. A missing first sample earns no time. A log that cannot be read cannot pass a clean-log requirement. Log backlog above 2 MB makes the plan inconclusive instead of silently dropping evidence. Partial lines, rotations, byte cursors, attempts and accepted events persist in SQLite. The detail page shows the most recent 200 evidence entries; older entries remain in the database.

Game reporter protocol

On game-process startup, fetch:

GET /api/telemetry?server=t5_zombie_theater_1

The response contains runs: [{ runId, betaId }]. Cache that list for the process lifetime, generate a unique bootId, and send each event once to every returned run. A run ID is an evidence correlation token, not an authentication credential. Do not refresh it during a match or relabel queued events with a newer run ID. Each beta-controlled restart creates a new run ID before requesting the restart. An unexpected process restart fetches the current run ID and uses a new boot ID.

POST JSON to /api/telemetry:

{
  "event": "round_change",
  "server": "t5_zombie_theater_1",
  "runId": "the-run-id-from-startup",
  "eventId": "boot-uuid:session-uuid:12",
  "bootId": "boot-uuid",
  "sessionId": "session-uuid",
  "sequence": 12,
  "round": 25,
  "players": 4
}

Send events in order:

1. boot: once per process, before all session events. sessionId can be omitted. Send sequence, round and players as 0.
2. session_start: a new unique session ID, sequence 0, current players/round. Send when actual gameplay begins, not when an idle map loads.
3. round_change: strictly increasing sequence within that session, reported round and player count. Emit on entry to each round.
4. feature: same sequence ordering, current players/round, plus name such as perk.purchased. Report one event per occurrence.
5. match_end: normal completion only, next sequence, final players/round. Crashes, forced shutdowns and idle map rotations must not emit this event.

IDs accept letters, numbers, _, ., :, and - (up to 128 characters). Feature names are limited to 120 characters. Numbers must be nonnegative safe integers. Retries reuse the same event ID and payload. Send sequentially: an older sequence arriving late is rejected, not retroactively credited.

Responses include test: { accepted, reason? }. Duplicates return HTTP 200 with accepted: false; malformed events return 400. Events for retired runs, unknown boots/sessions, already-ended sessions, or paused/closed tests are not credited. The existing unauthenticated/trusted-network deployment model is unchanged.

Legacy payloads without runId continue to support round_change for Fleet display. They do not alter RCON restart safety data or test evidence.

API and storage

POST /api/betas accepts testPlan, validated by parseBetaPlan in src/lib/beta-plan.ts. New plans always verify restart recovery before testing. GET /api/betas/:id includes the plan, outcome, per-server requirement progress and recent evidence under test.

POST /api/betas/:id/test accepts actions pause, resume, retry, or sign-off (with server, requirementId, and note). GET/POST /api/betas/presets lists/saves reusable plans. Saved presets are copied into a beta; later preset edits never change an active test.

beta_test_plans, beta_test_coverage, beta_test_runs, beta_test_evidence, and beta_test_presets hold configuration and results. beta_file_changes records which files were touched, including partially completed deployments. Legacy beta lifecycle states remain compatible with artifact holds and history; test outcomes are stored separately. Holds last until promotion or rollback.