Add configurable beta test plans and staged rollout
This commit is contained in:
@@ -0,0 +1,153 @@
|
||||
# Beta test plans
|
||||
|
||||
New betas use a saved plan rather than the two legacy wait flags. Existing
|
||||
betas and API callers without `testPlan` retain their original workflow.
|
||||
Schema additions are additive and run at startup; no active beta needs to be
|
||||
closed to install this feature.
|
||||
|
||||
## Requirements
|
||||
|
||||
Use **Add requirement** in Betas. Requirements on a qualifying server are ANDed:
|
||||
|
||||
| Requirement | Evidence |
|
||||
| --- | --- |
|
||||
| Reach round X | Session telemetry reports round >= X with the required player count |
|
||||
| Complete qualifying matches | Normal completed matches, optionally qualified by round, players and duration |
|
||||
| Populated minutes | Fresh RCON samples at both ends of a populated interval |
|
||||
| Player-hours | Populated minutes × the lower sampled player count / 60 |
|
||||
| Reach player count | Fresh healthy RCON sample |
|
||||
| Sustained population | Consecutive fresh healthy samples above the player threshold |
|
||||
| Continuous healthy minutes | Consecutive fresh container/RCON observations |
|
||||
| Maximum health outages | Transitions from observed healthy to unhealthy |
|
||||
| Maximum unexpected restarts | Distinct reported process boots within the same test run |
|
||||
| Maximum script errors | New console lines matching script errors/exceptions, unhandled exceptions or fatal errors |
|
||||
| Successful recovery cycles | Repeat the other requirements after each additional empty-server restart |
|
||||
| Feature event count | Named, session-scoped game events |
|
||||
| Tester checklist | Tester name and observation, entered on the beta detail page |
|
||||
|
||||
There is one round requirement: **reach round X**. For repeated high-round runs,
|
||||
use a qualifying-match count with a minimum round. The minimum player count must
|
||||
be present at the qualifying round observation in the same match. A match with
|
||||
many players and a different solo high-round match cannot combine to pass.
|
||||
|
||||
Unqualified match counts use incremental game logs (`InitGame`, a player join,
|
||||
then `ExitLevel: executed`). Round-qualified or multi-player-qualified matches
|
||||
use session telemetry. Each requirement uses one source, preventing duplicate
|
||||
counts. Match duration is measured from the observed join/session start. Log
|
||||
polling can undercount duration; it does not infer unobserved elapsed time.
|
||||
|
||||
Round, feature and unexpected-restart checks require the reporting protocol
|
||||
below. Legacy round reports still update Fleet but cannot prove beta progress.
|
||||
The game scripts themselves are uploaded artifacts, not source maintained in
|
||||
this repository; update their reporter to emit the protocol before selecting
|
||||
these requirements in production.
|
||||
|
||||
## Coverage, rollout and outcomes
|
||||
|
||||
Coverage can require all selected servers, a quorum, one per selected map,
|
||||
one per selected game, or one per selected game/map combination. Coverage
|
||||
metadata is pinned when the beta opens. Register game/map metadata first.
|
||||
|
||||
Set wave size to 1 for a canary, a larger number for batches, or 0 for all
|
||||
selected servers. Later waves reserve their artifact targets but receive no
|
||||
files until every server in the prior waves has passed. All waves must start
|
||||
testing before the final coverage rule can pass. Error-budget violations on
|
||||
any tested server block promotion, including with quorum coverage.
|
||||
|
||||
The test window starts when the beta opens, including time waiting for empty
|
||||
servers. Missing evidence at the deadline produces **inconclusive**, not a
|
||||
failure or an automatic rollback. Health and log monitoring continue after a
|
||||
pass until promotion; a deteriorating result revokes readiness.
|
||||
|
||||
For failures, select hold as failed, pause for investigation, or automatic
|
||||
rollback. Rollback restores only files touched by the beta. It can queue rolling
|
||||
restarts when servers empty, or leave restarts manual. Manual rollback uses the
|
||||
same selected restart policy. No beta auto-promotes.
|
||||
|
||||
Pause holds evidence evaluation and rollout. Resume retains evidence and the
|
||||
original deadline. A fresh attempt retires old evidence and restarts deployed
|
||||
servers when empty; it does not extend the deadline. Expired or incomplete-file
|
||||
deployments should be rolled back and reopened. Recovery cycles similarly
|
||||
require fresh milestones/checklists, while carrying error budgets forward.
|
||||
|
||||
Time accounting excludes intervals spanning monitoring gaps longer than 60
|
||||
seconds. A missing first sample earns no time. A log that cannot be read cannot
|
||||
pass a clean-log requirement. Log backlog above 2 MB makes the plan inconclusive
|
||||
instead of silently dropping evidence. Partial lines, rotations, byte cursors,
|
||||
attempts and accepted events persist in SQLite. The detail page shows the most
|
||||
recent 200 evidence entries; older entries remain in the database.
|
||||
|
||||
## Game reporter protocol
|
||||
|
||||
On **game-process startup**, fetch:
|
||||
|
||||
```http
|
||||
GET /api/telemetry?server=t5_zombie_theater_1
|
||||
```
|
||||
|
||||
The response contains `runs: [{ runId, betaId }]`. Cache that list for the process
|
||||
lifetime, generate a unique `bootId`, and send each event once to every returned
|
||||
run. A run ID is an evidence correlation token, not an authentication credential.
|
||||
Do not refresh it during a match or relabel queued events with a newer run ID.
|
||||
Each beta-controlled restart creates a new run ID before requesting the restart.
|
||||
An unexpected process restart fetches the current run ID and uses a new boot ID.
|
||||
|
||||
POST JSON to `/api/telemetry`:
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "round_change",
|
||||
"server": "t5_zombie_theater_1",
|
||||
"runId": "the-run-id-from-startup",
|
||||
"eventId": "boot-uuid:session-uuid:12",
|
||||
"bootId": "boot-uuid",
|
||||
"sessionId": "session-uuid",
|
||||
"sequence": 12,
|
||||
"round": 25,
|
||||
"players": 4
|
||||
}
|
||||
```
|
||||
|
||||
Send events in order:
|
||||
|
||||
1. `boot`: once per process, before all session events. `sessionId` can be
|
||||
omitted. Send `sequence`, `round` and `players` as 0.
|
||||
2. `session_start`: a new unique session ID, sequence 0, current players/round.
|
||||
Send when actual gameplay begins, not when an idle map loads.
|
||||
3. `round_change`: strictly increasing sequence within that session, reported
|
||||
round and player count. Emit on entry to each round.
|
||||
4. `feature`: same sequence ordering, current players/round, plus `name` such
|
||||
as `perk.purchased`. Report one event per occurrence.
|
||||
5. `match_end`: normal completion only, next sequence, final players/round.
|
||||
Crashes, forced shutdowns and idle map rotations must not emit this event.
|
||||
|
||||
IDs accept letters, numbers, `_`, `.`, `:`, and `-` (up to 128 characters).
|
||||
Feature names are limited to 120 characters. Numbers must be nonnegative safe
|
||||
integers. Retries reuse the same event ID and payload. Send sequentially: an
|
||||
older sequence arriving late is rejected, not retroactively credited.
|
||||
|
||||
Responses include `test: { accepted, reason? }`. Duplicates return HTTP 200 with
|
||||
`accepted: false`; malformed events return 400. Events for retired runs, unknown
|
||||
boots/sessions, already-ended sessions, or paused/closed tests are not credited.
|
||||
The existing unauthenticated/trusted-network deployment model is unchanged.
|
||||
|
||||
Legacy payloads without `runId` continue to support `round_change` for Fleet
|
||||
display. They do not alter RCON restart safety data or test evidence.
|
||||
|
||||
## API and storage
|
||||
|
||||
`POST /api/betas` accepts `testPlan`, validated by `parseBetaPlan` in
|
||||
`src/lib/beta-plan.ts`. New plans always verify restart recovery before testing.
|
||||
`GET /api/betas/:id` includes the plan, outcome, per-server requirement progress
|
||||
and recent evidence under `test`.
|
||||
|
||||
`POST /api/betas/:id/test` accepts actions `pause`, `resume`, `retry`, or
|
||||
`sign-off` (with `server`, `requirementId`, and `note`).
|
||||
`GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are copied
|
||||
into a beta; later preset edits never change an active test.
|
||||
|
||||
`beta_test_plans`, `beta_test_coverage`, `beta_test_runs`, `beta_test_evidence`,
|
||||
and `beta_test_presets` hold configuration and results. `beta_file_changes`
|
||||
records which files were touched, including partially completed deployments.
|
||||
Legacy beta lifecycle states remain compatible with artifact holds and history;
|
||||
test outcomes are stored separately. Holds last until promotion or rollback.
|
||||
Reference in New Issue
Block a user