Files
electric-cherry/docs/beta-test-plans.md
T

653 lines
29 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Beta test plans
New betas use a saved plan rather than the two legacy wait flags. Existing
betas and API callers without `testPlan` retain their original workflow.
Schema additions are additive and run at startup; no active beta needs to be
closed to install this feature.
## Requirements
Use **Add requirement** in Betas. Requirements on a qualifying server are ANDed:
| Requirement | Evidence |
| --- | --- |
| Reach round X | Session telemetry reports round >= X with the required player count |
| Complete qualifying matches | Normal completed matches, optionally qualified by round, players and duration |
| Populated minutes | Fresh RCON samples at both ends of a populated interval |
| Player-hours | Populated minutes × the lower sampled player count / 60 |
| Reach player count | Fresh healthy RCON sample |
| Sustained population | Consecutive fresh healthy samples above the player threshold |
| Continuous healthy minutes | Consecutive fresh container/RCON observations |
| Maximum health outages | Transitions from observed healthy to unhealthy |
| Maximum unexpected restarts | Distinct reported process boots within the same test run |
| Maximum script errors | New console lines matching script errors/exceptions, unhandled exceptions or fatal errors |
| Successful recovery cycles | Repeat the other requirements after each additional empty-server restart |
| Feature event count | Named, session-scoped game events |
| Tester checklist | Tester name and observation, entered on the beta detail page |
There is one round requirement: **reach round X**. For repeated high-round runs,
use a qualifying-match count with a minimum round. The minimum player count must
be present at the qualifying round observation in the same match. A match with
many players and a different solo high-round match cannot combine to pass.
Unqualified match counts use incremental game logs (`InitGame`, a player join,
then `ExitLevel: executed`). Round-qualified or multi-player-qualified matches
use session telemetry. Each requirement uses one source, preventing duplicate
counts. Match duration is measured from the observed join/session start. Log
polling can undercount duration; it does not infer unobserved elapsed time.
Round, feature and unexpected-restart checks require the reporting protocol
below. Legacy round reports still update Fleet but cannot prove beta progress.
The game scripts themselves are uploaded artifacts, not source maintained in
this repository; update their reporter to emit the protocol before selecting
these requirements in production.
## Coverage, rollout and outcomes
Coverage can require all selected servers, a quorum, one per selected map,
one per selected game, or one per selected game/map combination. Coverage
metadata is pinned when the beta opens. Register game/map metadata first.
Set wave size to 1 for a canary, a larger number for batches, or 0 for all
selected servers. Later waves reserve their artifact targets but receive no
files until every server in the prior waves has passed. All waves must start
testing before the final coverage rule can pass. Error-budget violations on
any tested server block promotion, including with quorum coverage.
The test window starts when the beta opens, including time waiting for empty
servers. Missing evidence at the deadline produces **inconclusive**, not a
failure or an automatic rollback. Health and log monitoring continue after a
pass until promotion; a deteriorating result revokes readiness.
For failures, select hold as failed, pause for investigation, or automatic
rollback. Rollback restores only files touched by the beta. It can queue rolling
restarts when servers empty, or leave restarts manual. Manual rollback uses the
same selected restart policy. No beta auto-promotes.
Pause holds evidence evaluation and rollout. Resume retains evidence and the
original deadline. A fresh attempt retires old evidence and restarts deployed
servers when empty; it does not extend the deadline. Expired or incomplete-file
deployments should be rolled back and reopened. Recovery cycles similarly
require fresh milestones/checklists, while carrying error budgets forward.
Time accounting excludes intervals spanning monitoring gaps longer than 60
seconds. A missing first sample earns no time. A log that cannot be read cannot
pass a clean-log requirement. Log backlog above 2 MB makes the plan inconclusive
instead of silently dropping evidence. Partial lines, rotations, byte cursors,
attempts and accepted events persist in SQLite. The detail page shows the most
recent 200 evidence entries; older entries remain in the database.
## Game reporter protocol
This section is a standalone integration guide for the T6 telemetry script.
It describes the **currently implemented API**, including its limitations.
Examples use `t6_zombie_town_1`; replace that with the exact, case-sensitive
Electric Cherry server registry name. The API base URL is the Electric Cherry
application, not Komodo or the game server's RCON address.
### 1. Two reporting modes: Fleet display and beta evidence
The reporter must work whether or not anybody is beta testing:
| Situation | Reporter behavior |
| --- | --- |
| No eligible beta run for this server | Keep sending ordinary `round_change` reports for Fleet. Do not send beta-only events. |
| One eligible run discovered at process startup | Send the beta event stream with that run's `runId`. Scoped `round_change` reports also update Fleet. |
| Several eligible runs discovered at startup | Send a copy of each beta event to each run, with separate acknowledgement/queue state per run. |
| Beta opened while an existing game process is running | Continue its existing reporting mode. The beta waits for the server to empty and restarts it; the new process discovers the new run. |
| Beta has passed and is awaiting promotion | Continue reporting. A passed test still accepts evidence until promotion or another outcome stops it. |
| Beta paused, failed, expired, promoted, rolled back, or run superseded | Stop crediting events to that inactive run; continue ordinary round reporting. See the suspension rules below. |
**No active beta is normal, not an error.** Fleet telemetry does not depend on a
beta, a run token, or a successful beta discovery request.
### 2. Startup discovery and an empty result
On **actual game-process startup**, request:
```http
GET /api/telemetry?server=t6_zombie_town_1
```
For a registered server without an eligible run, this returns **HTTP 200**:
```json
{
"runs": []
}
```
This can mean there is no beta, the server is not a beta target, its wave has not
started, the beta is waiting to restart it, or the test is not accepting evidence.
Legacy betas without a test plan are not returned either. The response is a list
of eligible **runs**, not a list of every beta in the application.
With an eligible run, the response is **HTTP 200**:
```json
{
"runs": [
{
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"betaId": 42
}
]
}
```
The response has `Cache-Control: no-store`. Do not assume it contains exactly one
entry, and do not choose only the first entry. `betaId` is informational; POST
requests use `runId`, not `betaId`, to identify the test attempt.
A missing or unknown `server` query parameter returns **404**, not an empty
success. Treat that as a configuration issue. The GET handler currently only
checks registry existence; POST additionally rejects a server marked destroyed.
Do not interpret successful discovery as permission to report for another server.
Cache the startup run list for this process lifetime. Electric Cherry creates a
new run before requesting each beta-controlled restart. That restart is the
boundary that makes gameplay belong to the newly deployed build. **Do not poll
for and adopt new run IDs in an old process, at a map reload, or mid-match.**
Never move queued events to a newer run ID.
If discovery fails, use bounded retries with backoff while still initializing,
before gameplay starts. If startup discovery cannot finish, run in Fleet-only
mode for that process and log the problem. Do not attach later gameplay to a run
discovered after that startup boundary. A new process/retry is needed for a new
safe beta attachment; the current API has no late-attachment handshake.
### 3. Reporting without a beta
Send this on each round change, even when `runs` is empty:
```http
POST /api/telemetry
Content-Type: application/json
```
```json
{
"event": "round_change",
"server": "t6_zombie_town_1",
"round": 17,
"players": 4
}
```
Example success response:
```json
{
"ok": true,
"telemetry": {
"server": "t6_zombie_town_1",
"round": 17,
"players": 4,
"receivedAt": "2026-09-18T12:00:00.000Z"
}
}
```
Omit `runId` entirely in this mode. `"runId": null` and `"runId": ""` are
invalid; they are not equivalent to omitting the field. The response has no
`test` property when `runId` was omitted.
Without `runId`, only `round_change` is supported. Do not send unscoped `boot`,
`session_start`, `feature`, or `match_end` requests: they return 400. Do not buffer
feature occurrences from normal operation to submit when a future beta opens.
Electric Cherry stores only the latest Fleet round report per server and stamps
its receipt time. Lower rounds are valid when a new match begins. These reports
do not change RCON player counts, restart safety decisions, or beta progress.
### 4. Identifier lifetimes and the T6 process boundary
| Field | Who creates it | Lifetime and rules |
| --- | --- | --- |
| `server` | Configuration | Exact registered name, unchanged across matches and process restarts. |
| `runId` | Electric Cherry | One server attempt/recovery phase. Use the value discovered at process startup. Treat it as opaque. |
| `bootId` | Reporter | Unique for each actual game-process lifetime. Keep it across matches, map reloads, and reporter reinitialization within that process. |
| `sessionId` | Reporter | Unique for each actual played match. Generate a new value for the next match, even if the players and map are unchanged. |
| `eventId` | Reporter | Unique for each logical event within a run, including across sessions/boots. A delivery retry reuses the exact same ID. |
| `sequence` | Reporter | Start at 0 for `session_start`; increment for every subsequent event in that session, across all event types. Reset only for a new session. |
Generate collision-resistant IDs, such as UUIDs. A boot ID plus a session ID and
counter is also suitable for event IDs if the combined string fits the length
limit. Example IDs below are illustrative; never hardcode them in the script.
The same event ID can be used for copies sent to different runs because
deduplication is scoped to `(runId, eventId)`.
**A GSC script's initialization is not necessarily a game-process startup.**
The T6 implementation must establish where its state survives match/map changes.
If script state resets on each map load, preserve process identity, the startup
run list, and delivery state in a process-lifetime facility or a helper that can
reliably identify the game process. A helper's own restart must not invent a new
game boot. Do not assume map-local globals provide process-lifetime storage.
If that distinction is unavailable, leave beta reporting disabled rather than
reporting a fabricated boot; ordinary Fleet round reports still work.
An unexpected game-process restart may discover the **same** run ID. Send a new
boot ID: the backend counts additional distinct boots in a run as unexpected
restarts. A planned recovery restart receives a **new** run ID and starts a new
evidence phase. Restarting only the Electric Cherry web app changes neither the
game boot nor the reporter's cached IDs.
### 5. Common request fields
All requests use JSON objects and `Content-Type: application/json`. Every
beta-scoped event includes these fields unless the table says otherwise:
| Field | Type | Constraints |
| --- | --- | --- |
| `event` | string | Exactly `boot`, `session_start`, `round_change`, `feature`, or `match_end`. |
| `server` | string | Registered server name, 1-255 characters, case-sensitive. |
| `runId` | string | Required for beta evidence; 1-128 characters from `A-Z a-z 0-9 _ . : -`. |
| `eventId` | string | Same identifier character/length limits. |
| `bootId` | string | Same identifier character/length limits. |
| `sessionId` | string | Same identifier limits; required except for `boot`. Omit for `boot`. |
| `sequence` | number | Nonnegative safe integer. Use 0 for `boot` and `session_start`. |
| `round` | number | Nonnegative safe integer; the actual player-facing round. Use 0 for `boot`. |
| `players` | number | Nonnegative safe integer; actual participating player count at the event. Use 0 for `boot`. |
| `name` | string | Required only for `feature`; 1-120 characters from the identifier character set. |
Numbers must be JSON numbers, not strings, fractions, `null`, or negative values.
The maximum safe integer is 9007199254740991. Sequence numbers need to increase;
the backend does not require them to be contiguous, but the sender should emit a
single ordered stream. Do not use a separate counter for each event type.
The examples below show excerpts from one process and match. Earlier round
reports are omitted, hence the sequence jumps to 25. The real script should send
each round change and increment its counter for every event it produces.
### 6. Full event examples
#### Process boot
Send once per discovered run at real game-process startup, before any session
events. A retry is the same request, not another boot with another ID.
```json
{
"event": "boot",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:boot",
"bootId": "boot-a81f",
"sequence": 0,
"round": 0,
"players": 0
}
```
#### Start of a played match
Send when actual gameplay begins with players, not merely when an empty map
loads. Retain this `sessionId` until that match ends or is abandoned.
```json
{
"event": "session_start",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:0",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 0,
"round": 1,
"players": 4
}
```
After this is acknowledged, report the initial round with `round_change` too.
The `round` on `session_start` does not itself satisfy a round requirement.
#### Reaching a round
Send on entry to each round, with the player count sampled at that event.
There is no separate "round completed" event.
```json
{
"event": "round_change",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:25",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 25,
"round": 25,
"players": 4
}
```
This can satisfy a round-25 check requiring four players. It also records the
qualifying round for a later completed-match check. A `match_end` reporting round
25 does not replace a missing `round_change` event for that round.
#### Feature: a successful perk purchase
The top-level event is always `"feature"`. The custom feature identifier goes in
`name`; it is not a new top-level event type.
```json
{
"event": "feature",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:26",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 26,
"round": 25,
"players": 4,
"name": "perk.purchased"
}
```
Emit when the purchase actually succeeds, not when a player opens a menu or
attempts an unaffordable purchase. One accepted event adds **one** to each
matching feature requirement whose minimum round/player qualifiers are met.
This event does not require waiting for the match to end.
For example, configure **Feature event count** with event name `perk.purchased`,
target 10, minimum round 20, and minimum players 4. The example above contributes
one occurrence. A purchase at round 19 or with only three players can still be
accepted into the event stream, but contributes zero to that requirement.
#### Feature: successful Pack-a-Punch completion
```json
{
"event": "feature",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:27",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 27,
"round": 25,
"players": 4,
"name": "weapon.pack_a_punch.completed"
}
```
#### Feature: successful execution of a custom mechanic
```json
{
"event": "feature",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:28",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 28,
"round": 25,
"players": 4,
"name": "mechanic.teleporter.completed"
}
```
These names are **suggested conventions**, not preinstalled T6 hooks. The script
must instrument the corresponding successful actions. Only emit a mechanic's
event on maps/builds where that mechanic actually exists.
Names are matched exactly and case-sensitively against the UI's **Reported event
name**. A requirement named `perk.purchased` with target 10 counts ten accepted
occurrences of that exact name; `perk.*` is not a wildcard.
For a specific perk, use a distinct name such as `perk.juggernog.purchased`.
The current API does not filter on a separate `perk`, `weapon`, or `playerId`
field. You may deliberately emit both a generic and a specific marker for one
action, but they must be separate events with separate event IDs and sequences;
each requirement still counts only its exact matching name.
There is no bulk-count field. `"count": 10`, `value`, `metadata`, `timestamp`,
`duration`, and other extra properties are not consumed or preserved as beta
signal fields. Send one event for each real occurrence. Keep any additional
diagnostics in script logs until the API explicitly supports them.
#### Normal match completion
```json
{
"event": "match_end",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:29",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 29,
"round": 25,
"players": 4
}
```
Send once when a played match ends normally, including a normal game-over.
Capture the actual terminal player count; the example's four players are not a
value to fabricate if players have already left. The backend retains earlier
round/player observations for qualification.
Do **not** send `match_end` for a crash, forced process shutdown, arbitrary map
rotation, or an idle map cycle. A generic script teardown callback alone is not
proof of normal match completion. There is no `session_abort` event at present;
abandon that session and use a new ID when actual gameplay next begins. Never
replay events for an abandoned session into the new one.
Accepted `match_end` closes the session. Later events for that session are not
accepted. Round-qualified or multi-player-qualified match requirements use this
stream; unqualified completed-match requirements use game logs instead, so the
same match is not counted twice for one requirement.
### 7. Responses: transport success is not evidence acceptance
An accepted `boot`, `session_start`, `feature`, or `match_end` returns HTTP 200:
```json
{
"ok": true,
"test": {
"accepted": true
}
}
```
An accepted scoped `round_change` additionally updates the Fleet display:
```json
{
"ok": true,
"telemetry": {
"server": "t6_zombie_town_1",
"round": 25,
"players": 4,
"receivedAt": "2026-09-18T12:25:00.000Z"
},
"test": {
"accepted": true
}
}
```
Acceptance means the event was recorded in the run's stream. It does not mean
the whole test passed, or even that a feature/round event met a configured
requirement's qualifiers.
A retry of an already accepted event returns HTTP 200:
```json
{
"ok": true,
"test": {
"accepted": false,
"reason": "Duplicate event"
}
}
```
Treat that exact duplicate response as delivery already acknowledged. Do not
generate a new ID to force the same action to count twice.
An inactive run returns HTTP 200:
```json
{
"ok": true,
"test": {
"accepted": false,
"reason": "Run is inactive or superseded; obtain the current run at process startup"
}
}
```
Invalid ordering or session identity also returns HTTP 200:
```json
{
"ok": true,
"test": {
"accepted": false,
"reason": "Duplicate, out-of-order, closed, or unknown session/boot"
}
}
```
That last response is **not** an acknowledgement: inspect the boot/session/
sequence handling. The API currently provides explanatory strings, not distinct
machine-readable error codes for every rejection. Do not treat every
`accepted: false` as a harmless duplicate or assume every inactive run is closed
permanently; paused runs use the same inactive response.
**Fleet updates are independent of beta acceptance.** A syntactically valid
scoped `round_change` still writes the latest Fleet round even when
`test.accepted` is false, including a duplicate or inactive-run rejection. Its
response contains both `telemetry` and `test`. Do not replay an old round backlog
after newer reports: it can move the Fleet display backwards. There is no
client-timestamp freshness check.
| HTTP result | Meaning and handling |
| --- | --- |
| 200, no `test` | Ordinary Fleet report accepted. |
| 200, `test.accepted: true` | Beta event accepted; advance that run's queue. |
| 200, exact `Duplicate event` reason | Already accepted earlier; advance that run's queue. |
| 200, inactive-run reason | Suspend beta delivery for that run; continue Fleet reporting. |
| 200, session/order rejection | Log an integration error and stop crediting that broken session. Do not invent a replacement session for an in-progress match. |
| 400 | Invalid JSON, fields, or event type; fix the payload instead of retrying unchanged forever. |
| 404 on POST | Server is unregistered or destroyed; check configuration/lifecycle. |
| 415 | Missing or incorrect JSON content type. |
| Network error, timeout, or 5xx | Delivery is uncertain; retry the same request with backoff within a bounded queue. |
### 8. Ordering, retries, pauses, and multiple runs
**Recommended sender behavior** (client implementation guidance, not extra API
features):
1. Keep delivery asynchronous so a web-service outage never blocks the game.
2. Send `boot` first for each run and wait for acceptance or an acknowledged
duplicate before delivering its `session_start`.
3. Serialize the remaining events per run. Do not allow a later event to overtake
an uncertain request. Use the same session counter for rounds, features, and
match completion.
4. On timeout, retry with exactly the same event ID, sequence, and payload. The
first request may have committed even if its response was lost.
5. Bound queue size, retry count, and event age. For example, back off through
1, 2, 5, and 10 seconds rather than retrying every frame. These intervals are
a suggested client policy, not server-enforced values.
6. If a long outage makes the queue stale, abandon beta evidence for that session
and keep the current Fleet round visible. Do not replay historical feature or
completion events hours later. Resume beta evidence at a later real match
only if the same startup-pinned run is still eligible and boot delivery is
established.
7. Keep acknowledgement state separately for every discovered run. Failure on
one run does not change another run's IDs or acknowledged sequence. Reuse the
logical event ID across copies, changing only `runId`.
The server stamps **receipt time**, not game event time. In particular, match
duration is measured between accepted `session_start` and `match_end` receipts.
Delayed replay can distort duration and no client-provided timestamp fixes it.
Keep beta events near real time; do not use them as a historical-import API.
While scoped round delivery is healthy, no extra unscoped copy is necessary.
If beta delivery is suspended or lagging, ordinary round reports can keep Fleet
current. Retire the affected session queue rather than later sending its old
rounds behind newer unscoped reports. A simple implementation may serialize
delivery across all run copies to avoid cross-run round reordering.
For paused/inactive runs, the simplest safe policy is Fleet-only reporting until
the next actual process startup. If automatic resume within the same process is
desired, the reporter may recheck **only IDs already pinned at startup**:
- Intersect discovery results with the original startup list. Never adopt an
unfamiliar run ID returned by a later GET.
- When a cached run becomes eligible again, resume at the **next real match**,
using the existing boot ID and a new session ID. Establish boot delivery first;
reusing the original boot request/event ID is safe if its acknowledgement was
lost.
- Discard events generated during suspension. Do not manufacture a new session
start in the middle of an existing match or replay paused gameplay.
- If startup returned `runs: []`, there are no pinned IDs to resume. Stay
Fleet-only until a real process restart. Similarly, a process started while a
test was paused may require an operator-requested fresh attempt after resume.
There is no polling interval or resume notification supplied by this API. The
reporter owns its retry/suspension policy. The rule about startup pinning must
remain true regardless of that policy.
### 9. T6 implementation handoff checklist
- Configure the Electric Cherry URL and exact server registry name.
- Preserve ordinary round reports when no beta is active or the service is
temporarily unreachable; never block gameplay waiting for HTTP.
- Identify actual game-process boots separately from GSC initialization and
map/match resets. Cache discovery results for the game process lifetime.
- Instrument real session start, round entry, normal game-over, and successful
feature actions. Confirm each chosen T6 callback's semantics; do not infer
normal completion from generic teardown.
- Use one sequence counter per session, unique event IDs, stable retry payloads,
and an ordered bounded queue with separate acknowledgements per run.
- Decide and document the feature names the script emits so beta creators can
enter those exact names. Extra JSON metadata cannot be used as a qualifier.
- Keep beta-only events scoped to startup-pinned runs. Do not backfill pre-beta,
paused, abandoned, or previous-process gameplay into a new run.
- Handle `ok` and `test.accepted` separately. Log rejections with enough context
to diagnose the run, boot, session, event ID, and sequence.
Suggested verification cases for the telemetry-script session:
| Case | Expected result |
| --- | --- |
| Startup returns an empty run list | Round reports continue; no beta-only POSTs. |
| Beta opens during an ordinary match | No new token adopted; beta reporting begins after the beta-controlled process restart. |
| Normal new match, same process | Same boot/run IDs; new session ID and sequence starting at 0. |
| Actual unexpected process restart | New boot ID; rediscover runs; no old queued session events replayed. |
| Retry after a lost response | Same event ID and payload; one evidence entry and a duplicate acknowledgement. |
| Round 25 with four players, then normal match end | Round evidence can pass immediately; a qualifying-match count advances only at normal completion. |
| Ten successful `perk.purchased` events | Ten uniquely identified feature events, counted only when each event meets the configured round/player qualifiers. |
| Crash or idle map reset | No fabricated `match_end`. |
| Paused, closed, or superseded run | Beta delivery suspended; Fleet remains current; never switch old queued events to a new token. |
| Two startup run IDs | Each receives the event once, with independent acknowledgements. |
| Reporter script reload, same game process | No new boot identity or fresh adoption of run IDs. |
The endpoint retains the application's existing trusted-network model. A `runId`
correlates evidence; it is **not** an authentication secret or proof of loaded
artifact bytes. Correct process/session attribution depends on the reporter
following the lifecycle rules above.
## API and storage
`POST /api/betas` accepts `testPlan`, validated by `parseBetaPlan` in
`src/lib/beta-plan.ts`. New plans always verify restart recovery before testing.
`GET /api/betas/:id` includes the plan, outcome, per-server requirement progress
and recent evidence under `test`.
`POST /api/betas/:id/test` accepts actions `pause`, `resume`, `retry`, or
`sign-off` (with `server`, `requirementId`, and `note`).
`GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are copied
into a beta; later preset edits never change an active test.
`beta_test_plans`, `beta_test_coverage`, `beta_test_runs`, `beta_test_evidence`,
and `beta_test_presets` hold configuration and results. `beta_file_changes`
records which files were touched, including partially completed deployments.
Legacy beta lifecycle states remain compatible with artifact holds and history;
test outcomes are stored separately. Holds last until promotion or rollback.