Simplify telemetry to three game events

This commit is contained in:
2026-09-18 15:35:21 -04:00
parent 6d903adb31
commit 5bb49a776e
29 changed files with 658 additions and 1314 deletions
+12 -24
View File
@@ -66,8 +66,7 @@ See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow.
Betas support configurable gameplay, participation, stability and manual-review Betas support configurable gameplay, participation, stability and manual-review
requirements, saved presets, coverage rules and staged rollout. See requirements, saved presets, coverage rules and staged rollout. See
[Beta test plans](docs/beta-test-plans.md) for usage and the session telemetry [Beta test plans](docs/beta-test-plans.md) for usage and the three-event game telemetry contract.
contract used by round and custom-feature checks.
File changes are staged sequentially. Their activation uses rolling restarts: File changes are staged sequentially. Their activation uses rolling restarts:
by default only **one server across the fleet** can restart or recover at a by default only **one server across the fleet** can restart or recover at a
@@ -229,32 +228,21 @@ Implementation references (checked September 2026):
- [T6 map/ruleset combinations](https://github.com/xerxes-at/T6ServerConfigs/blob/master/localappdata/Plutonium/storage/t6/dedicated_zm.cfg) - [T6 map/ruleset combinations](https://github.com/xerxes-at/T6ServerConfigs/blob/master/localappdata/Plutonium/storage/t6/dedicated_zm.cfg)
- [Plutonium GSC paths](https://plutonium.pw/docs/modding/loading-mods/) - [Plutonium GSC paths](https://plutonium.pw/docs/modding/loading-mods/)
## Round telemetry experiment ## Game telemetry
Send `POST /api/telemetry` with `Content-Type: application/json` to the Electric The server script sends JSON to `POST /api/telemetry`:
Cherry host (port 3000 in local development):
```bash - `game_start`: server, map, and all players (name and GUID).
curl http://localhost:3000/api/telemetry \ - `round_change`: server, map, round, and all players with kills, headshots, downs, revives, and score.
-H 'Content-Type: application/json' \ - `game_end`: the same fields as round change, with final stats.
-d '{"event":"round_change","round":17,"server":"t5_zombie_prototype_1","players":4}'
```
The server must already be registered in Electric Cherry. `round` and `players` These reports work with or without an active beta. Electric Cherry records
are required nonnegative integers. Success returns HTTP 200 with the full player data, updates Fleet, and applies gameplay progress to active
`{ "ok": true, "telemetry": { "server": "...", "round": 17, "players": 4, "receivedAt": "..." } }`. betas automatically. No discovery request or beta-specific fields are needed.
Invalid payloads return 400, unknown/destroyed servers 404, and non-JSON content
types 415. This endpoint uses the app's existing LAN/Tailscale access model;
there is no API key requirement.
The latest report is saved across app restarts and streamed immediately to the See [Game reporter protocol](docs/beta-test-plans.md#game-reporter-protocol) for
dashboard and server detail page, including after browser reconnection. The UI complete payload examples. Telemetry does not replace RCON's empty-server
labels it as the last reported round with its age: round-change events alone check before an automated restart.
cannot detect a stopped script or a match ending. Send the initial round on each
new match, then each round change. The latest received request wins (including
lower rounds for new matches); there is no event ordering or history yet.
Telemetry player counts are displayed with the report and never replace RCON
occupancy or affect automated restart gates.
## Running tests ## Running tests
+134 -621
View File
@@ -1,652 +1,165 @@
# Beta test plans # Beta test plans
New betas use a saved plan rather than the two legacy wait flags. Existing Create a plan in Betas, choose its servers, and deploy. The server's game script
betas and API callers without `testPlan` retain their original workflow. reports gameplay to Electric Cherry. The same reports update Fleet and any active
Schema additions are additive and run at startup; no active beta needs to be beta on that server.
closed to install this feature.
## Game reporter protocol
Send JSON to `POST /api/telemetry` with `Content-Type: application/json`.
There are exactly three events. Send them whether or not a beta is active.
| Event | Fields |
| --- | --- |
| `game_start` | `server`, `map`, `players` with each player's `name` and `guid` |
| `round_change` | `server`, `map`, `round`, `players` with each player's `name`, `guid`, `kills`, `headshots`, `downs`, `revives`, and `score` |
| `game_end` | `server`, `map`, `round`, `players` with each player's `name`, `guid`, `kills`, `headshots`, `downs`, `revives`, and `score` |
`server` is the exact registered Electric Cherry server name. `map` is the
current map. `players` is the full player array, including an empty array when
there are no players. Names and GUIDs are strings; keep GUIDs as strings to
preserve their precision. Round and stat values are nonnegative JSON integers.
### Game start
Send when the game starts. Round and stats are not required.
```json
{
"event": "game_start",
"server": "t6_zombie_town_1",
"map": "zm_transit",
"players": [
{ "name": "Josh", "guid": "12345678901234567" },
{ "name": "Alex", "guid": "23456789012345678" }
]
}
```
### Round change
Send the new round and each player's current stats.
```json
{
"event": "round_change",
"server": "t6_zombie_town_1",
"map": "zm_transit",
"round": 25,
"players": [
{ "name": "Josh", "guid": "12345678901234567", "kills": 420, "headshots": 160, "downs": 1, "revives": 3, "score": 32000 },
{ "name": "Alex", "guid": "23456789012345678", "kills": 380, "headshots": 140, "downs": 3, "revives": 1, "score": 28000 }
]
}
```
### Game end
Send the final round and each player's final stats when the game ends.
```json
{
"event": "game_end",
"server": "t6_zombie_town_1",
"map": "zm_transit",
"round": 26,
"players": [
{ "name": "Josh", "guid": "12345678901234567", "kills": 445, "headshots": 170, "downs": 2, "revives": 3, "score": 33500 },
{ "name": "Alex", "guid": "23456789012345678", "kills": 401, "headshots": 146, "downs": 4, "revives": 1, "score": 29750 }
]
}
```
Success returns HTTP 200 with `{ "ok": true, "telemetry": { ... } }`.
The returned telemetry contains the event and an Electric Cherry `receivedAt`
timestamp. Invalid JSON or fields return 400, an unknown/destroyed server 404,
and a non-JSON content type 415.
The script sends each event once, in gameplay order, directly to Electric Cherry.
There is no discovery request, beta token, boot/session ID, sequence number,
custom feature event, external helper, or separate beta acknowledgement.
Basic payload checks happen in the receiver.
Electric Cherry saves the latest complete report, including player stats, and
streams it to connected clients. A new start clears the previous round display;
an end closes the match. Receipt order determines the latest report. This small
protocol does not attempt to identify delayed events from earlier games or
provide exactly-once delivery across games.
The game scripts are uploaded artifacts, not source maintained in this
repository. Their reporter should emit the three payloads above.
## Requirements ## Requirements
Use **Add requirement** in Betas. Requirements on a qualifying server are ANDed: Requirements on a qualifying server are ANDed:
| Requirement | Evidence | | Requirement | Evidence |
| --- | --- | | --- | --- |
| Reach round X | Session telemetry reports round >= X with the required player count | | Reach round X | A round change or game end reports round >= X with the required player count |
| Complete qualifying matches | Normal completed matches, optionally qualified by round, players and duration | | Complete qualifying matches | Game end, optionally qualified by round, player count, and duration |
| Populated minutes | Fresh RCON samples at both ends of a populated interval | | Populated minutes | Time between consecutive game reports with enough players at both ends |
| Player-hours | Populated minutes × the lower sampled player count / 60 | | Player-hours | Populated minutes multiplied by the lower reported player count, divided by 60 |
| Reach player count | Fresh healthy RCON sample | | Reach player count | Player array length in a game report |
| Sustained population | Consecutive fresh healthy samples above the player threshold | | Sustained population | Consecutive report intervals above the player threshold within one game |
| Continuous healthy minutes | Consecutive fresh container/RCON observations | | Continuous healthy minutes | Consecutive fresh container/RCON observations |
| Maximum health outages | Transitions from observed healthy to unhealthy | | Maximum health outages | Transitions from observed healthy to unhealthy |
| Maximum unexpected restarts | Distinct reported process boots within the same test run |
| Maximum script errors | New console lines matching script errors/exceptions, unhandled exceptions or fatal errors |
| Successful recovery cycles | Repeat the other requirements after each additional empty-server restart | | Successful recovery cycles | Repeat the other requirements after each additional empty-server restart |
| Feature event count | Named, session-scoped game events | | Tester checklist | Tester name and observation entered on the beta detail page |
| Tester checklist | Tester name and observation, entered on the beta detail page |
There is one round requirement: **reach round X**. For repeated high-round runs, All gameplay requirements use the script reports. Match counts do not read game
use a qualifying-match count with a minimum round. The minimum player count must logs. The reported round and required player count must occur together in the
be present at the qualifying round observation in the same match. A match with same game; observations from different games cannot combine to qualify a match.
many players and a different solo high-round match cannot combine to pass. The final game-end report can supply that qualifying round. Consecutive duplicate
game ends do not add another completion.
Unqualified match counts use incremental game logs (`InitGame`, a player join, Match duration requires a game start and uses receipt times. Population time
then `ExitLevel: executed`). Round-qualified or multi-player-qualified matches advances only when another report arrives in the same game; it uses the lower
use session telemetry. Each requirement uses one source, preventing duplicate player count at the interval's endpoints. There is no heartbeat requirement or
counts. Match duration is measured from the observed join/session start. Log short interval limit between rounds. These are estimates between reported
polling can undercount duration; it does not infer unobserved elapsed time. snapshots, not continuous measurements of player presence.
Round, feature and unexpected-restart checks require the reporting protocol Health checks and safe restart gates use the existing container/RCON monitoring.
below. Legacy round reports still update Fleet but cannot prove beta progress. Health intervals spanning more than 60 seconds earn no healthy time.
The game scripts themselves are uploaded artifacts, not source maintained in
this repository; update their reporter to emit the protocol before selecting
these requirements in production.
## Coverage, rollout and outcomes ## Coverage, rollout and outcomes
Coverage can require all selected servers, a quorum, one per selected map, Coverage can require all selected servers, a quorum, one per selected map,
one per selected game, or one per selected game/map combination. Coverage one per selected game, or one per selected game/map combination. Coverage
metadata is pinned when the beta opens. Register game/map metadata first. metadata is pinned when the beta opens.
Set wave size to 1 for a canary, a larger number for batches, or 0 for all Set wave size to 1 for a canary, a larger number for batches, or 0 for all
selected servers. Later waves reserve their artifact targets but receive no selected servers. Later waves receive files after the previous waves pass.
files until every server in the prior waves has passed. All waves must start Health-outage budget violations on any tested server block promotion.
testing before the final coverage rule can pass. Error-budget violations on
any tested server block promotion, including with quorum coverage.
The test window starts when the beta opens, including time waiting for empty The test window starts when the beta opens, including time waiting for empty
servers. Missing evidence at the deadline produces **inconclusive**, not a servers. Missing evidence at the deadline produces **inconclusive**.
failure or an automatic rollback. Health and log monitoring continue after a Health monitoring continues after a pass until promotion.
pass until promotion; a deteriorating result revokes readiness.
For failures, select hold as failed, pause for investigation, or automatic For failures, select hold as failed, pause for investigation, or automatic
rollback. Rollback restores only files touched by the beta. It can queue rolling rollback. Rollback restores only files touched by the beta. Restarts can wait
restarts when servers empty, or leave restarts manual. Manual rollback uses the for empty servers or remain manual. Promotion is always manual.
same selected restart policy. No beta auto-promotes.
Pause holds evaluation and rollout. Resume retains completed progress and the
Pause holds evidence evaluation and rollout. Resume retains evidence and the original deadline, but starts new timing intervals. A fresh attempt resets
original deadline. A fresh attempt retires old evidence and restarts deployed progress and restarts deployed servers when empty. Recovery cycles repeat
servers when empty; it does not extend the deadline. Expired or incomplete-file gameplay milestones and checklist sign-offs while carrying health outages forward.
deployments should be rolled back and reopened. Recovery cycles similarly
require fresh milestones/checklists, while carrying error budgets forward. Plans, progress, and evidence persist in SQLite. The detail page shows the most
recent 200 evidence entries. On upgrading from the old reporter protocol,
Time accounting excludes intervals spanning monitoring gaps longer than 60 active progress resets. Unsupported checks in existing plans and saved presets
seconds. A missing first sample earns no time. A log that cannot be read cannot become manual review items; they cannot silently pass.
pass a clean-log requirement. Log backlog above 2 MB makes the plan inconclusive
instead of silently dropping evidence. Partial lines, rotations, byte cursors,
attempts and accepted events persist in SQLite. The detail page shows the most
recent 200 evidence entries; older entries remain in the database.
## Game reporter protocol
This section is a standalone integration guide for the T6 telemetry script.
It describes the **currently implemented API**, including its limitations.
Examples use `t6_zombie_town_1`; replace that with the exact, case-sensitive
Electric Cherry server registry name. The API base URL is the Electric Cherry
application, not Komodo or the game server's RCON address.
### 1. Two reporting modes: Fleet display and beta evidence
The reporter must work whether or not anybody is beta testing:
| Situation | Reporter behavior |
| --- | --- |
| No eligible beta run for this server | Keep sending ordinary `round_change` reports for Fleet. Do not send beta-only events. |
| One eligible run discovered at process startup | Send the beta event stream with that run's `runId`. Scoped `round_change` reports also update Fleet. |
| Several eligible runs discovered at startup | Send a copy of each beta event to each run, with separate acknowledgement/queue state per run. |
| Beta opened while an existing game process is running | Continue its existing reporting mode. The beta waits for the server to empty and restarts it; the new process discovers the new run. |
| Beta has passed and is awaiting promotion | Continue reporting. A passed test still accepts evidence until promotion or another outcome stops it. |
| Beta paused, failed, expired, promoted, rolled back, or run superseded | Stop crediting events to that inactive run; continue ordinary round reporting. See the suspension rules below. |
**No active beta is normal, not an error.** Fleet telemetry does not depend on a
beta, a run token, or a successful beta discovery request.
### 2. Startup discovery and an empty result
On **actual game-process startup**, request:
```http
GET /api/telemetry?server=t6_zombie_town_1
```
For a registered server without an eligible run, this returns **HTTP 200**:
```json
{
"runs": []
}
```
This can mean there is no beta, the server is not a beta target, its wave has not
started, the beta is waiting to restart it, or the test is not accepting evidence.
Legacy betas without a test plan are not returned either. The response is a list
of eligible **runs**, not a list of every beta in the application.
With an eligible run, the response is **HTTP 200**:
```json
{
"runs": [
{
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"betaId": 42
}
]
}
```
The response has `Cache-Control: no-store`. Do not assume it contains exactly one
entry, and do not choose only the first entry. `betaId` is informational; POST
requests use `runId`, not `betaId`, to identify the test attempt.
A missing or unknown `server` query parameter returns **404**, not an empty
success. Treat that as a configuration issue. The GET handler currently only
checks registry existence; POST additionally rejects a server marked destroyed.
Do not interpret successful discovery as permission to report for another server.
Cache the startup run list for this process lifetime. Electric Cherry creates a
new run before requesting each beta-controlled restart. That restart is the
boundary that makes gameplay belong to the newly deployed build. **Do not poll
for and adopt new run IDs in an old process, at a map reload, or mid-match.**
Never move queued events to a newer run ID.
If discovery fails, use bounded retries with backoff while still initializing,
before gameplay starts. If startup discovery cannot finish, run in Fleet-only
mode for that process and log the problem. Do not attach later gameplay to a run
discovered after that startup boundary. A new process/retry is needed for a new
safe beta attachment; the current API has no late-attachment handshake.
### 3. Reporting without a beta
Send this on each round change, even when `runs` is empty:
```http
POST /api/telemetry
Content-Type: application/json
```
```json
{
"event": "round_change",
"server": "t6_zombie_town_1",
"round": 17,
"players": 4
}
```
Example success response:
```json
{
"ok": true,
"telemetry": {
"server": "t6_zombie_town_1",
"round": 17,
"players": 4,
"receivedAt": "2026-09-18T12:00:00.000Z"
}
}
```
Omit `runId` entirely in this mode. `"runId": null` and `"runId": ""` are
invalid; they are not equivalent to omitting the field. The response has no
`test` property when `runId` was omitted.
Without `runId`, only `round_change` is supported. Do not send unscoped `boot`,
`session_start`, `feature`, or `match_end` requests: they return 400. Do not buffer
feature occurrences from normal operation to submit when a future beta opens.
Electric Cherry stores only the latest Fleet round report per server and stamps
its receipt time. Lower rounds are valid when a new match begins. These reports
do not change RCON player counts, restart safety decisions, or beta progress.
### 4. Identifier lifetimes and the T6 process boundary
| Field | Who creates it | Lifetime and rules |
| --- | --- | --- |
| `server` | Configuration | Exact registered name, unchanged across matches and process restarts. |
| `runId` | Electric Cherry | One server attempt/recovery phase. Use the value discovered at process startup. Treat it as opaque. |
| `bootId` | Reporter | Unique for each actual game-process lifetime. Keep it across matches, map reloads, and reporter reinitialization within that process. |
| `sessionId` | Reporter | Unique for each actual played match. Generate a new value for the next match, even if the players and map are unchanged. |
| `eventId` | Reporter | Unique for each logical event within a run, including across sessions/boots. A delivery retry reuses the exact same ID. |
| `sequence` | Reporter | Start at 0 for `session_start`; increment for every subsequent event in that session, across all event types. Reset only for a new session. |
Generate collision-resistant IDs, such as UUIDs. A boot ID plus a session ID and
counter is also suitable for event IDs if the combined string fits the length
limit. Example IDs below are illustrative; never hardcode them in the script.
The same event ID can be used for copies sent to different runs because
deduplication is scoped to `(runId, eventId)`.
**A GSC script's initialization is not necessarily a game-process startup.**
The T6 implementation must establish where its state survives match/map changes.
If script state resets on each map load, preserve process identity, the startup
run list, and delivery state in a process-lifetime facility or a helper that can
reliably identify the game process. A helper's own restart must not invent a new
game boot. Do not assume map-local globals provide process-lifetime storage.
If that distinction is unavailable, leave beta reporting disabled rather than
reporting a fabricated boot; ordinary Fleet round reports still work.
An unexpected game-process restart may discover the **same** run ID. Send a new
boot ID: the backend counts additional distinct boots in a run as unexpected
restarts. A planned recovery restart receives a **new** run ID and starts a new
evidence phase. Restarting only the Electric Cherry web app changes neither the
game boot nor the reporter's cached IDs.
### 5. Common request fields
All requests use JSON objects and `Content-Type: application/json`. Every
beta-scoped event includes these fields unless the table says otherwise:
| Field | Type | Constraints |
| --- | --- | --- |
| `event` | string | Exactly `boot`, `session_start`, `round_change`, `feature`, or `match_end`. |
| `server` | string | Registered server name, 1-255 characters, case-sensitive. |
| `runId` | string | Required for beta evidence; 1-128 characters from `A-Z a-z 0-9 _ . : -`. |
| `eventId` | string | Same identifier character/length limits. |
| `bootId` | string | Same identifier character/length limits. |
| `sessionId` | string | Same identifier limits; required except for `boot`. Omit for `boot`. |
| `sequence` | number | Nonnegative safe integer. Use 0 for `boot` and `session_start`. |
| `round` | number | Nonnegative safe integer; the actual player-facing round. Use 0 for `boot`. |
| `players` | number | Nonnegative safe integer; actual participating player count at the event. Use 0 for `boot`. |
| `name` | string | Required only for `feature`; 1-120 characters from the identifier character set. |
Numbers must be JSON numbers, not strings, fractions, `null`, or negative values.
The maximum safe integer is 9007199254740991. Sequence numbers need to increase;
the backend does not require them to be contiguous, but the sender should emit a
single ordered stream. Do not use a separate counter for each event type.
The examples below show excerpts from one process and match. Earlier round
reports are omitted, hence the sequence jumps to 25. The real script should send
each round change and increment its counter for every event it produces.
### 6. Full event examples
#### Process boot
Send once per discovered run at real game-process startup, before any session
events. A retry is the same request, not another boot with another ID.
```json
{
"event": "boot",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:boot",
"bootId": "boot-a81f",
"sequence": 0,
"round": 0,
"players": 0
}
```
#### Start of a played match
Send when actual gameplay begins with players, not merely when an empty map
loads. Retain this `sessionId` until that match ends or is abandoned.
```json
{
"event": "session_start",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:0",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 0,
"round": 1,
"players": 4
}
```
After this is acknowledged, report the initial round with `round_change` too.
The `round` on `session_start` does not itself satisfy a round requirement.
#### Reaching a round
Send on entry to each round, with the player count sampled at that event.
There is no separate "round completed" event.
```json
{
"event": "round_change",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:25",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 25,
"round": 25,
"players": 4
}
```
This can satisfy a round-25 check requiring four players. It also records the
qualifying round for a later completed-match check. A `match_end` reporting round
25 does not replace a missing `round_change` event for that round.
#### Feature: a successful perk purchase
The top-level event is always `"feature"`. The custom feature identifier goes in
`name`; it is not a new top-level event type.
```json
{
"event": "feature",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:26",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 26,
"round": 25,
"players": 4,
"name": "perk.purchased"
}
```
Emit when the purchase actually succeeds, not when a player opens a menu or
attempts an unaffordable purchase. One accepted event adds **one** to each
matching feature requirement whose minimum round/player qualifiers are met.
This event does not require waiting for the match to end.
For example, configure **Feature event count** with event name `perk.purchased`,
target 10, minimum round 20, and minimum players 4. The example above contributes
one occurrence. A purchase at round 19 or with only three players can still be
accepted into the event stream, but contributes zero to that requirement.
#### Feature: successful Pack-a-Punch completion
```json
{
"event": "feature",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:27",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 27,
"round": 25,
"players": 4,
"name": "weapon.pack_a_punch.completed"
}
```
#### Feature: successful execution of a custom mechanic
```json
{
"event": "feature",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:28",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 28,
"round": 25,
"players": 4,
"name": "mechanic.teleporter.completed"
}
```
These names are **suggested conventions**, not preinstalled T6 hooks. The script
must instrument the corresponding successful actions. Only emit a mechanic's
event on maps/builds where that mechanic actually exists.
Names are matched exactly and case-sensitively against the UI's **Reported event
name**. A requirement named `perk.purchased` with target 10 counts ten accepted
occurrences of that exact name; `perk.*` is not a wildcard.
For a specific perk, use a distinct name such as `perk.juggernog.purchased`.
The current API does not filter on a separate `perk`, `weapon`, or `playerId`
field. You may deliberately emit both a generic and a specific marker for one
action, but they must be separate events with separate event IDs and sequences;
each requirement still counts only its exact matching name.
There is no bulk-count field. `"count": 10`, `value`, `metadata`, `timestamp`,
`duration`, and other extra properties are not consumed or preserved as beta
signal fields. Send one event for each real occurrence. Keep any additional
diagnostics in script logs until the API explicitly supports them.
#### Normal match completion
```json
{
"event": "match_end",
"server": "t6_zombie_town_1",
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
"eventId": "boot-a81f:match-b92c:29",
"bootId": "boot-a81f",
"sessionId": "match-b92c",
"sequence": 29,
"round": 25,
"players": 4
}
```
Send once when a played match ends normally, including a normal game-over.
Capture the actual terminal player count; the example's four players are not a
value to fabricate if players have already left. The backend retains earlier
round/player observations for qualification.
Do **not** send `match_end` for a crash, forced process shutdown, arbitrary map
rotation, or an idle map cycle. A generic script teardown callback alone is not
proof of normal match completion. There is no `session_abort` event at present;
abandon that session and use a new ID when actual gameplay next begins. Never
replay events for an abandoned session into the new one.
Accepted `match_end` closes the session. Later events for that session are not
accepted. Round-qualified or multi-player-qualified match requirements use this
stream; unqualified completed-match requirements use game logs instead, so the
same match is not counted twice for one requirement.
### 7. Responses: transport success is not evidence acceptance
An accepted `boot`, `session_start`, `feature`, or `match_end` returns HTTP 200:
```json
{
"ok": true,
"test": {
"accepted": true
}
}
```
An accepted scoped `round_change` additionally updates the Fleet display:
```json
{
"ok": true,
"telemetry": {
"server": "t6_zombie_town_1",
"round": 25,
"players": 4,
"receivedAt": "2026-09-18T12:25:00.000Z"
},
"test": {
"accepted": true
}
}
```
Acceptance means the event was recorded in the run's stream. It does not mean
the whole test passed, or even that a feature/round event met a configured
requirement's qualifiers.
A retry of an already accepted event returns HTTP 200:
```json
{
"ok": true,
"test": {
"accepted": false,
"reason": "Duplicate event"
}
}
```
Treat that exact duplicate response as delivery already acknowledged. Do not
generate a new ID to force the same action to count twice.
An inactive run returns HTTP 200:
```json
{
"ok": true,
"test": {
"accepted": false,
"reason": "Run is inactive or superseded; obtain the current run at process startup"
}
}
```
Invalid ordering or session identity also returns HTTP 200:
```json
{
"ok": true,
"test": {
"accepted": false,
"reason": "Duplicate, out-of-order, closed, or unknown session/boot"
}
}
```
That last response is **not** an acknowledgement: inspect the boot/session/
sequence handling. The API currently provides explanatory strings, not distinct
machine-readable error codes for every rejection. Do not treat every
`accepted: false` as a harmless duplicate or assume every inactive run is closed
permanently; paused runs use the same inactive response.
**Fleet updates are independent of beta acceptance.** A syntactically valid
scoped `round_change` still writes the latest Fleet round even when
`test.accepted` is false, including a duplicate or inactive-run rejection. Its
response contains both `telemetry` and `test`. Do not replay an old round backlog
after newer reports: it can move the Fleet display backwards. There is no
client-timestamp freshness check.
| HTTP result | Meaning and handling |
| --- | --- |
| 200, no `test` | Ordinary Fleet report accepted. |
| 200, `test.accepted: true` | Beta event accepted; advance that run's queue. |
| 200, exact `Duplicate event` reason | Already accepted earlier; advance that run's queue. |
| 200, inactive-run reason | Suspend beta delivery for that run; continue Fleet reporting. |
| 200, session/order rejection | Log an integration error and stop crediting that broken session. Do not invent a replacement session for an in-progress match. |
| 400 | Invalid JSON, fields, or event type; fix the payload instead of retrying unchanged forever. |
| 404 on POST | Server is unregistered or destroyed; check configuration/lifecycle. |
| 415 | Missing or incorrect JSON content type. |
| Network error, timeout, or 5xx | Delivery is uncertain; retry the same request with backoff within a bounded queue. |
### 8. Ordering, retries, pauses, and multiple runs
**Recommended sender behavior** (client implementation guidance, not extra API
features):
1. Keep delivery asynchronous so a web-service outage never blocks the game.
2. Send `boot` first for each run and wait for acceptance or an acknowledged
duplicate before delivering its `session_start`.
3. Serialize the remaining events per run. Do not allow a later event to overtake
an uncertain request. Use the same session counter for rounds, features, and
match completion.
4. On timeout, retry with exactly the same event ID, sequence, and payload. The
first request may have committed even if its response was lost.
5. Bound queue size, retry count, and event age. For example, back off through
1, 2, 5, and 10 seconds rather than retrying every frame. These intervals are
a suggested client policy, not server-enforced values.
6. If a long outage makes the queue stale, abandon beta evidence for that session
and keep the current Fleet round visible. Do not replay historical feature or
completion events hours later. Resume beta evidence at a later real match
only if the same startup-pinned run is still eligible and boot delivery is
established.
7. Keep acknowledgement state separately for every discovered run. Failure on
one run does not change another run's IDs or acknowledged sequence. Reuse the
logical event ID across copies, changing only `runId`.
The server stamps **receipt time**, not game event time. In particular, match
duration is measured between accepted `session_start` and `match_end` receipts.
Delayed replay can distort duration and no client-provided timestamp fixes it.
Keep beta events near real time; do not use them as a historical-import API.
While scoped round delivery is healthy, no extra unscoped copy is necessary.
If beta delivery is suspended or lagging, ordinary round reports can keep Fleet
current. Retire the affected session queue rather than later sending its old
rounds behind newer unscoped reports. A simple implementation may serialize
delivery across all run copies to avoid cross-run round reordering.
For paused/inactive runs, the simplest safe policy is Fleet-only reporting until
the next actual process startup. If automatic resume within the same process is
desired, the reporter may recheck **only IDs already pinned at startup**:
- Intersect discovery results with the original startup list. Never adopt an
unfamiliar run ID returned by a later GET.
- When a cached run becomes eligible again, resume at the **next real match**,
using the existing boot ID and a new session ID. Establish boot delivery first;
reusing the original boot request/event ID is safe if its acknowledgement was
lost.
- Discard events generated during suspension. Do not manufacture a new session
start in the middle of an existing match or replay paused gameplay.
- If startup returned `runs: []`, there are no pinned IDs to resume. Stay
Fleet-only until a real process restart. Similarly, a process started while a
test was paused may require an operator-requested fresh attempt after resume.
There is no polling interval or resume notification supplied by this API. The
reporter owns its retry/suspension policy. The rule about startup pinning must
remain true regardless of that policy.
### 9. T6 implementation handoff checklist
- Configure the Electric Cherry URL and exact server registry name.
- Preserve ordinary round reports when no beta is active or the service is
temporarily unreachable; never block gameplay waiting for HTTP.
- Identify actual game-process boots separately from GSC initialization and
map/match resets. Cache discovery results for the game process lifetime.
- Instrument real session start, round entry, normal game-over, and successful
feature actions. Confirm each chosen T6 callback's semantics; do not infer
normal completion from generic teardown.
- Use one sequence counter per session, unique event IDs, stable retry payloads,
and an ordered bounded queue with separate acknowledgements per run.
- Decide and document the feature names the script emits so beta creators can
enter those exact names. Extra JSON metadata cannot be used as a qualifier.
- Keep beta-only events scoped to startup-pinned runs. Do not backfill pre-beta,
paused, abandoned, or previous-process gameplay into a new run.
- Handle `ok` and `test.accepted` separately. Log rejections with enough context
to diagnose the run, boot, session, event ID, and sequence.
Suggested verification cases for the telemetry-script session:
| Case | Expected result |
| --- | --- |
| Startup returns an empty run list | Round reports continue; no beta-only POSTs. |
| Beta opens during an ordinary match | No new token adopted; beta reporting begins after the beta-controlled process restart. |
| Normal new match, same process | Same boot/run IDs; new session ID and sequence starting at 0. |
| Actual unexpected process restart | New boot ID; rediscover runs; no old queued session events replayed. |
| Retry after a lost response | Same event ID and payload; one evidence entry and a duplicate acknowledgement. |
| Round 25 with four players, then normal match end | Round evidence can pass immediately; a qualifying-match count advances only at normal completion. |
| Ten successful `perk.purchased` events | Ten uniquely identified feature events, counted only when each event meets the configured round/player qualifiers. |
| Crash or idle map reset | No fabricated `match_end`. |
| Paused, closed, or superseded run | Beta delivery suspended; Fleet remains current; never switch old queued events to a new token. |
| Two startup run IDs | Each receives the event once, with independent acknowledgements. |
| Reporter script reload, same game process | No new boot identity or fresh adoption of run IDs. |
The endpoint retains the application's existing trusted-network model. A `runId`
correlates evidence; it is **not** an authentication secret or proof of loaded
artifact bytes. Correct process/session attribution depends on the reporter
following the lifecycle rules above.
## API and storage ## API and storage
`POST /api/betas` accepts `testPlan`, validated by `parseBetaPlan` in `POST /api/betas` accepts `testPlan`.
`src/lib/beta-plan.ts`. New plans always verify restart recovery before testing. `GET /api/betas/:id` includes the plan, outcome, per-server progress, and
`GET /api/betas/:id` includes the plan, outcome, per-server requirement progress recent evidence under `test`.
and recent evidence under `test`.
`POST /api/betas/:id/test` accepts actions `pause`, `resume`, `retry`, or `POST /api/betas/:id/test` accepts `pause`, `resume`, `retry`, or
`sign-off` (with `server`, `requirementId`, and `note`). `sign-off` (with `server`, `requirementId`, and `note`).
`GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are copied `GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are
into a beta; later preset edits never change an active test. copied into a beta; later edits do not change an active plan.
`beta_test_plans`, `beta_test_coverage`, `beta_test_runs`, `beta_test_evidence`, Internal progress records support attempts and recovery cycles. The reporter
and `beta_test_presets` hold configuration and results. `beta_file_changes` does not know about or supply these records. Legacy betas without a test plan
records which files were touched, including partially completed deployments. also use `game_end` for their match-completion gate.
Legacy beta lifecycle states remain compatible with artifact holds and history;
test outcomes are stored separately. Holds last until promotion or rollback.
+3 -6
View File
@@ -18,10 +18,7 @@ export function requirementSummary(r: BetaRequirement): string {
case 'sustained-population': return `Keep at least ${count(r.minPlayers, 'player')} online for ${duration(r.target)}`; case 'sustained-population': return `Keep at least ${count(r.minPlayers, 'player')} online for ${duration(r.target)}`;
case 'healthy-minutes': return `Stay continuously healthy for ${duration(r.target)}`; case 'healthy-minutes': return `Stay continuously healthy for ${duration(r.target)}`;
case 'failure-budget': return r.target === 0 ? 'Allow no health outages' : `Allow at most ${count(r.target, 'health outage')}`; case 'failure-budget': return r.target === 0 ? 'Allow no health outages' : `Allow at most ${count(r.target, 'health outage')}`;
case 'restart-budget': return r.target === 0 ? 'Allow no unexpected restarts' : `Allow at most ${count(r.target, 'unexpected restart')}`;
case 'clean-logs': return r.target === 0 ? 'Allow no detected script errors' : `Allow at most ${count(r.target, 'detected script error')}`;
case 'recovery': return `Repeat the checks after ${count(r.target, 'recovery restart')}`; case 'recovery': return `Repeat the checks after ${count(r.target, 'recovery restart')}`;
case 'feature': return `Observe ${r.event || 'a named feature event'} ${count(r.target, 'time')}${players}${r.minRound ? ` at round ${r.minRound} or later` : ''}`;
case 'manual': return r.label; case 'manual': return r.label;
} }
} }
@@ -29,9 +26,9 @@ export function coverageSummary(plan: BetaPlan): string {
return plan.scope === 'all' ? 'Every selected server' : plan.scope === 'quorum' ? `At least ${count(plan.quorum, 'selected server')}` return plan.scope === 'all' ? 'Every selected server' : plan.scope === 'quorum' ? `At least ${count(plan.quorum, 'selected server')}`
: plan.scope === 'map' ? 'One server per selected map' : plan.scope === 'game' ? 'One server per selected game' : 'One server per game / map'; : plan.scope === 'map' ? 'One server per selected map' : plan.scope === 'game' ? 'One server per selected game' : 'One server per game / map';
} }
export function needsSessionTelemetry(r: BetaRequirement): boolean { export function needsGameTelemetry(r: BetaRequirement): boolean {
return ['round', 'feature', 'restart-budget'].includes(r.kind) || (r.kind === 'matches' && (r.minRound > 0 || r.minPlayers > 1)); return ['round', 'matches', 'population', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind);
} }
export function requirementCategory(r: BetaRequirement): string { export function requirementCategory(r: BetaRequirement): string {
return r.kind === 'manual' ? 'Tester sign-off' : ['healthy-minutes', 'failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind) ? 'Continuous check' : 'Milestone'; return r.kind === 'manual' ? 'Tester sign-off' : ['healthy-minutes', 'failure-budget'].includes(r.kind) ? 'Continuous check' : 'Milestone';
} }
+9 -12
View File
@@ -1,5 +1,5 @@
export const requirementKinds = ['round', 'matches', 'populated-minutes', 'player-hours', 'population', export const requirementKinds = ['round', 'matches', 'populated-minutes', 'player-hours', 'population',
'sustained-population', 'healthy-minutes', 'failure-budget', 'restart-budget', 'clean-logs', 'recovery', 'feature', 'manual'] as const; 'sustained-population', 'healthy-minutes', 'failure-budget', 'recovery', 'manual'] as const;
export type RequirementKind = typeof requirementKinds[number]; export type RequirementKind = typeof requirementKinds[number];
export interface BetaRequirement { export interface BetaRequirement {
id: string; id: string;
@@ -9,7 +9,6 @@ export interface BetaRequirement {
minRound: number; minRound: number;
minPlayers: number; minPlayers: number;
minMinutes: number; minMinutes: number;
event: string;
} }
export interface BetaPlan { export interface BetaPlan {
version: 1; version: 1;
@@ -25,12 +24,11 @@ export const requirementLabels: Record<RequirementKind, string> = {
round: 'Reach round', matches: 'Complete qualifying matches', 'populated-minutes': 'Populated minutes', round: 'Reach round', matches: 'Complete qualifying matches', 'populated-minutes': 'Populated minutes',
'player-hours': 'Player-hours', population: 'Reach player count', 'sustained-population': 'Sustained population (minutes)', 'player-hours': 'Player-hours', population: 'Reach player count', 'sustained-population': 'Sustained population (minutes)',
'healthy-minutes': 'Continuous healthy minutes', 'failure-budget': 'Maximum health outages', 'healthy-minutes': 'Continuous healthy minutes', 'failure-budget': 'Maximum health outages',
'restart-budget': 'Maximum unexpected restarts', recovery: 'Successful recovery cycles', manual: 'Tester checklist'
'clean-logs': 'Maximum script errors', recovery: 'Successful recovery cycles', feature: 'Feature event count', manual: 'Tester checklist'
}; };
export function requirement(kind: RequirementKind, id: string): BetaRequirement { export function requirement(kind: RequirementKind, id: string): BetaRequirement {
return { id, kind, label: requirementLabels[kind], target: kind === 'round' ? 25 : ['failure-budget', 'restart-budget', 'clean-logs'].includes(kind) ? 0 : 1, return { id, kind, label: requirementLabels[kind], target: kind === 'round' ? 25 : kind === 'failure-budget' ? 0 : 1,
minRound: 0, minPlayers: 1, minMinutes: 0, event: '' }; minRound: 0, minPlayers: 1, minMinutes: 0 };
} }
export function defaultPlan(): BetaPlan { export function defaultPlan(): BetaPlan {
return { version: 1, requirements: [requirement('healthy-minutes', 'health')], scope: 'all', quorum: 1, return { version: 1, requirements: [requirement('healthy-minutes', 'health')], scope: 'all', quorum: 1,
@@ -44,7 +42,7 @@ export const builtInPlans: Array<{ name: string; plan: BetaPlan }> = [
] } }, ] } },
{ name: 'Long soak', plan: { ...defaultPlan(), timeoutHours: 72, requirements: [ { name: 'Long soak', plan: { ...defaultPlan(), timeoutHours: 72, requirements: [
{ ...requirement('healthy-minutes', 'health'), target: 360 }, { ...requirement('healthy-minutes', 'health'), target: 360 },
{ ...requirement('player-hours', 'participation'), target: 10 }, requirement('clean-logs', 'logs') { ...requirement('player-hours', 'participation'), target: 10 }, requirement('failure-budget', 'outages')
] } } ] } }
]; ];
@@ -68,13 +66,12 @@ export function parseBetaPlan(value: unknown): BetaPlan {
for (const key of ['target', 'minRound', 'minPlayers', 'minMinutes'] as const) { for (const key of ['target', 'minRound', 'minPlayers', 'minMinutes'] as const) {
if (typeof r[key] !== 'number' || !Number.isFinite(r[key]) || r[key] < 0 || r[key] > 1_000_000) throw new Error(`Invalid requirement ${key}`); if (typeof r[key] !== 'number' || !Number.isFinite(r[key]) || r[key] < 0 || r[key] > 1_000_000) throw new Error(`Invalid requirement ${key}`);
} }
if (!['failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind) && r.target <= 0) throw new Error('Requirement target must be positive'); if (r.kind !== 'failure-budget' && r.target <= 0) throw new Error('Requirement target must be positive');
if (['round', 'matches', 'population', 'failure-budget', 'restart-budget', 'clean-logs', 'recovery', 'feature', 'manual'].includes(r.kind) && !Number.isInteger(r.target)) throw new Error('Count targets must be integers'); if (['round', 'matches', 'population', 'failure-budget', 'recovery', 'manual'].includes(r.kind) && !Number.isInteger(r.target)) throw new Error('Count targets must be integers');
if (!Number.isInteger(r.minRound) || !Number.isInteger(r.minPlayers)) throw new Error('Round and player qualifiers must be integers'); if (!Number.isInteger(r.minRound) || !Number.isInteger(r.minPlayers)) throw new Error('Round and player qualifiers must be integers');
if (r.kind === 'manual' && r.target !== 1) throw new Error('Add a separate requirement for each checklist item'); if (r.kind === 'manual' && r.target !== 1) throw new Error('Add a separate requirement for each checklist item');
if (typeof r.event !== 'string' || r.event.length > 120 || (r.kind === 'feature' && !/^[a-zA-Z0-9_.:-]+$/.test(r.event))) throw new Error('Feature requirements need an event name');
return { id: r.id, kind: r.kind, label: r.label.trim(), target: r.target, minRound: r.minRound, return { id: r.id, kind: r.kind, label: r.label.trim(), target: r.target, minRound: r.minRound,
minPlayers: r.minPlayers, minMinutes: r.minMinutes, event: r.event }; minPlayers: r.minPlayers, minMinutes: r.minMinutes };
}); });
return { version: 1, requirements, scope: p.scope, quorum: p.quorum, timeoutHours: p.timeoutHours, return { version: 1, requirements, scope: p.scope, quorum: p.quorum, timeoutHours: p.timeoutHours,
failureAction: p.failureAction, rollbackRestart: p.rollbackRestart, waveSize: p.waveSize }; failureAction: p.failureAction, rollbackRestart: p.rollbackRestart, waveSize: p.waveSize };
@@ -85,6 +82,6 @@ export interface BetaTestView {
plan: BetaPlan; plan: BetaPlan;
outcome: string; outcome: string;
reason: string | null; reason: string | null;
servers: Array<{ server: string; runId: string; phase: number; passed: boolean; progress: RequirementProgress[] }>; servers: Array<{ server: string; phase: number; passed: boolean; progress: RequirementProgress[] }>;
evidence: Array<{ id: number; server: string; received_at: string; kind: string; payload: string }>; evidence: Array<{ id: number; server: string; received_at: string; kind: string; payload: string }>;
} }
+3 -3
View File
@@ -1,7 +1,7 @@
import { parseServerName, zombieMapBySlug } from './map-aliases.js'; import { parseServerName, zombieMapBySlug } from './map-aliases.js';
import { freshStatusTimestamp } from './server-freshness.js'; import { freshStatusTimestamp } from './server-freshness.js';
import type { ZombieGame } from './map-aliases.js'; import type { ZombieGame } from './map-aliases.js';
import type { RoundTelemetry } from './telemetry.js'; import type { GameTelemetry } from './telemetry.js';
import type { Health, RosterPlayer } from './ui/stores/servers.js'; import type { Health, RosterPlayer } from './ui/stores/servers.js';
import { mapTitle, serverTitle } from './ui/server-labels.js'; import { mapTitle, serverTitle } from './ui/server-labels.js';
@@ -63,8 +63,8 @@ export function fleetIdentity(server: FleetServer) {
}; };
} }
export function currentRound(server: FleetServer, report: RoundTelemetry | undefined, now: number): number | null { export function currentRound(server: FleetServer, report: GameTelemetry | undefined, now: number): number | null {
if (!report || !freshServer(server, now) || !server.active_since || !server.game_started_at if (!report || report.event !== 'round_change' || !freshServer(server, now) || !server.active_since || !server.game_started_at
|| server.game_ended_at || (server.last_players ?? 0) === 0) return null; || server.game_ended_at || (server.last_players ?? 0) === 0) return null;
const received = Date.parse(report.receivedAt); const received = Date.parse(report.receivedAt);
const age = now - received; const age = now - received;
+38
View File
@@ -3,6 +3,8 @@ import { mkdirSync } from 'node:fs';
import { dirname, resolve } from 'node:path'; import { dirname, resolve } from 'node:path';
import { config } from '../config.js'; import { config } from '../config.js';
import schemaSql from './schema.sql?raw'; import schemaSql from './schema.sql?raw';
import { requirement, type BetaPlan, type BetaRequirement } from '../../beta-plan.js';
import { emptyMetrics } from '../fleet/beta-evaluation.js';
let _db: Database.Database | null = null; let _db: Database.Database | null = null;
@@ -190,6 +192,40 @@ function assertSnapshotSchemeSafe(handle: Database.Database): void {
handle.prepare("INSERT INTO settings (key, value) VALUES ('snapshots_v1', '1')").run(); handle.prepare("INSERT INTO settings (key, value) VALUES ('snapshots_v1', '1')").run();
} }
// Preserve old plans as reviewable checklists instead of silently passing checks
// that the three-event protocol can no longer perform. Existing evidence stays
// in history; active progress starts fresh under the new gameplay contract.
function migrateGameTelemetry(handle: Database.Database): void {
if (handle.prepare("SELECT 1 FROM settings WHERE key = 'game_telemetry_v2'").get()) return;
handle.transaction(() => {
const convert = (raw: string): string => {
const plan = JSON.parse(raw) as BetaPlan;
plan.requirements = plan.requirements.map(r => {
const { event: _event, ...rest } = r as BetaRequirement & { event?: string };
return ['feature', 'restart-budget', 'clean-logs'].includes(r.kind)
? { ...requirement('manual', r.id), label: `Review: ${r.label}`.slice(0, 200) } : rest;
});
return JSON.stringify(plan);
};
const plans = handle.prepare('SELECT beta_id, plan FROM beta_test_plans').all() as Array<{ beta_id: number; plan: string }>;
for (const row of plans) handle.prepare('UPDATE beta_test_plans SET plan = ? WHERE beta_id = ?').run(convert(row.plan), row.beta_id);
const presets = handle.prepare('SELECT name, plan FROM beta_test_presets').all() as Array<{ name: string; plan: string }>;
for (const row of presets) handle.prepare('UPDATE beta_test_presets SET plan = ? WHERE name = ?').run(convert(row.plan), row.name);
handle.prepare(`UPDATE beta_test_runs SET metrics = ?, cursors = '{}' WHERE retired_at IS NULL
AND beta_id IN (SELECT id FROM betas WHERE state IN ('running','ready-promote'))`).run(JSON.stringify(emptyMetrics()));
handle.exec(`UPDATE beta_test_plans SET outcome = 'active', reason = 'Gameplay telemetry updated; collecting fresh script events'
WHERE outcome IN ('active','passed') AND beta_id IN (SELECT id FROM betas WHERE state IN ('running','ready-promote'));
UPDATE script_versions SET state = 'untested' WHERE state = 'tested'
AND id IN (SELECT bs.script_version_id FROM beta_scripts bs JOIN beta_test_plans p ON p.beta_id = bs.beta_id WHERE p.outcome = 'active')
AND id NOT IN (SELECT current_version_id FROM scripts WHERE current_version_id IS NOT NULL)
AND NOT EXISTS (SELECT 1 FROM beta_scripts bs JOIN betas b ON b.id = bs.beta_id
WHERE bs.script_version_id = script_versions.id AND b.state = 'promoted');
UPDATE betas SET state = 'running' WHERE state = 'ready-promote' AND id IN (SELECT beta_id FROM beta_test_plans WHERE outcome = 'active');
UPDATE beta_servers SET stage = 'observing' WHERE stage = 'ready' AND beta_id IN (SELECT beta_id FROM beta_test_plans WHERE outcome = 'active');
INSERT INTO settings (key, value) VALUES ('game_telemetry_v2', '1');`);
})();
}
export function db(): Database.Database { export function db(): Database.Database {
if (_db) return _db; if (_db) return _db;
@@ -212,6 +248,7 @@ export function db(): Database.Database {
migrateTable(handle, 'beta_scripts', BETA_SCRIPTS_COLUMNS); migrateTable(handle, 'beta_scripts', BETA_SCRIPTS_COLUMNS);
migrateTable(handle, 'deployment_scripts', DEPLOYMENT_SCRIPTS_COLUMNS); migrateTable(handle, 'deployment_scripts', DEPLOYMENT_SCRIPTS_COLUMNS);
migrateTable(handle, 'deployments', DEPLOYMENTS_COLUMNS); migrateTable(handle, 'deployments', DEPLOYMENTS_COLUMNS);
migrateTable(handle, 'server_telemetry', [{ name: 'payload', ddl: 'ALTER TABLE server_telemetry ADD COLUMN payload TEXT' }]);
// Kind-referencing indexes: safe here because migrateTable above just // Kind-referencing indexes: safe here because migrateTable above just
// ALTER'd the column into existence if it was missing. // ALTER'd the column into existence if it was missing.
handle.exec( handle.exec(
@@ -221,6 +258,7 @@ export function db(): Database.Database {
handle.exec(`CREATE INDEX IF NOT EXISTS idx_scripts_kind ON scripts(kind)`); handle.exec(`CREATE INDEX IF NOT EXISTS idx_scripts_kind ON scripts(kind)`);
backfillMultiScriptJoins(handle); backfillMultiScriptJoins(handle);
assertSnapshotSchemeSafe(handle); assertSnapshotSchemeSafe(handle);
migrateGameTelemetry(handle);
} catch (err) { } catch (err) {
handle.close(); handle.close();
throw err; throw err;
+4 -3
View File
@@ -39,8 +39,8 @@ CREATE TABLE IF NOT EXISTS servers (
-- and resume from the same anchor when the server comes back. -- and resume from the same anchor when the server comes back.
active_since TEXT, -- ISO ts of last 0 -> >0 player transition active_since TEXT, -- ISO ts of last 0 -> >0 player transition
inactive_since TEXT, -- ISO ts of last >0 -> 0 player transition inactive_since TEXT, -- ISO ts of last >0 -> 0 player transition
game_started_at TEXT, -- wall-clock ISO when we first saw a J; line for the current match cycle game_started_at TEXT, -- receipt time of the script's game_start
game_ended_at TEXT, -- wall-clock ISO when we first saw ExitLevel/ShutdownGame end it game_ended_at TEXT, -- receipt time of the script's game_end
game_cycle_key TEXT, -- opaque marker for the currently-tracked cycle (the first J; line after the latest InitGame); lets us detect same-players game restarts game_cycle_key TEXT, -- opaque marker for the currently-tracked cycle (the first J; line after the latest InitGame); lets us detect same-players game restarts
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now'))
); );
@@ -314,7 +314,8 @@ CREATE TABLE IF NOT EXISTS server_telemetry (
server TEXT PRIMARY KEY REFERENCES servers(name) ON DELETE CASCADE, server TEXT PRIMARY KEY REFERENCES servers(name) ON DELETE CASCADE,
round INTEGER NOT NULL, round INTEGER NOT NULL,
players INTEGER NOT NULL, players INTEGER NOT NULL,
received_at TEXT NOT NULL received_at TEXT NOT NULL,
payload TEXT
); );
-- Lifecycle history deliberately survives removal of the server registry row. -- Lifecycle history deliberately survives removal of the server registry row.
CREATE TABLE IF NOT EXISTS server_deployments ( CREATE TABLE IF NOT EXISTS server_deployments (
+46 -71
View File
@@ -1,108 +1,83 @@
import type { BetaPlan, RequirementProgress } from '../../beta-plan.js'; import type { BetaPlan, RequirementProgress } from '../../beta-plan.js';
import type { GameEvent } from '../../telemetry.js';
export interface SessionEvidence { export interface MatchEvidence {
started: number; map: string;
sequence: number; started?: number;
lastEvent: number;
players: number;
ended: boolean; ended: boolean;
rounds: Record<string, number>; rounds: Record<string, number>;
peakPlayers: number;
} }
export interface TestMetrics { export interface TestMetrics {
values: Record<string, number>; values: Record<string, number>;
sessions: Record<string, SessionEvidence>; match?: MatchEvidence;
manual: Record<string, string>; manual: Record<string, string>;
bootId?: string;
boots: string[];
restarts: number;
lastSample?: number; lastSample?: number;
wasHealthy?: boolean; wasHealthy?: boolean;
lastPlayers?: number;
outages: number; outages: number;
logErrors: number;
logAvailable?: boolean;
streaks: Record<string, number>; streaks: Record<string, number>;
} }
export const emptyMetrics = (): TestMetrics => ({ values: {}, sessions: {}, manual: {}, boots: [], restarts: 0, outages: 0, logErrors: 0, streaks: {} }); export const emptyMetrics = (): TestMetrics => ({ values: {}, manual: {}, outages: 0, streaks: {} });
export interface TestSignal { /** Gameplay comes only from the script. No reporter IDs or beta handshake. */
event: 'boot' | 'session_start' | 'round_change' | 'match_end' | 'feature'; export function applyGameEvent(plan: BetaPlan, metrics: TestMetrics, event: GameEvent, now: number): void {
bootId: string; const players = event.players.length;
sessionId: string; if (event.event === 'game_start' || !metrics.match || metrics.match.map !== event.map) {
sequence: number; metrics.match = { map: event.map, started: event.event === 'game_start' ? now : undefined,
round: number; lastEvent: now, players, ended: false, rounds: {} };
players: number; metrics.streaks = {};
name: string;
}
/** Events are accepted once, in session order; a match must start in this run. */
export function applySignal(plan: BetaPlan, metrics: TestMetrics, signal: TestSignal, now: number): boolean {
if (signal.event === 'boot') {
if (metrics.boots.includes(signal.bootId)) return false;
if (metrics.bootId) metrics.restarts++;
metrics.boots.push(signal.bootId);
metrics.bootId = signal.bootId;
return true;
} }
if (metrics.bootId !== signal.bootId) return false; const match = metrics.match;
const key = `${signal.bootId}:${signal.sessionId}`; if (match.ended) return;
if (signal.event === 'session_start') { const minutes = Math.max(0, now - match.lastEvent) / 60_000;
if (metrics.sessions[key]) return false;
metrics.sessions[key] = { started: now, sequence: signal.sequence, ended: false, rounds: {}, peakPlayers: signal.players };
return true;
}
const session = metrics.sessions[key];
if (!session || session.ended || signal.sequence <= session.sequence) return false;
session.sequence = signal.sequence;
session.peakPlayers = Math.max(session.peakPlayers, signal.players);
for (const r of plan.requirements) { for (const r of plan.requirements) {
const current = metrics.values[r.id] ?? 0; const current = metrics.values[r.id] ?? 0;
if (signal.event === 'round_change' && signal.players >= r.minPlayers) { const populated = players >= Math.max(1, r.minPlayers) && match.players >= Math.max(1, r.minPlayers);
session.rounds[r.id] = Math.max(session.rounds[r.id] ?? 0, signal.round); if (r.kind === 'population') metrics.values[r.id] = Math.max(current, players);
if (r.kind === 'round') metrics.values[r.id] = Math.max(current, signal.round); if (r.kind === 'populated-minutes' && populated) metrics.values[r.id] = current + minutes;
if (r.kind === 'player-hours' && populated) metrics.values[r.id] = current + minutes * Math.min(players, match.players) / 60;
if (r.kind === 'sustained-population') {
metrics.streaks[r.id] = populated ? (metrics.streaks[r.id] ?? 0) + minutes : 0;
metrics.values[r.id] = Math.max(current, metrics.streaks[r.id]);
} }
if (signal.event === 'match_end' && r.kind === 'matches' && (r.minRound > 0 || r.minPlayers > 1) if (event.event !== 'game_start' && players >= Math.max(1, r.minPlayers)) {
&& (session.rounds[r.id] ?? 0) >= r.minRound && session.peakPlayers >= r.minPlayers match.rounds[r.id] = Math.max(match.rounds[r.id] ?? 0, event.round);
&& now - session.started >= r.minMinutes * 60_000) metrics.values[r.id] = current + 1; if (r.kind === 'round') metrics.values[r.id] = Math.max(current, event.round);
if (signal.event === 'feature' && r.kind === 'feature' && signal.name === r.event
&& signal.players >= r.minPlayers && signal.round >= r.minRound) metrics.values[r.id] = current + 1;
} }
if (signal.event === 'match_end') session.ended = true; if (event.event === 'game_end' && r.kind === 'matches'
return true; && (match.rounds[r.id] ?? -1) >= r.minRound
&& (r.minMinutes === 0 || (match.started !== undefined && now - match.started >= r.minMinutes * 60_000))) {
metrics.values[r.id] = current + 1;
}
}
match.lastEvent = now;
match.players = players;
match.ended = event.event === 'game_end';
} }
/** Sample both ends of an interval. Monitoring gaps never earn playtime. */ /** Infrastructure health is separate from script-reported gameplay. */
export function applySample(plan: BetaPlan, m: TestMetrics, at: number, healthy: boolean, players: number): void { export function applySample(plan: BetaPlan, m: TestMetrics, at: number, healthy: boolean): void {
if (m.lastSample !== undefined && at <= m.lastSample) return; if (m.lastSample !== undefined && at <= m.lastSample) return;
const gap = m.lastSample === undefined ? Infinity : at - m.lastSample; const gap = m.lastSample === undefined ? Infinity : at - m.lastSample;
const contiguous = gap <= 60_000; const contiguous = gap <= 60_000;
const minutes = healthy && m.wasHealthy && contiguous ? gap / 60_000 : 0; const minutes = healthy && m.wasHealthy && contiguous ? gap / 60_000 : 0;
if (!healthy && m.wasHealthy === true) m.outages++; if (!healthy && m.wasHealthy === true) m.outages++;
for (const r of plan.requirements) { for (const r of plan.requirements) {
const value = m.values[r.id] ?? 0; if (r.kind === 'healthy-minutes') m.values[r.id] = healthy && contiguous && m.wasHealthy ? (m.values[r.id] ?? 0) + minutes : 0;
const populated = players >= Math.max(1, r.minPlayers) && (m.lastPlayers ?? 0) >= Math.max(1, r.minPlayers);
if (r.kind === 'populated-minutes' && populated) m.values[r.id] = value + minutes;
if (r.kind === 'player-hours' && populated) m.values[r.id] = value + minutes * Math.min(players, m.lastPlayers ?? 0) / 60;
if (r.kind === 'population' && healthy) m.values[r.id] = Math.max(value, players);
if (r.kind === 'healthy-minutes') m.values[r.id] = healthy && contiguous && m.wasHealthy ? value + minutes : 0;
if (r.kind === 'sustained-population') {
m.streaks[r.id] = healthy && populated && contiguous ? (m.streaks[r.id] ?? 0) + minutes : 0;
m.values[r.id] = Math.max(value, m.streaks[r.id]);
} }
} m.lastSample = at; m.wasHealthy = healthy;
m.lastSample = at; m.wasHealthy = healthy; m.lastPlayers = players;
} }
export function evaluateRequirements(plan: BetaPlan, metrics: TestMetrics, phase: number): RequirementProgress[] { export function evaluateRequirements(plan: BetaPlan, metrics: TestMetrics, phase: number): RequirementProgress[] {
return plan.requirements.map(r => { return plan.requirements.map(r => {
const budget = ['failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind); const budget = r.kind === 'failure-budget';
const current = r.kind === 'failure-budget' ? metrics.outages : r.kind === 'clean-logs' ? metrics.logErrors const current = budget ? metrics.outages : r.kind === 'manual' ? Number(!!metrics.manual[r.id])
: r.kind === 'restart-budget' ? metrics.restarts : r.kind === 'recovery' ? phase : metrics.values[r.id] ?? 0;
: r.kind === 'manual' ? Number(!!metrics.manual[r.id]) : r.kind === 'recovery' ? phase : metrics.values[r.id] ?? 0; return { id: r.id, label: r.label, current, target: r.target, passed: budget ? current <= r.target : current >= r.target,
const available = (r.kind !== 'clean-logs' || metrics.logAvailable === true) && (r.kind !== 'restart-budget' || !!metrics.bootId);
return { id: r.id, label: r.label, current, target: r.target, passed: available && (budget ? current <= r.target : current >= r.target),
failed: budget && current > r.target, failed: budget && current > r.target,
detail: r.kind === 'manual' ? metrics.manual[r.id] ?? 'Awaiting tester sign-off' detail: r.kind === 'manual' ? metrics.manual[r.id] ?? 'Awaiting tester sign-off'
: !available ? r.kind === 'restart-budget' ? 'Waiting for boot telemetry' : 'Waiting for a readable console log' : budget ? `${current} / ${r.target} allowed; monitored until promotion` : budget ? `${current} / ${r.target} allowed; monitored until promotion`
: `${Math.round(current * 100) / 100} / ${r.target}` }; : `${Math.round(current * 100) / 100} / ${r.target}` };
}); });
} }
+24 -125
View File
@@ -1,13 +1,11 @@
import { randomUUID } from 'node:crypto'; import { randomUUID } from 'node:crypto';
import { open, stat } from 'node:fs/promises';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { parseBetaPlan, type BetaPlan, type BetaTestView } from '../../beta-plan.js'; import { parseBetaPlan, type BetaPlan, type BetaTestView } from '../../beta-plan.js';
import { applySample, applySignal, emptyMetrics, evaluateRequirements, type TestMetrics, type TestSignal } from './beta-evaluation.js'; import type { GameTelemetry } from '../../telemetry.js';
import { gameLogKey, resolveLogPath, type LogKey } from './logs.js'; import { applySample, applyGameEvent, emptyMetrics, evaluateRequirements, type TestMetrics } from './beta-evaluation.js';
export interface TestRun { id: string; beta_id: number; server: string; phase: number; created_at: string; observing_at: string | null; metrics: string; cursors: string } export interface TestRun { id: string; beta_id: number; server: string; phase: number; created_at: string; observing_at: string | null; metrics: string }
export interface StoredPlan { beta_id: number; plan: string; outcome: string; reason: string | null } export interface StoredPlan { beta_id: number; plan: string; outcome: string; reason: string | null }
interface Cursor { position: number; identity: string; partial: string; anchor?: string; generation?: string; session?: string; joined?: boolean; started?: number }
const nowIso = () => new Date().toISOString(); const nowIso = () => new Date().toISOString();
export const storedPlan = (id: number) => db().prepare('SELECT * FROM beta_test_plans WHERE beta_id = ?').get(id) as StoredPlan | undefined; export const storedPlan = (id: number) => db().prepare('SELECT * FROM beta_test_plans WHERE beta_id = ?').get(id) as StoredPlan | undefined;
export const currentRun = (id: number, server: string) => db().prepare('SELECT * FROM beta_test_runs WHERE beta_id = ? AND server = ? AND retired_at IS NULL').get(id, server) as TestRun | undefined; export const currentRun = (id: number, server: string) => db().prepare('SELECT * FROM beta_test_runs WHERE beta_id = ? AND server = ? AND retired_at IS NULL').get(id, server) as TestRun | undefined;
@@ -25,146 +23,47 @@ export function evidence(run: TestRun, eventId: string, kind: string, payload: u
.run(run.id, eventId, nowIso(), kind, JSON.stringify(payload)).changes > 0; .run(run.id, eventId, nowIso(), kind, JSON.stringify(payload)).changes > 0;
} }
/** Snapshot log offsets BEFORE restart: pre-beta log entries can never qualify. */ /** Internal progress record; never sent to or required from the game script. */
export async function prepareRun(betaId: number, server: string, phase = 0): Promise<TestRun> { export async function prepareRun(betaId: number, server: string, phase = 0): Promise<TestRun> {
const cursors: Record<string, Cursor> = {};
for (const key of [gameLogKey(server), 'console'] as LogKey[]) {
const path = resolveLogPath(server, key);
const info = path ? await stat(path).catch(() => null) : null;
let anchor = '';
if (path && info?.size) {
const file = await open(path, 'r');
try {
const buffer = Buffer.alloc(Math.min(info.size, 128));
await file.read(buffer, 0, buffer.length, info.size - buffer.length);
anchor = buffer.toString('base64');
} finally { await file.close(); }
}
cursors[key] = { position: info?.size ?? 0, identity: info ? `${info.dev}:${info.ino}` : '', partial: '', anchor, generation: randomUUID() };
}
const id = randomUUID();
db().transaction(() => { db().transaction(() => {
db().prepare('UPDATE beta_test_runs SET retired_at = ? WHERE beta_id = ? AND server = ? AND retired_at IS NULL').run(nowIso(), betaId, server); db().prepare('UPDATE beta_test_runs SET retired_at = ? WHERE beta_id = ? AND server = ? AND retired_at IS NULL').run(nowIso(), betaId, server);
db().prepare('INSERT INTO beta_test_runs (id, beta_id, server, phase, created_at, metrics, cursors) VALUES (?, ?, ?, ?, ?, ?, ?)') db().prepare('INSERT INTO beta_test_runs (id, beta_id, server, phase, created_at, metrics) VALUES (?, ?, ?, ?, ?, ?)')
.run(id, betaId, server, phase, nowIso(), JSON.stringify(emptyMetrics()), JSON.stringify(cursors)); .run(randomUUID(), betaId, server, phase, nowIso(), JSON.stringify(emptyMetrics()));
})(); })();
return currentRun(betaId, server)!; return currentRun(betaId, server)!;
} }
export function acceptTestSignal(server: string, runId: string, eventId: string, signal: TestSignal): { accepted: boolean; reason?: string } { /** Route ordinary server events to every deployed, active beta automatically. */
return db().transaction(() => { export function recordGameEvent(event: GameTelemetry): void {
const run = db().prepare(`SELECT r.* FROM beta_test_runs r JOIN betas b ON b.id = r.beta_id const runs = db().prepare(`SELECT r.* FROM beta_test_runs r JOIN betas b ON b.id = r.beta_id
JOIN beta_test_plans p ON p.beta_id = b.id WHERE r.id = ? AND r.server = ? AND r.retired_at IS NULL JOIN beta_test_plans p ON p.beta_id = b.id
AND b.state IN ('running','ready-promote') AND p.outcome IN ('active','passed')`).get(runId, server) as TestRun | undefined; JOIN beta_servers s ON s.beta_id = b.id AND s.server = r.server
if (!run) return { accepted: false, reason: 'Run is inactive or superseded; obtain the current run at process startup' }; WHERE r.server = ? AND r.retired_at IS NULL AND b.state IN ('running','ready-promote')
if (db().prepare('SELECT 1 FROM beta_test_evidence WHERE run_id = ? AND event_id = ?').get(runId, `telemetry:${eventId}`)) return { accepted: false, reason: 'Duplicate event' }; AND p.outcome IN ('active','passed') AND s.stage IN ('awaiting-health','observing','ready')
AND s.restarted_at IS NOT NULL`).all(event.server) as TestRun[];
for (const run of runs) {
const plan = parseBetaPlan(JSON.parse(storedPlan(run.beta_id)!.plan)); const plan = parseBetaPlan(JSON.parse(storedPlan(run.beta_id)!.plan));
const metrics = readMetrics(run); const metrics = readMetrics(run);
if (!applySignal(plan, metrics, signal, Date.now())) return { accepted: false, reason: 'Duplicate, out-of-order, closed, or unknown session/boot' }; applyGameEvent(plan, metrics, event, Date.parse(event.receivedAt));
evidence(run, `telemetry:${eventId}`, signal.event, signal); evidence(run, randomUUID(), event.event, event);
saveMetrics(run, metrics); saveMetrics(run, metrics);
return { accepted: true };
})();
}
export function parseTestSignal(body: Record<string, unknown>): TestSignal {
if (!['boot', 'session_start', 'round_change', 'match_end', 'feature'].includes(String(body.event))) throw new Error('Unknown telemetry event');
for (const key of ['runId', 'eventId', 'bootId'] as const) {
if (typeof body[key] !== 'string' || !/^[a-zA-Z0-9_.:-]{1,128}$/.test(body[key] as string)) throw new Error(`${key} is required (1–128 identifier characters)`);
} }
if (body.event !== 'boot' && (typeof body.sessionId !== 'string' || !/^[a-zA-Z0-9_.:-]{1,128}$/.test(body.sessionId))) throw new Error('sessionId is required');
for (const key of ['sequence', 'round', 'players']) if (!Number.isSafeInteger(body[key]) || (body[key] as number) < 0) throw new Error(`${key} must be a nonnegative integer`);
if (body.event === 'feature' && (typeof body.name !== 'string' || !/^[a-zA-Z0-9_.:-]{1,120}$/.test(body.name))) throw new Error('Feature name is required');
return { event: body.event as TestSignal['event'], bootId: body.bootId as string, sessionId: body.sessionId as string ?? '',
sequence: body.sequence as number, round: body.round as number, players: body.players as number, name: body.name as string ?? '' };
} }
/** Bounded incremental reads, with persistent byte cursors and partial lines. */
export async function observeRun(run: TestRun, plan: BetaPlan): Promise<void> { export async function observeRun(run: TestRun, plan: BetaPlan): Promise<void> {
const cursors = JSON.parse(run.cursors) as Record<string, Cursor>; const m = readMetrics(run);
const observed: Array<{ key: LogKey; line: string; id: string }> = []; const server = db().prepare('SELECT last_checked_at, last_seen_at, last_error, container_state FROM servers WHERE name = ?').get(run.server) as {
let logAvailable = false; last_checked_at: string | null; last_seen_at: string | null; last_error: string | null; container_state: string | null };
for (const key of [gameLogKey(run.server), 'console'] as LogKey[]) {
const path = resolveLogPath(run.server, key);
const file = path ? await open(path, 'r').catch(() => null) : null;
if (!file) continue;
try {
const info = await file.stat();
if (key === 'console') logAvailable = true;
const identity = `${info.dev}:${info.ino}`;
let cursor = cursors[key] ?? { position: 0, identity, partial: '' };
let anchorChanged = false;
if (cursor.anchor && info.size >= cursor.position) {
const expected = Buffer.from(cursor.anchor, 'base64');
const actual = Buffer.alloc(expected.length);
await file.read(actual, 0, actual.length, cursor.position - actual.length);
anchorChanged = !actual.equals(expected);
}
if (cursor.identity !== identity || info.size < cursor.position || anchorChanged) {
cursor = { position: 0, identity, partial: '', generation: randomUUID() };
observed.push({ key, line: '', id: `rotation:${key}:${randomUUID()}` });
}
if (info.size - cursor.position > 2_000_000) {
setOutcome(run.beta_id, 'inconclusive', `${run.server}: log reader fell behind; evidence is incomplete`);
return;
}
const buffer = Buffer.alloc(info.size - cursor.position);
const { bytesRead } = await file.read(buffer, 0, buffer.length, cursor.position);
const combined = Buffer.concat([Buffer.from(cursor.partial, 'base64'), buffer.subarray(0, bytesRead)]);
let start = 0;
for (let i = 0; i < combined.length; i++) if (combined[i] === 10) {
observed.push({ key, line: combined.subarray(start, i).toString('utf8'), id: `${key}:${identity}:${cursor.generation ?? ''}:${cursor.position + i - Buffer.from(cursor.partial, 'base64').length}` });
start = i + 1;
}
cursor.position += bytesRead;
const anchor = Buffer.alloc(Math.min(cursor.position, 128));
await file.read(anchor, 0, anchor.length, cursor.position - anchor.length);
cursor.anchor = anchor.toString('base64');
cursor.partial = combined.subarray(start).toString('base64');
if (cursor.partial.length > 100_000) cursor.partial = ''; // malformed giant line cannot grow forever
cursor.identity = identity;
cursors[key] = cursor;
} finally { await file.close(); }
}
db().transaction(() => {
const fresh = currentRun(run.beta_id, run.server);
if (!fresh || fresh.id !== run.id) return;
const m = readMetrics(fresh);
m.logAvailable = logAvailable;
for (const entry of observed) {
const cursor = cursors[entry.key];
if (!entry.line) { evidence(run, entry.id, 'log-rotation', { log: entry.key }); continue; }
if (entry.key === 'console') {
if (/script (?:runtime )?error|unhandled exception|script exception|fatal error/i.test(entry.line) && evidence(run, entry.id, 'script-error', { line: entry.line.slice(0, 1000) })) m.logErrors++;
continue;
}
if (/InitGame:/.test(entry.line)) { cursor.session = entry.id; cursor.joined = false; cursor.started = undefined; }
if (cursor.session && /\sJ;[^;]*;\d+;/.test(entry.line) && !cursor.joined) { cursor.joined = true; cursor.started = Date.now(); }
if (/ExitLevel:\s*executed/.test(entry.line) && cursor.session && cursor.joined) {
// Unqualified match counts use logs. Round/population-qualified counts use session telemetry exclusively.
if (evidence(run, `match:${cursor.session}`, 'completed-match', { session: cursor.session, line: entry.line, minutes: (Date.now() - (cursor.started ?? Date.now())) / 60_000 })) {
for (const r of plan.requirements) if (r.kind === 'matches' && r.minRound === 0 && r.minPlayers <= 1
&& Date.now() - (cursor.started ?? Date.now()) >= r.minMinutes * 60_000) m.values[r.id] = (m.values[r.id] ?? 0) + 1;
}
cursor.joined = false;
}
if (/ShutdownGame:/.test(entry.line)) cursor.joined = false;
}
const server = db().prepare('SELECT last_checked_at, last_seen_at, last_players, last_error, container_state FROM servers WHERE name = ?').get(run.server) as {
last_checked_at: string | null; last_seen_at: string | null; last_players: number | null; last_error: string | null; container_state: string | null };
const sample = server.last_checked_at ? Date.parse(server.last_checked_at) : 0; const sample = server.last_checked_at ? Date.parse(server.last_checked_at) : 0;
const freshSample = sample >= Date.parse(run.observing_at ?? run.created_at) && Date.now() - sample <= 60_000; const freshSample = sample >= Date.parse(run.observing_at ?? run.created_at) && Date.now() - sample <= 60_000;
const healthy = freshSample && !server.last_error && server.container_state === 'running' && server.last_seen_at === server.last_checked_at; const healthy = freshSample && !server.last_error && server.container_state === 'running' && server.last_seen_at === server.last_checked_at;
const at = freshSample && server.container_state === 'running' ? sample : Date.now(); const at = freshSample && server.container_state === 'running' ? sample : Date.now();
// Startup requires fresh status; don't count an absent first observation as an outage.
const before = m.wasHealthy; const before = m.wasHealthy;
const lastSample = m.lastSample; const lastSample = m.lastSample;
applySample(plan, m, at, healthy, healthy ? server.last_players ?? 0 : 0); applySample(plan, m, at, healthy);
if (freshSample && sample !== lastSample) evidence(run, `sample:${sample}`, 'status-sample', { healthy, players: server.last_players, checkedAt: server.last_checked_at }); if (freshSample && sample !== lastSample) evidence(run, `sample:${sample}`, 'status-sample', { healthy, checkedAt: server.last_checked_at });
if (before !== m.wasHealthy) evidence(run, `health:${at}`, 'health', { healthy, players: server.last_players, reason: server.last_error }); if (before !== m.wasHealthy) evidence(run, `health:${at}`, 'health', { healthy, reason: server.last_error });
saveMetrics(run, m); saveMetrics(run, m);
db().prepare('UPDATE beta_test_runs SET cursors = ? WHERE id = ?').run(JSON.stringify(cursors), run.id);
})();
} }
export function testView(betaId: number): BetaTestView | null { export function testView(betaId: number): BetaTestView | null {
@@ -175,7 +74,7 @@ export function testView(betaId: number): BetaTestView | null {
return { plan, outcome: stored.outcome, reason: stored.reason, return { plan, outcome: stored.outcome, reason: stored.reason,
servers: runs.map(run => { servers: runs.map(run => {
const progress = evaluateRequirements(plan, readMetrics(run), run.phase); const progress = evaluateRequirements(plan, readMetrics(run), run.phase);
return { server: run.server, runId: run.id, phase: run.phase, passed: progress.every(r => r.passed), progress }; return { server: run.server, phase: run.phase, passed: progress.every(r => r.passed), progress };
}), }),
evidence: db().prepare(`SELECT e.id, r.server, e.received_at, e.kind, e.payload FROM beta_test_evidence e evidence: db().prepare(`SELECT e.id, r.server, e.received_at, e.kind, e.payload FROM beta_test_evidence e
JOIN beta_test_runs r ON r.id = e.run_id WHERE r.beta_id = ? ORDER BY e.id DESC LIMIT 200`).all(betaId) as BetaTestView['evidence'] }; JOIN beta_test_runs r ON r.id = e.run_id WHERE r.beta_id = ? ORDER BY e.id DESC LIMIT 200`).all(betaId) as BetaTestView['evidence'] };
-110
View File
@@ -1,110 +0,0 @@
import { resolveLogPath, tailLog, gameLogKey } from './logs.js';
// Match markers for T5 Plutonium's `logs/games_sp.log`. Verified against a live
// sample: each game session is bracketed by `InitGame:` and `ShutdownGame:`.
// A REAL completed match always emits `ExitLevel: executed` right before
// `ShutdownGame:`. Idle map cycles print `InitGame:` / `ShutdownGame:` at
// 0:00 with no ExitLevel — those don't count as a played match.
//
// So the reliable discriminator is `ExitLevel: executed`: seeing that line
// after `healthy_at` means a game actually ended normally (no crash — a crash
// would kill the container before ExitLevel fires and take RCON with it).
export const MATCH_START = /InitGame:/;
export const MATCH_END = /ExitLevel:\s*executed/;
export const MATCH_SHUTDOWN = /ShutdownGame:/;
// Player join line: `MM:SS J;<guid>;<slot>;<name>`. games_sp.log accumulates
// across restarts and contains many old J; lines; only the ones AFTER the
// latest InitGame belong to the current match cycle.
export const MATCH_JOIN = /\sJ;[^;]*;\d+;/;
/**
* Poll a server's games_sp.log tail for match markers. Returns whether a
* completed match end has been observed since `sinceIso` (the server's
* `healthy_at` timestamp).
*/
export async function scanForMatchEnd(
serverName: string,
sinceIso: string
): Promise<{ ended: boolean; started: boolean; reason?: string }> {
const path = resolveLogPath(serverName, gameLogKey(serverName));
if (!path) return { ended: false, started: false, reason: 'log path resolution failed' };
const tail = await tailLog(path, 2000);
if (!tail) return { ended: false, started: false, reason: 'log not written yet' };
if (tail.mtime < sinceIso) return { ended: false, started: false, reason: 'no log activity since healthy' };
let started = false;
let ended = false;
for (const line of tail.lines) {
if (MATCH_START.test(line)) started = true;
if (MATCH_END.test(line)) ended = true;
}
return { ended, started };
}
export type MatchScanKind = 'no-signal' | 'idle' | 'in-progress' | 'ended';
export interface MatchScanResult {
kind: MatchScanKind;
// Opaque identifier for the currently-observed match cycle. When kind is
// 'in-progress' or 'ended', this is the first J; line after the latest
// InitGame. Compare against persisted state to detect a fresh cycle
// (e.g. game auto-restart with same players — new J; line = new key).
cycleKey?: string;
}
/**
* Read the games_sp.log tail and classify the current match state. We do NOT
* infer any wall-clock timestamps from log content — the caller stamps
* game_started_at/game_ended_at with the observation time.
*
* - `in-progress`: latest InitGame has at least one J; join line after it
* and no ExitLevel/ShutdownGame between that J; and end-of-tail. `cycleKey`
* is set to the first J; line after the latest InitGame.
* - `ended`: latest InitGame has a J; join line followed by an ExitLevel
* or ShutdownGame. `cycleKey` identifies which cycle ended.
* - `idle`: no J; line after the latest InitGame — empty map loaded, no
* match happening.
* - `no-signal`: log missing / unreadable / no InitGame in tail.
*/
export async function scanMatchState(serverName: string): Promise<MatchScanResult> {
const path = resolveLogPath(serverName, gameLogKey(serverName));
if (!path) return { kind: 'no-signal' };
const tail = await tailLog(path, 2000);
if (!tail || tail.lines.length === 0) return { kind: 'no-signal' };
const lines = tail.lines;
// Find the LATEST InitGame — everything before it is prior sessions that
// this log accumulated. Only what comes after is the current cycle.
let lastInit = -1;
for (let i = lines.length - 1; i >= 0; i--) {
if (MATCH_START.test(lines[i])) {
lastInit = i;
break;
}
}
if (lastInit === -1) return { kind: 'no-signal' };
// Scan only the current cycle (everything after the latest InitGame).
let firstJoinIdx = -1;
let endIdx = -1;
for (let i = lastInit + 1; i < lines.length; i++) {
if (firstJoinIdx === -1 && MATCH_JOIN.test(lines[i])) {
firstJoinIdx = i;
}
if (MATCH_END.test(lines[i]) || MATCH_SHUTDOWN.test(lines[i])) {
endIdx = i;
break;
}
}
// No join in the current cycle -> nobody entered this match yet.
if (firstJoinIdx === -1) return { kind: 'idle' };
const cycleKey = lines[firstJoinIdx];
if (endIdx !== -1 && endIdx > firstJoinIdx) {
return { kind: 'ended', cycleKey };
}
return { kind: 'in-progress', cycleKey };
}
+19 -10
View File
@@ -1,15 +1,24 @@
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import type { RoundTelemetry } from '../../telemetry.js'; import type { GameTelemetry } from '../../telemetry.js';
export function listTelemetry(): RoundTelemetry[] { export function listTelemetry(): GameTelemetry[] {
return db().prepare(`SELECT t.server, t.round, t.players, t.received_at AS receivedAt const rows = db().prepare(`SELECT t.payload FROM server_telemetry t JOIN servers s ON s.name = t.server
FROM server_telemetry t JOIN servers s ON s.name = t.server WHERE s.destroyed_at IS NULL AND t.payload IS NOT NULL`).all() as Array<{ payload: string }>;
WHERE s.destroyed_at IS NULL`).all() as RoundTelemetry[]; return rows.map(row => JSON.parse(row.payload) as GameTelemetry);
} }
export function saveTelemetry(report: RoundTelemetry): void { export function saveTelemetry(report: GameTelemetry): void {
db().prepare(`INSERT INTO server_telemetry (server, round, players, received_at) db().prepare(`INSERT INTO server_telemetry (server, round, players, received_at, payload)
VALUES (?, ?, ?, ?) ON CONFLICT(server) DO UPDATE SET VALUES (?, ?, ?, ?, ?) ON CONFLICT(server) DO UPDATE SET
round = excluded.round, players = excluded.players, received_at = excluded.received_at`) round = excluded.round, players = excluded.players, received_at = excluded.received_at, payload = excluded.payload`)
.run(report.server, report.round, report.players, report.receivedAt); .run(report.server, report.event === 'game_start' ? 0 : report.round, report.players.length, report.receivedAt, JSON.stringify(report));
if (report.event === 'game_start') {
db().prepare('UPDATE servers SET game_started_at = ?, game_ended_at = NULL WHERE name = ?').run(report.receivedAt, report.server);
} else if (report.event === 'game_end') {
db().prepare('UPDATE servers SET game_ended_at = ? WHERE name = ?').run(report.receivedAt, report.server);
// Legacy betas also take match completion directly from the script.
db().prepare(`UPDATE beta_servers SET match_seen_at = ? WHERE server = ? AND stage = 'awaiting-match'
AND healthy_at <= ? AND beta_id IN (SELECT id FROM betas WHERE state = 'running')`)
.run(report.receivedAt, report.server, report.receivedAt);
}
} }
+1 -1
View File
@@ -112,7 +112,7 @@ async function advancePlan(stored: StoredPlan): Promise<void> {
if (oldRun) { if (oldRun) {
const previous = readMetrics(oldRun); const previous = readMetrics(oldRun);
const metrics = readMetrics(run); const metrics = readMetrics(run);
metrics.outages = previous.outages; metrics.logErrors = previous.logErrors; metrics.restarts = previous.restarts; metrics.outages = previous.outages;
saveMetrics(run, metrics); saveMetrics(run, metrics);
} }
const restart = await restartInRollout(server.server); const restart = await restartInRollout(server.server);
+3 -5
View File
@@ -3,7 +3,6 @@ import { emit } from './bus.js';
import type { BetaStage } from './bus.js'; import type { BetaStage } from './bus.js';
import { komodoConfigured, rconConfigured } from '../config.js'; import { komodoConfigured, rconConfigured } from '../config.js';
import { rconGetPlayerCount } from '../integrations/rcon.js'; import { rconGetPlayerCount } from '../integrations/rcon.js';
import { scanForMatchEnd } from '../fleet/match-log.js';
import { refreshRestartRollout, restartInRollout, rolloutHealth, rolloutWaitReason } from './restart-rollout.js'; import { refreshRestartRollout, restartInRollout, rolloutHealth, rolloutWaitReason } from './restart-rollout.js';
import { plannedBetaTick } from './beta-plan-runner.js'; import { plannedBetaTick } from './beta-plan-runner.js';
@@ -12,6 +11,7 @@ interface BetaServerActiveRow {
server: string; server: string;
stage: BetaStage; stage: BetaStage;
healthy_at: string | null; healthy_at: string | null;
match_seen_at: string | null;
wait_for_health: number; wait_for_health: number;
wait_for_match: number; wait_for_match: number;
} }
@@ -101,7 +101,7 @@ async function runBetaTick(): Promise<{ processed: number }> {
const rows = db() const rows = db()
.prepare( .prepare(
`SELECT bs.beta_id AS beta_id, bs.server AS server, bs.stage AS stage, `SELECT bs.beta_id AS beta_id, bs.server AS server, bs.stage AS stage,
bs.healthy_at AS healthy_at, bs.healthy_at AS healthy_at, bs.match_seen_at AS match_seen_at,
b.wait_for_health AS wait_for_health, b.wait_for_match AS wait_for_match b.wait_for_health AS wait_for_health, b.wait_for_match AS wait_for_match
FROM beta_servers bs FROM beta_servers bs
JOIN betas b ON b.id = bs.beta_id JOIN betas b ON b.id = bs.beta_id
@@ -181,9 +181,7 @@ async function advance(row: BetaServerActiveRow): Promise<void> {
return; return;
} }
case 'awaiting-match': { case 'awaiting-match': {
const since = row.healthy_at ?? nowIso(); if (!row.match_seen_at) return;
const result = await scanForMatchEnd(row.server, since);
if (!result.ended) return;
transition(row.beta_id, row.server, 'ready'); transition(row.beta_id, row.server, 'ready');
return; return;
} }
+2 -2
View File
@@ -1,5 +1,5 @@
import { EventEmitter } from 'node:events'; import { EventEmitter } from 'node:events';
import type { RoundTelemetry } from '../../telemetry.js'; import type { GameTelemetry } from '../../telemetry.js';
import type { RconEndpoint, RconFailure, RconPlayer } from '../integrations/rcon.js'; import type { RconEndpoint, RconFailure, RconPlayer } from '../integrations/rcon.js';
import type { Health } from '../fleet/health.js'; import type { Health } from '../fleet/health.js';
@@ -40,7 +40,7 @@ export type BetaStage =
export type BetaState = 'running' | 'ready-promote' | 'promoted' | 'rolled-back'; export type BetaState = 'running' | 'ready-promote' | 'promoted' | 'rolled-back';
export type PipelineEvent = export type PipelineEvent =
| { type: 'server-telemetry'; entries: RoundTelemetry[] } | { type: 'server-telemetry'; entries: GameTelemetry[] }
| { type: 'deployment-opened'; deploymentId: string; action: string; servers: string[] } | { type: 'deployment-opened'; deploymentId: string; action: string; servers: string[] }
| { type: 'server-changed'; deploymentId: string; server: string; files: number } | { type: 'server-changed'; deploymentId: string; server: string; files: number }
| { | {
+2 -1
View File
@@ -56,7 +56,8 @@ export async function restartInRollout(server: string): Promise<RolloutRestartRe
if (waitReason) return { kind: 'queued', message: waitReason }; if (waitReason) return { kind: 'queued', message: waitReason };
requestsInFlight.add(server); requestsInFlight.add(server);
db().prepare("UPDATE servers SET container_state = 'restarting', last_error = NULL, last_reply_snippet = NULL WHERE name = ?").run(server); db().prepare(`UPDATE servers SET container_state = 'restarting', last_error = NULL, last_reply_snippet = NULL,
game_started_at = NULL, game_ended_at = NULL WHERE name = ?`).run(server);
try { try {
const result = await komodoRestart(server); const result = await komodoRestart(server);
if (result.ok) return { kind: 'started' }; if (result.ok) return { kind: 'started' };
+2 -66
View File
@@ -10,7 +10,6 @@ import { komodoInspectHealthy } from '../integrations/komodo.js';
import { restartSettling } from './restart-rollout.js'; import { restartSettling } from './restart-rollout.js';
import { classifyHealth } from '../fleet/health.js'; import { classifyHealth } from '../fleet/health.js';
import { STATUS_REFRESH_MS } from '../../server-freshness.js'; import { STATUS_REFRESH_MS } from '../../server-freshness.js';
import { scanMatchState } from '../fleet/match-log.js';
const TICK_MS = 15_000; const TICK_MS = 15_000;
// Cap concurrent UDP RCON calls so a 15-server fleet doesn't blast the // Cap concurrent UDP RCON calls so a 15-server fleet doesn't blast the
@@ -44,61 +43,6 @@ async function runTick(): Promise<void> {
} catch (err) { } catch (err) {
console.error('[worker] beta-runner error:', err); console.error('[worker] beta-runner error:', err);
} }
try {
await scanMatches();
} catch (err) {
console.error('[worker] match-scan error:', err);
}
}
/**
* Read every server's games_sp.log tail to advance game_started_at /
* game_ended_at. Concurrency-capped so 15 servers don't hammer disk at once.
*/
async function scanMatches(): Promise<void> {
const servers = listServers();
if (servers.length === 0) return;
let cursor = 0;
async function pump(): Promise<void> {
while (true) {
const i = cursor++;
if (i >= servers.length) return;
const srv = servers[i];
const result = await scanMatchState(srv.name).catch((err) => {
console.error(`[worker] scanMatchState(${srv.name}) failed:`, err);
return { kind: 'no-signal' as const };
});
const persistedInProgress = !!srv.game_started_at && !srv.game_ended_at;
const persistedKey = srv.game_cycle_key;
const now = new Date().toISOString();
if (result.kind === 'in-progress' && result.cycleKey) {
// New cycle when persisted key differs (fresh start, or a game
// restart with the same players — new J; line = new key).
if (result.cycleKey !== persistedKey) {
setServerField(srv.name, 'game_started_at', now);
setServerField(srv.name, 'game_ended_at', null);
setServerField(srv.name, 'game_cycle_key', result.cycleKey);
} else if (!persistedInProgress) {
// Same key but previous cycle was ended — treat as new run
// (shouldn't happen in practice since key would differ,
// but be defensive).
setServerField(srv.name, 'game_started_at', now);
setServerField(srv.name, 'game_ended_at', null);
}
} else if (result.kind === 'ended' && result.cycleKey) {
// Only stamp the end if it's the cycle we've been tracking and
// we haven't already stamped it.
if (result.cycleKey === persistedKey && persistedInProgress) {
setServerField(srv.name, 'game_ended_at', now);
}
}
// 'idle' / 'no-signal': leave state alone. UI hides the timer when
// the server has no players (see ServerCard/detail page).
}
}
await Promise.all(Array.from({ length: STATUS_CONCURRENCY }, pump));
} }
/** /**
@@ -169,8 +113,8 @@ async function refreshStatus(): Promise<void> {
const nextPlayers = result.status.players; const nextPlayers = result.status.players;
let activeSince = srv.active_since; let activeSince = srv.active_since;
let inactiveSince = srv.inactive_since; let inactiveSince = srv.inactive_since;
let gameStartedAt = srv.game_started_at; const gameStartedAt = srv.game_started_at;
let gameEndedAt = srv.game_ended_at; const gameEndedAt = srv.game_ended_at;
if (nextPlayers > 0 && (activeSince === null || prevPlayers === 0 || prevPlayers === null)) { if (nextPlayers > 0 && (activeSince === null || prevPlayers === 0 || prevPlayers === null)) {
activeSince = now; activeSince = now;
inactiveSince = null; inactiveSince = null;
@@ -181,14 +125,6 @@ async function refreshStatus(): Promise<void> {
activeSince = null; activeSince = null;
setServerField(srv.name, 'inactive_since', inactiveSince); setServerField(srv.name, 'inactive_since', inactiveSince);
setServerField(srv.name, 'active_since', null); setServerField(srv.name, 'active_since', null);
// Game timer cannot run while inactive. Drop any tracked
// match state so the next active session anchors fresh
// when a new J; line fires.
gameStartedAt = null;
gameEndedAt = null;
setServerField(srv.name, 'game_started_at', null);
setServerField(srv.name, 'game_ended_at', null);
setServerField(srv.name, 'game_cycle_key', null);
} }
// Persist last-observed state. last_seen_at is the health anchor. // Persist last-observed state. last_seen_at is the health anchor.
setServerField(srv.name, 'last_players', nextPlayers); setServerField(srv.name, 'last_players', nextPlayers);
+48 -5
View File
@@ -1,6 +1,49 @@
export interface RoundTelemetry { export interface TelemetryPlayer {
server: string; name: string;
round: number; guid: string;
players: number; }
receivedAt: string;
export interface TelemetryPlayerStats extends TelemetryPlayer {
kills: number;
headshots: number;
downs: number;
revives: number;
score: number;
}
export type GameEvent = { server: string; map: string } & (
| { event: 'game_start'; players: TelemetryPlayer[] }
| { event: 'round_change' | 'game_end'; round: number; players: TelemetryPlayerStats[] }
);
export type GameTelemetry = GameEvent & { receivedAt: string };
/** Validate the three script payloads; beta state is never part of the protocol. */
export function parseGameEvent(value: unknown): GameEvent {
if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error('Payload must be a JSON object');
const body = value as Record<string, unknown>;
if (!['game_start', 'round_change', 'game_end'].includes(String(body.event))) throw new Error('event must be game_start, round_change, or game_end');
for (const key of ['server', 'map'] as const) {
if (typeof body[key] !== 'string' || !body[key].trim() || body[key].length > 255) throw new Error(`${key} must be a nonempty string (up to 255 characters)`);
}
if (!Array.isArray(body.players)) throw new Error('players must be an array');
const players = body.players.map((value, index) => {
if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error(`players[${index}] must be an object`);
const player = value as Record<string, unknown>;
for (const key of ['name', 'guid'] as const) {
if (typeof player[key] !== 'string' || !player[key].trim()) throw new Error(`players[${index}].${key} must be a nonempty string`);
}
const identity = { name: player.name as string, guid: player.guid as string };
if (body.event === 'game_start') return identity;
const stats = {} as Omit<TelemetryPlayerStats, keyof TelemetryPlayer>;
for (const key of ['kills', 'headshots', 'downs', 'revives', 'score'] as const) {
if (!Number.isSafeInteger(player[key]) || (player[key] as number) < 0) throw new Error(`players[${index}].${key} must be a nonnegative integer`);
stats[key] = player[key] as number;
}
return { ...identity, ...stats };
});
const common = { server: body.server as string, map: body.map as string };
if (body.event === 'game_start') return { ...common, event: 'game_start', players };
if (!Number.isSafeInteger(body.round) || (body.round as number) < 0) throw new Error('round must be a nonnegative integer');
return { ...common, event: body.event as 'round_change' | 'game_end', round: body.round as number, players: players as TelemetryPlayerStats[] };
} }
+11 -16
View File
@@ -2,7 +2,7 @@
import { onMount } from 'svelte'; import { onMount } from 'svelte';
import Button from '$ui/components/primitives/Button.svelte'; import Button from '$ui/components/primitives/Button.svelte';
import { builtInPlans, requirement, requirementKinds, requirementLabels, parseBetaPlan, type BetaPlan, type RequirementKind } from '$lib/beta-plan.js'; import { builtInPlans, requirement, requirementKinds, requirementLabels, parseBetaPlan, type BetaPlan, type RequirementKind } from '$lib/beta-plan.js';
import { coverageSummary, needsSessionTelemetry, presetDescriptions, requirementCategory, requirementSummary } from '$lib/beta-plan-summary.js'; import { coverageSummary, needsGameTelemetry, presetDescriptions, requirementCategory, requirementSummary } from '$lib/beta-plan-summary.js';
let { plan = $bindable<BetaPlan>() }: { plan: BetaPlan } = $props(); let { plan = $bindable<BetaPlan>() }: { plan: BetaPlan } = $props();
let presets = $state(builtInPlans); let presets = $state(builtInPlans);
let presetName = $state(''); let presetName = $state('');
@@ -102,8 +102,8 @@
<div><dt>If a check fails</dt><dd>{failureSummary}</dd></div> <div><dt>If a check fails</dt><dd>{failureSummary}</dd></div>
</dl> </dl>
<p class="plan-caption">Every requirement below must pass on each qualifying server. You can edit any check.</p> <p class="plan-caption">Every requirement below must pass on each qualifying server. You can edit any check.</p>
{#if plan.requirements.some(needsSessionTelemetry)} {#if plan.requirements.some(needsGameTelemetry)}
<p class="telemetry-note"><span aria-hidden="true">ⓘ</span> This plan needs session telemetry from the game reporter.</p> <p class="telemetry-note"><span aria-hidden="true">ⓘ</span> Gameplay progress comes from game start, round change, and game end reports.</p>
{/if} {/if}
<div class="requirements"> <div class="requirements">
@@ -120,31 +120,26 @@
<div class="fields"> <div class="fields">
<label>Description<input bind:value={r.label} maxlength="200" required /></label> <label>Description<input bind:value={r.label} maxlength="200" required /></label>
{#if r.kind !== 'manual'} {#if r.kind !== 'manual'}
<label>{['failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind) ? 'Maximum allowed' : 'Target'} <label>{r.kind === 'failure-budget' ? 'Maximum allowed' : 'Target'}
<input type="number" min={['failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind) ? 0 : ['healthy-minutes', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind) ? 0.01 : 1} step={['healthy-minutes', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind) ? 'any' : 1} bind:value={r.target} required /> <input type="number" min={r.kind === 'failure-budget' ? 0 : ['healthy-minutes', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind) ? 0.01 : 1} step={['healthy-minutes', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind) ? 'any' : 1} bind:value={r.target} required />
</label> </label>
{/if} {/if}
{#if ['round', 'matches', 'populated-minutes', 'player-hours', 'sustained-population', 'feature'].includes(r.kind)} {#if ['round', 'matches', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind)}
<label>Minimum players<input type="number" min="1" step="1" bind:value={r.minPlayers} required /></label> <label>Minimum players<input type="number" min="1" step="1" bind:value={r.minPlayers} required /></label>
{/if} {/if}
{#if r.kind === 'matches' || r.kind === 'feature'} {#if r.kind === 'matches'}
<label>Minimum round in the same session<input type="number" min="0" step="1" bind:value={r.minRound} required /></label> <label>Minimum round in the same game<input type="number" min="0" step="1" bind:value={r.minRound} required /></label>
{/if} {/if}
{#if r.kind === 'matches'} {#if r.kind === 'matches'}
<label>Minimum match duration (minutes)<input type="number" min="0" step="any" bind:value={r.minMinutes} required /></label> <label>Minimum match duration (minutes)<input type="number" min="0" step="any" bind:value={r.minMinutes} required /></label>
{/if} {/if}
{#if r.kind === 'feature'}
<label>Reported event name<input bind:value={r.event} placeholder="perk.purchased" maxlength="120" required /></label>
{/if}
</div> </div>
{#if r.kind === 'round' || r.kind === 'feature' || r.kind === 'restart-budget' || (r.kind === 'matches' && (r.minRound > 0 || r.minPlayers > 1))} {#if needsGameTelemetry(r)}
<p>Requires session telemetry from the game reporter. Progress waits for events from this beta’s current run.</p> <p>Uses game events reported by this server automatically.</p>
{:else if r.kind === 'recovery'} {:else if r.kind === 'recovery'}
<p>After the other requirements pass, restart when empty and repeat them. Every cycle requires fresh evidence and checklist sign-off.</p> <p>After the other requirements pass, restart when empty and repeat them. Every cycle requires fresh evidence and checklist sign-off.</p>
{:else if r.kind === 'failure-budget'} {:else if r.kind === 'failure-budget'}
<p>Counts transitions from healthy to unhealthy. Missing or stale monitoring resets continuous health and does not earn playtime.</p> <p>Counts transitions from healthy to unhealthy. Missing or stale monitoring resets continuous healthy time.</p>
{:else if r.kind === 'clean-logs'}
<p>Checks new console lines for script errors, script exceptions, unhandled exceptions and fatal errors. An unreadable log cannot pass.</p>
{:else if r.kind === 'manual'} {:else if r.kind === 'manual'}
<p>A tester records their name and observation on the beta detail page.</p> <p>A tester records their name and observation on the beta detail page.</p>
{/if} {/if}
@@ -44,10 +44,10 @@
{@const requirement = test.plan.requirements.find(r => r.id === progress.id)!} {@const requirement = test.plan.requirements.find(r => r.id === progress.id)!}
<div class="requirement" class:failed={progress.failed}> <div class="requirement" class:failed={progress.failed}>
<div><strong>{progress.passed ? '✓' : progress.failed ? '✕' : '○'} {progress.label}</strong><span>{progress.detail}</span></div> <div><strong>{progress.passed ? '✓' : progress.failed ? '✕' : '○'} {progress.label}</strong><span>{progress.detail}</span></div>
{#if ['round', 'matches', 'feature', 'populated-minutes', 'player-hours', 'sustained-population'].includes(requirement.kind)} {#if ['round', 'matches', 'populated-minutes', 'player-hours', 'sustained-population'].includes(requirement.kind)}
<span>Minimum {requirement.minPlayers} player(s){requirement.minRound ? `, round ${requirement.minRound} in the same session` : ''}{requirement.minMinutes ? `, ${requirement.minMinutes} minutes per match` : ''}{requirement.event ? ` · Event: ${requirement.event}` : ''}</span> <span>Minimum {requirement.minPlayers} player(s){requirement.minRound ? `, round ${requirement.minRound} in the same game` : ''}{requirement.minMinutes ? `, ${requirement.minMinutes} minutes per match` : ''}</span>
{/if} {/if}
{#if !['manual', 'failure-budget', 'restart-budget', 'clean-logs'].includes(requirement.kind)} {#if !['manual', 'failure-budget'].includes(requirement.kind)}
<progress aria-label={progress.label} value={Math.min(progress.current, progress.target)} max={progress.target}></progress> <progress aria-label={progress.label} value={Math.min(progress.current, progress.target)} max={progress.target}></progress>
{/if} {/if}
{#if requirement.kind === 'manual' && !progress.passed && !closed} {#if requirement.kind === 'manual' && !progress.passed && !closed}
+2 -2
View File
@@ -8,10 +8,10 @@
&& (!since || Date.parse(report.receivedAt) >= Date.parse(since))); && (!since || Date.parse(report.receivedAt) >= Date.parse(since)));
</script> </script>
{#if report && fresh} {#if report && report.event === 'round_change' && fresh}
<div class="round" aria-live="polite"> <div class="round" aria-live="polite">
<strong>Round {report.round}</strong> <strong>Round {report.round}</strong>
<span title={report.receivedAt}>Last reported {elapsedSince(report.receivedAt, $uiNow)} ago · {report.players} players</span> <span title={report.receivedAt}>Last reported {elapsedSince(report.receivedAt, $uiNow)} ago · {report.players.length} players</span>
</div> </div>
{/if} {/if}
+4 -4
View File
@@ -1,8 +1,8 @@
import { writable, readonly } from 'svelte/store'; import { writable, readonly } from 'svelte/store';
import { browser } from '$app/environment'; import { browser } from '$app/environment';
import type { RoundTelemetry } from '$lib/telemetry.js'; import type { GameTelemetry } from '$lib/telemetry.js';
export const telemetryByServer = writable<Map<string, RoundTelemetry>>(new Map()); export const telemetryByServer = writable<Map<string, GameTelemetry>>(new Map());
export interface RosterPlayer { export interface RosterPlayer {
num: number | null; num: number | null;
@@ -54,9 +54,9 @@ export function startServerStream(): (() => void) | undefined {
if (!browser || source) return; if (!browser || source) return;
source = new EventSource('/api/events/servers'); source = new EventSource('/api/events/servers');
source.addEventListener('server-telemetry', (evt) => { source.addEventListener('server-telemetry', (evt) => {
const data = JSON.parse((evt as MessageEvent).data) as { entries: RoundTelemetry[]; snapshot?: boolean }; const data = JSON.parse((evt as MessageEvent).data) as { entries: GameTelemetry[]; snapshot?: boolean };
telemetryByServer.update((previous) => { telemetryByServer.update((previous) => {
const next = data.snapshot ? new Map<string, RoundTelemetry>() : new Map(previous); const next = data.snapshot ? new Map<string, GameTelemetry>() : new Map(previous);
for (const entry of data.entries) next.set(entry.server, entry); for (const entry of data.entries) next.set(entry.server, entry);
return next; return next;
}); });
+10
View File
@@ -33,6 +33,16 @@ export async function POST({ params, request }: { params: { id: string }; reques
betaState(id, false); betaState(id, false);
} else if (body.action === 'resume') { } else if (body.action === 'resume') {
if (stored.outcome !== 'paused') throw new Error('Only paused plans can resume; failed or expired plans need a fresh attempt'); if (stored.outcome !== 'paused') throw new Error('Only paused plans can resume; failed or expired plans need a fresh attempt');
const runs = db().prepare('SELECT * FROM beta_test_runs WHERE beta_id = ? AND retired_at IS NULL').all(id) as Parameters<typeof readMetrics>[0][];
for (const run of runs) {
const metrics = readMetrics(run);
delete metrics.match;
delete metrics.lastSample;
delete metrics.wasHealthy;
metrics.streaks = {};
for (const r of plan.requirements) if (r.kind === 'healthy-minutes') metrics.values[r.id] = 0;
saveMetrics(run, metrics);
}
setOutcome(id, 'active', null); setOutcome(id, 'active', null);
} else if (body.action === 'retry') { } else if (body.action === 'retry') {
// Fresh attempts keep their predecessors as audit history. Preserve the original deadline: // Fresh attempts keep their predecessors as audit history. Preserve the original deadline:
+17 -78
View File
@@ -1,88 +1,27 @@
import { json } from '@sveltejs/kit'; import { json } from '@sveltejs/kit';
import { getServer, listServers } from '$lib/server/fleet/servers.js'; import { parseGameEvent, type GameTelemetry } from '$lib/telemetry.js';
import { getServer } from '$lib/server/fleet/servers.js';
import { saveTelemetry } from '$lib/server/fleet/telemetry.js'; import { saveTelemetry } from '$lib/server/fleet/telemetry.js';
import { emit } from '$lib/server/pipeline/bus.js'; import { recordGameEvent } from '$lib/server/fleet/beta-tests.js';
import { db } from '$lib/server/db/index.js'; import { db } from '$lib/server/db/index.js';
import { acceptTestSignal, parseTestSignal } from '$lib/server/fleet/beta-tests.js'; import { emit } from '$lib/server/pipeline/bus.js';
// Reporters fetch this once at process startup and keep the token for that boot.
export async function GET({ url }: { url: URL }) {
const server = url.searchParams.get('server');
if (!server || !getServer(server)) return json({ error: 'Unknown server' }, { status: 404 });
const runs = db().prepare(`SELECT r.id AS runId, r.beta_id AS betaId FROM beta_test_runs r
JOIN betas b ON b.id = r.beta_id JOIN beta_test_plans p ON p.beta_id = b.id
WHERE r.server = ? AND r.retired_at IS NULL AND b.state IN ('running','ready-promote')
AND p.outcome IN ('active','passed')`).all(server);
return json({ runs }, { headers: { 'Cache-Control': 'no-store' } });
}
export async function POST({ request }: { request: Request }) { export async function POST({ request }: { request: Request }) {
console.log('[telemetry] received POST /api/telemetry');
// Encode payloads as JSON so script-supplied newlines cannot forge log lines.
// Bound the preview to keep malformed requests from flooding container logs.
const raw = await request.text();
const context = {
contentType: request.headers.get('content-type'),
payloadPreview: raw.slice(0, 4096),
payloadTruncated: raw.length > 4096
};
function reject(status: number, error: string, details: Record<string, unknown> = {}) {
console.log(`[telemetry] rejected ${status}: ${JSON.stringify({ error, ...context, ...details })}`);
return json({ ok: false, error, details }, { status });
}
if (request.headers.get('content-type')?.split(';')[0].trim().toLowerCase() !== 'application/json') { if (request.headers.get('content-type')?.split(';')[0].trim().toLowerCase() !== 'application/json') {
return reject(415, 'Content-Type must be application/json', { receivedContentType: context.contentType }); return json({ ok: false, error: 'Content-Type must be application/json' }, { status: 415 });
} }
let body; let value: unknown;
try { try { value = await request.json(); }
body = JSON.parse(raw); catch { return json({ ok: false, error: 'Malformed JSON' }, { status: 400 }); }
} catch (error) { let telemetry: GameTelemetry;
return reject(400, 'Malformed JSON', { parseError: (error as Error).message }); try { telemetry = { ...parseGameEvent(value), receivedAt: new Date().toISOString() }; }
} catch (error) { return json({ ok: false, error: (error as Error).message }, { status: 400 }); }
if (!body || typeof body !== 'object' || Array.isArray(body)) { const server = getServer(telemetry.server);
return reject(400, 'Payload must be a JSON object'); if (!server || server.destroyed_at) return json({ ok: false, error: 'Server is not registered or has been destroyed' }, { status: 404 });
} db().transaction(() => {
const issues: string[] = [];
if (body.runId === undefined && body.event !== 'round_change') issues.push('Session telemetry requires runId');
if (typeof body.server !== 'string' || !body.server.length || body.server.length > 255) {
issues.push('server must be a registered server name (1–255 characters)');
}
for (const field of ['round', 'players']) {
if (!Number.isSafeInteger(body[field]) || body[field] < 0) {
issues.push(`${field} must be a nonnegative safe integer (JSON number, not a string)`);
}
}
if (issues.length) {
return reject(400, 'Invalid telemetry payload', { issues });
}
const server = getServer(body.server);
if (!server) {
const registeredServers = listServers().filter((entry) => !entry.destroyed_at).map((entry) => entry.name);
return reject(404, 'Server is not registered', {
receivedServer: body.server,
hint: 'Send the exact Electric Cherry server name in the server field; matching is case-sensitive.',
registeredServers
});
}
if (server.destroyed_at) {
return reject(404, 'Server has been destroyed', {
receivedServer: body.server, destroyedAt: server.destroyed_at
});
}
let test: { accepted: boolean; reason?: string } | undefined;
if (body.runId !== undefined) {
try {
const signal = parseTestSignal(body);
test = acceptTestSignal(body.server, body.runId, body.eventId, signal);
} catch (error) { return reject(400, (error as Error).message); }
if (body.event !== 'round_change') return json({ ok: true, test });
}
const telemetry = {
server: body.server as string, round: body.round as number,
players: body.players as number, receivedAt: new Date().toISOString()
};
saveTelemetry(telemetry); saveTelemetry(telemetry);
console.log(`[telemetry] accepted round_change ${JSON.stringify(telemetry)}`); recordGameEvent(telemetry);
})();
emit({ type: 'server-telemetry', entries: [telemetry] }); emit({ type: 'server-telemetry', entries: [telemetry] });
return json({ ok: true, telemetry, test }); return json({ ok: true, telemetry });
} }
+17
View File
@@ -0,0 +1,17 @@
import type { GameEvent, TelemetryPlayerStats } from '../../src/lib/telemetry.js';
export const roster = (count = 4): TelemetryPlayerStats[] => Array.from({ length: count }, (_, i) => ({
name: `Player ${i + 1}`, guid: `1234567890123456${i}`, kills: 100 + i,
headshots: 50, downs: 1, revives: 2, score: 5000
}));
export function gameEvent(event: GameEvent['event'], round = 25, players = 4, server = 't5_zombie_theater_1', map = 'zombie_theater'): GameEvent {
return event === 'game_start'
? { event, server, map, players: roster(players).map(({ name, guid }) => ({ name, guid })) }
: { event, server, map, round, players: roster(players) };
}
export async function postTelemetry(body: unknown, contentType = 'application/json') {
const { POST } = await import('../../src/routes/api/telemetry/+server.js');
return POST({ request: new Request('http://localhost/api/telemetry', {
method: 'POST', headers: { 'Content-Type': contentType }, body: JSON.stringify(body)
}) });
}
+61 -59
View File
@@ -1,100 +1,102 @@
import { describe, expect, it } from 'vitest'; import { describe, expect, it } from 'vitest';
import { defaultPlan, parseBetaPlan, requirement, type BetaPlan } from '../../src/lib/beta-plan.js'; import { defaultPlan, parseBetaPlan, requirement, type BetaPlan } from '../../src/lib/beta-plan.js';
import { applySample, applySignal, coveragePassed, emptyMetrics, evaluateRequirements, type TestSignal } from '../../src/lib/server/fleet/beta-evaluation.js'; import { applySample, applyGameEvent, coveragePassed, emptyMetrics, evaluateRequirements } from '../../src/lib/server/fleet/beta-evaluation.js';
import { gameEvent } from '../helpers/telemetry.js';
const signal = (event: TestSignal['event'], extra: Partial<TestSignal> = {}): TestSignal => ({ event, bootId: 'boot-1', sessionId: 'session-1', sequence: 0, round: 0, players: 4, name: '', ...extra });
const planWith = (...kinds: Parameters<typeof requirement>[0][]): BetaPlan => ({ ...defaultPlan(), requirements: kinds.map((k, i) => requirement(k, `r${i}`)) }); const planWith = (...kinds: Parameters<typeof requirement>[0][]): BetaPlan => ({ ...defaultPlan(), requirements: kinds.map((k, i) => requirement(k, `r${i}`)) });
describe('beta requirement evaluation', () => { describe('beta requirement evaluation', () => {
it('copies a reactive preset without cloning proxies or mutating the saved requirements', () => { it('copies reactive presets without mutating saved requirements', () => {
const source = defaultPlan(); const source = defaultPlan();
const proxied = new Proxy(source, { get(target, key, receiver) { const selected = parseBetaPlan(new Proxy(source, {}));
return key === 'requirements' ? new Proxy(target.requirements, {}) : Reflect.get(target, key, receiver);
} });
const selected = parseBetaPlan(proxied);
expect(selected).toEqual(source); expect(selected).toEqual(source);
selected.requirements[0].target = 15; selected.requirements[0].target = 15;
selected.requirements.push(requirement('round', 'round'));
expect(source.requirements).toHaveLength(1);
expect(source.requirements[0].target).toBe(1); expect(source.requirements[0].target).toBe(1);
}); });
it('validates plans without accepting unknown kinds, duplicate IDs, fractional counts or unsafe numbers', () => { it('rejects removed checks, duplicate IDs, fractional counts and unsafe numbers', () => {
expect(parseBetaPlan(defaultPlan())).toEqual(defaultPlan()); expect(parseBetaPlan(defaultPlan())).toEqual(defaultPlan());
expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [] })).toThrow(); expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [] })).toThrow();
expect(() => parseBetaPlan({ ...defaultPlan(), timeoutHours: NaN })).toThrow(); expect(() => parseBetaPlan({ ...defaultPlan(), timeoutHours: NaN })).toThrow();
expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [requirement('round', 'same'), requirement('round', 'same')] })).toThrow(); expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [requirement('round', 'same'), requirement('round', 'same')] })).toThrow();
for (const kind of ['feature', 'restart-budget', 'clean-logs']) {
expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [{ ...requirement('round', 'r'), kind }] })).toThrow();
}
expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [{ ...requirement('round', 'r'), target: 1.5 }] })).toThrow(); expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [{ ...requirement('round', 'r'), target: 1.5 }] })).toThrow();
expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [requirement('manual', 'constructor')] })).toThrow(); expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [requirement('manual', 'constructor')] })).toThrow();
}); });
it('counts only qualifying rounds and completed matches from the same session', () => { it('uses qualifying round and population together within the same game, including the final report', () => {
const plan = planWith('round', 'matches'); const plan = planWith('round', 'matches');
plan.requirements[0].minPlayers = 4; plan.requirements[0].minPlayers = 4;
plan.requirements[1] = { ...plan.requirements[1], target: 2, minRound: 25, minPlayers: 4, minMinutes: 1 }; plan.requirements[1] = { ...plan.requirements[1], target: 2, minRound: 25, minPlayers: 4, minMinutes: 1 };
const m = emptyMetrics(); const m = emptyMetrics();
expect(applySignal(plan, m, signal('round_change', { sequence: 1, round: 50 }), 0)).toBe(false); applyGameEvent(plan, m, gameEvent('game_start'), 0);
applySignal(plan, m, signal('boot'), 0); applyGameEvent(plan, m, gameEvent('round_change', 30, 1), 1000);
applySignal(plan, m, signal('session_start'), 0); applyGameEvent(plan, m, gameEvent('game_end', 30, 1), 61_000);
applySignal(plan, m, signal('round_change', { sequence: 1, round: 30, players: 1 }), 1000);
applySignal(plan, m, signal('match_end', { sequence: 2 }), 61_000);
expect(m.values.r0 ?? 0).toBe(0); expect(m.values.r0 ?? 0).toBe(0);
expect(m.values.r1 ?? 0).toBe(0); expect(m.values.r1 ?? 0).toBe(0);
applySignal(plan, m, signal('session_start', { sessionId: 'session-2' }), 62_000); applyGameEvent(plan, m, gameEvent('game_start'), 62_000);
applySignal(plan, m, signal('round_change', { sessionId: 'session-2', sequence: 1, round: 25 }), 63_000); applyGameEvent(plan, m, gameEvent('game_end', 25), 123_000);
applySignal(plan, m, signal('match_end', { sessionId: 'session-2', sequence: 2 }), 123_000);
expect(m.values).toMatchObject({ r0: 25, r1: 1 }); expect(m.values).toMatchObject({ r0: 25, r1: 1 });
expect(applySignal(plan, m, signal('match_end', { sessionId: 'session-2', sequence: 3 }), 124_000)).toBe(false); applyGameEvent(plan, m, gameEvent('game_end', 25), 124_000);
expect(m.values.r1).toBe(1);
}); });
it('rejects out-of-order events and retired boots, while counting actual boot changes once', () => { it('accepts ordinary rounds and game ends without boot events or a discovery handshake', () => {
const plan = planWith('feature', 'restart-budget'); const plan = planWith('round', 'matches');
plan.requirements[0].event = 'perk.purchased';
const m = emptyMetrics(); const m = emptyMetrics();
applySignal(plan, m, signal('boot'), 0); applyGameEvent(plan, m, gameEvent('round_change'), 0);
applySignal(plan, m, signal('session_start'), 0); applyGameEvent(plan, m, gameEvent('game_end'), 1000);
applySignal(plan, m, signal('feature', { sequence: 2, name: 'perk.purchased' }), 1); expect(m.values).toMatchObject({ r0: 25, r1: 1 });
expect(applySignal(plan, m, signal('feature', { sequence: 1, name: 'perk.purchased' }), 2)).toBe(false); applyGameEvent(plan, m, gameEvent('game_start'), 2000);
applySignal(plan, m, signal('boot', { bootId: 'boot-2' }), 3); applyGameEvent(plan, m, gameEvent('game_end', 1), 3000);
expect(applySignal(plan, m, signal('boot'), 4)).toBe(false); expect(m.values.r1).toBe(2);
expect(m.values.r0).toBe(1);
expect(evaluateRequirements(plan, m, 0)[1].failed).toBe(true);
expect(m.restarts).toBe(1);
}); });
it('does not double count unqualified matches across log and telemetry sources', () => { it('does not combine games or maps to satisfy duration and round qualifiers', () => {
const plan = planWith('matches'); const plan = planWith('matches');
plan.requirements[0] = { ...plan.requirements[0], minRound: 25, minMinutes: 1 };
const m = emptyMetrics(); const m = emptyMetrics();
applySignal(plan, m, signal('boot'), 0); applyGameEvent(plan, m, gameEvent('game_start'), 0);
applySignal(plan, m, signal('session_start'), 0); applyGameEvent(plan, m, gameEvent('round_change'), 1000);
applySignal(plan, m, signal('match_end', { sequence: 1 }), 1000); applyGameEvent(plan, m, gameEvent('game_start'), 60_000);
expect(m.values.r0).toBeUndefined(); // these requirements use incremental match logs applyGameEvent(plan, m, gameEvent('game_end', 1), 120_000);
expect(m.values.r0 ?? 0).toBe(0);
applyGameEvent(plan, m, gameEvent('game_end', 30, 4, 't5_zombie_theater_1', 'zombie_moon'), 180_000);
expect(m.values.r0 ?? 0).toBe(0);
}); });
it('integrates populated time and player-hours conservatively and excludes monitoring gaps', () => { it('derives participation only from game reports, even when rounds are many minutes apart', () => {
const plan = planWith('populated-minutes', 'player-hours', 'healthy-minutes', 'population', 'sustained-population', 'failure-budget'); const plan = planWith('populated-minutes', 'player-hours', 'population', 'sustained-population');
plan.requirements[4].minPlayers = 4; plan.requirements[3].minPlayers = 4;
const m = emptyMetrics(); const m = emptyMetrics();
applySample(plan, m, 0, true, 4); applySample(plan, m, 0, true);
applySample(plan, m, 30_000, true, 4); applySample(plan, m, 30_000, true);
applySample(plan, m, 60_000, true, 4); expect(m.values).toEqual({});
expect(m.values).toMatchObject({ r0: 1, r1: 4 / 60, r2: 1, r3: 4, r4: 1 }); applyGameEvent(plan, m, gameEvent('game_start'), 0);
applySample(plan, m, 300_000, true, 4); applyGameEvent(plan, m, gameEvent('round_change'), 600_000);
expect(m.values).toMatchObject({ r0: 10, r1: 40 / 60, r2: 4, r3: 10 });
applyGameEvent(plan, m, gameEvent('game_end', 26, 2), 1_200_000);
expect(m.values).toMatchObject({ r0: 20, r1: 1, r3: 10 });
applyGameEvent(plan, m, gameEvent('game_start'), 3_000_000);
expect(m.values.r0).toBe(20);
});
it('excludes monitoring gaps from healthy time and requires explicit manual sign-off', () => {
const plan = planWith('healthy-minutes', 'failure-budget', 'manual');
const m = emptyMetrics();
applySample(plan, m, 0, true);
applySample(plan, m, 60_000, true);
expect(m.values.r0).toBe(1); expect(m.values.r0).toBe(1);
expect(m.values.r2).toBe(0); applySample(plan, m, 300_000, true);
applySample(plan, m, 330_000, false, 0); expect(m.values.r0).toBe(0);
applySample(plan, m, 360_000, false, 0); applySample(plan, m, 330_000, false);
applySample(plan, m, 360_000, false);
expect(m.outages).toBe(1); expect(m.outages).toBe(1);
expect(evaluateRequirements(plan, m, 0)[5].failed).toBe(true); expect(evaluateRequirements(plan, m, 0)[1].failed).toBe(true);
}); expect(evaluateRequirements(plan, m, 0)[2].passed).toBe(false);
m.manual.r2 = 'Josh: HUD checked';
it('requires readable logs, boot evidence and explicit manual sign-off', () => { expect(evaluateRequirements(plan, m, 0)[2].passed).toBe(true);
const plan = planWith('clean-logs', 'restart-budget', 'manual');
const m = emptyMetrics();
expect(evaluateRequirements(plan, m, 0).every(r => !r.passed)).toBe(true);
m.logAvailable = true; m.bootId = 'boot'; m.manual.r2 = 'Josh: HUD checked';
expect(evaluateRequirements(plan, m, 0).every(r => r.passed)).toBe(true);
m.logErrors = 1;
expect(evaluateRequirements(plan, m, 0)[0].failed).toBe(true);
}); });
it('requires each selected coverage group even when many servers on one map pass', () => { it('requires each selected coverage group even when many servers on one map pass', () => {
+119 -54
View File
@@ -3,6 +3,7 @@ import { appendFile, mkdir, readFile, writeFile } from 'node:fs/promises';
import { existsSync } from 'node:fs'; import { existsSync } from 'node:fs';
import { join } from 'node:path'; import { join } from 'node:path';
import { newFixture, type Fixture } from '../helpers/fixture.js'; import { newFixture, type Fixture } from '../helpers/fixture.js';
import { gameEvent, postTelemetry } from '../helpers/telemetry.js';
import { defaultPlan, requirement, type BetaPlan } from '../../src/lib/beta-plan.js'; import { defaultPlan, requirement, type BetaPlan } from '../../src/lib/beta-plan.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({ komodoRestart: vi.fn(), komodoInspectHealthy: vi.fn() })); vi.mock('../../src/lib/server/integrations/komodo.js', () => ({ komodoRestart: vi.fn(), komodoInspectHealthy: vi.fn() }));
@@ -77,56 +78,50 @@ describe('durable beta test plans', () => {
expect(getBeta(id)?.state).toBe('ready-promote'); expect(getBeta(id)?.state).toBe('ready-promote');
}); });
it('ignores pre-beta log markers, handles partial lines, and persists cursors across app restarts', async () => { it('uses script events instead of logs and keeps match progress across app restarts', async () => {
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('matches', 'matches'), requirement('clean-logs', 'logs')] }); await postTelemetry(gameEvent('game_end'));
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('matches', 'matches')] });
await tick(); await tick(); await tick(); await tick();
const { getBeta } = await import('../../src/lib/server/fleet/beta.js');
const { testView } = await import('../../src/lib/server/fleet/beta-tests.js'); const { testView } = await import('../../src/lib/server/fleet/beta-tests.js');
expect(getBeta(id)?.state).toBe('running'); await appendFile(join(fixture.t5, names[0], 'logs/games_sp.log'), 'InitGame:\n 0:00 J;new;1;Josh\nExitLevel: executed\n');
expect(testView(id)?.servers[0].progress.map(p => p.current)).toEqual([0, 0]);
const log = join(fixture.t5, names[0], 'logs/games_sp.log');
await appendFile(log, 'InitGame:\n 0:00 J;new;1;Josh\nExitLevel: exe');
await tick(); await tick();
expect(getBeta(id)?.state).toBe('running'); expect(testView(id)?.servers[0].progress[0].current).toBe(0);
await postTelemetry(gameEvent('game_start'));
const { closeDb } = await import('../../src/lib/server/db/index.js'); const { closeDb } = await import('../../src/lib/server/db/index.js');
closeDb(); closeDb();
await appendFile(log, 'cuted\n'); await postTelemetry(gameEvent('game_end'));
await tick(); await tick(); await tick();
expect(getBeta(id)?.state).toBe('ready-promote');
expect(testView(id)?.servers[0].progress[0].current).toBe(1); expect(testView(id)?.servers[0].progress[0].current).toBe(1);
await appendFile(join(fixture.t5, names[0], 'logs/console.log'), 'script runtime error: undefined variable\n');
await tick();
expect(getBeta(id)?.state).toBe('running');
expect(testView(id)?.outcome).toBe('failed');
});
it('accepts durable run-scoped round evidence and rejects stale runs and duplicate events', async () => {
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('round', 'round')] });
await tick(); await tick();
const { acceptTestSignal, currentRun, testView } = await import('../../src/lib/server/fleet/beta-tests.js');
const run = currentRun(id, names[0])!;
const base = { bootId: 'boot', sessionId: 'session', sequence: 0, round: 0, players: 4, name: '' };
expect(acceptTestSignal(names[0], 'stale-run', 'old', { ...base, event: 'boot' }).accepted).toBe(false);
expect(acceptTestSignal(names[0], run.id, 'boot', { ...base, event: 'boot' }).accepted).toBe(true);
expect(acceptTestSignal(names[0], run.id, 'start', { ...base, event: 'session_start' }).accepted).toBe(true);
const round = { ...base, event: 'round_change' as const, sequence: 1, round: 25 };
expect(acceptTestSignal(names[0], run.id, 'round', round).accepted).toBe(true);
expect(acceptTestSignal(names[0], run.id, 'round', round).accepted).toBe(false);
await tick();
expect(testView(id)?.outcome).toBe('passed'); expect(testView(id)?.outcome).toBe('passed');
expect(testView(id)?.evidence.filter(e => e.kind === 'round_change')).toHaveLength(1);
}); });
it('resets evidence after planned recovery and only passes after testing the new process', async () => { it('automatically credits ordinary rounds only to active deployed beta servers', async () => {
const id = await openPlan({ ...defaultPlan(), waveSize: 1, requirements: [requirement('round', 'round')] }, names);
await postTelemetry(gameEvent('round_change'));
await tick(); await tick();
const { testView } = await import('../../src/lib/server/fleet/beta-tests.js');
expect(testView(id)?.servers[0].progress[0].current).toBe(0);
await postTelemetry(gameEvent('round_change', 99, 4, names[1]));
expect(testView(id)?.servers[0].progress[0].current).toBe(0);
await postTelemetry(gameEvent('round_change'));
expect(testView(id)?.servers[0].progress[0].current).toBe(25);
expect(testView(id)?.evidence.filter(e => e.kind === 'round_change')).toHaveLength(1);
await tick();
expect(existsSync(scriptPath(names[1]))).toBe(true);
});
it('resets evidence after planned recovery and consumes the same simple reports afterward', async () => {
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players'), requirement('recovery', 'recovery')] }); const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players'), requirement('recovery', 'recovery')] });
await tick(); await tick(); await tick(); await tick();
const { currentRun, testView } = await import('../../src/lib/server/fleet/beta-tests.js'); const { currentRun, testView } = await import('../../src/lib/server/fleet/beta-tests.js');
const first = currentRun(id, names[0])!; const first = currentRun(id, names[0])!;
expect(testView(id)?.outcome).toBe('active'); expect(testView(id)?.servers[0].progress[0].current).toBe(0);
await tick(); await postTelemetry(gameEvent('game_start'));
await tick(); await tick();
expect(currentRun(id, names[0])?.id).not.toBe(first.id); expect(currentRun(id, names[0])?.id).not.toBe(first.id);
expect(currentRun(id, names[0])?.phase).toBe(1); expect(currentRun(id, names[0])?.phase).toBe(1);
expect(testView(id)?.servers[0].progress[0].current).toBe(0); expect(testView(id)?.servers[0].progress[0].current).toBe(0);
await postTelemetry(gameEvent('game_start'));
await tick(); await tick();
expect(testView(id)?.outcome).toBe('passed'); expect(testView(id)?.outcome).toBe('passed');
}); });
@@ -143,6 +138,7 @@ describe('durable beta test plans', () => {
it('revokes readiness if health fails while awaiting manual promotion', async () => { it('revokes readiness if health fails while awaiting manual promotion', async () => {
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players'), requirement('failure-budget', 'outages')] }); const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players'), requirement('failure-budget', 'outages')] });
await tick(); await tick(); await tick(); await tick();
await postTelemetry(gameEvent('game_start')); await tick();
const { getBeta, promoteBeta } = await import('../../src/lib/server/fleet/beta.js'); const { getBeta, promoteBeta } = await import('../../src/lib/server/fleet/beta.js');
expect(getBeta(id)?.state).toBe('ready-promote'); expect(getBeta(id)?.state).toBe('ready-promote');
await tick(15, 0, false); await tick(15, 0, false);
@@ -152,10 +148,9 @@ describe('durable beta test plans', () => {
it('automatically rolls back only the deployed wave and queues safe rollback restarts', async () => { it('automatically rolls back only the deployed wave and queues safe rollback restarts', async () => {
await writeFile(scriptPath(names[0]), 'original'); await writeFile(scriptPath(names[0]), 'original');
const id = await openPlan({ ...defaultPlan(), waveSize: 1, failureAction: 'rollback', requirements: [requirement('round', 'round'), requirement('clean-logs', 'logs')] }, names); const id = await openPlan({ ...defaultPlan(), waveSize: 1, failureAction: 'rollback', requirements: [requirement('round', 'round'), requirement('failure-budget', 'outages')] }, names);
await tick(); await tick(); await tick(); await tick();
await appendFile(join(fixture.t5, names[0], 'logs/console.log'), 'fatal error: test\n'); await tick(15, 0, false);
await tick();
const { getBeta } = await import('../../src/lib/server/fleet/beta.js'); const { getBeta } = await import('../../src/lib/server/fleet/beta.js');
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
expect(getBeta(id)?.state).toBe('rolled-back'); expect(getBeta(id)?.state).toBe('rolled-back');
@@ -176,6 +171,7 @@ describe('durable beta test plans', () => {
it('retains artifact ownership while ready for promotion', async () => { it('retains artifact ownership while ready for promotion', async () => {
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players')] }); const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players')] });
await tick(); await tick(); await tick(); await tick();
await postTelemetry(gameEvent('game_start')); await tick();
const { heldByBeta, ownerOf } = await import('../../src/lib/server/fleet/servers.js'); const { heldByBeta, ownerOf } = await import('../../src/lib/server/fleet/servers.js');
expect(heldByBeta(names[0]).get('test.gsc')).toBe(id); expect(heldByBeta(names[0]).get('test.gsc')).toBe(id);
expect(ownerOf(names[0], 'test.gsc')).toBe(id); expect(ownerOf(names[0], 'test.gsc')).toBe(id);
@@ -185,32 +181,27 @@ describe('durable beta test plans', () => {
expect(ownerOf(names[0], 'test.gsc')).toBeNull(); expect(ownerOf(names[0], 'test.gsc')).toBeNull();
}); });
it('does not carry joins into a truncated log or deduplicate a genuinely new match after rotation', async () => { it('counts successive games with the same players and ignores repeated game ends', async () => {
const id = await openPlan({ ...defaultPlan(), requirements: [{ ...requirement('matches', 'matches'), target: 2 }] }); const id = await openPlan({ ...defaultPlan(), requirements: [{ ...requirement('matches', 'matches'), target: 2 }] });
await tick(); await tick(); await tick(); await tick();
const log = join(fixture.t5, names[0], 'logs/games_sp.log'); await postTelemetry(gameEvent('game_start'));
await appendFile(log, 'InitGame:\n 0:00 J;first;1;Josh\nExitLevel: executed\n'); await postTelemetry(gameEvent('game_end'));
await tick(); await postTelemetry(gameEvent('game_end'));
// Same inode, contents replaced with a larger file: anchor detects truncation/regrowth.
await writeFile(log, 'InitGame:\n 0:00 J;second;1;Josh\n' + 'padding\n'.repeat(30) + 'ExitLevel: executed\n');
await tick();
const { testView } = await import('../../src/lib/server/fleet/beta-tests.js'); const { testView } = await import('../../src/lib/server/fleet/beta-tests.js');
expect(testView(id)?.servers[0].progress[0].current).toBe(1);
await postTelemetry(gameEvent('game_start'));
await postTelemetry(gameEvent('game_end'));
await tick();
expect(testView(id)?.servers[0].progress[0].current).toBe(2); expect(testView(id)?.servers[0].progress[0].current).toBe(2);
expect(testView(id)?.outcome).toBe('passed'); expect(testView(id)?.outcome).toBe('passed');
}); });
it('exposes current run tokens, validates session payloads, and records checklist sign-off through the API', async () => { it('records checklist sign-off and starts a fresh attempt without changing script payloads', async () => {
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('round', 'round'), requirement('manual', 'hud')] }); const id = await openPlan({ ...defaultPlan(), requirements: [requirement('round', 'round'), requirement('manual', 'hud')] });
await tick(); await tick(); await tick(); await tick();
const telemetry = await import('../../src/routes/api/telemetry/+server.js'); const telemetry = await import('../../src/routes/api/telemetry/+server.js');
const context = await (await telemetry.GET({ url: new URL(`http://test/api/telemetry?server=${names[0]}`) })).json(); expect(telemetry).not.toHaveProperty('GET');
expect(context.runs).toHaveLength(1); expect(await (await postTelemetry(gameEvent('round_change'))).json()).not.toHaveProperty('test');
const body = { server: names[0], runId: context.runs[0].runId, eventId: 'boot', bootId: 'boot', sequence: 0, round: 0, players: 0, event: 'boot' };
const post = (payload: unknown) => telemetry.POST({ request: new Request('http://test/api/telemetry', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(payload) }) });
expect((await post({ ...body, eventId: null })).status).toBe(400);
expect(await (await post(body)).json()).toMatchObject({ test: { accepted: true } });
await post({ ...body, eventId: 'start', event: 'session_start', sessionId: 'game', players: 4 });
await post({ ...body, eventId: 'round', event: 'round_change', sessionId: 'game', sequence: 1, round: 25, players: 4 });
const controls = await import('../../src/routes/api/betas/[id]/test/+server.js'); const controls = await import('../../src/routes/api/betas/[id]/test/+server.js');
const result = await controls.POST({ params: { id: String(id) }, request: new Request('http://test', { method: 'POST', body: JSON.stringify({ action: 'sign-off', server: names[0], requirementId: 'hud', note: 'Josh: HUD verified' }) }) }); const result = await controls.POST({ params: { id: String(id) }, request: new Request('http://test', { method: 'POST', body: JSON.stringify({ action: 'sign-off', server: names[0], requirementId: 'hud', note: 'Josh: HUD verified' }) }) });
expect(result.status).toBe(200); expect(result.status).toBe(200);
@@ -219,8 +210,82 @@ describe('durable beta test plans', () => {
expect(testView(id)?.outcome).toBe('passed'); expect(testView(id)?.outcome).toBe('passed');
const retry = await controls.POST({ params: { id: String(id) }, request: new Request('http://test', { method: 'POST', body: JSON.stringify({ action: 'retry' }) }) }); const retry = await controls.POST({ params: { id: String(id) }, request: new Request('http://test', { method: 'POST', body: JSON.stringify({ action: 'retry' }) }) });
expect(retry.status).toBe(200); expect(retry.status).toBe(200);
expect(await (await post({ ...body, eventId: 'late' })).json()).toMatchObject({ test: { accepted: false } }); await postTelemetry(gameEvent('round_change'));
expect(testView(id)?.servers).toHaveLength(0); expect(testView(id)?.servers).toHaveLength(0);
await tick(); await tick();
expect(testView(id)?.servers[0].progress[0].current).toBe(0);
await postTelemetry(gameEvent('round_change'));
expect(testView(id)?.servers[0].progress[0].current).toBe(25);
});
it('keeps Fleet reporting during pauses without crediting paused gameplay or elapsed time', async () => {
const id = await openPlan({ ...defaultPlan(), requirements: [{ ...requirement('populated-minutes', 'time'), target: 60 }] });
await tick(); await tick();
await postTelemetry(gameEvent('game_start'));
vi.setSystemTime(Date.now() + 600_000);
await postTelemetry(gameEvent('round_change'));
const { testView } = await import('../../src/lib/server/fleet/beta-tests.js');
expect(testView(id)?.servers[0].progress[0].current).toBe(10);
const controls = await import('../../src/routes/api/betas/[id]/test/+server.js');
const control = (action: string) => controls.POST({ params: { id: String(id) }, request: new Request('http://test', {
method: 'POST', body: JSON.stringify({ action })
}) });
expect((await control('pause')).status).toBe(200);
vi.setSystemTime(Date.now() + 3_600_000);
await postTelemetry(gameEvent('game_end', 50));
const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js');
expect(listTelemetry()[0]).toMatchObject({ event: 'game_end', round: 50 });
expect(testView(id)?.servers[0].progress[0].current).toBe(10);
expect((await control('resume')).status).toBe(200);
await postTelemetry(gameEvent('round_change', 1));
expect(testView(id)?.servers[0].progress[0].current).toBe(10);
vi.setSystemTime(Date.now() + 600_000);
await postTelemetry(gameEvent('round_change', 2));
expect(testView(id)?.servers[0].progress[0].current).toBe(20);
});
it('migrates existing unsupported checks into review items and resets active progress once', async () => {
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('round', 'round')] });
await tick(); await tick();
await postTelemetry(gameEvent('round_change'));
await tick();
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const oldPlan = { ...defaultPlan(), requirements: [requirement('round', 'round'),
...['feature', 'restart-budget', 'clean-logs'].map((kind, i) => ({ ...requirement('manual', `old-${i}`), kind, event: 'perk.purchased' }))] };
db().prepare('UPDATE beta_test_plans SET plan = ? WHERE beta_id = ?').run(JSON.stringify(oldPlan), id);
db().prepare('INSERT INTO beta_test_presets (name, plan) VALUES (?, ?)').run('Old plan', JSON.stringify(oldPlan));
db().prepare("DELETE FROM settings WHERE key = 'game_telemetry_v2'").run();
closeDb();
const { testView } = await import('../../src/lib/server/fleet/beta-tests.js');
const { getBeta } = await import('../../src/lib/server/fleet/beta.js');
const view = testView(id)!;
expect(view.plan.requirements.map(r => r.kind)).toEqual(['round', 'manual', 'manual', 'manual']);
expect(view.plan.requirements.some(r => 'event' in r)).toBe(false);
expect(view.servers[0].progress.every(p => !p.passed)).toBe(true);
expect(view.evidence.some(e => e.kind === 'round_change')).toBe(true);
expect(getBeta(id)?.state).toBe('running');
const presets = await import('../../src/routes/api/betas/presets/+server.js');
const saved = (await presets.GET().json()).presets.find((p: { name: string }) => p.name === 'Old plan');
expect(saved.plan).toEqual(view.plan);
await postTelemetry(gameEvent('round_change'));
closeDb();
expect(testView(id)?.servers[0].progress[0].current).toBe(25);
});
it('uses game_end for the legacy match gate without reading logs', async () => {
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { openBeta, getBeta, listBetaServers } = await import('../../src/lib/server/fleet/beta.js');
const { script, version } = createScript({ name: 'Legacy beta', targetRelPath: 'legacy.gsc', content: Buffer.from('main() {}') });
const { betaId } = await openBeta({ scripts: [{ scriptId: script.id, scriptVersionId: version.id }], servers: [names[0]], waitForHealth: true, waitForMatch: true });
await tick(); await tick();
expect(listBetaServers(betaId)[0].stage).toBe('awaiting-match');
await appendFile(join(fixture.t5, names[0], 'logs/games_sp.log'), 'ExitLevel: executed\n');
await tick();
expect(getBeta(betaId)?.state).toBe('running');
await postTelemetry(gameEvent('game_end'));
await postTelemetry(gameEvent('game_start'));
await tick();
expect(getBeta(betaId)?.state).toBe('ready-promote');
}); });
it('stores validated presets as copies and rejects malformed plans', async () => { it('stores validated presets as copies and rejects malformed plans', async () => {
+2 -1
View File
@@ -1,5 +1,6 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { currentRound, fleetIdentity, matchesFleet, watchSatisfied, type FleetServer, type FleetOperation } from '../../src/lib/fleet-overview.js'; import { currentRound, fleetIdentity, matchesFleet, watchSatisfied, type FleetServer, type FleetOperation } from '../../src/lib/fleet-overview.js';
import { gameEvent } from '../helpers/telemetry.js';
import { newFixture, type Fixture } from '../helpers/fixture.js'; import { newFixture, type Fixture } from '../helpers/fixture.js';
const now = Date.parse('2026-09-18T12:00:00Z'); const now = Date.parse('2026-09-18T12:00:00Z');
@@ -42,7 +43,7 @@ describe('Fleet discovery and match freshness', () => {
expect(matchesFleet({ ...server, health: 'starting' }, '', 'attention', '', [], [], now)).toBe(false); expect(matchesFleet({ ...server, health: 'starting' }, '', 'attention', '', [], [], now)).toBe(false);
}); });
it('only spotlights fresh telemetry from the current occupied match', () => { it('only spotlights fresh telemetry from the current occupied match', () => {
const report = { server: server.name, round: 38, players: 4, receivedAt: iso(-1000) }; const report = { ...gameEvent('round_change', 38, 4, server.name), receivedAt: iso(-1000) };
expect(currentRound(server, report, now)).toBe(38); expect(currentRound(server, report, now)).toBe(38);
for (const change of [{ receivedAt: iso(-121_000) }, { receivedAt: iso(-61_000) }, { receivedAt: iso(60_000) }]) { for (const change of [{ receivedAt: iso(-121_000) }, { receivedAt: iso(-61_000) }, { receivedAt: iso(60_000) }]) {
expect(currentRound(server, { ...report, ...change }, now)).toBeNull(); expect(currentRound(server, { ...report, ...change }, now)).toBeNull();
+2 -3
View File
@@ -79,13 +79,12 @@ describe('T6 config lifecycle in a mixed fleet', () => {
}); });
it('reads the T6 game log and keeps log path traversal checks', async () => { it('reads the T6 game log and keeps log path traversal checks', async () => {
const { resolveLogPath, gameLogKey } = await import('../../src/lib/server/fleet/logs.js'); const { resolveLogPath, gameLogKey, tailLog } = await import('../../src/lib/server/fleet/logs.js');
const { scanMatchState } = await import('../../src/lib/server/fleet/match-log.js');
const logs = join(fixture.t6, town, 'logs'); const logs = join(fixture.t6, town, 'logs');
await mkdir(logs, { recursive: true }); await mkdir(logs, { recursive: true });
await writeFile(join(logs, 'games_zm.log'), '0:00 InitGame:\n0:01 J;123;0;Player\n'); await writeFile(join(logs, 'games_zm.log'), '0:00 InitGame:\n0:01 J;123;0;Player\n');
expect(resolveLogPath(town, gameLogKey(town))).toBe(resolve(logs, 'games_zm.log')); expect(resolveLogPath(town, gameLogKey(town))).toBe(resolve(logs, 'games_zm.log'));
expect(await scanMatchState(town)).toMatchObject({ kind: 'in-progress' }); expect((await tailLog(resolveLogPath(town, gameLogKey(town))!, 10))?.lines).toContain('0:01 J;123;0;Player');
expect(resolveLogPath('t6_/../../escape', 'games_zm')).toBeNull(); expect(resolveLogPath('t6_/../../escape', 'games_zm')).toBeNull();
}); });
}); });
+52 -21
View File
@@ -1,5 +1,6 @@
import { afterEach, beforeEach, expect, it, vi } from 'vitest'; import { afterEach, beforeEach, expect, it, vi } from 'vitest';
import { newFixture, type Fixture } from '../helpers/fixture.js'; import { newFixture, type Fixture } from '../helpers/fixture.js';
import { gameEvent, postTelemetry as post, roster } from '../helpers/telemetry.js';
let fixture: Fixture; let fixture: Fixture;
beforeEach(async () => { beforeEach(async () => {
@@ -10,42 +11,60 @@ beforeEach(async () => {
}); });
afterEach(async () => fixture.cleanup()); afterEach(async () => fixture.cleanup());
const payload = { event: 'round_change', server: 't5_zombie_theater_1', round: 17, players: 4 }; const payload = gameEvent('round_change', 17);
async function post(body: unknown, contentType = 'application/json') {
const { POST } = await import('../../src/routes/api/telemetry/+server.js');
return POST({ request: new Request('http://localhost/api/telemetry', {
method: 'POST', headers: { 'Content-Type': contentType }, body: JSON.stringify(body)
}) });
}
it('persists reports, broadcasts updates, and leaves RCON safety data untouched', async () => { it('persists full player stats, broadcasts updates, and leaves RCON safety data untouched', async () => {
const { subscribe } = await import('../../src/lib/server/pipeline/bus.js'); const { subscribe } = await import('../../src/lib/server/pipeline/bus.js');
const listener = vi.fn(); const listener = vi.fn();
const unsubscribe = subscribe(listener); const unsubscribe = subscribe(listener);
expect((await post(payload)).status).toBe(200); expect(await (await post(payload)).json()).toEqual({ ok: true, telemetry: { ...payload, receivedAt: expect.any(String) } });
expect(listener).toHaveBeenCalledWith(expect.objectContaining({ type: 'server-telemetry' })); expect(listener).toHaveBeenCalledWith({ type: 'server-telemetry', entries: [{ ...payload, receivedAt: expect.any(String) }] });
unsubscribe(); unsubscribe();
// A new match can legitimately report a lower round; retries overwrite one row.
await post({ ...payload, round: 1 });
await post({ ...payload, round: 1 }); await post({ ...payload, round: 1 });
const { closeDb } = await import('../../src/lib/server/db/index.js'); const { closeDb } = await import('../../src/lib/server/db/index.js');
closeDb(); closeDb();
const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js'); const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js');
expect(listTelemetry()).toEqual([expect.objectContaining({ round: 1, players: 4 })]); expect(listTelemetry()).toEqual([{ ...payload, round: 1, receivedAt: expect.any(String) }]);
const { getServer } = await import('../../src/lib/server/fleet/servers.js'); const { getServer } = await import('../../src/lib/server/fleet/servers.js');
expect(getServer(payload.server)?.last_players).toBeNull(); expect(getServer(payload.server)?.last_players).toBeNull();
const { GET } = await import('../../src/routes/api/events/servers/+server.js'); const { GET } = await import('../../src/routes/api/events/servers/+server.js');
const response = await GET(); const reader = (await GET()).body!.getReader();
const reader = response.body!.getReader(); await reader.read();
await reader.read(); // greeting const snapshot = new TextDecoder().decode((await reader.read()).value);
const snapshot = await reader.read(); expect(snapshot).toContain('"round":1');
expect(new TextDecoder().decode(snapshot.value)).toContain('"round":1'); expect(snapshot).toContain('"headshots":50');
expect(snapshot).toContain('"guid":"12345678901234560"');
await reader.cancel(); await reader.cancel();
}); });
it.each([null, [], {}, { ...payload, event: 'other' }, { ...payload, round: -1 }, it('accepts all three events without an active beta and updates match lifecycle', async () => {
{ ...payload, round: 1.5 }, { ...payload, round: '17' }, { ...payload, players: -1 }, const { getServer } = await import('../../src/lib/server/fleet/servers.js');
{ ...payload, players: null }])('rejects invalid telemetry: %j', async (body) => { const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js');
const start = gameEvent('game_start');
expect((await post(start)).status).toBe(200);
expect(listTelemetry()).toEqual([{ ...start, receivedAt: expect.any(String) }]);
expect(getServer(start.server)?.game_started_at).toBeTruthy();
expect(getServer(start.server)?.game_ended_at).toBeNull();
expect((await post(payload)).status).toBe(200);
const end = gameEvent('game_end', 18);
expect((await post(end)).status).toBe(200);
expect(getServer(start.server)?.game_ended_at).toBeTruthy();
expect(listTelemetry()[0]).toMatchObject(end);
await post(start);
expect(listTelemetry()[0]).not.toHaveProperty('round');
expect(getServer(start.server)?.game_ended_at).toBeNull();
expect((await post(gameEvent('game_start', 0, 0))).status).toBe(200);
expect((await post(gameEvent('game_end', 0, 0))).status).toBe(200);
});
it.each([null, [], {}, ...['boot', 'session_start', 'match_end', 'feature', 'other'].map(event => ({ ...payload, event })),
{ ...payload, map: '' }, { ...payload, map: undefined }, { ...payload, round: -1 },
{ ...payload, round: 1.5 }, { ...payload, round: '17' }, { ...payload, players: 4 }, { ...payload, players: null },
{ ...payload, players: [null] }, { ...payload, players: [{ name: 'Josh', guid: '123' }] },
...['name', 'guid', 'kills', 'headshots', 'downs', 'revives', 'score'].map(field => ({ ...payload, players: [{ ...roster(1)[0], [field]: null }] })),
{ ...payload, players: [{ ...roster(1)[0], guid: 12345678901234567 }] },
{ ...payload, players: [{ ...roster(1)[0], score: -1 }] }
])('rejects invalid telemetry: %j', async (body) => {
expect((await post(body)).status).toBe(400); expect((await post(body)).status).toBe(400);
}); });
@@ -63,3 +82,15 @@ it('rejects malformed JSON', async () => {
method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{' method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{'
}) })).status).toBe(400); }) })).status).toBe(400);
}); });
it('migrates old round-only storage without fabricating player identities', async () => {
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
db().exec('DROP TABLE server_telemetry; CREATE TABLE server_telemetry (server TEXT PRIMARY KEY, round INTEGER NOT NULL, players INTEGER NOT NULL, received_at TEXT NOT NULL)');
db().prepare('INSERT INTO server_telemetry VALUES (?, ?, ?, ?)').run(payload.server, 99, 4, new Date().toISOString());
closeDb();
const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js');
expect(listTelemetry()).toEqual([]);
await post(payload);
closeDb();
expect(listTelemetry()[0]).toMatchObject(payload);
});