diff --git a/README.md b/README.md index af3c463..a0cc1b8 100644 --- a/README.md +++ b/README.md @@ -66,8 +66,7 @@ See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow. Betas support configurable gameplay, participation, stability and manual-review requirements, saved presets, coverage rules and staged rollout. See -[Beta test plans](docs/beta-test-plans.md) for usage and the session telemetry -contract used by round and custom-feature checks. +[Beta test plans](docs/beta-test-plans.md) for usage and the three-event game telemetry contract. File changes are staged sequentially. Their activation uses rolling restarts: by default only **one server across the fleet** can restart or recover at a @@ -229,32 +228,21 @@ Implementation references (checked September 2026): - [T6 map/ruleset combinations](https://github.com/xerxes-at/T6ServerConfigs/blob/master/localappdata/Plutonium/storage/t6/dedicated_zm.cfg) - [Plutonium GSC paths](https://plutonium.pw/docs/modding/loading-mods/) -## Round telemetry experiment +## Game telemetry -Send `POST /api/telemetry` with `Content-Type: application/json` to the Electric -Cherry host (port 3000 in local development): +The server script sends JSON to `POST /api/telemetry`: -```bash -curl http://localhost:3000/api/telemetry \ - -H 'Content-Type: application/json' \ - -d '{"event":"round_change","round":17,"server":"t5_zombie_prototype_1","players":4}' -``` +- `game_start`: server, map, and all players (name and GUID). +- `round_change`: server, map, round, and all players with kills, headshots, downs, revives, and score. +- `game_end`: the same fields as round change, with final stats. -The server must already be registered in Electric Cherry. `round` and `players` -are required nonnegative integers. Success returns HTTP 200 with -`{ "ok": true, "telemetry": { "server": "...", "round": 17, "players": 4, "receivedAt": "..." } }`. -Invalid payloads return 400, unknown/destroyed servers 404, and non-JSON content -types 415. This endpoint uses the app's existing LAN/Tailscale access model; -there is no API key requirement. +These reports work with or without an active beta. Electric Cherry records +the full player data, updates Fleet, and applies gameplay progress to active +betas automatically. No discovery request or beta-specific fields are needed. -The latest report is saved across app restarts and streamed immediately to the -dashboard and server detail page, including after browser reconnection. The UI -labels it as the last reported round with its age: round-change events alone -cannot detect a stopped script or a match ending. Send the initial round on each -new match, then each round change. The latest received request wins (including -lower rounds for new matches); there is no event ordering or history yet. -Telemetry player counts are displayed with the report and never replace RCON -occupancy or affect automated restart gates. +See [Game reporter protocol](docs/beta-test-plans.md#game-reporter-protocol) for +complete payload examples. Telemetry does not replace RCON's empty-server +check before an automated restart. ## Running tests diff --git a/docs/beta-test-plans.md b/docs/beta-test-plans.md index 531d1e7..5eb8bf0 100644 --- a/docs/beta-test-plans.md +++ b/docs/beta-test-plans.md @@ -1,652 +1,165 @@ # Beta test plans -New betas use a saved plan rather than the two legacy wait flags. Existing -betas and API callers without `testPlan` retain their original workflow. -Schema additions are additive and run at startup; no active beta needs to be -closed to install this feature. +Create a plan in Betas, choose its servers, and deploy. The server's game script +reports gameplay to Electric Cherry. The same reports update Fleet and any active +beta on that server. + +## Game reporter protocol + +Send JSON to `POST /api/telemetry` with `Content-Type: application/json`. +There are exactly three events. Send them whether or not a beta is active. + +| Event | Fields | +| --- | --- | +| `game_start` | `server`, `map`, `players` with each player's `name` and `guid` | +| `round_change` | `server`, `map`, `round`, `players` with each player's `name`, `guid`, `kills`, `headshots`, `downs`, `revives`, and `score` | +| `game_end` | `server`, `map`, `round`, `players` with each player's `name`, `guid`, `kills`, `headshots`, `downs`, `revives`, and `score` | + +`server` is the exact registered Electric Cherry server name. `map` is the +current map. `players` is the full player array, including an empty array when +there are no players. Names and GUIDs are strings; keep GUIDs as strings to +preserve their precision. Round and stat values are nonnegative JSON integers. + +### Game start + +Send when the game starts. Round and stats are not required. + +```json +{ + "event": "game_start", + "server": "t6_zombie_town_1", + "map": "zm_transit", + "players": [ + { "name": "Josh", "guid": "12345678901234567" }, + { "name": "Alex", "guid": "23456789012345678" } + ] +} +``` + +### Round change + +Send the new round and each player's current stats. + +```json +{ + "event": "round_change", + "server": "t6_zombie_town_1", + "map": "zm_transit", + "round": 25, + "players": [ + { "name": "Josh", "guid": "12345678901234567", "kills": 420, "headshots": 160, "downs": 1, "revives": 3, "score": 32000 }, + { "name": "Alex", "guid": "23456789012345678", "kills": 380, "headshots": 140, "downs": 3, "revives": 1, "score": 28000 } + ] +} +``` + +### Game end + +Send the final round and each player's final stats when the game ends. + +```json +{ + "event": "game_end", + "server": "t6_zombie_town_1", + "map": "zm_transit", + "round": 26, + "players": [ + { "name": "Josh", "guid": "12345678901234567", "kills": 445, "headshots": 170, "downs": 2, "revives": 3, "score": 33500 }, + { "name": "Alex", "guid": "23456789012345678", "kills": 401, "headshots": 146, "downs": 4, "revives": 1, "score": 29750 } + ] +} +``` + +Success returns HTTP 200 with `{ "ok": true, "telemetry": { ... } }`. +The returned telemetry contains the event and an Electric Cherry `receivedAt` +timestamp. Invalid JSON or fields return 400, an unknown/destroyed server 404, +and a non-JSON content type 415. + +The script sends each event once, in gameplay order, directly to Electric Cherry. +There is no discovery request, beta token, boot/session ID, sequence number, +custom feature event, external helper, or separate beta acknowledgement. +Basic payload checks happen in the receiver. + +Electric Cherry saves the latest complete report, including player stats, and +streams it to connected clients. A new start clears the previous round display; +an end closes the match. Receipt order determines the latest report. This small +protocol does not attempt to identify delayed events from earlier games or +provide exactly-once delivery across games. + +The game scripts are uploaded artifacts, not source maintained in this +repository. Their reporter should emit the three payloads above. ## Requirements -Use **Add requirement** in Betas. Requirements on a qualifying server are ANDed: +Requirements on a qualifying server are ANDed: | Requirement | Evidence | | --- | --- | -| Reach round X | Session telemetry reports round >= X with the required player count | -| Complete qualifying matches | Normal completed matches, optionally qualified by round, players and duration | -| Populated minutes | Fresh RCON samples at both ends of a populated interval | -| Player-hours | Populated minutes × the lower sampled player count / 60 | -| Reach player count | Fresh healthy RCON sample | -| Sustained population | Consecutive fresh healthy samples above the player threshold | +| Reach round X | A round change or game end reports round >= X with the required player count | +| Complete qualifying matches | Game end, optionally qualified by round, player count, and duration | +| Populated minutes | Time between consecutive game reports with enough players at both ends | +| Player-hours | Populated minutes multiplied by the lower reported player count, divided by 60 | +| Reach player count | Player array length in a game report | +| Sustained population | Consecutive report intervals above the player threshold within one game | | Continuous healthy minutes | Consecutive fresh container/RCON observations | | Maximum health outages | Transitions from observed healthy to unhealthy | -| Maximum unexpected restarts | Distinct reported process boots within the same test run | -| Maximum script errors | New console lines matching script errors/exceptions, unhandled exceptions or fatal errors | | Successful recovery cycles | Repeat the other requirements after each additional empty-server restart | -| Feature event count | Named, session-scoped game events | -| Tester checklist | Tester name and observation, entered on the beta detail page | +| Tester checklist | Tester name and observation entered on the beta detail page | -There is one round requirement: **reach round X**. For repeated high-round runs, -use a qualifying-match count with a minimum round. The minimum player count must -be present at the qualifying round observation in the same match. A match with -many players and a different solo high-round match cannot combine to pass. +All gameplay requirements use the script reports. Match counts do not read game +logs. The reported round and required player count must occur together in the +same game; observations from different games cannot combine to qualify a match. +The final game-end report can supply that qualifying round. Consecutive duplicate +game ends do not add another completion. -Unqualified match counts use incremental game logs (`InitGame`, a player join, -then `ExitLevel: executed`). Round-qualified or multi-player-qualified matches -use session telemetry. Each requirement uses one source, preventing duplicate -counts. Match duration is measured from the observed join/session start. Log -polling can undercount duration; it does not infer unobserved elapsed time. +Match duration requires a game start and uses receipt times. Population time +advances only when another report arrives in the same game; it uses the lower +player count at the interval's endpoints. There is no heartbeat requirement or +short interval limit between rounds. These are estimates between reported +snapshots, not continuous measurements of player presence. -Round, feature and unexpected-restart checks require the reporting protocol -below. Legacy round reports still update Fleet but cannot prove beta progress. -The game scripts themselves are uploaded artifacts, not source maintained in -this repository; update their reporter to emit the protocol before selecting -these requirements in production. +Health checks and safe restart gates use the existing container/RCON monitoring. +Health intervals spanning more than 60 seconds earn no healthy time. ## Coverage, rollout and outcomes Coverage can require all selected servers, a quorum, one per selected map, one per selected game, or one per selected game/map combination. Coverage -metadata is pinned when the beta opens. Register game/map metadata first. +metadata is pinned when the beta opens. Set wave size to 1 for a canary, a larger number for batches, or 0 for all -selected servers. Later waves reserve their artifact targets but receive no -files until every server in the prior waves has passed. All waves must start -testing before the final coverage rule can pass. Error-budget violations on -any tested server block promotion, including with quorum coverage. +selected servers. Later waves receive files after the previous waves pass. +Health-outage budget violations on any tested server block promotion. The test window starts when the beta opens, including time waiting for empty -servers. Missing evidence at the deadline produces **inconclusive**, not a -failure or an automatic rollback. Health and log monitoring continue after a -pass until promotion; a deteriorating result revokes readiness. +servers. Missing evidence at the deadline produces **inconclusive**. +Health monitoring continues after a pass until promotion. For failures, select hold as failed, pause for investigation, or automatic -rollback. Rollback restores only files touched by the beta. It can queue rolling -restarts when servers empty, or leave restarts manual. Manual rollback uses the -same selected restart policy. No beta auto-promotes. - -Pause holds evidence evaluation and rollout. Resume retains evidence and the -original deadline. A fresh attempt retires old evidence and restarts deployed -servers when empty; it does not extend the deadline. Expired or incomplete-file -deployments should be rolled back and reopened. Recovery cycles similarly -require fresh milestones/checklists, while carrying error budgets forward. - -Time accounting excludes intervals spanning monitoring gaps longer than 60 -seconds. A missing first sample earns no time. A log that cannot be read cannot -pass a clean-log requirement. Log backlog above 2 MB makes the plan inconclusive -instead of silently dropping evidence. Partial lines, rotations, byte cursors, -attempts and accepted events persist in SQLite. The detail page shows the most -recent 200 evidence entries; older entries remain in the database. - -## Game reporter protocol - -This section is a standalone integration guide for the T6 telemetry script. -It describes the **currently implemented API**, including its limitations. -Examples use `t6_zombie_town_1`; replace that with the exact, case-sensitive -Electric Cherry server registry name. The API base URL is the Electric Cherry -application, not Komodo or the game server's RCON address. - -### 1. Two reporting modes: Fleet display and beta evidence - -The reporter must work whether or not anybody is beta testing: - -| Situation | Reporter behavior | -| --- | --- | -| No eligible beta run for this server | Keep sending ordinary `round_change` reports for Fleet. Do not send beta-only events. | -| One eligible run discovered at process startup | Send the beta event stream with that run's `runId`. Scoped `round_change` reports also update Fleet. | -| Several eligible runs discovered at startup | Send a copy of each beta event to each run, with separate acknowledgement/queue state per run. | -| Beta opened while an existing game process is running | Continue its existing reporting mode. The beta waits for the server to empty and restarts it; the new process discovers the new run. | -| Beta has passed and is awaiting promotion | Continue reporting. A passed test still accepts evidence until promotion or another outcome stops it. | -| Beta paused, failed, expired, promoted, rolled back, or run superseded | Stop crediting events to that inactive run; continue ordinary round reporting. See the suspension rules below. | - -**No active beta is normal, not an error.** Fleet telemetry does not depend on a -beta, a run token, or a successful beta discovery request. - -### 2. Startup discovery and an empty result - -On **actual game-process startup**, request: - -```http -GET /api/telemetry?server=t6_zombie_town_1 -``` - -For a registered server without an eligible run, this returns **HTTP 200**: - -```json -{ - "runs": [] -} -``` - -This can mean there is no beta, the server is not a beta target, its wave has not -started, the beta is waiting to restart it, or the test is not accepting evidence. -Legacy betas without a test plan are not returned either. The response is a list -of eligible **runs**, not a list of every beta in the application. - -With an eligible run, the response is **HTTP 200**: - -```json -{ - "runs": [ - { - "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", - "betaId": 42 - } - ] -} -``` - -The response has `Cache-Control: no-store`. Do not assume it contains exactly one -entry, and do not choose only the first entry. `betaId` is informational; POST -requests use `runId`, not `betaId`, to identify the test attempt. - -A missing or unknown `server` query parameter returns **404**, not an empty -success. Treat that as a configuration issue. The GET handler currently only -checks registry existence; POST additionally rejects a server marked destroyed. -Do not interpret successful discovery as permission to report for another server. - -Cache the startup run list for this process lifetime. Electric Cherry creates a -new run before requesting each beta-controlled restart. That restart is the -boundary that makes gameplay belong to the newly deployed build. **Do not poll -for and adopt new run IDs in an old process, at a map reload, or mid-match.** -Never move queued events to a newer run ID. - -If discovery fails, use bounded retries with backoff while still initializing, -before gameplay starts. If startup discovery cannot finish, run in Fleet-only -mode for that process and log the problem. Do not attach later gameplay to a run -discovered after that startup boundary. A new process/retry is needed for a new -safe beta attachment; the current API has no late-attachment handshake. - -### 3. Reporting without a beta - -Send this on each round change, even when `runs` is empty: - -```http -POST /api/telemetry -Content-Type: application/json -``` - -```json -{ - "event": "round_change", - "server": "t6_zombie_town_1", - "round": 17, - "players": 4 -} -``` - -Example success response: - -```json -{ - "ok": true, - "telemetry": { - "server": "t6_zombie_town_1", - "round": 17, - "players": 4, - "receivedAt": "2026-09-18T12:00:00.000Z" - } -} -``` - -Omit `runId` entirely in this mode. `"runId": null` and `"runId": ""` are -invalid; they are not equivalent to omitting the field. The response has no -`test` property when `runId` was omitted. - -Without `runId`, only `round_change` is supported. Do not send unscoped `boot`, -`session_start`, `feature`, or `match_end` requests: they return 400. Do not buffer -feature occurrences from normal operation to submit when a future beta opens. - -Electric Cherry stores only the latest Fleet round report per server and stamps -its receipt time. Lower rounds are valid when a new match begins. These reports -do not change RCON player counts, restart safety decisions, or beta progress. - -### 4. Identifier lifetimes and the T6 process boundary - -| Field | Who creates it | Lifetime and rules | -| --- | --- | --- | -| `server` | Configuration | Exact registered name, unchanged across matches and process restarts. | -| `runId` | Electric Cherry | One server attempt/recovery phase. Use the value discovered at process startup. Treat it as opaque. | -| `bootId` | Reporter | Unique for each actual game-process lifetime. Keep it across matches, map reloads, and reporter reinitialization within that process. | -| `sessionId` | Reporter | Unique for each actual played match. Generate a new value for the next match, even if the players and map are unchanged. | -| `eventId` | Reporter | Unique for each logical event within a run, including across sessions/boots. A delivery retry reuses the exact same ID. | -| `sequence` | Reporter | Start at 0 for `session_start`; increment for every subsequent event in that session, across all event types. Reset only for a new session. | - -Generate collision-resistant IDs, such as UUIDs. A boot ID plus a session ID and -counter is also suitable for event IDs if the combined string fits the length -limit. Example IDs below are illustrative; never hardcode them in the script. -The same event ID can be used for copies sent to different runs because -deduplication is scoped to `(runId, eventId)`. - -**A GSC script's initialization is not necessarily a game-process startup.** -The T6 implementation must establish where its state survives match/map changes. -If script state resets on each map load, preserve process identity, the startup -run list, and delivery state in a process-lifetime facility or a helper that can -reliably identify the game process. A helper's own restart must not invent a new -game boot. Do not assume map-local globals provide process-lifetime storage. -If that distinction is unavailable, leave beta reporting disabled rather than -reporting a fabricated boot; ordinary Fleet round reports still work. - -An unexpected game-process restart may discover the **same** run ID. Send a new -boot ID: the backend counts additional distinct boots in a run as unexpected -restarts. A planned recovery restart receives a **new** run ID and starts a new -evidence phase. Restarting only the Electric Cherry web app changes neither the -game boot nor the reporter's cached IDs. - -### 5. Common request fields - -All requests use JSON objects and `Content-Type: application/json`. Every -beta-scoped event includes these fields unless the table says otherwise: - -| Field | Type | Constraints | -| --- | --- | --- | -| `event` | string | Exactly `boot`, `session_start`, `round_change`, `feature`, or `match_end`. | -| `server` | string | Registered server name, 1-255 characters, case-sensitive. | -| `runId` | string | Required for beta evidence; 1-128 characters from `A-Z a-z 0-9 _ . : -`. | -| `eventId` | string | Same identifier character/length limits. | -| `bootId` | string | Same identifier character/length limits. | -| `sessionId` | string | Same identifier limits; required except for `boot`. Omit for `boot`. | -| `sequence` | number | Nonnegative safe integer. Use 0 for `boot` and `session_start`. | -| `round` | number | Nonnegative safe integer; the actual player-facing round. Use 0 for `boot`. | -| `players` | number | Nonnegative safe integer; actual participating player count at the event. Use 0 for `boot`. | -| `name` | string | Required only for `feature`; 1-120 characters from the identifier character set. | - -Numbers must be JSON numbers, not strings, fractions, `null`, or negative values. -The maximum safe integer is 9007199254740991. Sequence numbers need to increase; -the backend does not require them to be contiguous, but the sender should emit a -single ordered stream. Do not use a separate counter for each event type. - -The examples below show excerpts from one process and match. Earlier round -reports are omitted, hence the sequence jumps to 25. The real script should send -each round change and increment its counter for every event it produces. - -### 6. Full event examples - -#### Process boot - -Send once per discovered run at real game-process startup, before any session -events. A retry is the same request, not another boot with another ID. - -```json -{ - "event": "boot", - "server": "t6_zombie_town_1", - "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", - "eventId": "boot-a81f:boot", - "bootId": "boot-a81f", - "sequence": 0, - "round": 0, - "players": 0 -} -``` - -#### Start of a played match - -Send when actual gameplay begins with players, not merely when an empty map -loads. Retain this `sessionId` until that match ends or is abandoned. - -```json -{ - "event": "session_start", - "server": "t6_zombie_town_1", - "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", - "eventId": "boot-a81f:match-b92c:0", - "bootId": "boot-a81f", - "sessionId": "match-b92c", - "sequence": 0, - "round": 1, - "players": 4 -} -``` - -After this is acknowledged, report the initial round with `round_change` too. -The `round` on `session_start` does not itself satisfy a round requirement. - -#### Reaching a round - -Send on entry to each round, with the player count sampled at that event. -There is no separate "round completed" event. - -```json -{ - "event": "round_change", - "server": "t6_zombie_town_1", - "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", - "eventId": "boot-a81f:match-b92c:25", - "bootId": "boot-a81f", - "sessionId": "match-b92c", - "sequence": 25, - "round": 25, - "players": 4 -} -``` - -This can satisfy a round-25 check requiring four players. It also records the -qualifying round for a later completed-match check. A `match_end` reporting round -25 does not replace a missing `round_change` event for that round. - -#### Feature: a successful perk purchase - -The top-level event is always `"feature"`. The custom feature identifier goes in -`name`; it is not a new top-level event type. - -```json -{ - "event": "feature", - "server": "t6_zombie_town_1", - "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", - "eventId": "boot-a81f:match-b92c:26", - "bootId": "boot-a81f", - "sessionId": "match-b92c", - "sequence": 26, - "round": 25, - "players": 4, - "name": "perk.purchased" -} -``` - -Emit when the purchase actually succeeds, not when a player opens a menu or -attempts an unaffordable purchase. One accepted event adds **one** to each -matching feature requirement whose minimum round/player qualifiers are met. -This event does not require waiting for the match to end. - -For example, configure **Feature event count** with event name `perk.purchased`, -target 10, minimum round 20, and minimum players 4. The example above contributes -one occurrence. A purchase at round 19 or with only three players can still be -accepted into the event stream, but contributes zero to that requirement. - -#### Feature: successful Pack-a-Punch completion - -```json -{ - "event": "feature", - "server": "t6_zombie_town_1", - "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", - "eventId": "boot-a81f:match-b92c:27", - "bootId": "boot-a81f", - "sessionId": "match-b92c", - "sequence": 27, - "round": 25, - "players": 4, - "name": "weapon.pack_a_punch.completed" -} -``` - -#### Feature: successful execution of a custom mechanic - -```json -{ - "event": "feature", - "server": "t6_zombie_town_1", - "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", - "eventId": "boot-a81f:match-b92c:28", - "bootId": "boot-a81f", - "sessionId": "match-b92c", - "sequence": 28, - "round": 25, - "players": 4, - "name": "mechanic.teleporter.completed" -} -``` - -These names are **suggested conventions**, not preinstalled T6 hooks. The script -must instrument the corresponding successful actions. Only emit a mechanic's -event on maps/builds where that mechanic actually exists. - -Names are matched exactly and case-sensitively against the UI's **Reported event -name**. A requirement named `perk.purchased` with target 10 counts ten accepted -occurrences of that exact name; `perk.*` is not a wildcard. - -For a specific perk, use a distinct name such as `perk.juggernog.purchased`. -The current API does not filter on a separate `perk`, `weapon`, or `playerId` -field. You may deliberately emit both a generic and a specific marker for one -action, but they must be separate events with separate event IDs and sequences; -each requirement still counts only its exact matching name. - -There is no bulk-count field. `"count": 10`, `value`, `metadata`, `timestamp`, -`duration`, and other extra properties are not consumed or preserved as beta -signal fields. Send one event for each real occurrence. Keep any additional -diagnostics in script logs until the API explicitly supports them. - -#### Normal match completion - -```json -{ - "event": "match_end", - "server": "t6_zombie_town_1", - "runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88", - "eventId": "boot-a81f:match-b92c:29", - "bootId": "boot-a81f", - "sessionId": "match-b92c", - "sequence": 29, - "round": 25, - "players": 4 -} -``` - -Send once when a played match ends normally, including a normal game-over. -Capture the actual terminal player count; the example's four players are not a -value to fabricate if players have already left. The backend retains earlier -round/player observations for qualification. - -Do **not** send `match_end` for a crash, forced process shutdown, arbitrary map -rotation, or an idle map cycle. A generic script teardown callback alone is not -proof of normal match completion. There is no `session_abort` event at present; -abandon that session and use a new ID when actual gameplay next begins. Never -replay events for an abandoned session into the new one. - -Accepted `match_end` closes the session. Later events for that session are not -accepted. Round-qualified or multi-player-qualified match requirements use this -stream; unqualified completed-match requirements use game logs instead, so the -same match is not counted twice for one requirement. - -### 7. Responses: transport success is not evidence acceptance - -An accepted `boot`, `session_start`, `feature`, or `match_end` returns HTTP 200: - -```json -{ - "ok": true, - "test": { - "accepted": true - } -} -``` - -An accepted scoped `round_change` additionally updates the Fleet display: - -```json -{ - "ok": true, - "telemetry": { - "server": "t6_zombie_town_1", - "round": 25, - "players": 4, - "receivedAt": "2026-09-18T12:25:00.000Z" - }, - "test": { - "accepted": true - } -} -``` - -Acceptance means the event was recorded in the run's stream. It does not mean -the whole test passed, or even that a feature/round event met a configured -requirement's qualifiers. - -A retry of an already accepted event returns HTTP 200: - -```json -{ - "ok": true, - "test": { - "accepted": false, - "reason": "Duplicate event" - } -} -``` - -Treat that exact duplicate response as delivery already acknowledged. Do not -generate a new ID to force the same action to count twice. - -An inactive run returns HTTP 200: - -```json -{ - "ok": true, - "test": { - "accepted": false, - "reason": "Run is inactive or superseded; obtain the current run at process startup" - } -} -``` - -Invalid ordering or session identity also returns HTTP 200: - -```json -{ - "ok": true, - "test": { - "accepted": false, - "reason": "Duplicate, out-of-order, closed, or unknown session/boot" - } -} -``` - -That last response is **not** an acknowledgement: inspect the boot/session/ -sequence handling. The API currently provides explanatory strings, not distinct -machine-readable error codes for every rejection. Do not treat every -`accepted: false` as a harmless duplicate or assume every inactive run is closed -permanently; paused runs use the same inactive response. - -**Fleet updates are independent of beta acceptance.** A syntactically valid -scoped `round_change` still writes the latest Fleet round even when -`test.accepted` is false, including a duplicate or inactive-run rejection. Its -response contains both `telemetry` and `test`. Do not replay an old round backlog -after newer reports: it can move the Fleet display backwards. There is no -client-timestamp freshness check. - -| HTTP result | Meaning and handling | -| --- | --- | -| 200, no `test` | Ordinary Fleet report accepted. | -| 200, `test.accepted: true` | Beta event accepted; advance that run's queue. | -| 200, exact `Duplicate event` reason | Already accepted earlier; advance that run's queue. | -| 200, inactive-run reason | Suspend beta delivery for that run; continue Fleet reporting. | -| 200, session/order rejection | Log an integration error and stop crediting that broken session. Do not invent a replacement session for an in-progress match. | -| 400 | Invalid JSON, fields, or event type; fix the payload instead of retrying unchanged forever. | -| 404 on POST | Server is unregistered or destroyed; check configuration/lifecycle. | -| 415 | Missing or incorrect JSON content type. | -| Network error, timeout, or 5xx | Delivery is uncertain; retry the same request with backoff within a bounded queue. | - -### 8. Ordering, retries, pauses, and multiple runs - -**Recommended sender behavior** (client implementation guidance, not extra API -features): - -1. Keep delivery asynchronous so a web-service outage never blocks the game. -2. Send `boot` first for each run and wait for acceptance or an acknowledged - duplicate before delivering its `session_start`. -3. Serialize the remaining events per run. Do not allow a later event to overtake - an uncertain request. Use the same session counter for rounds, features, and - match completion. -4. On timeout, retry with exactly the same event ID, sequence, and payload. The - first request may have committed even if its response was lost. -5. Bound queue size, retry count, and event age. For example, back off through - 1, 2, 5, and 10 seconds rather than retrying every frame. These intervals are - a suggested client policy, not server-enforced values. -6. If a long outage makes the queue stale, abandon beta evidence for that session - and keep the current Fleet round visible. Do not replay historical feature or - completion events hours later. Resume beta evidence at a later real match - only if the same startup-pinned run is still eligible and boot delivery is - established. -7. Keep acknowledgement state separately for every discovered run. Failure on - one run does not change another run's IDs or acknowledged sequence. Reuse the - logical event ID across copies, changing only `runId`. - -The server stamps **receipt time**, not game event time. In particular, match -duration is measured between accepted `session_start` and `match_end` receipts. -Delayed replay can distort duration and no client-provided timestamp fixes it. -Keep beta events near real time; do not use them as a historical-import API. - -While scoped round delivery is healthy, no extra unscoped copy is necessary. -If beta delivery is suspended or lagging, ordinary round reports can keep Fleet -current. Retire the affected session queue rather than later sending its old -rounds behind newer unscoped reports. A simple implementation may serialize -delivery across all run copies to avoid cross-run round reordering. - -For paused/inactive runs, the simplest safe policy is Fleet-only reporting until -the next actual process startup. If automatic resume within the same process is -desired, the reporter may recheck **only IDs already pinned at startup**: - -- Intersect discovery results with the original startup list. Never adopt an - unfamiliar run ID returned by a later GET. -- When a cached run becomes eligible again, resume at the **next real match**, - using the existing boot ID and a new session ID. Establish boot delivery first; - reusing the original boot request/event ID is safe if its acknowledgement was - lost. -- Discard events generated during suspension. Do not manufacture a new session - start in the middle of an existing match or replay paused gameplay. -- If startup returned `runs: []`, there are no pinned IDs to resume. Stay - Fleet-only until a real process restart. Similarly, a process started while a - test was paused may require an operator-requested fresh attempt after resume. - -There is no polling interval or resume notification supplied by this API. The -reporter owns its retry/suspension policy. The rule about startup pinning must -remain true regardless of that policy. - -### 9. T6 implementation handoff checklist - -- Configure the Electric Cherry URL and exact server registry name. -- Preserve ordinary round reports when no beta is active or the service is - temporarily unreachable; never block gameplay waiting for HTTP. -- Identify actual game-process boots separately from GSC initialization and - map/match resets. Cache discovery results for the game process lifetime. -- Instrument real session start, round entry, normal game-over, and successful - feature actions. Confirm each chosen T6 callback's semantics; do not infer - normal completion from generic teardown. -- Use one sequence counter per session, unique event IDs, stable retry payloads, - and an ordered bounded queue with separate acknowledgements per run. -- Decide and document the feature names the script emits so beta creators can - enter those exact names. Extra JSON metadata cannot be used as a qualifier. -- Keep beta-only events scoped to startup-pinned runs. Do not backfill pre-beta, - paused, abandoned, or previous-process gameplay into a new run. -- Handle `ok` and `test.accepted` separately. Log rejections with enough context - to diagnose the run, boot, session, event ID, and sequence. - -Suggested verification cases for the telemetry-script session: - -| Case | Expected result | -| --- | --- | -| Startup returns an empty run list | Round reports continue; no beta-only POSTs. | -| Beta opens during an ordinary match | No new token adopted; beta reporting begins after the beta-controlled process restart. | -| Normal new match, same process | Same boot/run IDs; new session ID and sequence starting at 0. | -| Actual unexpected process restart | New boot ID; rediscover runs; no old queued session events replayed. | -| Retry after a lost response | Same event ID and payload; one evidence entry and a duplicate acknowledgement. | -| Round 25 with four players, then normal match end | Round evidence can pass immediately; a qualifying-match count advances only at normal completion. | -| Ten successful `perk.purchased` events | Ten uniquely identified feature events, counted only when each event meets the configured round/player qualifiers. | -| Crash or idle map reset | No fabricated `match_end`. | -| Paused, closed, or superseded run | Beta delivery suspended; Fleet remains current; never switch old queued events to a new token. | -| Two startup run IDs | Each receives the event once, with independent acknowledgements. | -| Reporter script reload, same game process | No new boot identity or fresh adoption of run IDs. | - -The endpoint retains the application's existing trusted-network model. A `runId` -correlates evidence; it is **not** an authentication secret or proof of loaded -artifact bytes. Correct process/session attribution depends on the reporter -following the lifecycle rules above. +rollback. Rollback restores only files touched by the beta. Restarts can wait +for empty servers or remain manual. Promotion is always manual. + +Pause holds evaluation and rollout. Resume retains completed progress and the +original deadline, but starts new timing intervals. A fresh attempt resets +progress and restarts deployed servers when empty. Recovery cycles repeat +gameplay milestones and checklist sign-offs while carrying health outages forward. + +Plans, progress, and evidence persist in SQLite. The detail page shows the most +recent 200 evidence entries. On upgrading from the old reporter protocol, +active progress resets. Unsupported checks in existing plans and saved presets +become manual review items; they cannot silently pass. ## API and storage -`POST /api/betas` accepts `testPlan`, validated by `parseBetaPlan` in -`src/lib/beta-plan.ts`. New plans always verify restart recovery before testing. -`GET /api/betas/:id` includes the plan, outcome, per-server requirement progress -and recent evidence under `test`. +`POST /api/betas` accepts `testPlan`. +`GET /api/betas/:id` includes the plan, outcome, per-server progress, and +recent evidence under `test`. -`POST /api/betas/:id/test` accepts actions `pause`, `resume`, `retry`, or +`POST /api/betas/:id/test` accepts `pause`, `resume`, `retry`, or `sign-off` (with `server`, `requirementId`, and `note`). -`GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are copied -into a beta; later preset edits never change an active test. +`GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are +copied into a beta; later edits do not change an active plan. -`beta_test_plans`, `beta_test_coverage`, `beta_test_runs`, `beta_test_evidence`, -and `beta_test_presets` hold configuration and results. `beta_file_changes` -records which files were touched, including partially completed deployments. -Legacy beta lifecycle states remain compatible with artifact holds and history; -test outcomes are stored separately. Holds last until promotion or rollback. +Internal progress records support attempts and recovery cycles. The reporter +does not know about or supply these records. Legacy betas without a test plan +also use `game_end` for their match-completion gate. diff --git a/src/lib/beta-plan-summary.ts b/src/lib/beta-plan-summary.ts index 98bcdfb..4545932 100644 --- a/src/lib/beta-plan-summary.ts +++ b/src/lib/beta-plan-summary.ts @@ -18,10 +18,7 @@ export function requirementSummary(r: BetaRequirement): string { case 'sustained-population': return `Keep at least ${count(r.minPlayers, 'player')} online for ${duration(r.target)}`; case 'healthy-minutes': return `Stay continuously healthy for ${duration(r.target)}`; case 'failure-budget': return r.target === 0 ? 'Allow no health outages' : `Allow at most ${count(r.target, 'health outage')}`; - case 'restart-budget': return r.target === 0 ? 'Allow no unexpected restarts' : `Allow at most ${count(r.target, 'unexpected restart')}`; - case 'clean-logs': return r.target === 0 ? 'Allow no detected script errors' : `Allow at most ${count(r.target, 'detected script error')}`; case 'recovery': return `Repeat the checks after ${count(r.target, 'recovery restart')}`; - case 'feature': return `Observe ${r.event || 'a named feature event'} ${count(r.target, 'time')}${players}${r.minRound ? ` at round ${r.minRound} or later` : ''}`; case 'manual': return r.label; } } @@ -29,9 +26,9 @@ export function coverageSummary(plan: BetaPlan): string { return plan.scope === 'all' ? 'Every selected server' : plan.scope === 'quorum' ? `At least ${count(plan.quorum, 'selected server')}` : plan.scope === 'map' ? 'One server per selected map' : plan.scope === 'game' ? 'One server per selected game' : 'One server per game / map'; } -export function needsSessionTelemetry(r: BetaRequirement): boolean { - return ['round', 'feature', 'restart-budget'].includes(r.kind) || (r.kind === 'matches' && (r.minRound > 0 || r.minPlayers > 1)); +export function needsGameTelemetry(r: BetaRequirement): boolean { + return ['round', 'matches', 'population', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind); } export function requirementCategory(r: BetaRequirement): string { - return r.kind === 'manual' ? 'Tester sign-off' : ['healthy-minutes', 'failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind) ? 'Continuous check' : 'Milestone'; + return r.kind === 'manual' ? 'Tester sign-off' : ['healthy-minutes', 'failure-budget'].includes(r.kind) ? 'Continuous check' : 'Milestone'; } diff --git a/src/lib/beta-plan.ts b/src/lib/beta-plan.ts index f7e9fd7..65bd775 100644 --- a/src/lib/beta-plan.ts +++ b/src/lib/beta-plan.ts @@ -1,5 +1,5 @@ export const requirementKinds = ['round', 'matches', 'populated-minutes', 'player-hours', 'population', - 'sustained-population', 'healthy-minutes', 'failure-budget', 'restart-budget', 'clean-logs', 'recovery', 'feature', 'manual'] as const; + 'sustained-population', 'healthy-minutes', 'failure-budget', 'recovery', 'manual'] as const; export type RequirementKind = typeof requirementKinds[number]; export interface BetaRequirement { id: string; @@ -9,7 +9,6 @@ export interface BetaRequirement { minRound: number; minPlayers: number; minMinutes: number; - event: string; } export interface BetaPlan { version: 1; @@ -25,12 +24,11 @@ export const requirementLabels: Record = { round: 'Reach round', matches: 'Complete qualifying matches', 'populated-minutes': 'Populated minutes', 'player-hours': 'Player-hours', population: 'Reach player count', 'sustained-population': 'Sustained population (minutes)', 'healthy-minutes': 'Continuous healthy minutes', 'failure-budget': 'Maximum health outages', - 'restart-budget': 'Maximum unexpected restarts', - 'clean-logs': 'Maximum script errors', recovery: 'Successful recovery cycles', feature: 'Feature event count', manual: 'Tester checklist' + recovery: 'Successful recovery cycles', manual: 'Tester checklist' }; export function requirement(kind: RequirementKind, id: string): BetaRequirement { - return { id, kind, label: requirementLabels[kind], target: kind === 'round' ? 25 : ['failure-budget', 'restart-budget', 'clean-logs'].includes(kind) ? 0 : 1, - minRound: 0, minPlayers: 1, minMinutes: 0, event: '' }; + return { id, kind, label: requirementLabels[kind], target: kind === 'round' ? 25 : kind === 'failure-budget' ? 0 : 1, + minRound: 0, minPlayers: 1, minMinutes: 0 }; } export function defaultPlan(): BetaPlan { return { version: 1, requirements: [requirement('healthy-minutes', 'health')], scope: 'all', quorum: 1, @@ -44,7 +42,7 @@ export const builtInPlans: Array<{ name: string; plan: BetaPlan }> = [ ] } }, { name: 'Long soak', plan: { ...defaultPlan(), timeoutHours: 72, requirements: [ { ...requirement('healthy-minutes', 'health'), target: 360 }, - { ...requirement('player-hours', 'participation'), target: 10 }, requirement('clean-logs', 'logs') + { ...requirement('player-hours', 'participation'), target: 10 }, requirement('failure-budget', 'outages') ] } } ]; @@ -68,13 +66,12 @@ export function parseBetaPlan(value: unknown): BetaPlan { for (const key of ['target', 'minRound', 'minPlayers', 'minMinutes'] as const) { if (typeof r[key] !== 'number' || !Number.isFinite(r[key]) || r[key] < 0 || r[key] > 1_000_000) throw new Error(`Invalid requirement ${key}`); } - if (!['failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind) && r.target <= 0) throw new Error('Requirement target must be positive'); - if (['round', 'matches', 'population', 'failure-budget', 'restart-budget', 'clean-logs', 'recovery', 'feature', 'manual'].includes(r.kind) && !Number.isInteger(r.target)) throw new Error('Count targets must be integers'); + if (r.kind !== 'failure-budget' && r.target <= 0) throw new Error('Requirement target must be positive'); + if (['round', 'matches', 'population', 'failure-budget', 'recovery', 'manual'].includes(r.kind) && !Number.isInteger(r.target)) throw new Error('Count targets must be integers'); if (!Number.isInteger(r.minRound) || !Number.isInteger(r.minPlayers)) throw new Error('Round and player qualifiers must be integers'); if (r.kind === 'manual' && r.target !== 1) throw new Error('Add a separate requirement for each checklist item'); - if (typeof r.event !== 'string' || r.event.length > 120 || (r.kind === 'feature' && !/^[a-zA-Z0-9_.:-]+$/.test(r.event))) throw new Error('Feature requirements need an event name'); return { id: r.id, kind: r.kind, label: r.label.trim(), target: r.target, minRound: r.minRound, - minPlayers: r.minPlayers, minMinutes: r.minMinutes, event: r.event }; + minPlayers: r.minPlayers, minMinutes: r.minMinutes }; }); return { version: 1, requirements, scope: p.scope, quorum: p.quorum, timeoutHours: p.timeoutHours, failureAction: p.failureAction, rollbackRestart: p.rollbackRestart, waveSize: p.waveSize }; @@ -85,6 +82,6 @@ export interface BetaTestView { plan: BetaPlan; outcome: string; reason: string | null; - servers: Array<{ server: string; runId: string; phase: number; passed: boolean; progress: RequirementProgress[] }>; + servers: Array<{ server: string; phase: number; passed: boolean; progress: RequirementProgress[] }>; evidence: Array<{ id: number; server: string; received_at: string; kind: string; payload: string }>; } diff --git a/src/lib/fleet-overview.ts b/src/lib/fleet-overview.ts index 825e2f9..d128466 100644 --- a/src/lib/fleet-overview.ts +++ b/src/lib/fleet-overview.ts @@ -1,7 +1,7 @@ import { parseServerName, zombieMapBySlug } from './map-aliases.js'; import { freshStatusTimestamp } from './server-freshness.js'; import type { ZombieGame } from './map-aliases.js'; -import type { RoundTelemetry } from './telemetry.js'; +import type { GameTelemetry } from './telemetry.js'; import type { Health, RosterPlayer } from './ui/stores/servers.js'; import { mapTitle, serverTitle } from './ui/server-labels.js'; @@ -63,8 +63,8 @@ export function fleetIdentity(server: FleetServer) { }; } -export function currentRound(server: FleetServer, report: RoundTelemetry | undefined, now: number): number | null { - if (!report || !freshServer(server, now) || !server.active_since || !server.game_started_at +export function currentRound(server: FleetServer, report: GameTelemetry | undefined, now: number): number | null { + if (!report || report.event !== 'round_change' || !freshServer(server, now) || !server.active_since || !server.game_started_at || server.game_ended_at || (server.last_players ?? 0) === 0) return null; const received = Date.parse(report.receivedAt); const age = now - received; diff --git a/src/lib/server/db/index.ts b/src/lib/server/db/index.ts index ec13947..ff33da7 100644 --- a/src/lib/server/db/index.ts +++ b/src/lib/server/db/index.ts @@ -3,6 +3,8 @@ import { mkdirSync } from 'node:fs'; import { dirname, resolve } from 'node:path'; import { config } from '../config.js'; import schemaSql from './schema.sql?raw'; +import { requirement, type BetaPlan, type BetaRequirement } from '../../beta-plan.js'; +import { emptyMetrics } from '../fleet/beta-evaluation.js'; let _db: Database.Database | null = null; @@ -190,6 +192,40 @@ function assertSnapshotSchemeSafe(handle: Database.Database): void { handle.prepare("INSERT INTO settings (key, value) VALUES ('snapshots_v1', '1')").run(); } +// Preserve old plans as reviewable checklists instead of silently passing checks +// that the three-event protocol can no longer perform. Existing evidence stays +// in history; active progress starts fresh under the new gameplay contract. +function migrateGameTelemetry(handle: Database.Database): void { + if (handle.prepare("SELECT 1 FROM settings WHERE key = 'game_telemetry_v2'").get()) return; + handle.transaction(() => { + const convert = (raw: string): string => { + const plan = JSON.parse(raw) as BetaPlan; + plan.requirements = plan.requirements.map(r => { + const { event: _event, ...rest } = r as BetaRequirement & { event?: string }; + return ['feature', 'restart-budget', 'clean-logs'].includes(r.kind) + ? { ...requirement('manual', r.id), label: `Review: ${r.label}`.slice(0, 200) } : rest; + }); + return JSON.stringify(plan); + }; + const plans = handle.prepare('SELECT beta_id, plan FROM beta_test_plans').all() as Array<{ beta_id: number; plan: string }>; + for (const row of plans) handle.prepare('UPDATE beta_test_plans SET plan = ? WHERE beta_id = ?').run(convert(row.plan), row.beta_id); + const presets = handle.prepare('SELECT name, plan FROM beta_test_presets').all() as Array<{ name: string; plan: string }>; + for (const row of presets) handle.prepare('UPDATE beta_test_presets SET plan = ? WHERE name = ?').run(convert(row.plan), row.name); + handle.prepare(`UPDATE beta_test_runs SET metrics = ?, cursors = '{}' WHERE retired_at IS NULL + AND beta_id IN (SELECT id FROM betas WHERE state IN ('running','ready-promote'))`).run(JSON.stringify(emptyMetrics())); + handle.exec(`UPDATE beta_test_plans SET outcome = 'active', reason = 'Gameplay telemetry updated; collecting fresh script events' + WHERE outcome IN ('active','passed') AND beta_id IN (SELECT id FROM betas WHERE state IN ('running','ready-promote')); + UPDATE script_versions SET state = 'untested' WHERE state = 'tested' + AND id IN (SELECT bs.script_version_id FROM beta_scripts bs JOIN beta_test_plans p ON p.beta_id = bs.beta_id WHERE p.outcome = 'active') + AND id NOT IN (SELECT current_version_id FROM scripts WHERE current_version_id IS NOT NULL) + AND NOT EXISTS (SELECT 1 FROM beta_scripts bs JOIN betas b ON b.id = bs.beta_id + WHERE bs.script_version_id = script_versions.id AND b.state = 'promoted'); + UPDATE betas SET state = 'running' WHERE state = 'ready-promote' AND id IN (SELECT beta_id FROM beta_test_plans WHERE outcome = 'active'); + UPDATE beta_servers SET stage = 'observing' WHERE stage = 'ready' AND beta_id IN (SELECT beta_id FROM beta_test_plans WHERE outcome = 'active'); + INSERT INTO settings (key, value) VALUES ('game_telemetry_v2', '1');`); + })(); +} + export function db(): Database.Database { if (_db) return _db; @@ -212,6 +248,7 @@ export function db(): Database.Database { migrateTable(handle, 'beta_scripts', BETA_SCRIPTS_COLUMNS); migrateTable(handle, 'deployment_scripts', DEPLOYMENT_SCRIPTS_COLUMNS); migrateTable(handle, 'deployments', DEPLOYMENTS_COLUMNS); + migrateTable(handle, 'server_telemetry', [{ name: 'payload', ddl: 'ALTER TABLE server_telemetry ADD COLUMN payload TEXT' }]); // Kind-referencing indexes: safe here because migrateTable above just // ALTER'd the column into existence if it was missing. handle.exec( @@ -221,6 +258,7 @@ export function db(): Database.Database { handle.exec(`CREATE INDEX IF NOT EXISTS idx_scripts_kind ON scripts(kind)`); backfillMultiScriptJoins(handle); assertSnapshotSchemeSafe(handle); + migrateGameTelemetry(handle); } catch (err) { handle.close(); throw err; diff --git a/src/lib/server/db/schema.sql b/src/lib/server/db/schema.sql index 9836178..ed78c2e 100644 --- a/src/lib/server/db/schema.sql +++ b/src/lib/server/db/schema.sql @@ -39,8 +39,8 @@ CREATE TABLE IF NOT EXISTS servers ( -- and resume from the same anchor when the server comes back. active_since TEXT, -- ISO ts of last 0 -> >0 player transition inactive_since TEXT, -- ISO ts of last >0 -> 0 player transition - game_started_at TEXT, -- wall-clock ISO when we first saw a J; line for the current match cycle - game_ended_at TEXT, -- wall-clock ISO when we first saw ExitLevel/ShutdownGame end it + game_started_at TEXT, -- receipt time of the script's game_start + game_ended_at TEXT, -- receipt time of the script's game_end game_cycle_key TEXT, -- opaque marker for the currently-tracked cycle (the first J; line after the latest InitGame); lets us detect same-players game restarts created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ); @@ -314,7 +314,8 @@ CREATE TABLE IF NOT EXISTS server_telemetry ( server TEXT PRIMARY KEY REFERENCES servers(name) ON DELETE CASCADE, round INTEGER NOT NULL, players INTEGER NOT NULL, - received_at TEXT NOT NULL + received_at TEXT NOT NULL, + payload TEXT ); -- Lifecycle history deliberately survives removal of the server registry row. CREATE TABLE IF NOT EXISTS server_deployments ( diff --git a/src/lib/server/fleet/beta-evaluation.ts b/src/lib/server/fleet/beta-evaluation.ts index ab94827..3299ef6 100644 --- a/src/lib/server/fleet/beta-evaluation.ts +++ b/src/lib/server/fleet/beta-evaluation.ts @@ -1,108 +1,83 @@ import type { BetaPlan, RequirementProgress } from '../../beta-plan.js'; +import type { GameEvent } from '../../telemetry.js'; -export interface SessionEvidence { - started: number; - sequence: number; +export interface MatchEvidence { + map: string; + started?: number; + lastEvent: number; + players: number; ended: boolean; rounds: Record; - peakPlayers: number; } export interface TestMetrics { values: Record; - sessions: Record; + match?: MatchEvidence; manual: Record; - bootId?: string; - boots: string[]; - restarts: number; lastSample?: number; wasHealthy?: boolean; - lastPlayers?: number; outages: number; - logErrors: number; - logAvailable?: boolean; streaks: Record; } -export const emptyMetrics = (): TestMetrics => ({ values: {}, sessions: {}, manual: {}, boots: [], restarts: 0, outages: 0, logErrors: 0, streaks: {} }); +export const emptyMetrics = (): TestMetrics => ({ values: {}, manual: {}, outages: 0, streaks: {} }); -export interface TestSignal { - event: 'boot' | 'session_start' | 'round_change' | 'match_end' | 'feature'; - bootId: string; - sessionId: string; - sequence: number; - round: number; - players: number; - name: string; -} - -/** Events are accepted once, in session order; a match must start in this run. */ -export function applySignal(plan: BetaPlan, metrics: TestMetrics, signal: TestSignal, now: number): boolean { - if (signal.event === 'boot') { - if (metrics.boots.includes(signal.bootId)) return false; - if (metrics.bootId) metrics.restarts++; - metrics.boots.push(signal.bootId); - metrics.bootId = signal.bootId; - return true; +/** Gameplay comes only from the script. No reporter IDs or beta handshake. */ +export function applyGameEvent(plan: BetaPlan, metrics: TestMetrics, event: GameEvent, now: number): void { + const players = event.players.length; + if (event.event === 'game_start' || !metrics.match || metrics.match.map !== event.map) { + metrics.match = { map: event.map, started: event.event === 'game_start' ? now : undefined, + lastEvent: now, players, ended: false, rounds: {} }; + metrics.streaks = {}; } - if (metrics.bootId !== signal.bootId) return false; - const key = `${signal.bootId}:${signal.sessionId}`; - if (signal.event === 'session_start') { - if (metrics.sessions[key]) return false; - metrics.sessions[key] = { started: now, sequence: signal.sequence, ended: false, rounds: {}, peakPlayers: signal.players }; - return true; - } - const session = metrics.sessions[key]; - if (!session || session.ended || signal.sequence <= session.sequence) return false; - session.sequence = signal.sequence; - session.peakPlayers = Math.max(session.peakPlayers, signal.players); + const match = metrics.match; + if (match.ended) return; + const minutes = Math.max(0, now - match.lastEvent) / 60_000; for (const r of plan.requirements) { const current = metrics.values[r.id] ?? 0; - if (signal.event === 'round_change' && signal.players >= r.minPlayers) { - session.rounds[r.id] = Math.max(session.rounds[r.id] ?? 0, signal.round); - if (r.kind === 'round') metrics.values[r.id] = Math.max(current, signal.round); + const populated = players >= Math.max(1, r.minPlayers) && match.players >= Math.max(1, r.minPlayers); + if (r.kind === 'population') metrics.values[r.id] = Math.max(current, players); + if (r.kind === 'populated-minutes' && populated) metrics.values[r.id] = current + minutes; + if (r.kind === 'player-hours' && populated) metrics.values[r.id] = current + minutes * Math.min(players, match.players) / 60; + if (r.kind === 'sustained-population') { + metrics.streaks[r.id] = populated ? (metrics.streaks[r.id] ?? 0) + minutes : 0; + metrics.values[r.id] = Math.max(current, metrics.streaks[r.id]); + } + if (event.event !== 'game_start' && players >= Math.max(1, r.minPlayers)) { + match.rounds[r.id] = Math.max(match.rounds[r.id] ?? 0, event.round); + if (r.kind === 'round') metrics.values[r.id] = Math.max(current, event.round); + } + if (event.event === 'game_end' && r.kind === 'matches' + && (match.rounds[r.id] ?? -1) >= r.minRound + && (r.minMinutes === 0 || (match.started !== undefined && now - match.started >= r.minMinutes * 60_000))) { + metrics.values[r.id] = current + 1; } - if (signal.event === 'match_end' && r.kind === 'matches' && (r.minRound > 0 || r.minPlayers > 1) - && (session.rounds[r.id] ?? 0) >= r.minRound && session.peakPlayers >= r.minPlayers - && now - session.started >= r.minMinutes * 60_000) metrics.values[r.id] = current + 1; - if (signal.event === 'feature' && r.kind === 'feature' && signal.name === r.event - && signal.players >= r.minPlayers && signal.round >= r.minRound) metrics.values[r.id] = current + 1; } - if (signal.event === 'match_end') session.ended = true; - return true; + match.lastEvent = now; + match.players = players; + match.ended = event.event === 'game_end'; } -/** Sample both ends of an interval. Monitoring gaps never earn playtime. */ -export function applySample(plan: BetaPlan, m: TestMetrics, at: number, healthy: boolean, players: number): void { +/** Infrastructure health is separate from script-reported gameplay. */ +export function applySample(plan: BetaPlan, m: TestMetrics, at: number, healthy: boolean): void { if (m.lastSample !== undefined && at <= m.lastSample) return; const gap = m.lastSample === undefined ? Infinity : at - m.lastSample; const contiguous = gap <= 60_000; const minutes = healthy && m.wasHealthy && contiguous ? gap / 60_000 : 0; if (!healthy && m.wasHealthy === true) m.outages++; for (const r of plan.requirements) { - const value = m.values[r.id] ?? 0; - const populated = players >= Math.max(1, r.minPlayers) && (m.lastPlayers ?? 0) >= Math.max(1, r.minPlayers); - if (r.kind === 'populated-minutes' && populated) m.values[r.id] = value + minutes; - if (r.kind === 'player-hours' && populated) m.values[r.id] = value + minutes * Math.min(players, m.lastPlayers ?? 0) / 60; - if (r.kind === 'population' && healthy) m.values[r.id] = Math.max(value, players); - if (r.kind === 'healthy-minutes') m.values[r.id] = healthy && contiguous && m.wasHealthy ? value + minutes : 0; - if (r.kind === 'sustained-population') { - m.streaks[r.id] = healthy && populated && contiguous ? (m.streaks[r.id] ?? 0) + minutes : 0; - m.values[r.id] = Math.max(value, m.streaks[r.id]); - } + if (r.kind === 'healthy-minutes') m.values[r.id] = healthy && contiguous && m.wasHealthy ? (m.values[r.id] ?? 0) + minutes : 0; } - m.lastSample = at; m.wasHealthy = healthy; m.lastPlayers = players; + m.lastSample = at; m.wasHealthy = healthy; } export function evaluateRequirements(plan: BetaPlan, metrics: TestMetrics, phase: number): RequirementProgress[] { return plan.requirements.map(r => { - const budget = ['failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind); - const current = r.kind === 'failure-budget' ? metrics.outages : r.kind === 'clean-logs' ? metrics.logErrors - : r.kind === 'restart-budget' ? metrics.restarts - : r.kind === 'manual' ? Number(!!metrics.manual[r.id]) : r.kind === 'recovery' ? phase : metrics.values[r.id] ?? 0; - const available = (r.kind !== 'clean-logs' || metrics.logAvailable === true) && (r.kind !== 'restart-budget' || !!metrics.bootId); - return { id: r.id, label: r.label, current, target: r.target, passed: available && (budget ? current <= r.target : current >= r.target), + const budget = r.kind === 'failure-budget'; + const current = budget ? metrics.outages : r.kind === 'manual' ? Number(!!metrics.manual[r.id]) + : r.kind === 'recovery' ? phase : metrics.values[r.id] ?? 0; + return { id: r.id, label: r.label, current, target: r.target, passed: budget ? current <= r.target : current >= r.target, failed: budget && current > r.target, detail: r.kind === 'manual' ? metrics.manual[r.id] ?? 'Awaiting tester sign-off' - : !available ? r.kind === 'restart-budget' ? 'Waiting for boot telemetry' : 'Waiting for a readable console log' : budget ? `${current} / ${r.target} allowed; monitored until promotion` + : budget ? `${current} / ${r.target} allowed; monitored until promotion` : `${Math.round(current * 100) / 100} / ${r.target}` }; }); } diff --git a/src/lib/server/fleet/beta-tests.ts b/src/lib/server/fleet/beta-tests.ts index c978f77..fb08fa0 100644 --- a/src/lib/server/fleet/beta-tests.ts +++ b/src/lib/server/fleet/beta-tests.ts @@ -1,13 +1,11 @@ import { randomUUID } from 'node:crypto'; -import { open, stat } from 'node:fs/promises'; import { db } from '../db/index.js'; import { parseBetaPlan, type BetaPlan, type BetaTestView } from '../../beta-plan.js'; -import { applySample, applySignal, emptyMetrics, evaluateRequirements, type TestMetrics, type TestSignal } from './beta-evaluation.js'; -import { gameLogKey, resolveLogPath, type LogKey } from './logs.js'; +import type { GameTelemetry } from '../../telemetry.js'; +import { applySample, applyGameEvent, emptyMetrics, evaluateRequirements, type TestMetrics } from './beta-evaluation.js'; -export interface TestRun { id: string; beta_id: number; server: string; phase: number; created_at: string; observing_at: string | null; metrics: string; cursors: string } +export interface TestRun { id: string; beta_id: number; server: string; phase: number; created_at: string; observing_at: string | null; metrics: string } export interface StoredPlan { beta_id: number; plan: string; outcome: string; reason: string | null } -interface Cursor { position: number; identity: string; partial: string; anchor?: string; generation?: string; session?: string; joined?: boolean; started?: number } const nowIso = () => new Date().toISOString(); export const storedPlan = (id: number) => db().prepare('SELECT * FROM beta_test_plans WHERE beta_id = ?').get(id) as StoredPlan | undefined; export const currentRun = (id: number, server: string) => db().prepare('SELECT * FROM beta_test_runs WHERE beta_id = ? AND server = ? AND retired_at IS NULL').get(id, server) as TestRun | undefined; @@ -25,146 +23,47 @@ export function evidence(run: TestRun, eventId: string, kind: string, payload: u .run(run.id, eventId, nowIso(), kind, JSON.stringify(payload)).changes > 0; } -/** Snapshot log offsets BEFORE restart: pre-beta log entries can never qualify. */ +/** Internal progress record; never sent to or required from the game script. */ export async function prepareRun(betaId: number, server: string, phase = 0): Promise { - const cursors: Record = {}; - for (const key of [gameLogKey(server), 'console'] as LogKey[]) { - const path = resolveLogPath(server, key); - const info = path ? await stat(path).catch(() => null) : null; - let anchor = ''; - if (path && info?.size) { - const file = await open(path, 'r'); - try { - const buffer = Buffer.alloc(Math.min(info.size, 128)); - await file.read(buffer, 0, buffer.length, info.size - buffer.length); - anchor = buffer.toString('base64'); - } finally { await file.close(); } - } - cursors[key] = { position: info?.size ?? 0, identity: info ? `${info.dev}:${info.ino}` : '', partial: '', anchor, generation: randomUUID() }; - } - const id = randomUUID(); db().transaction(() => { db().prepare('UPDATE beta_test_runs SET retired_at = ? WHERE beta_id = ? AND server = ? AND retired_at IS NULL').run(nowIso(), betaId, server); - db().prepare('INSERT INTO beta_test_runs (id, beta_id, server, phase, created_at, metrics, cursors) VALUES (?, ?, ?, ?, ?, ?, ?)') - .run(id, betaId, server, phase, nowIso(), JSON.stringify(emptyMetrics()), JSON.stringify(cursors)); + db().prepare('INSERT INTO beta_test_runs (id, beta_id, server, phase, created_at, metrics) VALUES (?, ?, ?, ?, ?, ?)') + .run(randomUUID(), betaId, server, phase, nowIso(), JSON.stringify(emptyMetrics())); })(); return currentRun(betaId, server)!; } -export function acceptTestSignal(server: string, runId: string, eventId: string, signal: TestSignal): { accepted: boolean; reason?: string } { - return db().transaction(() => { - const run = db().prepare(`SELECT r.* FROM beta_test_runs r JOIN betas b ON b.id = r.beta_id - JOIN beta_test_plans p ON p.beta_id = b.id WHERE r.id = ? AND r.server = ? AND r.retired_at IS NULL - AND b.state IN ('running','ready-promote') AND p.outcome IN ('active','passed')`).get(runId, server) as TestRun | undefined; - if (!run) return { accepted: false, reason: 'Run is inactive or superseded; obtain the current run at process startup' }; - if (db().prepare('SELECT 1 FROM beta_test_evidence WHERE run_id = ? AND event_id = ?').get(runId, `telemetry:${eventId}`)) return { accepted: false, reason: 'Duplicate event' }; +/** Route ordinary server events to every deployed, active beta automatically. */ +export function recordGameEvent(event: GameTelemetry): void { + const runs = db().prepare(`SELECT r.* FROM beta_test_runs r JOIN betas b ON b.id = r.beta_id + JOIN beta_test_plans p ON p.beta_id = b.id + JOIN beta_servers s ON s.beta_id = b.id AND s.server = r.server + WHERE r.server = ? AND r.retired_at IS NULL AND b.state IN ('running','ready-promote') + AND p.outcome IN ('active','passed') AND s.stage IN ('awaiting-health','observing','ready') + AND s.restarted_at IS NOT NULL`).all(event.server) as TestRun[]; + for (const run of runs) { const plan = parseBetaPlan(JSON.parse(storedPlan(run.beta_id)!.plan)); const metrics = readMetrics(run); - if (!applySignal(plan, metrics, signal, Date.now())) return { accepted: false, reason: 'Duplicate, out-of-order, closed, or unknown session/boot' }; - evidence(run, `telemetry:${eventId}`, signal.event, signal); + applyGameEvent(plan, metrics, event, Date.parse(event.receivedAt)); + evidence(run, randomUUID(), event.event, event); saveMetrics(run, metrics); - return { accepted: true }; - })(); -} - -export function parseTestSignal(body: Record): TestSignal { - if (!['boot', 'session_start', 'round_change', 'match_end', 'feature'].includes(String(body.event))) throw new Error('Unknown telemetry event'); - for (const key of ['runId', 'eventId', 'bootId'] as const) { - if (typeof body[key] !== 'string' || !/^[a-zA-Z0-9_.:-]{1,128}$/.test(body[key] as string)) throw new Error(`${key} is required (1–128 identifier characters)`); } - if (body.event !== 'boot' && (typeof body.sessionId !== 'string' || !/^[a-zA-Z0-9_.:-]{1,128}$/.test(body.sessionId))) throw new Error('sessionId is required'); - for (const key of ['sequence', 'round', 'players']) if (!Number.isSafeInteger(body[key]) || (body[key] as number) < 0) throw new Error(`${key} must be a nonnegative integer`); - if (body.event === 'feature' && (typeof body.name !== 'string' || !/^[a-zA-Z0-9_.:-]{1,120}$/.test(body.name))) throw new Error('Feature name is required'); - return { event: body.event as TestSignal['event'], bootId: body.bootId as string, sessionId: body.sessionId as string ?? '', - sequence: body.sequence as number, round: body.round as number, players: body.players as number, name: body.name as string ?? '' }; } -/** Bounded incremental reads, with persistent byte cursors and partial lines. */ export async function observeRun(run: TestRun, plan: BetaPlan): Promise { - const cursors = JSON.parse(run.cursors) as Record; - const observed: Array<{ key: LogKey; line: string; id: string }> = []; - let logAvailable = false; - for (const key of [gameLogKey(run.server), 'console'] as LogKey[]) { - const path = resolveLogPath(run.server, key); - const file = path ? await open(path, 'r').catch(() => null) : null; - if (!file) continue; - try { - const info = await file.stat(); - if (key === 'console') logAvailable = true; - const identity = `${info.dev}:${info.ino}`; - let cursor = cursors[key] ?? { position: 0, identity, partial: '' }; - let anchorChanged = false; - if (cursor.anchor && info.size >= cursor.position) { - const expected = Buffer.from(cursor.anchor, 'base64'); - const actual = Buffer.alloc(expected.length); - await file.read(actual, 0, actual.length, cursor.position - actual.length); - anchorChanged = !actual.equals(expected); - } - if (cursor.identity !== identity || info.size < cursor.position || anchorChanged) { - cursor = { position: 0, identity, partial: '', generation: randomUUID() }; - observed.push({ key, line: '', id: `rotation:${key}:${randomUUID()}` }); - } - if (info.size - cursor.position > 2_000_000) { - setOutcome(run.beta_id, 'inconclusive', `${run.server}: log reader fell behind; evidence is incomplete`); - return; - } - const buffer = Buffer.alloc(info.size - cursor.position); - const { bytesRead } = await file.read(buffer, 0, buffer.length, cursor.position); - const combined = Buffer.concat([Buffer.from(cursor.partial, 'base64'), buffer.subarray(0, bytesRead)]); - let start = 0; - for (let i = 0; i < combined.length; i++) if (combined[i] === 10) { - observed.push({ key, line: combined.subarray(start, i).toString('utf8'), id: `${key}:${identity}:${cursor.generation ?? ''}:${cursor.position + i - Buffer.from(cursor.partial, 'base64').length}` }); - start = i + 1; - } - cursor.position += bytesRead; - const anchor = Buffer.alloc(Math.min(cursor.position, 128)); - await file.read(anchor, 0, anchor.length, cursor.position - anchor.length); - cursor.anchor = anchor.toString('base64'); - cursor.partial = combined.subarray(start).toString('base64'); - if (cursor.partial.length > 100_000) cursor.partial = ''; // malformed giant line cannot grow forever - cursor.identity = identity; - cursors[key] = cursor; - } finally { await file.close(); } - } - db().transaction(() => { - const fresh = currentRun(run.beta_id, run.server); - if (!fresh || fresh.id !== run.id) return; - const m = readMetrics(fresh); - m.logAvailable = logAvailable; - for (const entry of observed) { - const cursor = cursors[entry.key]; - if (!entry.line) { evidence(run, entry.id, 'log-rotation', { log: entry.key }); continue; } - if (entry.key === 'console') { - if (/script (?:runtime )?error|unhandled exception|script exception|fatal error/i.test(entry.line) && evidence(run, entry.id, 'script-error', { line: entry.line.slice(0, 1000) })) m.logErrors++; - continue; - } - if (/InitGame:/.test(entry.line)) { cursor.session = entry.id; cursor.joined = false; cursor.started = undefined; } - if (cursor.session && /\sJ;[^;]*;\d+;/.test(entry.line) && !cursor.joined) { cursor.joined = true; cursor.started = Date.now(); } - if (/ExitLevel:\s*executed/.test(entry.line) && cursor.session && cursor.joined) { - // Unqualified match counts use logs. Round/population-qualified counts use session telemetry exclusively. - if (evidence(run, `match:${cursor.session}`, 'completed-match', { session: cursor.session, line: entry.line, minutes: (Date.now() - (cursor.started ?? Date.now())) / 60_000 })) { - for (const r of plan.requirements) if (r.kind === 'matches' && r.minRound === 0 && r.minPlayers <= 1 - && Date.now() - (cursor.started ?? Date.now()) >= r.minMinutes * 60_000) m.values[r.id] = (m.values[r.id] ?? 0) + 1; - } - cursor.joined = false; - } - if (/ShutdownGame:/.test(entry.line)) cursor.joined = false; - } - const server = db().prepare('SELECT last_checked_at, last_seen_at, last_players, last_error, container_state FROM servers WHERE name = ?').get(run.server) as { - last_checked_at: string | null; last_seen_at: string | null; last_players: number | null; last_error: string | null; container_state: string | null }; - const sample = server.last_checked_at ? Date.parse(server.last_checked_at) : 0; - const freshSample = sample >= Date.parse(run.observing_at ?? run.created_at) && Date.now() - sample <= 60_000; - const healthy = freshSample && !server.last_error && server.container_state === 'running' && server.last_seen_at === server.last_checked_at; - const at = freshSample && server.container_state === 'running' ? sample : Date.now(); - // Startup requires fresh status; don't count an absent first observation as an outage. - const before = m.wasHealthy; - const lastSample = m.lastSample; - applySample(plan, m, at, healthy, healthy ? server.last_players ?? 0 : 0); - if (freshSample && sample !== lastSample) evidence(run, `sample:${sample}`, 'status-sample', { healthy, players: server.last_players, checkedAt: server.last_checked_at }); - if (before !== m.wasHealthy) evidence(run, `health:${at}`, 'health', { healthy, players: server.last_players, reason: server.last_error }); - saveMetrics(run, m); - db().prepare('UPDATE beta_test_runs SET cursors = ? WHERE id = ?').run(JSON.stringify(cursors), run.id); - })(); + const m = readMetrics(run); + const server = db().prepare('SELECT last_checked_at, last_seen_at, last_error, container_state FROM servers WHERE name = ?').get(run.server) as { + last_checked_at: string | null; last_seen_at: string | null; last_error: string | null; container_state: string | null }; + const sample = server.last_checked_at ? Date.parse(server.last_checked_at) : 0; + const freshSample = sample >= Date.parse(run.observing_at ?? run.created_at) && Date.now() - sample <= 60_000; + const healthy = freshSample && !server.last_error && server.container_state === 'running' && server.last_seen_at === server.last_checked_at; + const at = freshSample && server.container_state === 'running' ? sample : Date.now(); + const before = m.wasHealthy; + const lastSample = m.lastSample; + applySample(plan, m, at, healthy); + if (freshSample && sample !== lastSample) evidence(run, `sample:${sample}`, 'status-sample', { healthy, checkedAt: server.last_checked_at }); + if (before !== m.wasHealthy) evidence(run, `health:${at}`, 'health', { healthy, reason: server.last_error }); + saveMetrics(run, m); } export function testView(betaId: number): BetaTestView | null { @@ -175,7 +74,7 @@ export function testView(betaId: number): BetaTestView | null { return { plan, outcome: stored.outcome, reason: stored.reason, servers: runs.map(run => { const progress = evaluateRequirements(plan, readMetrics(run), run.phase); - return { server: run.server, runId: run.id, phase: run.phase, passed: progress.every(r => r.passed), progress }; + return { server: run.server, phase: run.phase, passed: progress.every(r => r.passed), progress }; }), evidence: db().prepare(`SELECT e.id, r.server, e.received_at, e.kind, e.payload FROM beta_test_evidence e JOIN beta_test_runs r ON r.id = e.run_id WHERE r.beta_id = ? ORDER BY e.id DESC LIMIT 200`).all(betaId) as BetaTestView['evidence'] }; diff --git a/src/lib/server/fleet/match-log.ts b/src/lib/server/fleet/match-log.ts deleted file mode 100644 index f359e6e..0000000 --- a/src/lib/server/fleet/match-log.ts +++ /dev/null @@ -1,110 +0,0 @@ -import { resolveLogPath, tailLog, gameLogKey } from './logs.js'; - -// Match markers for T5 Plutonium's `logs/games_sp.log`. Verified against a live -// sample: each game session is bracketed by `InitGame:` and `ShutdownGame:`. -// A REAL completed match always emits `ExitLevel: executed` right before -// `ShutdownGame:`. Idle map cycles print `InitGame:` / `ShutdownGame:` at -// 0:00 with no ExitLevel — those don't count as a played match. -// -// So the reliable discriminator is `ExitLevel: executed`: seeing that line -// after `healthy_at` means a game actually ended normally (no crash — a crash -// would kill the container before ExitLevel fires and take RCON with it). -export const MATCH_START = /InitGame:/; -export const MATCH_END = /ExitLevel:\s*executed/; -export const MATCH_SHUTDOWN = /ShutdownGame:/; -// Player join line: `MM:SS J;;;`. games_sp.log accumulates -// across restarts and contains many old J; lines; only the ones AFTER the -// latest InitGame belong to the current match cycle. -export const MATCH_JOIN = /\sJ;[^;]*;\d+;/; - -/** - * Poll a server's games_sp.log tail for match markers. Returns whether a - * completed match end has been observed since `sinceIso` (the server's - * `healthy_at` timestamp). - */ -export async function scanForMatchEnd( - serverName: string, - sinceIso: string -): Promise<{ ended: boolean; started: boolean; reason?: string }> { - const path = resolveLogPath(serverName, gameLogKey(serverName)); - if (!path) return { ended: false, started: false, reason: 'log path resolution failed' }; - - const tail = await tailLog(path, 2000); - if (!tail) return { ended: false, started: false, reason: 'log not written yet' }; - if (tail.mtime < sinceIso) return { ended: false, started: false, reason: 'no log activity since healthy' }; - - let started = false; - let ended = false; - for (const line of tail.lines) { - if (MATCH_START.test(line)) started = true; - if (MATCH_END.test(line)) ended = true; - } - return { ended, started }; -} - -export type MatchScanKind = 'no-signal' | 'idle' | 'in-progress' | 'ended'; - -export interface MatchScanResult { - kind: MatchScanKind; - // Opaque identifier for the currently-observed match cycle. When kind is - // 'in-progress' or 'ended', this is the first J; line after the latest - // InitGame. Compare against persisted state to detect a fresh cycle - // (e.g. game auto-restart with same players — new J; line = new key). - cycleKey?: string; -} - -/** - * Read the games_sp.log tail and classify the current match state. We do NOT - * infer any wall-clock timestamps from log content — the caller stamps - * game_started_at/game_ended_at with the observation time. - * - * - `in-progress`: latest InitGame has at least one J; join line after it - * and no ExitLevel/ShutdownGame between that J; and end-of-tail. `cycleKey` - * is set to the first J; line after the latest InitGame. - * - `ended`: latest InitGame has a J; join line followed by an ExitLevel - * or ShutdownGame. `cycleKey` identifies which cycle ended. - * - `idle`: no J; line after the latest InitGame — empty map loaded, no - * match happening. - * - `no-signal`: log missing / unreadable / no InitGame in tail. - */ -export async function scanMatchState(serverName: string): Promise { - const path = resolveLogPath(serverName, gameLogKey(serverName)); - if (!path) return { kind: 'no-signal' }; - - const tail = await tailLog(path, 2000); - if (!tail || tail.lines.length === 0) return { kind: 'no-signal' }; - - const lines = tail.lines; - // Find the LATEST InitGame — everything before it is prior sessions that - // this log accumulated. Only what comes after is the current cycle. - let lastInit = -1; - for (let i = lines.length - 1; i >= 0; i--) { - if (MATCH_START.test(lines[i])) { - lastInit = i; - break; - } - } - if (lastInit === -1) return { kind: 'no-signal' }; - - // Scan only the current cycle (everything after the latest InitGame). - let firstJoinIdx = -1; - let endIdx = -1; - for (let i = lastInit + 1; i < lines.length; i++) { - if (firstJoinIdx === -1 && MATCH_JOIN.test(lines[i])) { - firstJoinIdx = i; - } - if (MATCH_END.test(lines[i]) || MATCH_SHUTDOWN.test(lines[i])) { - endIdx = i; - break; - } - } - - // No join in the current cycle -> nobody entered this match yet. - if (firstJoinIdx === -1) return { kind: 'idle' }; - - const cycleKey = lines[firstJoinIdx]; - if (endIdx !== -1 && endIdx > firstJoinIdx) { - return { kind: 'ended', cycleKey }; - } - return { kind: 'in-progress', cycleKey }; -} diff --git a/src/lib/server/fleet/telemetry.ts b/src/lib/server/fleet/telemetry.ts index 3d64ef3..26e8099 100644 --- a/src/lib/server/fleet/telemetry.ts +++ b/src/lib/server/fleet/telemetry.ts @@ -1,15 +1,24 @@ import { db } from '../db/index.js'; -import type { RoundTelemetry } from '../../telemetry.js'; +import type { GameTelemetry } from '../../telemetry.js'; -export function listTelemetry(): RoundTelemetry[] { - return db().prepare(`SELECT t.server, t.round, t.players, t.received_at AS receivedAt - FROM server_telemetry t JOIN servers s ON s.name = t.server - WHERE s.destroyed_at IS NULL`).all() as RoundTelemetry[]; +export function listTelemetry(): GameTelemetry[] { + const rows = db().prepare(`SELECT t.payload FROM server_telemetry t JOIN servers s ON s.name = t.server + WHERE s.destroyed_at IS NULL AND t.payload IS NOT NULL`).all() as Array<{ payload: string }>; + return rows.map(row => JSON.parse(row.payload) as GameTelemetry); } -export function saveTelemetry(report: RoundTelemetry): void { - db().prepare(`INSERT INTO server_telemetry (server, round, players, received_at) - VALUES (?, ?, ?, ?) ON CONFLICT(server) DO UPDATE SET - round = excluded.round, players = excluded.players, received_at = excluded.received_at`) - .run(report.server, report.round, report.players, report.receivedAt); +export function saveTelemetry(report: GameTelemetry): void { + db().prepare(`INSERT INTO server_telemetry (server, round, players, received_at, payload) + VALUES (?, ?, ?, ?, ?) ON CONFLICT(server) DO UPDATE SET + round = excluded.round, players = excluded.players, received_at = excluded.received_at, payload = excluded.payload`) + .run(report.server, report.event === 'game_start' ? 0 : report.round, report.players.length, report.receivedAt, JSON.stringify(report)); + if (report.event === 'game_start') { + db().prepare('UPDATE servers SET game_started_at = ?, game_ended_at = NULL WHERE name = ?').run(report.receivedAt, report.server); + } else if (report.event === 'game_end') { + db().prepare('UPDATE servers SET game_ended_at = ? WHERE name = ?').run(report.receivedAt, report.server); + // Legacy betas also take match completion directly from the script. + db().prepare(`UPDATE beta_servers SET match_seen_at = ? WHERE server = ? AND stage = 'awaiting-match' + AND healthy_at <= ? AND beta_id IN (SELECT id FROM betas WHERE state = 'running')`) + .run(report.receivedAt, report.server, report.receivedAt); + } } diff --git a/src/lib/server/pipeline/beta-plan-runner.ts b/src/lib/server/pipeline/beta-plan-runner.ts index 03f3b47..77b159d 100644 --- a/src/lib/server/pipeline/beta-plan-runner.ts +++ b/src/lib/server/pipeline/beta-plan-runner.ts @@ -112,7 +112,7 @@ async function advancePlan(stored: StoredPlan): Promise { if (oldRun) { const previous = readMetrics(oldRun); const metrics = readMetrics(run); - metrics.outages = previous.outages; metrics.logErrors = previous.logErrors; metrics.restarts = previous.restarts; + metrics.outages = previous.outages; saveMetrics(run, metrics); } const restart = await restartInRollout(server.server); diff --git a/src/lib/server/pipeline/beta-runner.ts b/src/lib/server/pipeline/beta-runner.ts index d67acc6..802dc51 100644 --- a/src/lib/server/pipeline/beta-runner.ts +++ b/src/lib/server/pipeline/beta-runner.ts @@ -3,7 +3,6 @@ import { emit } from './bus.js'; import type { BetaStage } from './bus.js'; import { komodoConfigured, rconConfigured } from '../config.js'; import { rconGetPlayerCount } from '../integrations/rcon.js'; -import { scanForMatchEnd } from '../fleet/match-log.js'; import { refreshRestartRollout, restartInRollout, rolloutHealth, rolloutWaitReason } from './restart-rollout.js'; import { plannedBetaTick } from './beta-plan-runner.js'; @@ -12,6 +11,7 @@ interface BetaServerActiveRow { server: string; stage: BetaStage; healthy_at: string | null; + match_seen_at: string | null; wait_for_health: number; wait_for_match: number; } @@ -101,7 +101,7 @@ async function runBetaTick(): Promise<{ processed: number }> { const rows = db() .prepare( `SELECT bs.beta_id AS beta_id, bs.server AS server, bs.stage AS stage, - bs.healthy_at AS healthy_at, + bs.healthy_at AS healthy_at, bs.match_seen_at AS match_seen_at, b.wait_for_health AS wait_for_health, b.wait_for_match AS wait_for_match FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id @@ -181,9 +181,7 @@ async function advance(row: BetaServerActiveRow): Promise { return; } case 'awaiting-match': { - const since = row.healthy_at ?? nowIso(); - const result = await scanForMatchEnd(row.server, since); - if (!result.ended) return; + if (!row.match_seen_at) return; transition(row.beta_id, row.server, 'ready'); return; } diff --git a/src/lib/server/pipeline/bus.ts b/src/lib/server/pipeline/bus.ts index cd9bd88..f8b27c7 100644 --- a/src/lib/server/pipeline/bus.ts +++ b/src/lib/server/pipeline/bus.ts @@ -1,5 +1,5 @@ import { EventEmitter } from 'node:events'; -import type { RoundTelemetry } from '../../telemetry.js'; +import type { GameTelemetry } from '../../telemetry.js'; import type { RconEndpoint, RconFailure, RconPlayer } from '../integrations/rcon.js'; import type { Health } from '../fleet/health.js'; @@ -40,7 +40,7 @@ export type BetaStage = export type BetaState = 'running' | 'ready-promote' | 'promoted' | 'rolled-back'; export type PipelineEvent = - | { type: 'server-telemetry'; entries: RoundTelemetry[] } + | { type: 'server-telemetry'; entries: GameTelemetry[] } | { type: 'deployment-opened'; deploymentId: string; action: string; servers: string[] } | { type: 'server-changed'; deploymentId: string; server: string; files: number } | { diff --git a/src/lib/server/pipeline/restart-rollout.ts b/src/lib/server/pipeline/restart-rollout.ts index e939658..2e0f9c8 100644 --- a/src/lib/server/pipeline/restart-rollout.ts +++ b/src/lib/server/pipeline/restart-rollout.ts @@ -56,7 +56,8 @@ export async function restartInRollout(server: string): Promise { } catch (err) { console.error('[worker] beta-runner error:', err); } - try { - await scanMatches(); - } catch (err) { - console.error('[worker] match-scan error:', err); - } -} - -/** - * Read every server's games_sp.log tail to advance game_started_at / - * game_ended_at. Concurrency-capped so 15 servers don't hammer disk at once. - */ -async function scanMatches(): Promise { - const servers = listServers(); - if (servers.length === 0) return; - - let cursor = 0; - async function pump(): Promise { - while (true) { - const i = cursor++; - if (i >= servers.length) return; - const srv = servers[i]; - const result = await scanMatchState(srv.name).catch((err) => { - console.error(`[worker] scanMatchState(${srv.name}) failed:`, err); - return { kind: 'no-signal' as const }; - }); - const persistedInProgress = !!srv.game_started_at && !srv.game_ended_at; - const persistedKey = srv.game_cycle_key; - const now = new Date().toISOString(); - - if (result.kind === 'in-progress' && result.cycleKey) { - // New cycle when persisted key differs (fresh start, or a game - // restart with the same players — new J; line = new key). - if (result.cycleKey !== persistedKey) { - setServerField(srv.name, 'game_started_at', now); - setServerField(srv.name, 'game_ended_at', null); - setServerField(srv.name, 'game_cycle_key', result.cycleKey); - } else if (!persistedInProgress) { - // Same key but previous cycle was ended — treat as new run - // (shouldn't happen in practice since key would differ, - // but be defensive). - setServerField(srv.name, 'game_started_at', now); - setServerField(srv.name, 'game_ended_at', null); - } - } else if (result.kind === 'ended' && result.cycleKey) { - // Only stamp the end if it's the cycle we've been tracking and - // we haven't already stamped it. - if (result.cycleKey === persistedKey && persistedInProgress) { - setServerField(srv.name, 'game_ended_at', now); - } - } - // 'idle' / 'no-signal': leave state alone. UI hides the timer when - // the server has no players (see ServerCard/detail page). - } - } - await Promise.all(Array.from({ length: STATUS_CONCURRENCY }, pump)); } /** @@ -169,8 +113,8 @@ async function refreshStatus(): Promise { const nextPlayers = result.status.players; let activeSince = srv.active_since; let inactiveSince = srv.inactive_since; - let gameStartedAt = srv.game_started_at; - let gameEndedAt = srv.game_ended_at; + const gameStartedAt = srv.game_started_at; + const gameEndedAt = srv.game_ended_at; if (nextPlayers > 0 && (activeSince === null || prevPlayers === 0 || prevPlayers === null)) { activeSince = now; inactiveSince = null; @@ -181,14 +125,6 @@ async function refreshStatus(): Promise { activeSince = null; setServerField(srv.name, 'inactive_since', inactiveSince); setServerField(srv.name, 'active_since', null); - // Game timer cannot run while inactive. Drop any tracked - // match state so the next active session anchors fresh - // when a new J; line fires. - gameStartedAt = null; - gameEndedAt = null; - setServerField(srv.name, 'game_started_at', null); - setServerField(srv.name, 'game_ended_at', null); - setServerField(srv.name, 'game_cycle_key', null); } // Persist last-observed state. last_seen_at is the health anchor. setServerField(srv.name, 'last_players', nextPlayers); diff --git a/src/lib/telemetry.ts b/src/lib/telemetry.ts index e1c2f3c..90a707f 100644 --- a/src/lib/telemetry.ts +++ b/src/lib/telemetry.ts @@ -1,6 +1,49 @@ -export interface RoundTelemetry { - server: string; - round: number; - players: number; - receivedAt: string; +export interface TelemetryPlayer { + name: string; + guid: string; +} + +export interface TelemetryPlayerStats extends TelemetryPlayer { + kills: number; + headshots: number; + downs: number; + revives: number; + score: number; +} + +export type GameEvent = { server: string; map: string } & ( + | { event: 'game_start'; players: TelemetryPlayer[] } + | { event: 'round_change' | 'game_end'; round: number; players: TelemetryPlayerStats[] } +); + +export type GameTelemetry = GameEvent & { receivedAt: string }; + +/** Validate the three script payloads; beta state is never part of the protocol. */ +export function parseGameEvent(value: unknown): GameEvent { + if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error('Payload must be a JSON object'); + const body = value as Record; + if (!['game_start', 'round_change', 'game_end'].includes(String(body.event))) throw new Error('event must be game_start, round_change, or game_end'); + for (const key of ['server', 'map'] as const) { + if (typeof body[key] !== 'string' || !body[key].trim() || body[key].length > 255) throw new Error(`${key} must be a nonempty string (up to 255 characters)`); + } + if (!Array.isArray(body.players)) throw new Error('players must be an array'); + const players = body.players.map((value, index) => { + if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error(`players[${index}] must be an object`); + const player = value as Record; + for (const key of ['name', 'guid'] as const) { + if (typeof player[key] !== 'string' || !player[key].trim()) throw new Error(`players[${index}].${key} must be a nonempty string`); + } + const identity = { name: player.name as string, guid: player.guid as string }; + if (body.event === 'game_start') return identity; + const stats = {} as Omit; + for (const key of ['kills', 'headshots', 'downs', 'revives', 'score'] as const) { + if (!Number.isSafeInteger(player[key]) || (player[key] as number) < 0) throw new Error(`players[${index}].${key} must be a nonnegative integer`); + stats[key] = player[key] as number; + } + return { ...identity, ...stats }; + }); + const common = { server: body.server as string, map: body.map as string }; + if (body.event === 'game_start') return { ...common, event: 'game_start', players }; + if (!Number.isSafeInteger(body.round) || (body.round as number) < 0) throw new Error('round must be a nonnegative integer'); + return { ...common, event: body.event as 'round_change' | 'game_end', round: body.round as number, players: players as TelemetryPlayerStats[] }; } diff --git a/src/lib/ui/components/BetaPlanBuilder.svelte b/src/lib/ui/components/BetaPlanBuilder.svelte index c8f499e..824e28b 100644 --- a/src/lib/ui/components/BetaPlanBuilder.svelte +++ b/src/lib/ui/components/BetaPlanBuilder.svelte @@ -2,7 +2,7 @@ import { onMount } from 'svelte'; import Button from '$ui/components/primitives/Button.svelte'; import { builtInPlans, requirement, requirementKinds, requirementLabels, parseBetaPlan, type BetaPlan, type RequirementKind } from '$lib/beta-plan.js'; - import { coverageSummary, needsSessionTelemetry, presetDescriptions, requirementCategory, requirementSummary } from '$lib/beta-plan-summary.js'; + import { coverageSummary, needsGameTelemetry, presetDescriptions, requirementCategory, requirementSummary } from '$lib/beta-plan-summary.js'; let { plan = $bindable() }: { plan: BetaPlan } = $props(); let presets = $state(builtInPlans); let presetName = $state(''); @@ -102,8 +102,8 @@
If a check fails
{failureSummary}

Every requirement below must pass on each qualifying server. You can edit any check.

- {#if plan.requirements.some(needsSessionTelemetry)} -

This plan needs session telemetry from the game reporter.

+ {#if plan.requirements.some(needsGameTelemetry)} +

Gameplay progress comes from game start, round change, and game end reports.

{/if}
@@ -120,31 +120,26 @@
{#if r.kind !== 'manual'} -
- {#if r.kind === 'round' || r.kind === 'feature' || r.kind === 'restart-budget' || (r.kind === 'matches' && (r.minRound > 0 || r.minPlayers > 1))} -

Requires session telemetry from the game reporter. Progress waits for events from this beta’s current run.

+ {#if needsGameTelemetry(r)} +

Uses game events reported by this server automatically.

{:else if r.kind === 'recovery'}

After the other requirements pass, restart when empty and repeat them. Every cycle requires fresh evidence and checklist sign-off.

{:else if r.kind === 'failure-budget'} -

Counts transitions from healthy to unhealthy. Missing or stale monitoring resets continuous health and does not earn playtime.

- {:else if r.kind === 'clean-logs'} -

Checks new console lines for script errors, script exceptions, unhandled exceptions and fatal errors. An unreadable log cannot pass.

+

Counts transitions from healthy to unhealthy. Missing or stale monitoring resets continuous healthy time.

{:else if r.kind === 'manual'}

A tester records their name and observation on the beta detail page.

{/if} diff --git a/src/lib/ui/components/BetaTestProgress.svelte b/src/lib/ui/components/BetaTestProgress.svelte index 659d26d..2988a18 100644 --- a/src/lib/ui/components/BetaTestProgress.svelte +++ b/src/lib/ui/components/BetaTestProgress.svelte @@ -44,10 +44,10 @@ {@const requirement = test.plan.requirements.find(r => r.id === progress.id)!}
{progress.passed ? '✓' : progress.failed ? '✕' : '○'} {progress.label}{progress.detail}
- {#if ['round', 'matches', 'feature', 'populated-minutes', 'player-hours', 'sustained-population'].includes(requirement.kind)} - Minimum {requirement.minPlayers} player(s){requirement.minRound ? `, round ${requirement.minRound} in the same session` : ''}{requirement.minMinutes ? `, ${requirement.minMinutes} minutes per match` : ''}{requirement.event ? ` · Event: ${requirement.event}` : ''} + {#if ['round', 'matches', 'populated-minutes', 'player-hours', 'sustained-population'].includes(requirement.kind)} + Minimum {requirement.minPlayers} player(s){requirement.minRound ? `, round ${requirement.minRound} in the same game` : ''}{requirement.minMinutes ? `, ${requirement.minMinutes} minutes per match` : ''} {/if} - {#if !['manual', 'failure-budget', 'restart-budget', 'clean-logs'].includes(requirement.kind)} + {#if !['manual', 'failure-budget'].includes(requirement.kind)} {/if} {#if requirement.kind === 'manual' && !progress.passed && !closed} diff --git a/src/lib/ui/components/RoundTelemetry.svelte b/src/lib/ui/components/RoundTelemetry.svelte index 6941926..3f90751 100644 --- a/src/lib/ui/components/RoundTelemetry.svelte +++ b/src/lib/ui/components/RoundTelemetry.svelte @@ -8,10 +8,10 @@ && (!since || Date.parse(report.receivedAt) >= Date.parse(since))); -{#if report && fresh} +{#if report && report.event === 'round_change' && fresh}
Round {report.round} - Last reported {elapsedSince(report.receivedAt, $uiNow)} ago · {report.players} players + Last reported {elapsedSince(report.receivedAt, $uiNow)} ago · {report.players.length} players
{/if} diff --git a/src/lib/ui/stores/servers.ts b/src/lib/ui/stores/servers.ts index 9eab1b7..8143e01 100644 --- a/src/lib/ui/stores/servers.ts +++ b/src/lib/ui/stores/servers.ts @@ -1,8 +1,8 @@ import { writable, readonly } from 'svelte/store'; import { browser } from '$app/environment'; -import type { RoundTelemetry } from '$lib/telemetry.js'; +import type { GameTelemetry } from '$lib/telemetry.js'; -export const telemetryByServer = writable>(new Map()); +export const telemetryByServer = writable>(new Map()); export interface RosterPlayer { num: number | null; @@ -54,9 +54,9 @@ export function startServerStream(): (() => void) | undefined { if (!browser || source) return; source = new EventSource('/api/events/servers'); source.addEventListener('server-telemetry', (evt) => { - const data = JSON.parse((evt as MessageEvent).data) as { entries: RoundTelemetry[]; snapshot?: boolean }; + const data = JSON.parse((evt as MessageEvent).data) as { entries: GameTelemetry[]; snapshot?: boolean }; telemetryByServer.update((previous) => { - const next = data.snapshot ? new Map() : new Map(previous); + const next = data.snapshot ? new Map() : new Map(previous); for (const entry of data.entries) next.set(entry.server, entry); return next; }); diff --git a/src/routes/api/betas/[id]/test/+server.ts b/src/routes/api/betas/[id]/test/+server.ts index 22f98c9..795acd3 100644 --- a/src/routes/api/betas/[id]/test/+server.ts +++ b/src/routes/api/betas/[id]/test/+server.ts @@ -33,6 +33,16 @@ export async function POST({ params, request }: { params: { id: string }; reques betaState(id, false); } else if (body.action === 'resume') { if (stored.outcome !== 'paused') throw new Error('Only paused plans can resume; failed or expired plans need a fresh attempt'); + const runs = db().prepare('SELECT * FROM beta_test_runs WHERE beta_id = ? AND retired_at IS NULL').all(id) as Parameters[0][]; + for (const run of runs) { + const metrics = readMetrics(run); + delete metrics.match; + delete metrics.lastSample; + delete metrics.wasHealthy; + metrics.streaks = {}; + for (const r of plan.requirements) if (r.kind === 'healthy-minutes') metrics.values[r.id] = 0; + saveMetrics(run, metrics); + } setOutcome(id, 'active', null); } else if (body.action === 'retry') { // Fresh attempts keep their predecessors as audit history. Preserve the original deadline: diff --git a/src/routes/api/telemetry/+server.ts b/src/routes/api/telemetry/+server.ts index e0aa035..ca7a48b 100644 --- a/src/routes/api/telemetry/+server.ts +++ b/src/routes/api/telemetry/+server.ts @@ -1,88 +1,27 @@ import { json } from '@sveltejs/kit'; -import { getServer, listServers } from '$lib/server/fleet/servers.js'; +import { parseGameEvent, type GameTelemetry } from '$lib/telemetry.js'; +import { getServer } from '$lib/server/fleet/servers.js'; import { saveTelemetry } from '$lib/server/fleet/telemetry.js'; -import { emit } from '$lib/server/pipeline/bus.js'; +import { recordGameEvent } from '$lib/server/fleet/beta-tests.js'; import { db } from '$lib/server/db/index.js'; -import { acceptTestSignal, parseTestSignal } from '$lib/server/fleet/beta-tests.js'; - -// Reporters fetch this once at process startup and keep the token for that boot. -export async function GET({ url }: { url: URL }) { - const server = url.searchParams.get('server'); - if (!server || !getServer(server)) return json({ error: 'Unknown server' }, { status: 404 }); - const runs = db().prepare(`SELECT r.id AS runId, r.beta_id AS betaId FROM beta_test_runs r - JOIN betas b ON b.id = r.beta_id JOIN beta_test_plans p ON p.beta_id = b.id - WHERE r.server = ? AND r.retired_at IS NULL AND b.state IN ('running','ready-promote') - AND p.outcome IN ('active','passed')`).all(server); - return json({ runs }, { headers: { 'Cache-Control': 'no-store' } }); -} +import { emit } from '$lib/server/pipeline/bus.js'; export async function POST({ request }: { request: Request }) { - console.log('[telemetry] received POST /api/telemetry'); - // Encode payloads as JSON so script-supplied newlines cannot forge log lines. - // Bound the preview to keep malformed requests from flooding container logs. - const raw = await request.text(); - const context = { - contentType: request.headers.get('content-type'), - payloadPreview: raw.slice(0, 4096), - payloadTruncated: raw.length > 4096 - }; - function reject(status: number, error: string, details: Record = {}) { - console.log(`[telemetry] rejected ${status}: ${JSON.stringify({ error, ...context, ...details })}`); - return json({ ok: false, error, details }, { status }); - } if (request.headers.get('content-type')?.split(';')[0].trim().toLowerCase() !== 'application/json') { - return reject(415, 'Content-Type must be application/json', { receivedContentType: context.contentType }); + return json({ ok: false, error: 'Content-Type must be application/json' }, { status: 415 }); } - let body; - try { - body = JSON.parse(raw); - } catch (error) { - return reject(400, 'Malformed JSON', { parseError: (error as Error).message }); - } - if (!body || typeof body !== 'object' || Array.isArray(body)) { - return reject(400, 'Payload must be a JSON object'); - } - const issues: string[] = []; - if (body.runId === undefined && body.event !== 'round_change') issues.push('Session telemetry requires runId'); - if (typeof body.server !== 'string' || !body.server.length || body.server.length > 255) { - issues.push('server must be a registered server name (1–255 characters)'); - } - for (const field of ['round', 'players']) { - if (!Number.isSafeInteger(body[field]) || body[field] < 0) { - issues.push(`${field} must be a nonnegative safe integer (JSON number, not a string)`); - } - } - if (issues.length) { - return reject(400, 'Invalid telemetry payload', { issues }); - } - const server = getServer(body.server); - if (!server) { - const registeredServers = listServers().filter((entry) => !entry.destroyed_at).map((entry) => entry.name); - return reject(404, 'Server is not registered', { - receivedServer: body.server, - hint: 'Send the exact Electric Cherry server name in the server field; matching is case-sensitive.', - registeredServers - }); - } - if (server.destroyed_at) { - return reject(404, 'Server has been destroyed', { - receivedServer: body.server, destroyedAt: server.destroyed_at - }); - } - let test: { accepted: boolean; reason?: string } | undefined; - if (body.runId !== undefined) { - try { - const signal = parseTestSignal(body); - test = acceptTestSignal(body.server, body.runId, body.eventId, signal); - } catch (error) { return reject(400, (error as Error).message); } - if (body.event !== 'round_change') return json({ ok: true, test }); - } - const telemetry = { - server: body.server as string, round: body.round as number, - players: body.players as number, receivedAt: new Date().toISOString() - }; - saveTelemetry(telemetry); - console.log(`[telemetry] accepted round_change ${JSON.stringify(telemetry)}`); + let value: unknown; + try { value = await request.json(); } + catch { return json({ ok: false, error: 'Malformed JSON' }, { status: 400 }); } + let telemetry: GameTelemetry; + try { telemetry = { ...parseGameEvent(value), receivedAt: new Date().toISOString() }; } + catch (error) { return json({ ok: false, error: (error as Error).message }, { status: 400 }); } + const server = getServer(telemetry.server); + if (!server || server.destroyed_at) return json({ ok: false, error: 'Server is not registered or has been destroyed' }, { status: 404 }); + db().transaction(() => { + saveTelemetry(telemetry); + recordGameEvent(telemetry); + })(); emit({ type: 'server-telemetry', entries: [telemetry] }); - return json({ ok: true, telemetry, test }); + return json({ ok: true, telemetry }); } diff --git a/tests/helpers/telemetry.ts b/tests/helpers/telemetry.ts new file mode 100644 index 0000000..2d16b0a --- /dev/null +++ b/tests/helpers/telemetry.ts @@ -0,0 +1,17 @@ +import type { GameEvent, TelemetryPlayerStats } from '../../src/lib/telemetry.js'; + +export const roster = (count = 4): TelemetryPlayerStats[] => Array.from({ length: count }, (_, i) => ({ + name: `Player ${i + 1}`, guid: `1234567890123456${i}`, kills: 100 + i, + headshots: 50, downs: 1, revives: 2, score: 5000 +})); +export function gameEvent(event: GameEvent['event'], round = 25, players = 4, server = 't5_zombie_theater_1', map = 'zombie_theater'): GameEvent { + return event === 'game_start' + ? { event, server, map, players: roster(players).map(({ name, guid }) => ({ name, guid })) } + : { event, server, map, round, players: roster(players) }; +} +export async function postTelemetry(body: unknown, contentType = 'application/json') { + const { POST } = await import('../../src/routes/api/telemetry/+server.js'); + return POST({ request: new Request('http://localhost/api/telemetry', { + method: 'POST', headers: { 'Content-Type': contentType }, body: JSON.stringify(body) + }) }); +} diff --git a/tests/unit/beta-evaluation.test.ts b/tests/unit/beta-evaluation.test.ts index 06362b0..1362e30 100644 --- a/tests/unit/beta-evaluation.test.ts +++ b/tests/unit/beta-evaluation.test.ts @@ -1,100 +1,102 @@ import { describe, expect, it } from 'vitest'; import { defaultPlan, parseBetaPlan, requirement, type BetaPlan } from '../../src/lib/beta-plan.js'; -import { applySample, applySignal, coveragePassed, emptyMetrics, evaluateRequirements, type TestSignal } from '../../src/lib/server/fleet/beta-evaluation.js'; +import { applySample, applyGameEvent, coveragePassed, emptyMetrics, evaluateRequirements } from '../../src/lib/server/fleet/beta-evaluation.js'; +import { gameEvent } from '../helpers/telemetry.js'; -const signal = (event: TestSignal['event'], extra: Partial = {}): TestSignal => ({ event, bootId: 'boot-1', sessionId: 'session-1', sequence: 0, round: 0, players: 4, name: '', ...extra }); const planWith = (...kinds: Parameters[0][]): BetaPlan => ({ ...defaultPlan(), requirements: kinds.map((k, i) => requirement(k, `r${i}`)) }); describe('beta requirement evaluation', () => { - it('copies a reactive preset without cloning proxies or mutating the saved requirements', () => { + it('copies reactive presets without mutating saved requirements', () => { const source = defaultPlan(); - const proxied = new Proxy(source, { get(target, key, receiver) { - return key === 'requirements' ? new Proxy(target.requirements, {}) : Reflect.get(target, key, receiver); - } }); - const selected = parseBetaPlan(proxied); + const selected = parseBetaPlan(new Proxy(source, {})); expect(selected).toEqual(source); selected.requirements[0].target = 15; - selected.requirements.push(requirement('round', 'round')); - expect(source.requirements).toHaveLength(1); expect(source.requirements[0].target).toBe(1); }); - it('validates plans without accepting unknown kinds, duplicate IDs, fractional counts or unsafe numbers', () => { + it('rejects removed checks, duplicate IDs, fractional counts and unsafe numbers', () => { expect(parseBetaPlan(defaultPlan())).toEqual(defaultPlan()); expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [] })).toThrow(); expect(() => parseBetaPlan({ ...defaultPlan(), timeoutHours: NaN })).toThrow(); expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [requirement('round', 'same'), requirement('round', 'same')] })).toThrow(); + for (const kind of ['feature', 'restart-budget', 'clean-logs']) { + expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [{ ...requirement('round', 'r'), kind }] })).toThrow(); + } expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [{ ...requirement('round', 'r'), target: 1.5 }] })).toThrow(); expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [requirement('manual', 'constructor')] })).toThrow(); }); - it('counts only qualifying rounds and completed matches from the same session', () => { + it('uses qualifying round and population together within the same game, including the final report', () => { const plan = planWith('round', 'matches'); plan.requirements[0].minPlayers = 4; plan.requirements[1] = { ...plan.requirements[1], target: 2, minRound: 25, minPlayers: 4, minMinutes: 1 }; const m = emptyMetrics(); - expect(applySignal(plan, m, signal('round_change', { sequence: 1, round: 50 }), 0)).toBe(false); - applySignal(plan, m, signal('boot'), 0); - applySignal(plan, m, signal('session_start'), 0); - applySignal(plan, m, signal('round_change', { sequence: 1, round: 30, players: 1 }), 1000); - applySignal(plan, m, signal('match_end', { sequence: 2 }), 61_000); + applyGameEvent(plan, m, gameEvent('game_start'), 0); + applyGameEvent(plan, m, gameEvent('round_change', 30, 1), 1000); + applyGameEvent(plan, m, gameEvent('game_end', 30, 1), 61_000); expect(m.values.r0 ?? 0).toBe(0); expect(m.values.r1 ?? 0).toBe(0); - applySignal(plan, m, signal('session_start', { sessionId: 'session-2' }), 62_000); - applySignal(plan, m, signal('round_change', { sessionId: 'session-2', sequence: 1, round: 25 }), 63_000); - applySignal(plan, m, signal('match_end', { sessionId: 'session-2', sequence: 2 }), 123_000); + applyGameEvent(plan, m, gameEvent('game_start'), 62_000); + applyGameEvent(plan, m, gameEvent('game_end', 25), 123_000); expect(m.values).toMatchObject({ r0: 25, r1: 1 }); - expect(applySignal(plan, m, signal('match_end', { sessionId: 'session-2', sequence: 3 }), 124_000)).toBe(false); + applyGameEvent(plan, m, gameEvent('game_end', 25), 124_000); + expect(m.values.r1).toBe(1); }); - it('rejects out-of-order events and retired boots, while counting actual boot changes once', () => { - const plan = planWith('feature', 'restart-budget'); - plan.requirements[0].event = 'perk.purchased'; + it('accepts ordinary rounds and game ends without boot events or a discovery handshake', () => { + const plan = planWith('round', 'matches'); const m = emptyMetrics(); - applySignal(plan, m, signal('boot'), 0); - applySignal(plan, m, signal('session_start'), 0); - applySignal(plan, m, signal('feature', { sequence: 2, name: 'perk.purchased' }), 1); - expect(applySignal(plan, m, signal('feature', { sequence: 1, name: 'perk.purchased' }), 2)).toBe(false); - applySignal(plan, m, signal('boot', { bootId: 'boot-2' }), 3); - expect(applySignal(plan, m, signal('boot'), 4)).toBe(false); - expect(m.values.r0).toBe(1); - expect(evaluateRequirements(plan, m, 0)[1].failed).toBe(true); - expect(m.restarts).toBe(1); + applyGameEvent(plan, m, gameEvent('round_change'), 0); + applyGameEvent(plan, m, gameEvent('game_end'), 1000); + expect(m.values).toMatchObject({ r0: 25, r1: 1 }); + applyGameEvent(plan, m, gameEvent('game_start'), 2000); + applyGameEvent(plan, m, gameEvent('game_end', 1), 3000); + expect(m.values.r1).toBe(2); }); - it('does not double count unqualified matches across log and telemetry sources', () => { + it('does not combine games or maps to satisfy duration and round qualifiers', () => { const plan = planWith('matches'); + plan.requirements[0] = { ...plan.requirements[0], minRound: 25, minMinutes: 1 }; const m = emptyMetrics(); - applySignal(plan, m, signal('boot'), 0); - applySignal(plan, m, signal('session_start'), 0); - applySignal(plan, m, signal('match_end', { sequence: 1 }), 1000); - expect(m.values.r0).toBeUndefined(); // these requirements use incremental match logs + applyGameEvent(plan, m, gameEvent('game_start'), 0); + applyGameEvent(plan, m, gameEvent('round_change'), 1000); + applyGameEvent(plan, m, gameEvent('game_start'), 60_000); + applyGameEvent(plan, m, gameEvent('game_end', 1), 120_000); + expect(m.values.r0 ?? 0).toBe(0); + applyGameEvent(plan, m, gameEvent('game_end', 30, 4, 't5_zombie_theater_1', 'zombie_moon'), 180_000); + expect(m.values.r0 ?? 0).toBe(0); }); - it('integrates populated time and player-hours conservatively and excludes monitoring gaps', () => { - const plan = planWith('populated-minutes', 'player-hours', 'healthy-minutes', 'population', 'sustained-population', 'failure-budget'); - plan.requirements[4].minPlayers = 4; + it('derives participation only from game reports, even when rounds are many minutes apart', () => { + const plan = planWith('populated-minutes', 'player-hours', 'population', 'sustained-population'); + plan.requirements[3].minPlayers = 4; const m = emptyMetrics(); - applySample(plan, m, 0, true, 4); - applySample(plan, m, 30_000, true, 4); - applySample(plan, m, 60_000, true, 4); - expect(m.values).toMatchObject({ r0: 1, r1: 4 / 60, r2: 1, r3: 4, r4: 1 }); - applySample(plan, m, 300_000, true, 4); + applySample(plan, m, 0, true); + applySample(plan, m, 30_000, true); + expect(m.values).toEqual({}); + applyGameEvent(plan, m, gameEvent('game_start'), 0); + applyGameEvent(plan, m, gameEvent('round_change'), 600_000); + expect(m.values).toMatchObject({ r0: 10, r1: 40 / 60, r2: 4, r3: 10 }); + applyGameEvent(plan, m, gameEvent('game_end', 26, 2), 1_200_000); + expect(m.values).toMatchObject({ r0: 20, r1: 1, r3: 10 }); + applyGameEvent(plan, m, gameEvent('game_start'), 3_000_000); + expect(m.values.r0).toBe(20); + }); + + it('excludes monitoring gaps from healthy time and requires explicit manual sign-off', () => { + const plan = planWith('healthy-minutes', 'failure-budget', 'manual'); + const m = emptyMetrics(); + applySample(plan, m, 0, true); + applySample(plan, m, 60_000, true); expect(m.values.r0).toBe(1); - expect(m.values.r2).toBe(0); - applySample(plan, m, 330_000, false, 0); - applySample(plan, m, 360_000, false, 0); + applySample(plan, m, 300_000, true); + expect(m.values.r0).toBe(0); + applySample(plan, m, 330_000, false); + applySample(plan, m, 360_000, false); expect(m.outages).toBe(1); - expect(evaluateRequirements(plan, m, 0)[5].failed).toBe(true); - }); - - it('requires readable logs, boot evidence and explicit manual sign-off', () => { - const plan = planWith('clean-logs', 'restart-budget', 'manual'); - const m = emptyMetrics(); - expect(evaluateRequirements(plan, m, 0).every(r => !r.passed)).toBe(true); - m.logAvailable = true; m.bootId = 'boot'; m.manual.r2 = 'Josh: HUD checked'; - expect(evaluateRequirements(plan, m, 0).every(r => r.passed)).toBe(true); - m.logErrors = 1; - expect(evaluateRequirements(plan, m, 0)[0].failed).toBe(true); + expect(evaluateRequirements(plan, m, 0)[1].failed).toBe(true); + expect(evaluateRequirements(plan, m, 0)[2].passed).toBe(false); + m.manual.r2 = 'Josh: HUD checked'; + expect(evaluateRequirements(plan, m, 0)[2].passed).toBe(true); }); it('requires each selected coverage group even when many servers on one map pass', () => { diff --git a/tests/unit/beta-plans.test.ts b/tests/unit/beta-plans.test.ts index 408f255..796f93b 100644 --- a/tests/unit/beta-plans.test.ts +++ b/tests/unit/beta-plans.test.ts @@ -3,6 +3,7 @@ import { appendFile, mkdir, readFile, writeFile } from 'node:fs/promises'; import { existsSync } from 'node:fs'; import { join } from 'node:path'; import { newFixture, type Fixture } from '../helpers/fixture.js'; +import { gameEvent, postTelemetry } from '../helpers/telemetry.js'; import { defaultPlan, requirement, type BetaPlan } from '../../src/lib/beta-plan.js'; vi.mock('../../src/lib/server/integrations/komodo.js', () => ({ komodoRestart: vi.fn(), komodoInspectHealthy: vi.fn() })); @@ -77,56 +78,50 @@ describe('durable beta test plans', () => { expect(getBeta(id)?.state).toBe('ready-promote'); }); - it('ignores pre-beta log markers, handles partial lines, and persists cursors across app restarts', async () => { - const id = await openPlan({ ...defaultPlan(), requirements: [requirement('matches', 'matches'), requirement('clean-logs', 'logs')] }); + it('uses script events instead of logs and keeps match progress across app restarts', async () => { + await postTelemetry(gameEvent('game_end')); + const id = await openPlan({ ...defaultPlan(), requirements: [requirement('matches', 'matches')] }); await tick(); await tick(); - const { getBeta } = await import('../../src/lib/server/fleet/beta.js'); const { testView } = await import('../../src/lib/server/fleet/beta-tests.js'); - expect(getBeta(id)?.state).toBe('running'); - expect(testView(id)?.servers[0].progress.map(p => p.current)).toEqual([0, 0]); - const log = join(fixture.t5, names[0], 'logs/games_sp.log'); - await appendFile(log, 'InitGame:\n 0:00 J;new;1;Josh\nExitLevel: exe'); + await appendFile(join(fixture.t5, names[0], 'logs/games_sp.log'), 'InitGame:\n 0:00 J;new;1;Josh\nExitLevel: executed\n'); await tick(); - expect(getBeta(id)?.state).toBe('running'); + expect(testView(id)?.servers[0].progress[0].current).toBe(0); + await postTelemetry(gameEvent('game_start')); const { closeDb } = await import('../../src/lib/server/db/index.js'); closeDb(); - await appendFile(log, 'cuted\n'); - await tick(); await tick(); - expect(getBeta(id)?.state).toBe('ready-promote'); + await postTelemetry(gameEvent('game_end')); + await tick(); expect(testView(id)?.servers[0].progress[0].current).toBe(1); - await appendFile(join(fixture.t5, names[0], 'logs/console.log'), 'script runtime error: undefined variable\n'); - await tick(); - expect(getBeta(id)?.state).toBe('running'); - expect(testView(id)?.outcome).toBe('failed'); - }); - - it('accepts durable run-scoped round evidence and rejects stale runs and duplicate events', async () => { - const id = await openPlan({ ...defaultPlan(), requirements: [requirement('round', 'round')] }); - await tick(); await tick(); - const { acceptTestSignal, currentRun, testView } = await import('../../src/lib/server/fleet/beta-tests.js'); - const run = currentRun(id, names[0])!; - const base = { bootId: 'boot', sessionId: 'session', sequence: 0, round: 0, players: 4, name: '' }; - expect(acceptTestSignal(names[0], 'stale-run', 'old', { ...base, event: 'boot' }).accepted).toBe(false); - expect(acceptTestSignal(names[0], run.id, 'boot', { ...base, event: 'boot' }).accepted).toBe(true); - expect(acceptTestSignal(names[0], run.id, 'start', { ...base, event: 'session_start' }).accepted).toBe(true); - const round = { ...base, event: 'round_change' as const, sequence: 1, round: 25 }; - expect(acceptTestSignal(names[0], run.id, 'round', round).accepted).toBe(true); - expect(acceptTestSignal(names[0], run.id, 'round', round).accepted).toBe(false); - await tick(); expect(testView(id)?.outcome).toBe('passed'); - expect(testView(id)?.evidence.filter(e => e.kind === 'round_change')).toHaveLength(1); }); - it('resets evidence after planned recovery and only passes after testing the new process', async () => { + it('automatically credits ordinary rounds only to active deployed beta servers', async () => { + const id = await openPlan({ ...defaultPlan(), waveSize: 1, requirements: [requirement('round', 'round')] }, names); + await postTelemetry(gameEvent('round_change')); + await tick(); await tick(); + const { testView } = await import('../../src/lib/server/fleet/beta-tests.js'); + expect(testView(id)?.servers[0].progress[0].current).toBe(0); + await postTelemetry(gameEvent('round_change', 99, 4, names[1])); + expect(testView(id)?.servers[0].progress[0].current).toBe(0); + await postTelemetry(gameEvent('round_change')); + expect(testView(id)?.servers[0].progress[0].current).toBe(25); + expect(testView(id)?.evidence.filter(e => e.kind === 'round_change')).toHaveLength(1); + await tick(); + expect(existsSync(scriptPath(names[1]))).toBe(true); + }); + + it('resets evidence after planned recovery and consumes the same simple reports afterward', async () => { const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players'), requirement('recovery', 'recovery')] }); await tick(); await tick(); const { currentRun, testView } = await import('../../src/lib/server/fleet/beta-tests.js'); const first = currentRun(id, names[0])!; - expect(testView(id)?.outcome).toBe('active'); - await tick(); + expect(testView(id)?.servers[0].progress[0].current).toBe(0); + await postTelemetry(gameEvent('game_start')); + await tick(); await tick(); expect(currentRun(id, names[0])?.id).not.toBe(first.id); expect(currentRun(id, names[0])?.phase).toBe(1); expect(testView(id)?.servers[0].progress[0].current).toBe(0); + await postTelemetry(gameEvent('game_start')); await tick(); expect(testView(id)?.outcome).toBe('passed'); }); @@ -143,6 +138,7 @@ describe('durable beta test plans', () => { it('revokes readiness if health fails while awaiting manual promotion', async () => { const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players'), requirement('failure-budget', 'outages')] }); await tick(); await tick(); + await postTelemetry(gameEvent('game_start')); await tick(); const { getBeta, promoteBeta } = await import('../../src/lib/server/fleet/beta.js'); expect(getBeta(id)?.state).toBe('ready-promote'); await tick(15, 0, false); @@ -152,10 +148,9 @@ describe('durable beta test plans', () => { it('automatically rolls back only the deployed wave and queues safe rollback restarts', async () => { await writeFile(scriptPath(names[0]), 'original'); - const id = await openPlan({ ...defaultPlan(), waveSize: 1, failureAction: 'rollback', requirements: [requirement('round', 'round'), requirement('clean-logs', 'logs')] }, names); + const id = await openPlan({ ...defaultPlan(), waveSize: 1, failureAction: 'rollback', requirements: [requirement('round', 'round'), requirement('failure-budget', 'outages')] }, names); await tick(); await tick(); - await appendFile(join(fixture.t5, names[0], 'logs/console.log'), 'fatal error: test\n'); - await tick(); + await tick(15, 0, false); const { getBeta } = await import('../../src/lib/server/fleet/beta.js'); const { db } = await import('../../src/lib/server/db/index.js'); expect(getBeta(id)?.state).toBe('rolled-back'); @@ -176,6 +171,7 @@ describe('durable beta test plans', () => { it('retains artifact ownership while ready for promotion', async () => { const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players')] }); await tick(); await tick(); + await postTelemetry(gameEvent('game_start')); await tick(); const { heldByBeta, ownerOf } = await import('../../src/lib/server/fleet/servers.js'); expect(heldByBeta(names[0]).get('test.gsc')).toBe(id); expect(ownerOf(names[0], 'test.gsc')).toBe(id); @@ -185,32 +181,27 @@ describe('durable beta test plans', () => { expect(ownerOf(names[0], 'test.gsc')).toBeNull(); }); - it('does not carry joins into a truncated log or deduplicate a genuinely new match after rotation', async () => { + it('counts successive games with the same players and ignores repeated game ends', async () => { const id = await openPlan({ ...defaultPlan(), requirements: [{ ...requirement('matches', 'matches'), target: 2 }] }); await tick(); await tick(); - const log = join(fixture.t5, names[0], 'logs/games_sp.log'); - await appendFile(log, 'InitGame:\n 0:00 J;first;1;Josh\nExitLevel: executed\n'); - await tick(); - // Same inode, contents replaced with a larger file: anchor detects truncation/regrowth. - await writeFile(log, 'InitGame:\n 0:00 J;second;1;Josh\n' + 'padding\n'.repeat(30) + 'ExitLevel: executed\n'); - await tick(); + await postTelemetry(gameEvent('game_start')); + await postTelemetry(gameEvent('game_end')); + await postTelemetry(gameEvent('game_end')); const { testView } = await import('../../src/lib/server/fleet/beta-tests.js'); + expect(testView(id)?.servers[0].progress[0].current).toBe(1); + await postTelemetry(gameEvent('game_start')); + await postTelemetry(gameEvent('game_end')); + await tick(); expect(testView(id)?.servers[0].progress[0].current).toBe(2); expect(testView(id)?.outcome).toBe('passed'); }); - it('exposes current run tokens, validates session payloads, and records checklist sign-off through the API', async () => { + it('records checklist sign-off and starts a fresh attempt without changing script payloads', async () => { const id = await openPlan({ ...defaultPlan(), requirements: [requirement('round', 'round'), requirement('manual', 'hud')] }); await tick(); await tick(); const telemetry = await import('../../src/routes/api/telemetry/+server.js'); - const context = await (await telemetry.GET({ url: new URL(`http://test/api/telemetry?server=${names[0]}`) })).json(); - expect(context.runs).toHaveLength(1); - const body = { server: names[0], runId: context.runs[0].runId, eventId: 'boot', bootId: 'boot', sequence: 0, round: 0, players: 0, event: 'boot' }; - const post = (payload: unknown) => telemetry.POST({ request: new Request('http://test/api/telemetry', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(payload) }) }); - expect((await post({ ...body, eventId: null })).status).toBe(400); - expect(await (await post(body)).json()).toMatchObject({ test: { accepted: true } }); - await post({ ...body, eventId: 'start', event: 'session_start', sessionId: 'game', players: 4 }); - await post({ ...body, eventId: 'round', event: 'round_change', sessionId: 'game', sequence: 1, round: 25, players: 4 }); + expect(telemetry).not.toHaveProperty('GET'); + expect(await (await postTelemetry(gameEvent('round_change'))).json()).not.toHaveProperty('test'); const controls = await import('../../src/routes/api/betas/[id]/test/+server.js'); const result = await controls.POST({ params: { id: String(id) }, request: new Request('http://test', { method: 'POST', body: JSON.stringify({ action: 'sign-off', server: names[0], requirementId: 'hud', note: 'Josh: HUD verified' }) }) }); expect(result.status).toBe(200); @@ -219,8 +210,82 @@ describe('durable beta test plans', () => { expect(testView(id)?.outcome).toBe('passed'); const retry = await controls.POST({ params: { id: String(id) }, request: new Request('http://test', { method: 'POST', body: JSON.stringify({ action: 'retry' }) }) }); expect(retry.status).toBe(200); - expect(await (await post({ ...body, eventId: 'late' })).json()).toMatchObject({ test: { accepted: false } }); + await postTelemetry(gameEvent('round_change')); expect(testView(id)?.servers).toHaveLength(0); + await tick(); await tick(); + expect(testView(id)?.servers[0].progress[0].current).toBe(0); + await postTelemetry(gameEvent('round_change')); + expect(testView(id)?.servers[0].progress[0].current).toBe(25); + }); + + it('keeps Fleet reporting during pauses without crediting paused gameplay or elapsed time', async () => { + const id = await openPlan({ ...defaultPlan(), requirements: [{ ...requirement('populated-minutes', 'time'), target: 60 }] }); + await tick(); await tick(); + await postTelemetry(gameEvent('game_start')); + vi.setSystemTime(Date.now() + 600_000); + await postTelemetry(gameEvent('round_change')); + const { testView } = await import('../../src/lib/server/fleet/beta-tests.js'); + expect(testView(id)?.servers[0].progress[0].current).toBe(10); + const controls = await import('../../src/routes/api/betas/[id]/test/+server.js'); + const control = (action: string) => controls.POST({ params: { id: String(id) }, request: new Request('http://test', { + method: 'POST', body: JSON.stringify({ action }) + }) }); + expect((await control('pause')).status).toBe(200); + vi.setSystemTime(Date.now() + 3_600_000); + await postTelemetry(gameEvent('game_end', 50)); + const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js'); + expect(listTelemetry()[0]).toMatchObject({ event: 'game_end', round: 50 }); + expect(testView(id)?.servers[0].progress[0].current).toBe(10); + expect((await control('resume')).status).toBe(200); + await postTelemetry(gameEvent('round_change', 1)); + expect(testView(id)?.servers[0].progress[0].current).toBe(10); + vi.setSystemTime(Date.now() + 600_000); + await postTelemetry(gameEvent('round_change', 2)); + expect(testView(id)?.servers[0].progress[0].current).toBe(20); + }); + + it('migrates existing unsupported checks into review items and resets active progress once', async () => { + const id = await openPlan({ ...defaultPlan(), requirements: [requirement('round', 'round')] }); + await tick(); await tick(); + await postTelemetry(gameEvent('round_change')); + await tick(); + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + const oldPlan = { ...defaultPlan(), requirements: [requirement('round', 'round'), + ...['feature', 'restart-budget', 'clean-logs'].map((kind, i) => ({ ...requirement('manual', `old-${i}`), kind, event: 'perk.purchased' }))] }; + db().prepare('UPDATE beta_test_plans SET plan = ? WHERE beta_id = ?').run(JSON.stringify(oldPlan), id); + db().prepare('INSERT INTO beta_test_presets (name, plan) VALUES (?, ?)').run('Old plan', JSON.stringify(oldPlan)); + db().prepare("DELETE FROM settings WHERE key = 'game_telemetry_v2'").run(); + closeDb(); + const { testView } = await import('../../src/lib/server/fleet/beta-tests.js'); + const { getBeta } = await import('../../src/lib/server/fleet/beta.js'); + const view = testView(id)!; + expect(view.plan.requirements.map(r => r.kind)).toEqual(['round', 'manual', 'manual', 'manual']); + expect(view.plan.requirements.some(r => 'event' in r)).toBe(false); + expect(view.servers[0].progress.every(p => !p.passed)).toBe(true); + expect(view.evidence.some(e => e.kind === 'round_change')).toBe(true); + expect(getBeta(id)?.state).toBe('running'); + const presets = await import('../../src/routes/api/betas/presets/+server.js'); + const saved = (await presets.GET().json()).presets.find((p: { name: string }) => p.name === 'Old plan'); + expect(saved.plan).toEqual(view.plan); + await postTelemetry(gameEvent('round_change')); + closeDb(); + expect(testView(id)?.servers[0].progress[0].current).toBe(25); + }); + + it('uses game_end for the legacy match gate without reading logs', async () => { + const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); + const { openBeta, getBeta, listBetaServers } = await import('../../src/lib/server/fleet/beta.js'); + const { script, version } = createScript({ name: 'Legacy beta', targetRelPath: 'legacy.gsc', content: Buffer.from('main() {}') }); + const { betaId } = await openBeta({ scripts: [{ scriptId: script.id, scriptVersionId: version.id }], servers: [names[0]], waitForHealth: true, waitForMatch: true }); + await tick(); await tick(); + expect(listBetaServers(betaId)[0].stage).toBe('awaiting-match'); + await appendFile(join(fixture.t5, names[0], 'logs/games_sp.log'), 'ExitLevel: executed\n'); + await tick(); + expect(getBeta(betaId)?.state).toBe('running'); + await postTelemetry(gameEvent('game_end')); + await postTelemetry(gameEvent('game_start')); + await tick(); + expect(getBeta(betaId)?.state).toBe('ready-promote'); }); it('stores validated presets as copies and rejects malformed plans', async () => { diff --git a/tests/unit/fleet-overview.test.ts b/tests/unit/fleet-overview.test.ts index 20f3485..6971740 100644 --- a/tests/unit/fleet-overview.test.ts +++ b/tests/unit/fleet-overview.test.ts @@ -1,5 +1,6 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { currentRound, fleetIdentity, matchesFleet, watchSatisfied, type FleetServer, type FleetOperation } from '../../src/lib/fleet-overview.js'; +import { gameEvent } from '../helpers/telemetry.js'; import { newFixture, type Fixture } from '../helpers/fixture.js'; const now = Date.parse('2026-09-18T12:00:00Z'); @@ -42,7 +43,7 @@ describe('Fleet discovery and match freshness', () => { expect(matchesFleet({ ...server, health: 'starting' }, '', 'attention', '', [], [], now)).toBe(false); }); it('only spotlights fresh telemetry from the current occupied match', () => { - const report = { server: server.name, round: 38, players: 4, receivedAt: iso(-1000) }; + const report = { ...gameEvent('round_change', 38, 4, server.name), receivedAt: iso(-1000) }; expect(currentRound(server, report, now)).toBe(38); for (const change of [{ receivedAt: iso(-121_000) }, { receivedAt: iso(-61_000) }, { receivedAt: iso(60_000) }]) { expect(currentRound(server, { ...report, ...change }, now)).toBeNull(); diff --git a/tests/unit/t6-config-flow.test.ts b/tests/unit/t6-config-flow.test.ts index 089f838..015a75f 100644 --- a/tests/unit/t6-config-flow.test.ts +++ b/tests/unit/t6-config-flow.test.ts @@ -79,13 +79,12 @@ describe('T6 config lifecycle in a mixed fleet', () => { }); it('reads the T6 game log and keeps log path traversal checks', async () => { - const { resolveLogPath, gameLogKey } = await import('../../src/lib/server/fleet/logs.js'); - const { scanMatchState } = await import('../../src/lib/server/fleet/match-log.js'); + const { resolveLogPath, gameLogKey, tailLog } = await import('../../src/lib/server/fleet/logs.js'); const logs = join(fixture.t6, town, 'logs'); await mkdir(logs, { recursive: true }); await writeFile(join(logs, 'games_zm.log'), '0:00 InitGame:\n0:01 J;123;0;Player\n'); expect(resolveLogPath(town, gameLogKey(town))).toBe(resolve(logs, 'games_zm.log')); - expect(await scanMatchState(town)).toMatchObject({ kind: 'in-progress' }); + expect((await tailLog(resolveLogPath(town, gameLogKey(town))!, 10))?.lines).toContain('0:01 J;123;0;Player'); expect(resolveLogPath('t6_/../../escape', 'games_zm')).toBeNull(); }); }); diff --git a/tests/unit/telemetry.test.ts b/tests/unit/telemetry.test.ts index dff3c7a..179bd30 100644 --- a/tests/unit/telemetry.test.ts +++ b/tests/unit/telemetry.test.ts @@ -1,5 +1,6 @@ import { afterEach, beforeEach, expect, it, vi } from 'vitest'; import { newFixture, type Fixture } from '../helpers/fixture.js'; +import { gameEvent, postTelemetry as post, roster } from '../helpers/telemetry.js'; let fixture: Fixture; beforeEach(async () => { @@ -10,42 +11,60 @@ beforeEach(async () => { }); afterEach(async () => fixture.cleanup()); -const payload = { event: 'round_change', server: 't5_zombie_theater_1', round: 17, players: 4 }; -async function post(body: unknown, contentType = 'application/json') { - const { POST } = await import('../../src/routes/api/telemetry/+server.js'); - return POST({ request: new Request('http://localhost/api/telemetry', { - method: 'POST', headers: { 'Content-Type': contentType }, body: JSON.stringify(body) - }) }); -} +const payload = gameEvent('round_change', 17); -it('persists reports, broadcasts updates, and leaves RCON safety data untouched', async () => { +it('persists full player stats, broadcasts updates, and leaves RCON safety data untouched', async () => { const { subscribe } = await import('../../src/lib/server/pipeline/bus.js'); const listener = vi.fn(); const unsubscribe = subscribe(listener); - expect((await post(payload)).status).toBe(200); - expect(listener).toHaveBeenCalledWith(expect.objectContaining({ type: 'server-telemetry' })); + expect(await (await post(payload)).json()).toEqual({ ok: true, telemetry: { ...payload, receivedAt: expect.any(String) } }); + expect(listener).toHaveBeenCalledWith({ type: 'server-telemetry', entries: [{ ...payload, receivedAt: expect.any(String) }] }); unsubscribe(); - // A new match can legitimately report a lower round; retries overwrite one row. - await post({ ...payload, round: 1 }); await post({ ...payload, round: 1 }); const { closeDb } = await import('../../src/lib/server/db/index.js'); closeDb(); const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js'); - expect(listTelemetry()).toEqual([expect.objectContaining({ round: 1, players: 4 })]); + expect(listTelemetry()).toEqual([{ ...payload, round: 1, receivedAt: expect.any(String) }]); const { getServer } = await import('../../src/lib/server/fleet/servers.js'); expect(getServer(payload.server)?.last_players).toBeNull(); const { GET } = await import('../../src/routes/api/events/servers/+server.js'); - const response = await GET(); - const reader = response.body!.getReader(); - await reader.read(); // greeting - const snapshot = await reader.read(); - expect(new TextDecoder().decode(snapshot.value)).toContain('"round":1'); + const reader = (await GET()).body!.getReader(); + await reader.read(); + const snapshot = new TextDecoder().decode((await reader.read()).value); + expect(snapshot).toContain('"round":1'); + expect(snapshot).toContain('"headshots":50'); + expect(snapshot).toContain('"guid":"12345678901234560"'); await reader.cancel(); }); -it.each([null, [], {}, { ...payload, event: 'other' }, { ...payload, round: -1 }, - { ...payload, round: 1.5 }, { ...payload, round: '17' }, { ...payload, players: -1 }, - { ...payload, players: null }])('rejects invalid telemetry: %j', async (body) => { +it('accepts all three events without an active beta and updates match lifecycle', async () => { + const { getServer } = await import('../../src/lib/server/fleet/servers.js'); + const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js'); + const start = gameEvent('game_start'); + expect((await post(start)).status).toBe(200); + expect(listTelemetry()).toEqual([{ ...start, receivedAt: expect.any(String) }]); + expect(getServer(start.server)?.game_started_at).toBeTruthy(); + expect(getServer(start.server)?.game_ended_at).toBeNull(); + expect((await post(payload)).status).toBe(200); + const end = gameEvent('game_end', 18); + expect((await post(end)).status).toBe(200); + expect(getServer(start.server)?.game_ended_at).toBeTruthy(); + expect(listTelemetry()[0]).toMatchObject(end); + await post(start); + expect(listTelemetry()[0]).not.toHaveProperty('round'); + expect(getServer(start.server)?.game_ended_at).toBeNull(); + expect((await post(gameEvent('game_start', 0, 0))).status).toBe(200); + expect((await post(gameEvent('game_end', 0, 0))).status).toBe(200); +}); + +it.each([null, [], {}, ...['boot', 'session_start', 'match_end', 'feature', 'other'].map(event => ({ ...payload, event })), + { ...payload, map: '' }, { ...payload, map: undefined }, { ...payload, round: -1 }, + { ...payload, round: 1.5 }, { ...payload, round: '17' }, { ...payload, players: 4 }, { ...payload, players: null }, + { ...payload, players: [null] }, { ...payload, players: [{ name: 'Josh', guid: '123' }] }, + ...['name', 'guid', 'kills', 'headshots', 'downs', 'revives', 'score'].map(field => ({ ...payload, players: [{ ...roster(1)[0], [field]: null }] })), + { ...payload, players: [{ ...roster(1)[0], guid: 12345678901234567 }] }, + { ...payload, players: [{ ...roster(1)[0], score: -1 }] } +])('rejects invalid telemetry: %j', async (body) => { expect((await post(body)).status).toBe(400); }); @@ -63,3 +82,15 @@ it('rejects malformed JSON', async () => { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{' }) })).status).toBe(400); }); + +it('migrates old round-only storage without fabricating player identities', async () => { + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + db().exec('DROP TABLE server_telemetry; CREATE TABLE server_telemetry (server TEXT PRIMARY KEY, round INTEGER NOT NULL, players INTEGER NOT NULL, received_at TEXT NOT NULL)'); + db().prepare('INSERT INTO server_telemetry VALUES (?, ?, ?, ?)').run(payload.server, 99, 4, new Date().toISOString()); + closeDb(); + const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js'); + expect(listTelemetry()).toEqual([]); + await post(payload); + closeDb(); + expect(listTelemetry()[0]).toMatchObject(payload); +});