Simplify telemetry to three game events
This commit is contained in:
@@ -66,8 +66,7 @@ See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow.
|
||||
|
||||
Betas support configurable gameplay, participation, stability and manual-review
|
||||
requirements, saved presets, coverage rules and staged rollout. See
|
||||
[Beta test plans](docs/beta-test-plans.md) for usage and the session telemetry
|
||||
contract used by round and custom-feature checks.
|
||||
[Beta test plans](docs/beta-test-plans.md) for usage and the three-event game telemetry contract.
|
||||
|
||||
File changes are staged sequentially. Their activation uses rolling restarts:
|
||||
by default only **one server across the fleet** can restart or recover at a
|
||||
@@ -229,32 +228,21 @@ Implementation references (checked September 2026):
|
||||
- [T6 map/ruleset combinations](https://github.com/xerxes-at/T6ServerConfigs/blob/master/localappdata/Plutonium/storage/t6/dedicated_zm.cfg)
|
||||
- [Plutonium GSC paths](https://plutonium.pw/docs/modding/loading-mods/)
|
||||
|
||||
## Round telemetry experiment
|
||||
## Game telemetry
|
||||
|
||||
Send `POST /api/telemetry` with `Content-Type: application/json` to the Electric
|
||||
Cherry host (port 3000 in local development):
|
||||
The server script sends JSON to `POST /api/telemetry`:
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000/api/telemetry \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"event":"round_change","round":17,"server":"t5_zombie_prototype_1","players":4}'
|
||||
```
|
||||
- `game_start`: server, map, and all players (name and GUID).
|
||||
- `round_change`: server, map, round, and all players with kills, headshots, downs, revives, and score.
|
||||
- `game_end`: the same fields as round change, with final stats.
|
||||
|
||||
The server must already be registered in Electric Cherry. `round` and `players`
|
||||
are required nonnegative integers. Success returns HTTP 200 with
|
||||
`{ "ok": true, "telemetry": { "server": "...", "round": 17, "players": 4, "receivedAt": "..." } }`.
|
||||
Invalid payloads return 400, unknown/destroyed servers 404, and non-JSON content
|
||||
types 415. This endpoint uses the app's existing LAN/Tailscale access model;
|
||||
there is no API key requirement.
|
||||
These reports work with or without an active beta. Electric Cherry records
|
||||
the full player data, updates Fleet, and applies gameplay progress to active
|
||||
betas automatically. No discovery request or beta-specific fields are needed.
|
||||
|
||||
The latest report is saved across app restarts and streamed immediately to the
|
||||
dashboard and server detail page, including after browser reconnection. The UI
|
||||
labels it as the last reported round with its age: round-change events alone
|
||||
cannot detect a stopped script or a match ending. Send the initial round on each
|
||||
new match, then each round change. The latest received request wins (including
|
||||
lower rounds for new matches); there is no event ordering or history yet.
|
||||
Telemetry player counts are displayed with the report and never replace RCON
|
||||
occupancy or affect automated restart gates.
|
||||
See [Game reporter protocol](docs/beta-test-plans.md#game-reporter-protocol) for
|
||||
complete payload examples. Telemetry does not replace RCON's empty-server
|
||||
check before an automated restart.
|
||||
|
||||
## Running tests
|
||||
|
||||
|
||||
+134
-621
@@ -1,652 +1,165 @@
|
||||
# Beta test plans
|
||||
|
||||
New betas use a saved plan rather than the two legacy wait flags. Existing
|
||||
betas and API callers without `testPlan` retain their original workflow.
|
||||
Schema additions are additive and run at startup; no active beta needs to be
|
||||
closed to install this feature.
|
||||
Create a plan in Betas, choose its servers, and deploy. The server's game script
|
||||
reports gameplay to Electric Cherry. The same reports update Fleet and any active
|
||||
beta on that server.
|
||||
|
||||
## Game reporter protocol
|
||||
|
||||
Send JSON to `POST /api/telemetry` with `Content-Type: application/json`.
|
||||
There are exactly three events. Send them whether or not a beta is active.
|
||||
|
||||
| Event | Fields |
|
||||
| --- | --- |
|
||||
| `game_start` | `server`, `map`, `players` with each player's `name` and `guid` |
|
||||
| `round_change` | `server`, `map`, `round`, `players` with each player's `name`, `guid`, `kills`, `headshots`, `downs`, `revives`, and `score` |
|
||||
| `game_end` | `server`, `map`, `round`, `players` with each player's `name`, `guid`, `kills`, `headshots`, `downs`, `revives`, and `score` |
|
||||
|
||||
`server` is the exact registered Electric Cherry server name. `map` is the
|
||||
current map. `players` is the full player array, including an empty array when
|
||||
there are no players. Names and GUIDs are strings; keep GUIDs as strings to
|
||||
preserve their precision. Round and stat values are nonnegative JSON integers.
|
||||
|
||||
### Game start
|
||||
|
||||
Send when the game starts. Round and stats are not required.
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "game_start",
|
||||
"server": "t6_zombie_town_1",
|
||||
"map": "zm_transit",
|
||||
"players": [
|
||||
{ "name": "Josh", "guid": "12345678901234567" },
|
||||
{ "name": "Alex", "guid": "23456789012345678" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Round change
|
||||
|
||||
Send the new round and each player's current stats.
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "round_change",
|
||||
"server": "t6_zombie_town_1",
|
||||
"map": "zm_transit",
|
||||
"round": 25,
|
||||
"players": [
|
||||
{ "name": "Josh", "guid": "12345678901234567", "kills": 420, "headshots": 160, "downs": 1, "revives": 3, "score": 32000 },
|
||||
{ "name": "Alex", "guid": "23456789012345678", "kills": 380, "headshots": 140, "downs": 3, "revives": 1, "score": 28000 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Game end
|
||||
|
||||
Send the final round and each player's final stats when the game ends.
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "game_end",
|
||||
"server": "t6_zombie_town_1",
|
||||
"map": "zm_transit",
|
||||
"round": 26,
|
||||
"players": [
|
||||
{ "name": "Josh", "guid": "12345678901234567", "kills": 445, "headshots": 170, "downs": 2, "revives": 3, "score": 33500 },
|
||||
{ "name": "Alex", "guid": "23456789012345678", "kills": 401, "headshots": 146, "downs": 4, "revives": 1, "score": 29750 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Success returns HTTP 200 with `{ "ok": true, "telemetry": { ... } }`.
|
||||
The returned telemetry contains the event and an Electric Cherry `receivedAt`
|
||||
timestamp. Invalid JSON or fields return 400, an unknown/destroyed server 404,
|
||||
and a non-JSON content type 415.
|
||||
|
||||
The script sends each event once, in gameplay order, directly to Electric Cherry.
|
||||
There is no discovery request, beta token, boot/session ID, sequence number,
|
||||
custom feature event, external helper, or separate beta acknowledgement.
|
||||
Basic payload checks happen in the receiver.
|
||||
|
||||
Electric Cherry saves the latest complete report, including player stats, and
|
||||
streams it to connected clients. A new start clears the previous round display;
|
||||
an end closes the match. Receipt order determines the latest report. This small
|
||||
protocol does not attempt to identify delayed events from earlier games or
|
||||
provide exactly-once delivery across games.
|
||||
|
||||
The game scripts are uploaded artifacts, not source maintained in this
|
||||
repository. Their reporter should emit the three payloads above.
|
||||
|
||||
## Requirements
|
||||
|
||||
Use **Add requirement** in Betas. Requirements on a qualifying server are ANDed:
|
||||
Requirements on a qualifying server are ANDed:
|
||||
|
||||
| Requirement | Evidence |
|
||||
| --- | --- |
|
||||
| Reach round X | Session telemetry reports round >= X with the required player count |
|
||||
| Complete qualifying matches | Normal completed matches, optionally qualified by round, players and duration |
|
||||
| Populated minutes | Fresh RCON samples at both ends of a populated interval |
|
||||
| Player-hours | Populated minutes × the lower sampled player count / 60 |
|
||||
| Reach player count | Fresh healthy RCON sample |
|
||||
| Sustained population | Consecutive fresh healthy samples above the player threshold |
|
||||
| Reach round X | A round change or game end reports round >= X with the required player count |
|
||||
| Complete qualifying matches | Game end, optionally qualified by round, player count, and duration |
|
||||
| Populated minutes | Time between consecutive game reports with enough players at both ends |
|
||||
| Player-hours | Populated minutes multiplied by the lower reported player count, divided by 60 |
|
||||
| Reach player count | Player array length in a game report |
|
||||
| Sustained population | Consecutive report intervals above the player threshold within one game |
|
||||
| Continuous healthy minutes | Consecutive fresh container/RCON observations |
|
||||
| Maximum health outages | Transitions from observed healthy to unhealthy |
|
||||
| Maximum unexpected restarts | Distinct reported process boots within the same test run |
|
||||
| Maximum script errors | New console lines matching script errors/exceptions, unhandled exceptions or fatal errors |
|
||||
| Successful recovery cycles | Repeat the other requirements after each additional empty-server restart |
|
||||
| Feature event count | Named, session-scoped game events |
|
||||
| Tester checklist | Tester name and observation, entered on the beta detail page |
|
||||
| Tester checklist | Tester name and observation entered on the beta detail page |
|
||||
|
||||
There is one round requirement: **reach round X**. For repeated high-round runs,
|
||||
use a qualifying-match count with a minimum round. The minimum player count must
|
||||
be present at the qualifying round observation in the same match. A match with
|
||||
many players and a different solo high-round match cannot combine to pass.
|
||||
All gameplay requirements use the script reports. Match counts do not read game
|
||||
logs. The reported round and required player count must occur together in the
|
||||
same game; observations from different games cannot combine to qualify a match.
|
||||
The final game-end report can supply that qualifying round. Consecutive duplicate
|
||||
game ends do not add another completion.
|
||||
|
||||
Unqualified match counts use incremental game logs (`InitGame`, a player join,
|
||||
then `ExitLevel: executed`). Round-qualified or multi-player-qualified matches
|
||||
use session telemetry. Each requirement uses one source, preventing duplicate
|
||||
counts. Match duration is measured from the observed join/session start. Log
|
||||
polling can undercount duration; it does not infer unobserved elapsed time.
|
||||
Match duration requires a game start and uses receipt times. Population time
|
||||
advances only when another report arrives in the same game; it uses the lower
|
||||
player count at the interval's endpoints. There is no heartbeat requirement or
|
||||
short interval limit between rounds. These are estimates between reported
|
||||
snapshots, not continuous measurements of player presence.
|
||||
|
||||
Round, feature and unexpected-restart checks require the reporting protocol
|
||||
below. Legacy round reports still update Fleet but cannot prove beta progress.
|
||||
The game scripts themselves are uploaded artifacts, not source maintained in
|
||||
this repository; update their reporter to emit the protocol before selecting
|
||||
these requirements in production.
|
||||
Health checks and safe restart gates use the existing container/RCON monitoring.
|
||||
Health intervals spanning more than 60 seconds earn no healthy time.
|
||||
|
||||
## Coverage, rollout and outcomes
|
||||
|
||||
Coverage can require all selected servers, a quorum, one per selected map,
|
||||
one per selected game, or one per selected game/map combination. Coverage
|
||||
metadata is pinned when the beta opens. Register game/map metadata first.
|
||||
metadata is pinned when the beta opens.
|
||||
|
||||
Set wave size to 1 for a canary, a larger number for batches, or 0 for all
|
||||
selected servers. Later waves reserve their artifact targets but receive no
|
||||
files until every server in the prior waves has passed. All waves must start
|
||||
testing before the final coverage rule can pass. Error-budget violations on
|
||||
any tested server block promotion, including with quorum coverage.
|
||||
selected servers. Later waves receive files after the previous waves pass.
|
||||
Health-outage budget violations on any tested server block promotion.
|
||||
|
||||
The test window starts when the beta opens, including time waiting for empty
|
||||
servers. Missing evidence at the deadline produces **inconclusive**, not a
|
||||
failure or an automatic rollback. Health and log monitoring continue after a
|
||||
pass until promotion; a deteriorating result revokes readiness.
|
||||
servers. Missing evidence at the deadline produces **inconclusive**.
|
||||
Health monitoring continues after a pass until promotion.
|
||||
|
||||
For failures, select hold as failed, pause for investigation, or automatic
|
||||
rollback. Rollback restores only files touched by the beta. It can queue rolling
|
||||
restarts when servers empty, or leave restarts manual. Manual rollback uses the
|
||||
same selected restart policy. No beta auto-promotes.
|
||||
|
||||
Pause holds evidence evaluation and rollout. Resume retains evidence and the
|
||||
original deadline. A fresh attempt retires old evidence and restarts deployed
|
||||
servers when empty; it does not extend the deadline. Expired or incomplete-file
|
||||
deployments should be rolled back and reopened. Recovery cycles similarly
|
||||
require fresh milestones/checklists, while carrying error budgets forward.
|
||||
|
||||
Time accounting excludes intervals spanning monitoring gaps longer than 60
|
||||
seconds. A missing first sample earns no time. A log that cannot be read cannot
|
||||
pass a clean-log requirement. Log backlog above 2 MB makes the plan inconclusive
|
||||
instead of silently dropping evidence. Partial lines, rotations, byte cursors,
|
||||
attempts and accepted events persist in SQLite. The detail page shows the most
|
||||
recent 200 evidence entries; older entries remain in the database.
|
||||
|
||||
## Game reporter protocol
|
||||
|
||||
This section is a standalone integration guide for the T6 telemetry script.
|
||||
It describes the **currently implemented API**, including its limitations.
|
||||
Examples use `t6_zombie_town_1`; replace that with the exact, case-sensitive
|
||||
Electric Cherry server registry name. The API base URL is the Electric Cherry
|
||||
application, not Komodo or the game server's RCON address.
|
||||
|
||||
### 1. Two reporting modes: Fleet display and beta evidence
|
||||
|
||||
The reporter must work whether or not anybody is beta testing:
|
||||
|
||||
| Situation | Reporter behavior |
|
||||
| --- | --- |
|
||||
| No eligible beta run for this server | Keep sending ordinary `round_change` reports for Fleet. Do not send beta-only events. |
|
||||
| One eligible run discovered at process startup | Send the beta event stream with that run's `runId`. Scoped `round_change` reports also update Fleet. |
|
||||
| Several eligible runs discovered at startup | Send a copy of each beta event to each run, with separate acknowledgement/queue state per run. |
|
||||
| Beta opened while an existing game process is running | Continue its existing reporting mode. The beta waits for the server to empty and restarts it; the new process discovers the new run. |
|
||||
| Beta has passed and is awaiting promotion | Continue reporting. A passed test still accepts evidence until promotion or another outcome stops it. |
|
||||
| Beta paused, failed, expired, promoted, rolled back, or run superseded | Stop crediting events to that inactive run; continue ordinary round reporting. See the suspension rules below. |
|
||||
|
||||
**No active beta is normal, not an error.** Fleet telemetry does not depend on a
|
||||
beta, a run token, or a successful beta discovery request.
|
||||
|
||||
### 2. Startup discovery and an empty result
|
||||
|
||||
On **actual game-process startup**, request:
|
||||
|
||||
```http
|
||||
GET /api/telemetry?server=t6_zombie_town_1
|
||||
```
|
||||
|
||||
For a registered server without an eligible run, this returns **HTTP 200**:
|
||||
|
||||
```json
|
||||
{
|
||||
"runs": []
|
||||
}
|
||||
```
|
||||
|
||||
This can mean there is no beta, the server is not a beta target, its wave has not
|
||||
started, the beta is waiting to restart it, or the test is not accepting evidence.
|
||||
Legacy betas without a test plan are not returned either. The response is a list
|
||||
of eligible **runs**, not a list of every beta in the application.
|
||||
|
||||
With an eligible run, the response is **HTTP 200**:
|
||||
|
||||
```json
|
||||
{
|
||||
"runs": [
|
||||
{
|
||||
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
|
||||
"betaId": 42
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The response has `Cache-Control: no-store`. Do not assume it contains exactly one
|
||||
entry, and do not choose only the first entry. `betaId` is informational; POST
|
||||
requests use `runId`, not `betaId`, to identify the test attempt.
|
||||
|
||||
A missing or unknown `server` query parameter returns **404**, not an empty
|
||||
success. Treat that as a configuration issue. The GET handler currently only
|
||||
checks registry existence; POST additionally rejects a server marked destroyed.
|
||||
Do not interpret successful discovery as permission to report for another server.
|
||||
|
||||
Cache the startup run list for this process lifetime. Electric Cherry creates a
|
||||
new run before requesting each beta-controlled restart. That restart is the
|
||||
boundary that makes gameplay belong to the newly deployed build. **Do not poll
|
||||
for and adopt new run IDs in an old process, at a map reload, or mid-match.**
|
||||
Never move queued events to a newer run ID.
|
||||
|
||||
If discovery fails, use bounded retries with backoff while still initializing,
|
||||
before gameplay starts. If startup discovery cannot finish, run in Fleet-only
|
||||
mode for that process and log the problem. Do not attach later gameplay to a run
|
||||
discovered after that startup boundary. A new process/retry is needed for a new
|
||||
safe beta attachment; the current API has no late-attachment handshake.
|
||||
|
||||
### 3. Reporting without a beta
|
||||
|
||||
Send this on each round change, even when `runs` is empty:
|
||||
|
||||
```http
|
||||
POST /api/telemetry
|
||||
Content-Type: application/json
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "round_change",
|
||||
"server": "t6_zombie_town_1",
|
||||
"round": 17,
|
||||
"players": 4
|
||||
}
|
||||
```
|
||||
|
||||
Example success response:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"telemetry": {
|
||||
"server": "t6_zombie_town_1",
|
||||
"round": 17,
|
||||
"players": 4,
|
||||
"receivedAt": "2026-09-18T12:00:00.000Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Omit `runId` entirely in this mode. `"runId": null` and `"runId": ""` are
|
||||
invalid; they are not equivalent to omitting the field. The response has no
|
||||
`test` property when `runId` was omitted.
|
||||
|
||||
Without `runId`, only `round_change` is supported. Do not send unscoped `boot`,
|
||||
`session_start`, `feature`, or `match_end` requests: they return 400. Do not buffer
|
||||
feature occurrences from normal operation to submit when a future beta opens.
|
||||
|
||||
Electric Cherry stores only the latest Fleet round report per server and stamps
|
||||
its receipt time. Lower rounds are valid when a new match begins. These reports
|
||||
do not change RCON player counts, restart safety decisions, or beta progress.
|
||||
|
||||
### 4. Identifier lifetimes and the T6 process boundary
|
||||
|
||||
| Field | Who creates it | Lifetime and rules |
|
||||
| --- | --- | --- |
|
||||
| `server` | Configuration | Exact registered name, unchanged across matches and process restarts. |
|
||||
| `runId` | Electric Cherry | One server attempt/recovery phase. Use the value discovered at process startup. Treat it as opaque. |
|
||||
| `bootId` | Reporter | Unique for each actual game-process lifetime. Keep it across matches, map reloads, and reporter reinitialization within that process. |
|
||||
| `sessionId` | Reporter | Unique for each actual played match. Generate a new value for the next match, even if the players and map are unchanged. |
|
||||
| `eventId` | Reporter | Unique for each logical event within a run, including across sessions/boots. A delivery retry reuses the exact same ID. |
|
||||
| `sequence` | Reporter | Start at 0 for `session_start`; increment for every subsequent event in that session, across all event types. Reset only for a new session. |
|
||||
|
||||
Generate collision-resistant IDs, such as UUIDs. A boot ID plus a session ID and
|
||||
counter is also suitable for event IDs if the combined string fits the length
|
||||
limit. Example IDs below are illustrative; never hardcode them in the script.
|
||||
The same event ID can be used for copies sent to different runs because
|
||||
deduplication is scoped to `(runId, eventId)`.
|
||||
|
||||
**A GSC script's initialization is not necessarily a game-process startup.**
|
||||
The T6 implementation must establish where its state survives match/map changes.
|
||||
If script state resets on each map load, preserve process identity, the startup
|
||||
run list, and delivery state in a process-lifetime facility or a helper that can
|
||||
reliably identify the game process. A helper's own restart must not invent a new
|
||||
game boot. Do not assume map-local globals provide process-lifetime storage.
|
||||
If that distinction is unavailable, leave beta reporting disabled rather than
|
||||
reporting a fabricated boot; ordinary Fleet round reports still work.
|
||||
|
||||
An unexpected game-process restart may discover the **same** run ID. Send a new
|
||||
boot ID: the backend counts additional distinct boots in a run as unexpected
|
||||
restarts. A planned recovery restart receives a **new** run ID and starts a new
|
||||
evidence phase. Restarting only the Electric Cherry web app changes neither the
|
||||
game boot nor the reporter's cached IDs.
|
||||
|
||||
### 5. Common request fields
|
||||
|
||||
All requests use JSON objects and `Content-Type: application/json`. Every
|
||||
beta-scoped event includes these fields unless the table says otherwise:
|
||||
|
||||
| Field | Type | Constraints |
|
||||
| --- | --- | --- |
|
||||
| `event` | string | Exactly `boot`, `session_start`, `round_change`, `feature`, or `match_end`. |
|
||||
| `server` | string | Registered server name, 1-255 characters, case-sensitive. |
|
||||
| `runId` | string | Required for beta evidence; 1-128 characters from `A-Z a-z 0-9 _ . : -`. |
|
||||
| `eventId` | string | Same identifier character/length limits. |
|
||||
| `bootId` | string | Same identifier character/length limits. |
|
||||
| `sessionId` | string | Same identifier limits; required except for `boot`. Omit for `boot`. |
|
||||
| `sequence` | number | Nonnegative safe integer. Use 0 for `boot` and `session_start`. |
|
||||
| `round` | number | Nonnegative safe integer; the actual player-facing round. Use 0 for `boot`. |
|
||||
| `players` | number | Nonnegative safe integer; actual participating player count at the event. Use 0 for `boot`. |
|
||||
| `name` | string | Required only for `feature`; 1-120 characters from the identifier character set. |
|
||||
|
||||
Numbers must be JSON numbers, not strings, fractions, `null`, or negative values.
|
||||
The maximum safe integer is 9007199254740991. Sequence numbers need to increase;
|
||||
the backend does not require them to be contiguous, but the sender should emit a
|
||||
single ordered stream. Do not use a separate counter for each event type.
|
||||
|
||||
The examples below show excerpts from one process and match. Earlier round
|
||||
reports are omitted, hence the sequence jumps to 25. The real script should send
|
||||
each round change and increment its counter for every event it produces.
|
||||
|
||||
### 6. Full event examples
|
||||
|
||||
#### Process boot
|
||||
|
||||
Send once per discovered run at real game-process startup, before any session
|
||||
events. A retry is the same request, not another boot with another ID.
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "boot",
|
||||
"server": "t6_zombie_town_1",
|
||||
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
|
||||
"eventId": "boot-a81f:boot",
|
||||
"bootId": "boot-a81f",
|
||||
"sequence": 0,
|
||||
"round": 0,
|
||||
"players": 0
|
||||
}
|
||||
```
|
||||
|
||||
#### Start of a played match
|
||||
|
||||
Send when actual gameplay begins with players, not merely when an empty map
|
||||
loads. Retain this `sessionId` until that match ends or is abandoned.
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "session_start",
|
||||
"server": "t6_zombie_town_1",
|
||||
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
|
||||
"eventId": "boot-a81f:match-b92c:0",
|
||||
"bootId": "boot-a81f",
|
||||
"sessionId": "match-b92c",
|
||||
"sequence": 0,
|
||||
"round": 1,
|
||||
"players": 4
|
||||
}
|
||||
```
|
||||
|
||||
After this is acknowledged, report the initial round with `round_change` too.
|
||||
The `round` on `session_start` does not itself satisfy a round requirement.
|
||||
|
||||
#### Reaching a round
|
||||
|
||||
Send on entry to each round, with the player count sampled at that event.
|
||||
There is no separate "round completed" event.
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "round_change",
|
||||
"server": "t6_zombie_town_1",
|
||||
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
|
||||
"eventId": "boot-a81f:match-b92c:25",
|
||||
"bootId": "boot-a81f",
|
||||
"sessionId": "match-b92c",
|
||||
"sequence": 25,
|
||||
"round": 25,
|
||||
"players": 4
|
||||
}
|
||||
```
|
||||
|
||||
This can satisfy a round-25 check requiring four players. It also records the
|
||||
qualifying round for a later completed-match check. A `match_end` reporting round
|
||||
25 does not replace a missing `round_change` event for that round.
|
||||
|
||||
#### Feature: a successful perk purchase
|
||||
|
||||
The top-level event is always `"feature"`. The custom feature identifier goes in
|
||||
`name`; it is not a new top-level event type.
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "feature",
|
||||
"server": "t6_zombie_town_1",
|
||||
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
|
||||
"eventId": "boot-a81f:match-b92c:26",
|
||||
"bootId": "boot-a81f",
|
||||
"sessionId": "match-b92c",
|
||||
"sequence": 26,
|
||||
"round": 25,
|
||||
"players": 4,
|
||||
"name": "perk.purchased"
|
||||
}
|
||||
```
|
||||
|
||||
Emit when the purchase actually succeeds, not when a player opens a menu or
|
||||
attempts an unaffordable purchase. One accepted event adds **one** to each
|
||||
matching feature requirement whose minimum round/player qualifiers are met.
|
||||
This event does not require waiting for the match to end.
|
||||
|
||||
For example, configure **Feature event count** with event name `perk.purchased`,
|
||||
target 10, minimum round 20, and minimum players 4. The example above contributes
|
||||
one occurrence. A purchase at round 19 or with only three players can still be
|
||||
accepted into the event stream, but contributes zero to that requirement.
|
||||
|
||||
#### Feature: successful Pack-a-Punch completion
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "feature",
|
||||
"server": "t6_zombie_town_1",
|
||||
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
|
||||
"eventId": "boot-a81f:match-b92c:27",
|
||||
"bootId": "boot-a81f",
|
||||
"sessionId": "match-b92c",
|
||||
"sequence": 27,
|
||||
"round": 25,
|
||||
"players": 4,
|
||||
"name": "weapon.pack_a_punch.completed"
|
||||
}
|
||||
```
|
||||
|
||||
#### Feature: successful execution of a custom mechanic
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "feature",
|
||||
"server": "t6_zombie_town_1",
|
||||
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
|
||||
"eventId": "boot-a81f:match-b92c:28",
|
||||
"bootId": "boot-a81f",
|
||||
"sessionId": "match-b92c",
|
||||
"sequence": 28,
|
||||
"round": 25,
|
||||
"players": 4,
|
||||
"name": "mechanic.teleporter.completed"
|
||||
}
|
||||
```
|
||||
|
||||
These names are **suggested conventions**, not preinstalled T6 hooks. The script
|
||||
must instrument the corresponding successful actions. Only emit a mechanic's
|
||||
event on maps/builds where that mechanic actually exists.
|
||||
|
||||
Names are matched exactly and case-sensitively against the UI's **Reported event
|
||||
name**. A requirement named `perk.purchased` with target 10 counts ten accepted
|
||||
occurrences of that exact name; `perk.*` is not a wildcard.
|
||||
|
||||
For a specific perk, use a distinct name such as `perk.juggernog.purchased`.
|
||||
The current API does not filter on a separate `perk`, `weapon`, or `playerId`
|
||||
field. You may deliberately emit both a generic and a specific marker for one
|
||||
action, but they must be separate events with separate event IDs and sequences;
|
||||
each requirement still counts only its exact matching name.
|
||||
|
||||
There is no bulk-count field. `"count": 10`, `value`, `metadata`, `timestamp`,
|
||||
`duration`, and other extra properties are not consumed or preserved as beta
|
||||
signal fields. Send one event for each real occurrence. Keep any additional
|
||||
diagnostics in script logs until the API explicitly supports them.
|
||||
|
||||
#### Normal match completion
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "match_end",
|
||||
"server": "t6_zombie_town_1",
|
||||
"runId": "4d0d91f9-39d2-4793-a9a3-4054c2a91c88",
|
||||
"eventId": "boot-a81f:match-b92c:29",
|
||||
"bootId": "boot-a81f",
|
||||
"sessionId": "match-b92c",
|
||||
"sequence": 29,
|
||||
"round": 25,
|
||||
"players": 4
|
||||
}
|
||||
```
|
||||
|
||||
Send once when a played match ends normally, including a normal game-over.
|
||||
Capture the actual terminal player count; the example's four players are not a
|
||||
value to fabricate if players have already left. The backend retains earlier
|
||||
round/player observations for qualification.
|
||||
|
||||
Do **not** send `match_end` for a crash, forced process shutdown, arbitrary map
|
||||
rotation, or an idle map cycle. A generic script teardown callback alone is not
|
||||
proof of normal match completion. There is no `session_abort` event at present;
|
||||
abandon that session and use a new ID when actual gameplay next begins. Never
|
||||
replay events for an abandoned session into the new one.
|
||||
|
||||
Accepted `match_end` closes the session. Later events for that session are not
|
||||
accepted. Round-qualified or multi-player-qualified match requirements use this
|
||||
stream; unqualified completed-match requirements use game logs instead, so the
|
||||
same match is not counted twice for one requirement.
|
||||
|
||||
### 7. Responses: transport success is not evidence acceptance
|
||||
|
||||
An accepted `boot`, `session_start`, `feature`, or `match_end` returns HTTP 200:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"test": {
|
||||
"accepted": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
An accepted scoped `round_change` additionally updates the Fleet display:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"telemetry": {
|
||||
"server": "t6_zombie_town_1",
|
||||
"round": 25,
|
||||
"players": 4,
|
||||
"receivedAt": "2026-09-18T12:25:00.000Z"
|
||||
},
|
||||
"test": {
|
||||
"accepted": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Acceptance means the event was recorded in the run's stream. It does not mean
|
||||
the whole test passed, or even that a feature/round event met a configured
|
||||
requirement's qualifiers.
|
||||
|
||||
A retry of an already accepted event returns HTTP 200:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"test": {
|
||||
"accepted": false,
|
||||
"reason": "Duplicate event"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Treat that exact duplicate response as delivery already acknowledged. Do not
|
||||
generate a new ID to force the same action to count twice.
|
||||
|
||||
An inactive run returns HTTP 200:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"test": {
|
||||
"accepted": false,
|
||||
"reason": "Run is inactive or superseded; obtain the current run at process startup"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Invalid ordering or session identity also returns HTTP 200:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"test": {
|
||||
"accepted": false,
|
||||
"reason": "Duplicate, out-of-order, closed, or unknown session/boot"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
That last response is **not** an acknowledgement: inspect the boot/session/
|
||||
sequence handling. The API currently provides explanatory strings, not distinct
|
||||
machine-readable error codes for every rejection. Do not treat every
|
||||
`accepted: false` as a harmless duplicate or assume every inactive run is closed
|
||||
permanently; paused runs use the same inactive response.
|
||||
|
||||
**Fleet updates are independent of beta acceptance.** A syntactically valid
|
||||
scoped `round_change` still writes the latest Fleet round even when
|
||||
`test.accepted` is false, including a duplicate or inactive-run rejection. Its
|
||||
response contains both `telemetry` and `test`. Do not replay an old round backlog
|
||||
after newer reports: it can move the Fleet display backwards. There is no
|
||||
client-timestamp freshness check.
|
||||
|
||||
| HTTP result | Meaning and handling |
|
||||
| --- | --- |
|
||||
| 200, no `test` | Ordinary Fleet report accepted. |
|
||||
| 200, `test.accepted: true` | Beta event accepted; advance that run's queue. |
|
||||
| 200, exact `Duplicate event` reason | Already accepted earlier; advance that run's queue. |
|
||||
| 200, inactive-run reason | Suspend beta delivery for that run; continue Fleet reporting. |
|
||||
| 200, session/order rejection | Log an integration error and stop crediting that broken session. Do not invent a replacement session for an in-progress match. |
|
||||
| 400 | Invalid JSON, fields, or event type; fix the payload instead of retrying unchanged forever. |
|
||||
| 404 on POST | Server is unregistered or destroyed; check configuration/lifecycle. |
|
||||
| 415 | Missing or incorrect JSON content type. |
|
||||
| Network error, timeout, or 5xx | Delivery is uncertain; retry the same request with backoff within a bounded queue. |
|
||||
|
||||
### 8. Ordering, retries, pauses, and multiple runs
|
||||
|
||||
**Recommended sender behavior** (client implementation guidance, not extra API
|
||||
features):
|
||||
|
||||
1. Keep delivery asynchronous so a web-service outage never blocks the game.
|
||||
2. Send `boot` first for each run and wait for acceptance or an acknowledged
|
||||
duplicate before delivering its `session_start`.
|
||||
3. Serialize the remaining events per run. Do not allow a later event to overtake
|
||||
an uncertain request. Use the same session counter for rounds, features, and
|
||||
match completion.
|
||||
4. On timeout, retry with exactly the same event ID, sequence, and payload. The
|
||||
first request may have committed even if its response was lost.
|
||||
5. Bound queue size, retry count, and event age. For example, back off through
|
||||
1, 2, 5, and 10 seconds rather than retrying every frame. These intervals are
|
||||
a suggested client policy, not server-enforced values.
|
||||
6. If a long outage makes the queue stale, abandon beta evidence for that session
|
||||
and keep the current Fleet round visible. Do not replay historical feature or
|
||||
completion events hours later. Resume beta evidence at a later real match
|
||||
only if the same startup-pinned run is still eligible and boot delivery is
|
||||
established.
|
||||
7. Keep acknowledgement state separately for every discovered run. Failure on
|
||||
one run does not change another run's IDs or acknowledged sequence. Reuse the
|
||||
logical event ID across copies, changing only `runId`.
|
||||
|
||||
The server stamps **receipt time**, not game event time. In particular, match
|
||||
duration is measured between accepted `session_start` and `match_end` receipts.
|
||||
Delayed replay can distort duration and no client-provided timestamp fixes it.
|
||||
Keep beta events near real time; do not use them as a historical-import API.
|
||||
|
||||
While scoped round delivery is healthy, no extra unscoped copy is necessary.
|
||||
If beta delivery is suspended or lagging, ordinary round reports can keep Fleet
|
||||
current. Retire the affected session queue rather than later sending its old
|
||||
rounds behind newer unscoped reports. A simple implementation may serialize
|
||||
delivery across all run copies to avoid cross-run round reordering.
|
||||
|
||||
For paused/inactive runs, the simplest safe policy is Fleet-only reporting until
|
||||
the next actual process startup. If automatic resume within the same process is
|
||||
desired, the reporter may recheck **only IDs already pinned at startup**:
|
||||
|
||||
- Intersect discovery results with the original startup list. Never adopt an
|
||||
unfamiliar run ID returned by a later GET.
|
||||
- When a cached run becomes eligible again, resume at the **next real match**,
|
||||
using the existing boot ID and a new session ID. Establish boot delivery first;
|
||||
reusing the original boot request/event ID is safe if its acknowledgement was
|
||||
lost.
|
||||
- Discard events generated during suspension. Do not manufacture a new session
|
||||
start in the middle of an existing match or replay paused gameplay.
|
||||
- If startup returned `runs: []`, there are no pinned IDs to resume. Stay
|
||||
Fleet-only until a real process restart. Similarly, a process started while a
|
||||
test was paused may require an operator-requested fresh attempt after resume.
|
||||
|
||||
There is no polling interval or resume notification supplied by this API. The
|
||||
reporter owns its retry/suspension policy. The rule about startup pinning must
|
||||
remain true regardless of that policy.
|
||||
|
||||
### 9. T6 implementation handoff checklist
|
||||
|
||||
- Configure the Electric Cherry URL and exact server registry name.
|
||||
- Preserve ordinary round reports when no beta is active or the service is
|
||||
temporarily unreachable; never block gameplay waiting for HTTP.
|
||||
- Identify actual game-process boots separately from GSC initialization and
|
||||
map/match resets. Cache discovery results for the game process lifetime.
|
||||
- Instrument real session start, round entry, normal game-over, and successful
|
||||
feature actions. Confirm each chosen T6 callback's semantics; do not infer
|
||||
normal completion from generic teardown.
|
||||
- Use one sequence counter per session, unique event IDs, stable retry payloads,
|
||||
and an ordered bounded queue with separate acknowledgements per run.
|
||||
- Decide and document the feature names the script emits so beta creators can
|
||||
enter those exact names. Extra JSON metadata cannot be used as a qualifier.
|
||||
- Keep beta-only events scoped to startup-pinned runs. Do not backfill pre-beta,
|
||||
paused, abandoned, or previous-process gameplay into a new run.
|
||||
- Handle `ok` and `test.accepted` separately. Log rejections with enough context
|
||||
to diagnose the run, boot, session, event ID, and sequence.
|
||||
|
||||
Suggested verification cases for the telemetry-script session:
|
||||
|
||||
| Case | Expected result |
|
||||
| --- | --- |
|
||||
| Startup returns an empty run list | Round reports continue; no beta-only POSTs. |
|
||||
| Beta opens during an ordinary match | No new token adopted; beta reporting begins after the beta-controlled process restart. |
|
||||
| Normal new match, same process | Same boot/run IDs; new session ID and sequence starting at 0. |
|
||||
| Actual unexpected process restart | New boot ID; rediscover runs; no old queued session events replayed. |
|
||||
| Retry after a lost response | Same event ID and payload; one evidence entry and a duplicate acknowledgement. |
|
||||
| Round 25 with four players, then normal match end | Round evidence can pass immediately; a qualifying-match count advances only at normal completion. |
|
||||
| Ten successful `perk.purchased` events | Ten uniquely identified feature events, counted only when each event meets the configured round/player qualifiers. |
|
||||
| Crash or idle map reset | No fabricated `match_end`. |
|
||||
| Paused, closed, or superseded run | Beta delivery suspended; Fleet remains current; never switch old queued events to a new token. |
|
||||
| Two startup run IDs | Each receives the event once, with independent acknowledgements. |
|
||||
| Reporter script reload, same game process | No new boot identity or fresh adoption of run IDs. |
|
||||
|
||||
The endpoint retains the application's existing trusted-network model. A `runId`
|
||||
correlates evidence; it is **not** an authentication secret or proof of loaded
|
||||
artifact bytes. Correct process/session attribution depends on the reporter
|
||||
following the lifecycle rules above.
|
||||
rollback. Rollback restores only files touched by the beta. Restarts can wait
|
||||
for empty servers or remain manual. Promotion is always manual.
|
||||
|
||||
Pause holds evaluation and rollout. Resume retains completed progress and the
|
||||
original deadline, but starts new timing intervals. A fresh attempt resets
|
||||
progress and restarts deployed servers when empty. Recovery cycles repeat
|
||||
gameplay milestones and checklist sign-offs while carrying health outages forward.
|
||||
|
||||
Plans, progress, and evidence persist in SQLite. The detail page shows the most
|
||||
recent 200 evidence entries. On upgrading from the old reporter protocol,
|
||||
active progress resets. Unsupported checks in existing plans and saved presets
|
||||
become manual review items; they cannot silently pass.
|
||||
|
||||
## API and storage
|
||||
|
||||
`POST /api/betas` accepts `testPlan`, validated by `parseBetaPlan` in
|
||||
`src/lib/beta-plan.ts`. New plans always verify restart recovery before testing.
|
||||
`GET /api/betas/:id` includes the plan, outcome, per-server requirement progress
|
||||
and recent evidence under `test`.
|
||||
`POST /api/betas` accepts `testPlan`.
|
||||
`GET /api/betas/:id` includes the plan, outcome, per-server progress, and
|
||||
recent evidence under `test`.
|
||||
|
||||
`POST /api/betas/:id/test` accepts actions `pause`, `resume`, `retry`, or
|
||||
`POST /api/betas/:id/test` accepts `pause`, `resume`, `retry`, or
|
||||
`sign-off` (with `server`, `requirementId`, and `note`).
|
||||
`GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are copied
|
||||
into a beta; later preset edits never change an active test.
|
||||
`GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are
|
||||
copied into a beta; later edits do not change an active plan.
|
||||
|
||||
`beta_test_plans`, `beta_test_coverage`, `beta_test_runs`, `beta_test_evidence`,
|
||||
and `beta_test_presets` hold configuration and results. `beta_file_changes`
|
||||
records which files were touched, including partially completed deployments.
|
||||
Legacy beta lifecycle states remain compatible with artifact holds and history;
|
||||
test outcomes are stored separately. Holds last until promotion or rollback.
|
||||
Internal progress records support attempts and recovery cycles. The reporter
|
||||
does not know about or supply these records. Legacy betas without a test plan
|
||||
also use `game_end` for their match-completion gate.
|
||||
|
||||
@@ -18,10 +18,7 @@ export function requirementSummary(r: BetaRequirement): string {
|
||||
case 'sustained-population': return `Keep at least ${count(r.minPlayers, 'player')} online for ${duration(r.target)}`;
|
||||
case 'healthy-minutes': return `Stay continuously healthy for ${duration(r.target)}`;
|
||||
case 'failure-budget': return r.target === 0 ? 'Allow no health outages' : `Allow at most ${count(r.target, 'health outage')}`;
|
||||
case 'restart-budget': return r.target === 0 ? 'Allow no unexpected restarts' : `Allow at most ${count(r.target, 'unexpected restart')}`;
|
||||
case 'clean-logs': return r.target === 0 ? 'Allow no detected script errors' : `Allow at most ${count(r.target, 'detected script error')}`;
|
||||
case 'recovery': return `Repeat the checks after ${count(r.target, 'recovery restart')}`;
|
||||
case 'feature': return `Observe ${r.event || 'a named feature event'} ${count(r.target, 'time')}${players}${r.minRound ? ` at round ${r.minRound} or later` : ''}`;
|
||||
case 'manual': return r.label;
|
||||
}
|
||||
}
|
||||
@@ -29,9 +26,9 @@ export function coverageSummary(plan: BetaPlan): string {
|
||||
return plan.scope === 'all' ? 'Every selected server' : plan.scope === 'quorum' ? `At least ${count(plan.quorum, 'selected server')}`
|
||||
: plan.scope === 'map' ? 'One server per selected map' : plan.scope === 'game' ? 'One server per selected game' : 'One server per game / map';
|
||||
}
|
||||
export function needsSessionTelemetry(r: BetaRequirement): boolean {
|
||||
return ['round', 'feature', 'restart-budget'].includes(r.kind) || (r.kind === 'matches' && (r.minRound > 0 || r.minPlayers > 1));
|
||||
export function needsGameTelemetry(r: BetaRequirement): boolean {
|
||||
return ['round', 'matches', 'population', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind);
|
||||
}
|
||||
export function requirementCategory(r: BetaRequirement): string {
|
||||
return r.kind === 'manual' ? 'Tester sign-off' : ['healthy-minutes', 'failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind) ? 'Continuous check' : 'Milestone';
|
||||
return r.kind === 'manual' ? 'Tester sign-off' : ['healthy-minutes', 'failure-budget'].includes(r.kind) ? 'Continuous check' : 'Milestone';
|
||||
}
|
||||
|
||||
+9
-12
@@ -1,5 +1,5 @@
|
||||
export const requirementKinds = ['round', 'matches', 'populated-minutes', 'player-hours', 'population',
|
||||
'sustained-population', 'healthy-minutes', 'failure-budget', 'restart-budget', 'clean-logs', 'recovery', 'feature', 'manual'] as const;
|
||||
'sustained-population', 'healthy-minutes', 'failure-budget', 'recovery', 'manual'] as const;
|
||||
export type RequirementKind = typeof requirementKinds[number];
|
||||
export interface BetaRequirement {
|
||||
id: string;
|
||||
@@ -9,7 +9,6 @@ export interface BetaRequirement {
|
||||
minRound: number;
|
||||
minPlayers: number;
|
||||
minMinutes: number;
|
||||
event: string;
|
||||
}
|
||||
export interface BetaPlan {
|
||||
version: 1;
|
||||
@@ -25,12 +24,11 @@ export const requirementLabels: Record<RequirementKind, string> = {
|
||||
round: 'Reach round', matches: 'Complete qualifying matches', 'populated-minutes': 'Populated minutes',
|
||||
'player-hours': 'Player-hours', population: 'Reach player count', 'sustained-population': 'Sustained population (minutes)',
|
||||
'healthy-minutes': 'Continuous healthy minutes', 'failure-budget': 'Maximum health outages',
|
||||
'restart-budget': 'Maximum unexpected restarts',
|
||||
'clean-logs': 'Maximum script errors', recovery: 'Successful recovery cycles', feature: 'Feature event count', manual: 'Tester checklist'
|
||||
recovery: 'Successful recovery cycles', manual: 'Tester checklist'
|
||||
};
|
||||
export function requirement(kind: RequirementKind, id: string): BetaRequirement {
|
||||
return { id, kind, label: requirementLabels[kind], target: kind === 'round' ? 25 : ['failure-budget', 'restart-budget', 'clean-logs'].includes(kind) ? 0 : 1,
|
||||
minRound: 0, minPlayers: 1, minMinutes: 0, event: '' };
|
||||
return { id, kind, label: requirementLabels[kind], target: kind === 'round' ? 25 : kind === 'failure-budget' ? 0 : 1,
|
||||
minRound: 0, minPlayers: 1, minMinutes: 0 };
|
||||
}
|
||||
export function defaultPlan(): BetaPlan {
|
||||
return { version: 1, requirements: [requirement('healthy-minutes', 'health')], scope: 'all', quorum: 1,
|
||||
@@ -44,7 +42,7 @@ export const builtInPlans: Array<{ name: string; plan: BetaPlan }> = [
|
||||
] } },
|
||||
{ name: 'Long soak', plan: { ...defaultPlan(), timeoutHours: 72, requirements: [
|
||||
{ ...requirement('healthy-minutes', 'health'), target: 360 },
|
||||
{ ...requirement('player-hours', 'participation'), target: 10 }, requirement('clean-logs', 'logs')
|
||||
{ ...requirement('player-hours', 'participation'), target: 10 }, requirement('failure-budget', 'outages')
|
||||
] } }
|
||||
];
|
||||
|
||||
@@ -68,13 +66,12 @@ export function parseBetaPlan(value: unknown): BetaPlan {
|
||||
for (const key of ['target', 'minRound', 'minPlayers', 'minMinutes'] as const) {
|
||||
if (typeof r[key] !== 'number' || !Number.isFinite(r[key]) || r[key] < 0 || r[key] > 1_000_000) throw new Error(`Invalid requirement ${key}`);
|
||||
}
|
||||
if (!['failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind) && r.target <= 0) throw new Error('Requirement target must be positive');
|
||||
if (['round', 'matches', 'population', 'failure-budget', 'restart-budget', 'clean-logs', 'recovery', 'feature', 'manual'].includes(r.kind) && !Number.isInteger(r.target)) throw new Error('Count targets must be integers');
|
||||
if (r.kind !== 'failure-budget' && r.target <= 0) throw new Error('Requirement target must be positive');
|
||||
if (['round', 'matches', 'population', 'failure-budget', 'recovery', 'manual'].includes(r.kind) && !Number.isInteger(r.target)) throw new Error('Count targets must be integers');
|
||||
if (!Number.isInteger(r.minRound) || !Number.isInteger(r.minPlayers)) throw new Error('Round and player qualifiers must be integers');
|
||||
if (r.kind === 'manual' && r.target !== 1) throw new Error('Add a separate requirement for each checklist item');
|
||||
if (typeof r.event !== 'string' || r.event.length > 120 || (r.kind === 'feature' && !/^[a-zA-Z0-9_.:-]+$/.test(r.event))) throw new Error('Feature requirements need an event name');
|
||||
return { id: r.id, kind: r.kind, label: r.label.trim(), target: r.target, minRound: r.minRound,
|
||||
minPlayers: r.minPlayers, minMinutes: r.minMinutes, event: r.event };
|
||||
minPlayers: r.minPlayers, minMinutes: r.minMinutes };
|
||||
});
|
||||
return { version: 1, requirements, scope: p.scope, quorum: p.quorum, timeoutHours: p.timeoutHours,
|
||||
failureAction: p.failureAction, rollbackRestart: p.rollbackRestart, waveSize: p.waveSize };
|
||||
@@ -85,6 +82,6 @@ export interface BetaTestView {
|
||||
plan: BetaPlan;
|
||||
outcome: string;
|
||||
reason: string | null;
|
||||
servers: Array<{ server: string; runId: string; phase: number; passed: boolean; progress: RequirementProgress[] }>;
|
||||
servers: Array<{ server: string; phase: number; passed: boolean; progress: RequirementProgress[] }>;
|
||||
evidence: Array<{ id: number; server: string; received_at: string; kind: string; payload: string }>;
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { parseServerName, zombieMapBySlug } from './map-aliases.js';
|
||||
import { freshStatusTimestamp } from './server-freshness.js';
|
||||
import type { ZombieGame } from './map-aliases.js';
|
||||
import type { RoundTelemetry } from './telemetry.js';
|
||||
import type { GameTelemetry } from './telemetry.js';
|
||||
import type { Health, RosterPlayer } from './ui/stores/servers.js';
|
||||
import { mapTitle, serverTitle } from './ui/server-labels.js';
|
||||
|
||||
@@ -63,8 +63,8 @@ export function fleetIdentity(server: FleetServer) {
|
||||
};
|
||||
}
|
||||
|
||||
export function currentRound(server: FleetServer, report: RoundTelemetry | undefined, now: number): number | null {
|
||||
if (!report || !freshServer(server, now) || !server.active_since || !server.game_started_at
|
||||
export function currentRound(server: FleetServer, report: GameTelemetry | undefined, now: number): number | null {
|
||||
if (!report || report.event !== 'round_change' || !freshServer(server, now) || !server.active_since || !server.game_started_at
|
||||
|| server.game_ended_at || (server.last_players ?? 0) === 0) return null;
|
||||
const received = Date.parse(report.receivedAt);
|
||||
const age = now - received;
|
||||
|
||||
@@ -3,6 +3,8 @@ import { mkdirSync } from 'node:fs';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
import { config } from '../config.js';
|
||||
import schemaSql from './schema.sql?raw';
|
||||
import { requirement, type BetaPlan, type BetaRequirement } from '../../beta-plan.js';
|
||||
import { emptyMetrics } from '../fleet/beta-evaluation.js';
|
||||
|
||||
let _db: Database.Database | null = null;
|
||||
|
||||
@@ -190,6 +192,40 @@ function assertSnapshotSchemeSafe(handle: Database.Database): void {
|
||||
handle.prepare("INSERT INTO settings (key, value) VALUES ('snapshots_v1', '1')").run();
|
||||
}
|
||||
|
||||
// Preserve old plans as reviewable checklists instead of silently passing checks
|
||||
// that the three-event protocol can no longer perform. Existing evidence stays
|
||||
// in history; active progress starts fresh under the new gameplay contract.
|
||||
function migrateGameTelemetry(handle: Database.Database): void {
|
||||
if (handle.prepare("SELECT 1 FROM settings WHERE key = 'game_telemetry_v2'").get()) return;
|
||||
handle.transaction(() => {
|
||||
const convert = (raw: string): string => {
|
||||
const plan = JSON.parse(raw) as BetaPlan;
|
||||
plan.requirements = plan.requirements.map(r => {
|
||||
const { event: _event, ...rest } = r as BetaRequirement & { event?: string };
|
||||
return ['feature', 'restart-budget', 'clean-logs'].includes(r.kind)
|
||||
? { ...requirement('manual', r.id), label: `Review: ${r.label}`.slice(0, 200) } : rest;
|
||||
});
|
||||
return JSON.stringify(plan);
|
||||
};
|
||||
const plans = handle.prepare('SELECT beta_id, plan FROM beta_test_plans').all() as Array<{ beta_id: number; plan: string }>;
|
||||
for (const row of plans) handle.prepare('UPDATE beta_test_plans SET plan = ? WHERE beta_id = ?').run(convert(row.plan), row.beta_id);
|
||||
const presets = handle.prepare('SELECT name, plan FROM beta_test_presets').all() as Array<{ name: string; plan: string }>;
|
||||
for (const row of presets) handle.prepare('UPDATE beta_test_presets SET plan = ? WHERE name = ?').run(convert(row.plan), row.name);
|
||||
handle.prepare(`UPDATE beta_test_runs SET metrics = ?, cursors = '{}' WHERE retired_at IS NULL
|
||||
AND beta_id IN (SELECT id FROM betas WHERE state IN ('running','ready-promote'))`).run(JSON.stringify(emptyMetrics()));
|
||||
handle.exec(`UPDATE beta_test_plans SET outcome = 'active', reason = 'Gameplay telemetry updated; collecting fresh script events'
|
||||
WHERE outcome IN ('active','passed') AND beta_id IN (SELECT id FROM betas WHERE state IN ('running','ready-promote'));
|
||||
UPDATE script_versions SET state = 'untested' WHERE state = 'tested'
|
||||
AND id IN (SELECT bs.script_version_id FROM beta_scripts bs JOIN beta_test_plans p ON p.beta_id = bs.beta_id WHERE p.outcome = 'active')
|
||||
AND id NOT IN (SELECT current_version_id FROM scripts WHERE current_version_id IS NOT NULL)
|
||||
AND NOT EXISTS (SELECT 1 FROM beta_scripts bs JOIN betas b ON b.id = bs.beta_id
|
||||
WHERE bs.script_version_id = script_versions.id AND b.state = 'promoted');
|
||||
UPDATE betas SET state = 'running' WHERE state = 'ready-promote' AND id IN (SELECT beta_id FROM beta_test_plans WHERE outcome = 'active');
|
||||
UPDATE beta_servers SET stage = 'observing' WHERE stage = 'ready' AND beta_id IN (SELECT beta_id FROM beta_test_plans WHERE outcome = 'active');
|
||||
INSERT INTO settings (key, value) VALUES ('game_telemetry_v2', '1');`);
|
||||
})();
|
||||
}
|
||||
|
||||
export function db(): Database.Database {
|
||||
if (_db) return _db;
|
||||
|
||||
@@ -212,6 +248,7 @@ export function db(): Database.Database {
|
||||
migrateTable(handle, 'beta_scripts', BETA_SCRIPTS_COLUMNS);
|
||||
migrateTable(handle, 'deployment_scripts', DEPLOYMENT_SCRIPTS_COLUMNS);
|
||||
migrateTable(handle, 'deployments', DEPLOYMENTS_COLUMNS);
|
||||
migrateTable(handle, 'server_telemetry', [{ name: 'payload', ddl: 'ALTER TABLE server_telemetry ADD COLUMN payload TEXT' }]);
|
||||
// Kind-referencing indexes: safe here because migrateTable above just
|
||||
// ALTER'd the column into existence if it was missing.
|
||||
handle.exec(
|
||||
@@ -221,6 +258,7 @@ export function db(): Database.Database {
|
||||
handle.exec(`CREATE INDEX IF NOT EXISTS idx_scripts_kind ON scripts(kind)`);
|
||||
backfillMultiScriptJoins(handle);
|
||||
assertSnapshotSchemeSafe(handle);
|
||||
migrateGameTelemetry(handle);
|
||||
} catch (err) {
|
||||
handle.close();
|
||||
throw err;
|
||||
|
||||
@@ -39,8 +39,8 @@ CREATE TABLE IF NOT EXISTS servers (
|
||||
-- and resume from the same anchor when the server comes back.
|
||||
active_since TEXT, -- ISO ts of last 0 -> >0 player transition
|
||||
inactive_since TEXT, -- ISO ts of last >0 -> 0 player transition
|
||||
game_started_at TEXT, -- wall-clock ISO when we first saw a J; line for the current match cycle
|
||||
game_ended_at TEXT, -- wall-clock ISO when we first saw ExitLevel/ShutdownGame end it
|
||||
game_started_at TEXT, -- receipt time of the script's game_start
|
||||
game_ended_at TEXT, -- receipt time of the script's game_end
|
||||
game_cycle_key TEXT, -- opaque marker for the currently-tracked cycle (the first J; line after the latest InitGame); lets us detect same-players game restarts
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now'))
|
||||
);
|
||||
@@ -314,7 +314,8 @@ CREATE TABLE IF NOT EXISTS server_telemetry (
|
||||
server TEXT PRIMARY KEY REFERENCES servers(name) ON DELETE CASCADE,
|
||||
round INTEGER NOT NULL,
|
||||
players INTEGER NOT NULL,
|
||||
received_at TEXT NOT NULL
|
||||
received_at TEXT NOT NULL,
|
||||
payload TEXT
|
||||
);
|
||||
-- Lifecycle history deliberately survives removal of the server registry row.
|
||||
CREATE TABLE IF NOT EXISTS server_deployments (
|
||||
|
||||
@@ -1,108 +1,83 @@
|
||||
import type { BetaPlan, RequirementProgress } from '../../beta-plan.js';
|
||||
import type { GameEvent } from '../../telemetry.js';
|
||||
|
||||
export interface SessionEvidence {
|
||||
started: number;
|
||||
sequence: number;
|
||||
export interface MatchEvidence {
|
||||
map: string;
|
||||
started?: number;
|
||||
lastEvent: number;
|
||||
players: number;
|
||||
ended: boolean;
|
||||
rounds: Record<string, number>;
|
||||
peakPlayers: number;
|
||||
}
|
||||
export interface TestMetrics {
|
||||
values: Record<string, number>;
|
||||
sessions: Record<string, SessionEvidence>;
|
||||
match?: MatchEvidence;
|
||||
manual: Record<string, string>;
|
||||
bootId?: string;
|
||||
boots: string[];
|
||||
restarts: number;
|
||||
lastSample?: number;
|
||||
wasHealthy?: boolean;
|
||||
lastPlayers?: number;
|
||||
outages: number;
|
||||
logErrors: number;
|
||||
logAvailable?: boolean;
|
||||
streaks: Record<string, number>;
|
||||
}
|
||||
export const emptyMetrics = (): TestMetrics => ({ values: {}, sessions: {}, manual: {}, boots: [], restarts: 0, outages: 0, logErrors: 0, streaks: {} });
|
||||
export const emptyMetrics = (): TestMetrics => ({ values: {}, manual: {}, outages: 0, streaks: {} });
|
||||
|
||||
export interface TestSignal {
|
||||
event: 'boot' | 'session_start' | 'round_change' | 'match_end' | 'feature';
|
||||
bootId: string;
|
||||
sessionId: string;
|
||||
sequence: number;
|
||||
round: number;
|
||||
players: number;
|
||||
name: string;
|
||||
}
|
||||
|
||||
/** Events are accepted once, in session order; a match must start in this run. */
|
||||
export function applySignal(plan: BetaPlan, metrics: TestMetrics, signal: TestSignal, now: number): boolean {
|
||||
if (signal.event === 'boot') {
|
||||
if (metrics.boots.includes(signal.bootId)) return false;
|
||||
if (metrics.bootId) metrics.restarts++;
|
||||
metrics.boots.push(signal.bootId);
|
||||
metrics.bootId = signal.bootId;
|
||||
return true;
|
||||
/** Gameplay comes only from the script. No reporter IDs or beta handshake. */
|
||||
export function applyGameEvent(plan: BetaPlan, metrics: TestMetrics, event: GameEvent, now: number): void {
|
||||
const players = event.players.length;
|
||||
if (event.event === 'game_start' || !metrics.match || metrics.match.map !== event.map) {
|
||||
metrics.match = { map: event.map, started: event.event === 'game_start' ? now : undefined,
|
||||
lastEvent: now, players, ended: false, rounds: {} };
|
||||
metrics.streaks = {};
|
||||
}
|
||||
if (metrics.bootId !== signal.bootId) return false;
|
||||
const key = `${signal.bootId}:${signal.sessionId}`;
|
||||
if (signal.event === 'session_start') {
|
||||
if (metrics.sessions[key]) return false;
|
||||
metrics.sessions[key] = { started: now, sequence: signal.sequence, ended: false, rounds: {}, peakPlayers: signal.players };
|
||||
return true;
|
||||
}
|
||||
const session = metrics.sessions[key];
|
||||
if (!session || session.ended || signal.sequence <= session.sequence) return false;
|
||||
session.sequence = signal.sequence;
|
||||
session.peakPlayers = Math.max(session.peakPlayers, signal.players);
|
||||
const match = metrics.match;
|
||||
if (match.ended) return;
|
||||
const minutes = Math.max(0, now - match.lastEvent) / 60_000;
|
||||
for (const r of plan.requirements) {
|
||||
const current = metrics.values[r.id] ?? 0;
|
||||
if (signal.event === 'round_change' && signal.players >= r.minPlayers) {
|
||||
session.rounds[r.id] = Math.max(session.rounds[r.id] ?? 0, signal.round);
|
||||
if (r.kind === 'round') metrics.values[r.id] = Math.max(current, signal.round);
|
||||
const populated = players >= Math.max(1, r.minPlayers) && match.players >= Math.max(1, r.minPlayers);
|
||||
if (r.kind === 'population') metrics.values[r.id] = Math.max(current, players);
|
||||
if (r.kind === 'populated-minutes' && populated) metrics.values[r.id] = current + minutes;
|
||||
if (r.kind === 'player-hours' && populated) metrics.values[r.id] = current + minutes * Math.min(players, match.players) / 60;
|
||||
if (r.kind === 'sustained-population') {
|
||||
metrics.streaks[r.id] = populated ? (metrics.streaks[r.id] ?? 0) + minutes : 0;
|
||||
metrics.values[r.id] = Math.max(current, metrics.streaks[r.id]);
|
||||
}
|
||||
if (signal.event === 'match_end' && r.kind === 'matches' && (r.minRound > 0 || r.minPlayers > 1)
|
||||
&& (session.rounds[r.id] ?? 0) >= r.minRound && session.peakPlayers >= r.minPlayers
|
||||
&& now - session.started >= r.minMinutes * 60_000) metrics.values[r.id] = current + 1;
|
||||
if (signal.event === 'feature' && r.kind === 'feature' && signal.name === r.event
|
||||
&& signal.players >= r.minPlayers && signal.round >= r.minRound) metrics.values[r.id] = current + 1;
|
||||
if (event.event !== 'game_start' && players >= Math.max(1, r.minPlayers)) {
|
||||
match.rounds[r.id] = Math.max(match.rounds[r.id] ?? 0, event.round);
|
||||
if (r.kind === 'round') metrics.values[r.id] = Math.max(current, event.round);
|
||||
}
|
||||
if (signal.event === 'match_end') session.ended = true;
|
||||
return true;
|
||||
if (event.event === 'game_end' && r.kind === 'matches'
|
||||
&& (match.rounds[r.id] ?? -1) >= r.minRound
|
||||
&& (r.minMinutes === 0 || (match.started !== undefined && now - match.started >= r.minMinutes * 60_000))) {
|
||||
metrics.values[r.id] = current + 1;
|
||||
}
|
||||
}
|
||||
match.lastEvent = now;
|
||||
match.players = players;
|
||||
match.ended = event.event === 'game_end';
|
||||
}
|
||||
|
||||
/** Sample both ends of an interval. Monitoring gaps never earn playtime. */
|
||||
export function applySample(plan: BetaPlan, m: TestMetrics, at: number, healthy: boolean, players: number): void {
|
||||
/** Infrastructure health is separate from script-reported gameplay. */
|
||||
export function applySample(plan: BetaPlan, m: TestMetrics, at: number, healthy: boolean): void {
|
||||
if (m.lastSample !== undefined && at <= m.lastSample) return;
|
||||
const gap = m.lastSample === undefined ? Infinity : at - m.lastSample;
|
||||
const contiguous = gap <= 60_000;
|
||||
const minutes = healthy && m.wasHealthy && contiguous ? gap / 60_000 : 0;
|
||||
if (!healthy && m.wasHealthy === true) m.outages++;
|
||||
for (const r of plan.requirements) {
|
||||
const value = m.values[r.id] ?? 0;
|
||||
const populated = players >= Math.max(1, r.minPlayers) && (m.lastPlayers ?? 0) >= Math.max(1, r.minPlayers);
|
||||
if (r.kind === 'populated-minutes' && populated) m.values[r.id] = value + minutes;
|
||||
if (r.kind === 'player-hours' && populated) m.values[r.id] = value + minutes * Math.min(players, m.lastPlayers ?? 0) / 60;
|
||||
if (r.kind === 'population' && healthy) m.values[r.id] = Math.max(value, players);
|
||||
if (r.kind === 'healthy-minutes') m.values[r.id] = healthy && contiguous && m.wasHealthy ? value + minutes : 0;
|
||||
if (r.kind === 'sustained-population') {
|
||||
m.streaks[r.id] = healthy && populated && contiguous ? (m.streaks[r.id] ?? 0) + minutes : 0;
|
||||
m.values[r.id] = Math.max(value, m.streaks[r.id]);
|
||||
if (r.kind === 'healthy-minutes') m.values[r.id] = healthy && contiguous && m.wasHealthy ? (m.values[r.id] ?? 0) + minutes : 0;
|
||||
}
|
||||
}
|
||||
m.lastSample = at; m.wasHealthy = healthy; m.lastPlayers = players;
|
||||
m.lastSample = at; m.wasHealthy = healthy;
|
||||
}
|
||||
|
||||
export function evaluateRequirements(plan: BetaPlan, metrics: TestMetrics, phase: number): RequirementProgress[] {
|
||||
return plan.requirements.map(r => {
|
||||
const budget = ['failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind);
|
||||
const current = r.kind === 'failure-budget' ? metrics.outages : r.kind === 'clean-logs' ? metrics.logErrors
|
||||
: r.kind === 'restart-budget' ? metrics.restarts
|
||||
: r.kind === 'manual' ? Number(!!metrics.manual[r.id]) : r.kind === 'recovery' ? phase : metrics.values[r.id] ?? 0;
|
||||
const available = (r.kind !== 'clean-logs' || metrics.logAvailable === true) && (r.kind !== 'restart-budget' || !!metrics.bootId);
|
||||
return { id: r.id, label: r.label, current, target: r.target, passed: available && (budget ? current <= r.target : current >= r.target),
|
||||
const budget = r.kind === 'failure-budget';
|
||||
const current = budget ? metrics.outages : r.kind === 'manual' ? Number(!!metrics.manual[r.id])
|
||||
: r.kind === 'recovery' ? phase : metrics.values[r.id] ?? 0;
|
||||
return { id: r.id, label: r.label, current, target: r.target, passed: budget ? current <= r.target : current >= r.target,
|
||||
failed: budget && current > r.target,
|
||||
detail: r.kind === 'manual' ? metrics.manual[r.id] ?? 'Awaiting tester sign-off'
|
||||
: !available ? r.kind === 'restart-budget' ? 'Waiting for boot telemetry' : 'Waiting for a readable console log' : budget ? `${current} / ${r.target} allowed; monitored until promotion`
|
||||
: budget ? `${current} / ${r.target} allowed; monitored until promotion`
|
||||
: `${Math.round(current * 100) / 100} / ${r.target}` };
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1,13 +1,11 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { open, stat } from 'node:fs/promises';
|
||||
import { db } from '../db/index.js';
|
||||
import { parseBetaPlan, type BetaPlan, type BetaTestView } from '../../beta-plan.js';
|
||||
import { applySample, applySignal, emptyMetrics, evaluateRequirements, type TestMetrics, type TestSignal } from './beta-evaluation.js';
|
||||
import { gameLogKey, resolveLogPath, type LogKey } from './logs.js';
|
||||
import type { GameTelemetry } from '../../telemetry.js';
|
||||
import { applySample, applyGameEvent, emptyMetrics, evaluateRequirements, type TestMetrics } from './beta-evaluation.js';
|
||||
|
||||
export interface TestRun { id: string; beta_id: number; server: string; phase: number; created_at: string; observing_at: string | null; metrics: string; cursors: string }
|
||||
export interface TestRun { id: string; beta_id: number; server: string; phase: number; created_at: string; observing_at: string | null; metrics: string }
|
||||
export interface StoredPlan { beta_id: number; plan: string; outcome: string; reason: string | null }
|
||||
interface Cursor { position: number; identity: string; partial: string; anchor?: string; generation?: string; session?: string; joined?: boolean; started?: number }
|
||||
const nowIso = () => new Date().toISOString();
|
||||
export const storedPlan = (id: number) => db().prepare('SELECT * FROM beta_test_plans WHERE beta_id = ?').get(id) as StoredPlan | undefined;
|
||||
export const currentRun = (id: number, server: string) => db().prepare('SELECT * FROM beta_test_runs WHERE beta_id = ? AND server = ? AND retired_at IS NULL').get(id, server) as TestRun | undefined;
|
||||
@@ -25,146 +23,47 @@ export function evidence(run: TestRun, eventId: string, kind: string, payload: u
|
||||
.run(run.id, eventId, nowIso(), kind, JSON.stringify(payload)).changes > 0;
|
||||
}
|
||||
|
||||
/** Snapshot log offsets BEFORE restart: pre-beta log entries can never qualify. */
|
||||
/** Internal progress record; never sent to or required from the game script. */
|
||||
export async function prepareRun(betaId: number, server: string, phase = 0): Promise<TestRun> {
|
||||
const cursors: Record<string, Cursor> = {};
|
||||
for (const key of [gameLogKey(server), 'console'] as LogKey[]) {
|
||||
const path = resolveLogPath(server, key);
|
||||
const info = path ? await stat(path).catch(() => null) : null;
|
||||
let anchor = '';
|
||||
if (path && info?.size) {
|
||||
const file = await open(path, 'r');
|
||||
try {
|
||||
const buffer = Buffer.alloc(Math.min(info.size, 128));
|
||||
await file.read(buffer, 0, buffer.length, info.size - buffer.length);
|
||||
anchor = buffer.toString('base64');
|
||||
} finally { await file.close(); }
|
||||
}
|
||||
cursors[key] = { position: info?.size ?? 0, identity: info ? `${info.dev}:${info.ino}` : '', partial: '', anchor, generation: randomUUID() };
|
||||
}
|
||||
const id = randomUUID();
|
||||
db().transaction(() => {
|
||||
db().prepare('UPDATE beta_test_runs SET retired_at = ? WHERE beta_id = ? AND server = ? AND retired_at IS NULL').run(nowIso(), betaId, server);
|
||||
db().prepare('INSERT INTO beta_test_runs (id, beta_id, server, phase, created_at, metrics, cursors) VALUES (?, ?, ?, ?, ?, ?, ?)')
|
||||
.run(id, betaId, server, phase, nowIso(), JSON.stringify(emptyMetrics()), JSON.stringify(cursors));
|
||||
db().prepare('INSERT INTO beta_test_runs (id, beta_id, server, phase, created_at, metrics) VALUES (?, ?, ?, ?, ?, ?)')
|
||||
.run(randomUUID(), betaId, server, phase, nowIso(), JSON.stringify(emptyMetrics()));
|
||||
})();
|
||||
return currentRun(betaId, server)!;
|
||||
}
|
||||
|
||||
export function acceptTestSignal(server: string, runId: string, eventId: string, signal: TestSignal): { accepted: boolean; reason?: string } {
|
||||
return db().transaction(() => {
|
||||
const run = db().prepare(`SELECT r.* FROM beta_test_runs r JOIN betas b ON b.id = r.beta_id
|
||||
JOIN beta_test_plans p ON p.beta_id = b.id WHERE r.id = ? AND r.server = ? AND r.retired_at IS NULL
|
||||
AND b.state IN ('running','ready-promote') AND p.outcome IN ('active','passed')`).get(runId, server) as TestRun | undefined;
|
||||
if (!run) return { accepted: false, reason: 'Run is inactive or superseded; obtain the current run at process startup' };
|
||||
if (db().prepare('SELECT 1 FROM beta_test_evidence WHERE run_id = ? AND event_id = ?').get(runId, `telemetry:${eventId}`)) return { accepted: false, reason: 'Duplicate event' };
|
||||
/** Route ordinary server events to every deployed, active beta automatically. */
|
||||
export function recordGameEvent(event: GameTelemetry): void {
|
||||
const runs = db().prepare(`SELECT r.* FROM beta_test_runs r JOIN betas b ON b.id = r.beta_id
|
||||
JOIN beta_test_plans p ON p.beta_id = b.id
|
||||
JOIN beta_servers s ON s.beta_id = b.id AND s.server = r.server
|
||||
WHERE r.server = ? AND r.retired_at IS NULL AND b.state IN ('running','ready-promote')
|
||||
AND p.outcome IN ('active','passed') AND s.stage IN ('awaiting-health','observing','ready')
|
||||
AND s.restarted_at IS NOT NULL`).all(event.server) as TestRun[];
|
||||
for (const run of runs) {
|
||||
const plan = parseBetaPlan(JSON.parse(storedPlan(run.beta_id)!.plan));
|
||||
const metrics = readMetrics(run);
|
||||
if (!applySignal(plan, metrics, signal, Date.now())) return { accepted: false, reason: 'Duplicate, out-of-order, closed, or unknown session/boot' };
|
||||
evidence(run, `telemetry:${eventId}`, signal.event, signal);
|
||||
applyGameEvent(plan, metrics, event, Date.parse(event.receivedAt));
|
||||
evidence(run, randomUUID(), event.event, event);
|
||||
saveMetrics(run, metrics);
|
||||
return { accepted: true };
|
||||
})();
|
||||
}
|
||||
|
||||
export function parseTestSignal(body: Record<string, unknown>): TestSignal {
|
||||
if (!['boot', 'session_start', 'round_change', 'match_end', 'feature'].includes(String(body.event))) throw new Error('Unknown telemetry event');
|
||||
for (const key of ['runId', 'eventId', 'bootId'] as const) {
|
||||
if (typeof body[key] !== 'string' || !/^[a-zA-Z0-9_.:-]{1,128}$/.test(body[key] as string)) throw new Error(`${key} is required (1–128 identifier characters)`);
|
||||
}
|
||||
if (body.event !== 'boot' && (typeof body.sessionId !== 'string' || !/^[a-zA-Z0-9_.:-]{1,128}$/.test(body.sessionId))) throw new Error('sessionId is required');
|
||||
for (const key of ['sequence', 'round', 'players']) if (!Number.isSafeInteger(body[key]) || (body[key] as number) < 0) throw new Error(`${key} must be a nonnegative integer`);
|
||||
if (body.event === 'feature' && (typeof body.name !== 'string' || !/^[a-zA-Z0-9_.:-]{1,120}$/.test(body.name))) throw new Error('Feature name is required');
|
||||
return { event: body.event as TestSignal['event'], bootId: body.bootId as string, sessionId: body.sessionId as string ?? '',
|
||||
sequence: body.sequence as number, round: body.round as number, players: body.players as number, name: body.name as string ?? '' };
|
||||
}
|
||||
|
||||
/** Bounded incremental reads, with persistent byte cursors and partial lines. */
|
||||
export async function observeRun(run: TestRun, plan: BetaPlan): Promise<void> {
|
||||
const cursors = JSON.parse(run.cursors) as Record<string, Cursor>;
|
||||
const observed: Array<{ key: LogKey; line: string; id: string }> = [];
|
||||
let logAvailable = false;
|
||||
for (const key of [gameLogKey(run.server), 'console'] as LogKey[]) {
|
||||
const path = resolveLogPath(run.server, key);
|
||||
const file = path ? await open(path, 'r').catch(() => null) : null;
|
||||
if (!file) continue;
|
||||
try {
|
||||
const info = await file.stat();
|
||||
if (key === 'console') logAvailable = true;
|
||||
const identity = `${info.dev}:${info.ino}`;
|
||||
let cursor = cursors[key] ?? { position: 0, identity, partial: '' };
|
||||
let anchorChanged = false;
|
||||
if (cursor.anchor && info.size >= cursor.position) {
|
||||
const expected = Buffer.from(cursor.anchor, 'base64');
|
||||
const actual = Buffer.alloc(expected.length);
|
||||
await file.read(actual, 0, actual.length, cursor.position - actual.length);
|
||||
anchorChanged = !actual.equals(expected);
|
||||
}
|
||||
if (cursor.identity !== identity || info.size < cursor.position || anchorChanged) {
|
||||
cursor = { position: 0, identity, partial: '', generation: randomUUID() };
|
||||
observed.push({ key, line: '', id: `rotation:${key}:${randomUUID()}` });
|
||||
}
|
||||
if (info.size - cursor.position > 2_000_000) {
|
||||
setOutcome(run.beta_id, 'inconclusive', `${run.server}: log reader fell behind; evidence is incomplete`);
|
||||
return;
|
||||
}
|
||||
const buffer = Buffer.alloc(info.size - cursor.position);
|
||||
const { bytesRead } = await file.read(buffer, 0, buffer.length, cursor.position);
|
||||
const combined = Buffer.concat([Buffer.from(cursor.partial, 'base64'), buffer.subarray(0, bytesRead)]);
|
||||
let start = 0;
|
||||
for (let i = 0; i < combined.length; i++) if (combined[i] === 10) {
|
||||
observed.push({ key, line: combined.subarray(start, i).toString('utf8'), id: `${key}:${identity}:${cursor.generation ?? ''}:${cursor.position + i - Buffer.from(cursor.partial, 'base64').length}` });
|
||||
start = i + 1;
|
||||
}
|
||||
cursor.position += bytesRead;
|
||||
const anchor = Buffer.alloc(Math.min(cursor.position, 128));
|
||||
await file.read(anchor, 0, anchor.length, cursor.position - anchor.length);
|
||||
cursor.anchor = anchor.toString('base64');
|
||||
cursor.partial = combined.subarray(start).toString('base64');
|
||||
if (cursor.partial.length > 100_000) cursor.partial = ''; // malformed giant line cannot grow forever
|
||||
cursor.identity = identity;
|
||||
cursors[key] = cursor;
|
||||
} finally { await file.close(); }
|
||||
}
|
||||
db().transaction(() => {
|
||||
const fresh = currentRun(run.beta_id, run.server);
|
||||
if (!fresh || fresh.id !== run.id) return;
|
||||
const m = readMetrics(fresh);
|
||||
m.logAvailable = logAvailable;
|
||||
for (const entry of observed) {
|
||||
const cursor = cursors[entry.key];
|
||||
if (!entry.line) { evidence(run, entry.id, 'log-rotation', { log: entry.key }); continue; }
|
||||
if (entry.key === 'console') {
|
||||
if (/script (?:runtime )?error|unhandled exception|script exception|fatal error/i.test(entry.line) && evidence(run, entry.id, 'script-error', { line: entry.line.slice(0, 1000) })) m.logErrors++;
|
||||
continue;
|
||||
}
|
||||
if (/InitGame:/.test(entry.line)) { cursor.session = entry.id; cursor.joined = false; cursor.started = undefined; }
|
||||
if (cursor.session && /\sJ;[^;]*;\d+;/.test(entry.line) && !cursor.joined) { cursor.joined = true; cursor.started = Date.now(); }
|
||||
if (/ExitLevel:\s*executed/.test(entry.line) && cursor.session && cursor.joined) {
|
||||
// Unqualified match counts use logs. Round/population-qualified counts use session telemetry exclusively.
|
||||
if (evidence(run, `match:${cursor.session}`, 'completed-match', { session: cursor.session, line: entry.line, minutes: (Date.now() - (cursor.started ?? Date.now())) / 60_000 })) {
|
||||
for (const r of plan.requirements) if (r.kind === 'matches' && r.minRound === 0 && r.minPlayers <= 1
|
||||
&& Date.now() - (cursor.started ?? Date.now()) >= r.minMinutes * 60_000) m.values[r.id] = (m.values[r.id] ?? 0) + 1;
|
||||
}
|
||||
cursor.joined = false;
|
||||
}
|
||||
if (/ShutdownGame:/.test(entry.line)) cursor.joined = false;
|
||||
}
|
||||
const server = db().prepare('SELECT last_checked_at, last_seen_at, last_players, last_error, container_state FROM servers WHERE name = ?').get(run.server) as {
|
||||
last_checked_at: string | null; last_seen_at: string | null; last_players: number | null; last_error: string | null; container_state: string | null };
|
||||
const m = readMetrics(run);
|
||||
const server = db().prepare('SELECT last_checked_at, last_seen_at, last_error, container_state FROM servers WHERE name = ?').get(run.server) as {
|
||||
last_checked_at: string | null; last_seen_at: string | null; last_error: string | null; container_state: string | null };
|
||||
const sample = server.last_checked_at ? Date.parse(server.last_checked_at) : 0;
|
||||
const freshSample = sample >= Date.parse(run.observing_at ?? run.created_at) && Date.now() - sample <= 60_000;
|
||||
const healthy = freshSample && !server.last_error && server.container_state === 'running' && server.last_seen_at === server.last_checked_at;
|
||||
const at = freshSample && server.container_state === 'running' ? sample : Date.now();
|
||||
// Startup requires fresh status; don't count an absent first observation as an outage.
|
||||
const before = m.wasHealthy;
|
||||
const lastSample = m.lastSample;
|
||||
applySample(plan, m, at, healthy, healthy ? server.last_players ?? 0 : 0);
|
||||
if (freshSample && sample !== lastSample) evidence(run, `sample:${sample}`, 'status-sample', { healthy, players: server.last_players, checkedAt: server.last_checked_at });
|
||||
if (before !== m.wasHealthy) evidence(run, `health:${at}`, 'health', { healthy, players: server.last_players, reason: server.last_error });
|
||||
applySample(plan, m, at, healthy);
|
||||
if (freshSample && sample !== lastSample) evidence(run, `sample:${sample}`, 'status-sample', { healthy, checkedAt: server.last_checked_at });
|
||||
if (before !== m.wasHealthy) evidence(run, `health:${at}`, 'health', { healthy, reason: server.last_error });
|
||||
saveMetrics(run, m);
|
||||
db().prepare('UPDATE beta_test_runs SET cursors = ? WHERE id = ?').run(JSON.stringify(cursors), run.id);
|
||||
})();
|
||||
}
|
||||
|
||||
export function testView(betaId: number): BetaTestView | null {
|
||||
@@ -175,7 +74,7 @@ export function testView(betaId: number): BetaTestView | null {
|
||||
return { plan, outcome: stored.outcome, reason: stored.reason,
|
||||
servers: runs.map(run => {
|
||||
const progress = evaluateRequirements(plan, readMetrics(run), run.phase);
|
||||
return { server: run.server, runId: run.id, phase: run.phase, passed: progress.every(r => r.passed), progress };
|
||||
return { server: run.server, phase: run.phase, passed: progress.every(r => r.passed), progress };
|
||||
}),
|
||||
evidence: db().prepare(`SELECT e.id, r.server, e.received_at, e.kind, e.payload FROM beta_test_evidence e
|
||||
JOIN beta_test_runs r ON r.id = e.run_id WHERE r.beta_id = ? ORDER BY e.id DESC LIMIT 200`).all(betaId) as BetaTestView['evidence'] };
|
||||
|
||||
@@ -1,110 +0,0 @@
|
||||
import { resolveLogPath, tailLog, gameLogKey } from './logs.js';
|
||||
|
||||
// Match markers for T5 Plutonium's `logs/games_sp.log`. Verified against a live
|
||||
// sample: each game session is bracketed by `InitGame:` and `ShutdownGame:`.
|
||||
// A REAL completed match always emits `ExitLevel: executed` right before
|
||||
// `ShutdownGame:`. Idle map cycles print `InitGame:` / `ShutdownGame:` at
|
||||
// 0:00 with no ExitLevel — those don't count as a played match.
|
||||
//
|
||||
// So the reliable discriminator is `ExitLevel: executed`: seeing that line
|
||||
// after `healthy_at` means a game actually ended normally (no crash — a crash
|
||||
// would kill the container before ExitLevel fires and take RCON with it).
|
||||
export const MATCH_START = /InitGame:/;
|
||||
export const MATCH_END = /ExitLevel:\s*executed/;
|
||||
export const MATCH_SHUTDOWN = /ShutdownGame:/;
|
||||
// Player join line: `MM:SS J;<guid>;<slot>;<name>`. games_sp.log accumulates
|
||||
// across restarts and contains many old J; lines; only the ones AFTER the
|
||||
// latest InitGame belong to the current match cycle.
|
||||
export const MATCH_JOIN = /\sJ;[^;]*;\d+;/;
|
||||
|
||||
/**
|
||||
* Poll a server's games_sp.log tail for match markers. Returns whether a
|
||||
* completed match end has been observed since `sinceIso` (the server's
|
||||
* `healthy_at` timestamp).
|
||||
*/
|
||||
export async function scanForMatchEnd(
|
||||
serverName: string,
|
||||
sinceIso: string
|
||||
): Promise<{ ended: boolean; started: boolean; reason?: string }> {
|
||||
const path = resolveLogPath(serverName, gameLogKey(serverName));
|
||||
if (!path) return { ended: false, started: false, reason: 'log path resolution failed' };
|
||||
|
||||
const tail = await tailLog(path, 2000);
|
||||
if (!tail) return { ended: false, started: false, reason: 'log not written yet' };
|
||||
if (tail.mtime < sinceIso) return { ended: false, started: false, reason: 'no log activity since healthy' };
|
||||
|
||||
let started = false;
|
||||
let ended = false;
|
||||
for (const line of tail.lines) {
|
||||
if (MATCH_START.test(line)) started = true;
|
||||
if (MATCH_END.test(line)) ended = true;
|
||||
}
|
||||
return { ended, started };
|
||||
}
|
||||
|
||||
export type MatchScanKind = 'no-signal' | 'idle' | 'in-progress' | 'ended';
|
||||
|
||||
export interface MatchScanResult {
|
||||
kind: MatchScanKind;
|
||||
// Opaque identifier for the currently-observed match cycle. When kind is
|
||||
// 'in-progress' or 'ended', this is the first J; line after the latest
|
||||
// InitGame. Compare against persisted state to detect a fresh cycle
|
||||
// (e.g. game auto-restart with same players — new J; line = new key).
|
||||
cycleKey?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the games_sp.log tail and classify the current match state. We do NOT
|
||||
* infer any wall-clock timestamps from log content — the caller stamps
|
||||
* game_started_at/game_ended_at with the observation time.
|
||||
*
|
||||
* - `in-progress`: latest InitGame has at least one J; join line after it
|
||||
* and no ExitLevel/ShutdownGame between that J; and end-of-tail. `cycleKey`
|
||||
* is set to the first J; line after the latest InitGame.
|
||||
* - `ended`: latest InitGame has a J; join line followed by an ExitLevel
|
||||
* or ShutdownGame. `cycleKey` identifies which cycle ended.
|
||||
* - `idle`: no J; line after the latest InitGame — empty map loaded, no
|
||||
* match happening.
|
||||
* - `no-signal`: log missing / unreadable / no InitGame in tail.
|
||||
*/
|
||||
export async function scanMatchState(serverName: string): Promise<MatchScanResult> {
|
||||
const path = resolveLogPath(serverName, gameLogKey(serverName));
|
||||
if (!path) return { kind: 'no-signal' };
|
||||
|
||||
const tail = await tailLog(path, 2000);
|
||||
if (!tail || tail.lines.length === 0) return { kind: 'no-signal' };
|
||||
|
||||
const lines = tail.lines;
|
||||
// Find the LATEST InitGame — everything before it is prior sessions that
|
||||
// this log accumulated. Only what comes after is the current cycle.
|
||||
let lastInit = -1;
|
||||
for (let i = lines.length - 1; i >= 0; i--) {
|
||||
if (MATCH_START.test(lines[i])) {
|
||||
lastInit = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (lastInit === -1) return { kind: 'no-signal' };
|
||||
|
||||
// Scan only the current cycle (everything after the latest InitGame).
|
||||
let firstJoinIdx = -1;
|
||||
let endIdx = -1;
|
||||
for (let i = lastInit + 1; i < lines.length; i++) {
|
||||
if (firstJoinIdx === -1 && MATCH_JOIN.test(lines[i])) {
|
||||
firstJoinIdx = i;
|
||||
}
|
||||
if (MATCH_END.test(lines[i]) || MATCH_SHUTDOWN.test(lines[i])) {
|
||||
endIdx = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// No join in the current cycle -> nobody entered this match yet.
|
||||
if (firstJoinIdx === -1) return { kind: 'idle' };
|
||||
|
||||
const cycleKey = lines[firstJoinIdx];
|
||||
if (endIdx !== -1 && endIdx > firstJoinIdx) {
|
||||
return { kind: 'ended', cycleKey };
|
||||
}
|
||||
return { kind: 'in-progress', cycleKey };
|
||||
}
|
||||
@@ -1,15 +1,24 @@
|
||||
import { db } from '../db/index.js';
|
||||
import type { RoundTelemetry } from '../../telemetry.js';
|
||||
import type { GameTelemetry } from '../../telemetry.js';
|
||||
|
||||
export function listTelemetry(): RoundTelemetry[] {
|
||||
return db().prepare(`SELECT t.server, t.round, t.players, t.received_at AS receivedAt
|
||||
FROM server_telemetry t JOIN servers s ON s.name = t.server
|
||||
WHERE s.destroyed_at IS NULL`).all() as RoundTelemetry[];
|
||||
export function listTelemetry(): GameTelemetry[] {
|
||||
const rows = db().prepare(`SELECT t.payload FROM server_telemetry t JOIN servers s ON s.name = t.server
|
||||
WHERE s.destroyed_at IS NULL AND t.payload IS NOT NULL`).all() as Array<{ payload: string }>;
|
||||
return rows.map(row => JSON.parse(row.payload) as GameTelemetry);
|
||||
}
|
||||
|
||||
export function saveTelemetry(report: RoundTelemetry): void {
|
||||
db().prepare(`INSERT INTO server_telemetry (server, round, players, received_at)
|
||||
VALUES (?, ?, ?, ?) ON CONFLICT(server) DO UPDATE SET
|
||||
round = excluded.round, players = excluded.players, received_at = excluded.received_at`)
|
||||
.run(report.server, report.round, report.players, report.receivedAt);
|
||||
export function saveTelemetry(report: GameTelemetry): void {
|
||||
db().prepare(`INSERT INTO server_telemetry (server, round, players, received_at, payload)
|
||||
VALUES (?, ?, ?, ?, ?) ON CONFLICT(server) DO UPDATE SET
|
||||
round = excluded.round, players = excluded.players, received_at = excluded.received_at, payload = excluded.payload`)
|
||||
.run(report.server, report.event === 'game_start' ? 0 : report.round, report.players.length, report.receivedAt, JSON.stringify(report));
|
||||
if (report.event === 'game_start') {
|
||||
db().prepare('UPDATE servers SET game_started_at = ?, game_ended_at = NULL WHERE name = ?').run(report.receivedAt, report.server);
|
||||
} else if (report.event === 'game_end') {
|
||||
db().prepare('UPDATE servers SET game_ended_at = ? WHERE name = ?').run(report.receivedAt, report.server);
|
||||
// Legacy betas also take match completion directly from the script.
|
||||
db().prepare(`UPDATE beta_servers SET match_seen_at = ? WHERE server = ? AND stage = 'awaiting-match'
|
||||
AND healthy_at <= ? AND beta_id IN (SELECT id FROM betas WHERE state = 'running')`)
|
||||
.run(report.receivedAt, report.server, report.receivedAt);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -112,7 +112,7 @@ async function advancePlan(stored: StoredPlan): Promise<void> {
|
||||
if (oldRun) {
|
||||
const previous = readMetrics(oldRun);
|
||||
const metrics = readMetrics(run);
|
||||
metrics.outages = previous.outages; metrics.logErrors = previous.logErrors; metrics.restarts = previous.restarts;
|
||||
metrics.outages = previous.outages;
|
||||
saveMetrics(run, metrics);
|
||||
}
|
||||
const restart = await restartInRollout(server.server);
|
||||
|
||||
@@ -3,7 +3,6 @@ import { emit } from './bus.js';
|
||||
import type { BetaStage } from './bus.js';
|
||||
import { komodoConfigured, rconConfigured } from '../config.js';
|
||||
import { rconGetPlayerCount } from '../integrations/rcon.js';
|
||||
import { scanForMatchEnd } from '../fleet/match-log.js';
|
||||
import { refreshRestartRollout, restartInRollout, rolloutHealth, rolloutWaitReason } from './restart-rollout.js';
|
||||
import { plannedBetaTick } from './beta-plan-runner.js';
|
||||
|
||||
@@ -12,6 +11,7 @@ interface BetaServerActiveRow {
|
||||
server: string;
|
||||
stage: BetaStage;
|
||||
healthy_at: string | null;
|
||||
match_seen_at: string | null;
|
||||
wait_for_health: number;
|
||||
wait_for_match: number;
|
||||
}
|
||||
@@ -101,7 +101,7 @@ async function runBetaTick(): Promise<{ processed: number }> {
|
||||
const rows = db()
|
||||
.prepare(
|
||||
`SELECT bs.beta_id AS beta_id, bs.server AS server, bs.stage AS stage,
|
||||
bs.healthy_at AS healthy_at,
|
||||
bs.healthy_at AS healthy_at, bs.match_seen_at AS match_seen_at,
|
||||
b.wait_for_health AS wait_for_health, b.wait_for_match AS wait_for_match
|
||||
FROM beta_servers bs
|
||||
JOIN betas b ON b.id = bs.beta_id
|
||||
@@ -181,9 +181,7 @@ async function advance(row: BetaServerActiveRow): Promise<void> {
|
||||
return;
|
||||
}
|
||||
case 'awaiting-match': {
|
||||
const since = row.healthy_at ?? nowIso();
|
||||
const result = await scanForMatchEnd(row.server, since);
|
||||
if (!result.ended) return;
|
||||
if (!row.match_seen_at) return;
|
||||
transition(row.beta_id, row.server, 'ready');
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { EventEmitter } from 'node:events';
|
||||
import type { RoundTelemetry } from '../../telemetry.js';
|
||||
import type { GameTelemetry } from '../../telemetry.js';
|
||||
import type { RconEndpoint, RconFailure, RconPlayer } from '../integrations/rcon.js';
|
||||
import type { Health } from '../fleet/health.js';
|
||||
|
||||
@@ -40,7 +40,7 @@ export type BetaStage =
|
||||
export type BetaState = 'running' | 'ready-promote' | 'promoted' | 'rolled-back';
|
||||
|
||||
export type PipelineEvent =
|
||||
| { type: 'server-telemetry'; entries: RoundTelemetry[] }
|
||||
| { type: 'server-telemetry'; entries: GameTelemetry[] }
|
||||
| { type: 'deployment-opened'; deploymentId: string; action: string; servers: string[] }
|
||||
| { type: 'server-changed'; deploymentId: string; server: string; files: number }
|
||||
| {
|
||||
|
||||
@@ -56,7 +56,8 @@ export async function restartInRollout(server: string): Promise<RolloutRestartRe
|
||||
if (waitReason) return { kind: 'queued', message: waitReason };
|
||||
|
||||
requestsInFlight.add(server);
|
||||
db().prepare("UPDATE servers SET container_state = 'restarting', last_error = NULL, last_reply_snippet = NULL WHERE name = ?").run(server);
|
||||
db().prepare(`UPDATE servers SET container_state = 'restarting', last_error = NULL, last_reply_snippet = NULL,
|
||||
game_started_at = NULL, game_ended_at = NULL WHERE name = ?`).run(server);
|
||||
try {
|
||||
const result = await komodoRestart(server);
|
||||
if (result.ok) return { kind: 'started' };
|
||||
|
||||
@@ -10,7 +10,6 @@ import { komodoInspectHealthy } from '../integrations/komodo.js';
|
||||
import { restartSettling } from './restart-rollout.js';
|
||||
import { classifyHealth } from '../fleet/health.js';
|
||||
import { STATUS_REFRESH_MS } from '../../server-freshness.js';
|
||||
import { scanMatchState } from '../fleet/match-log.js';
|
||||
|
||||
const TICK_MS = 15_000;
|
||||
// Cap concurrent UDP RCON calls so a 15-server fleet doesn't blast the
|
||||
@@ -44,61 +43,6 @@ async function runTick(): Promise<void> {
|
||||
} catch (err) {
|
||||
console.error('[worker] beta-runner error:', err);
|
||||
}
|
||||
try {
|
||||
await scanMatches();
|
||||
} catch (err) {
|
||||
console.error('[worker] match-scan error:', err);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read every server's games_sp.log tail to advance game_started_at /
|
||||
* game_ended_at. Concurrency-capped so 15 servers don't hammer disk at once.
|
||||
*/
|
||||
async function scanMatches(): Promise<void> {
|
||||
const servers = listServers();
|
||||
if (servers.length === 0) return;
|
||||
|
||||
let cursor = 0;
|
||||
async function pump(): Promise<void> {
|
||||
while (true) {
|
||||
const i = cursor++;
|
||||
if (i >= servers.length) return;
|
||||
const srv = servers[i];
|
||||
const result = await scanMatchState(srv.name).catch((err) => {
|
||||
console.error(`[worker] scanMatchState(${srv.name}) failed:`, err);
|
||||
return { kind: 'no-signal' as const };
|
||||
});
|
||||
const persistedInProgress = !!srv.game_started_at && !srv.game_ended_at;
|
||||
const persistedKey = srv.game_cycle_key;
|
||||
const now = new Date().toISOString();
|
||||
|
||||
if (result.kind === 'in-progress' && result.cycleKey) {
|
||||
// New cycle when persisted key differs (fresh start, or a game
|
||||
// restart with the same players — new J; line = new key).
|
||||
if (result.cycleKey !== persistedKey) {
|
||||
setServerField(srv.name, 'game_started_at', now);
|
||||
setServerField(srv.name, 'game_ended_at', null);
|
||||
setServerField(srv.name, 'game_cycle_key', result.cycleKey);
|
||||
} else if (!persistedInProgress) {
|
||||
// Same key but previous cycle was ended — treat as new run
|
||||
// (shouldn't happen in practice since key would differ,
|
||||
// but be defensive).
|
||||
setServerField(srv.name, 'game_started_at', now);
|
||||
setServerField(srv.name, 'game_ended_at', null);
|
||||
}
|
||||
} else if (result.kind === 'ended' && result.cycleKey) {
|
||||
// Only stamp the end if it's the cycle we've been tracking and
|
||||
// we haven't already stamped it.
|
||||
if (result.cycleKey === persistedKey && persistedInProgress) {
|
||||
setServerField(srv.name, 'game_ended_at', now);
|
||||
}
|
||||
}
|
||||
// 'idle' / 'no-signal': leave state alone. UI hides the timer when
|
||||
// the server has no players (see ServerCard/detail page).
|
||||
}
|
||||
}
|
||||
await Promise.all(Array.from({ length: STATUS_CONCURRENCY }, pump));
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -169,8 +113,8 @@ async function refreshStatus(): Promise<void> {
|
||||
const nextPlayers = result.status.players;
|
||||
let activeSince = srv.active_since;
|
||||
let inactiveSince = srv.inactive_since;
|
||||
let gameStartedAt = srv.game_started_at;
|
||||
let gameEndedAt = srv.game_ended_at;
|
||||
const gameStartedAt = srv.game_started_at;
|
||||
const gameEndedAt = srv.game_ended_at;
|
||||
if (nextPlayers > 0 && (activeSince === null || prevPlayers === 0 || prevPlayers === null)) {
|
||||
activeSince = now;
|
||||
inactiveSince = null;
|
||||
@@ -181,14 +125,6 @@ async function refreshStatus(): Promise<void> {
|
||||
activeSince = null;
|
||||
setServerField(srv.name, 'inactive_since', inactiveSince);
|
||||
setServerField(srv.name, 'active_since', null);
|
||||
// Game timer cannot run while inactive. Drop any tracked
|
||||
// match state so the next active session anchors fresh
|
||||
// when a new J; line fires.
|
||||
gameStartedAt = null;
|
||||
gameEndedAt = null;
|
||||
setServerField(srv.name, 'game_started_at', null);
|
||||
setServerField(srv.name, 'game_ended_at', null);
|
||||
setServerField(srv.name, 'game_cycle_key', null);
|
||||
}
|
||||
// Persist last-observed state. last_seen_at is the health anchor.
|
||||
setServerField(srv.name, 'last_players', nextPlayers);
|
||||
|
||||
+48
-5
@@ -1,6 +1,49 @@
|
||||
export interface RoundTelemetry {
|
||||
server: string;
|
||||
round: number;
|
||||
players: number;
|
||||
receivedAt: string;
|
||||
export interface TelemetryPlayer {
|
||||
name: string;
|
||||
guid: string;
|
||||
}
|
||||
|
||||
export interface TelemetryPlayerStats extends TelemetryPlayer {
|
||||
kills: number;
|
||||
headshots: number;
|
||||
downs: number;
|
||||
revives: number;
|
||||
score: number;
|
||||
}
|
||||
|
||||
export type GameEvent = { server: string; map: string } & (
|
||||
| { event: 'game_start'; players: TelemetryPlayer[] }
|
||||
| { event: 'round_change' | 'game_end'; round: number; players: TelemetryPlayerStats[] }
|
||||
);
|
||||
|
||||
export type GameTelemetry = GameEvent & { receivedAt: string };
|
||||
|
||||
/** Validate the three script payloads; beta state is never part of the protocol. */
|
||||
export function parseGameEvent(value: unknown): GameEvent {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error('Payload must be a JSON object');
|
||||
const body = value as Record<string, unknown>;
|
||||
if (!['game_start', 'round_change', 'game_end'].includes(String(body.event))) throw new Error('event must be game_start, round_change, or game_end');
|
||||
for (const key of ['server', 'map'] as const) {
|
||||
if (typeof body[key] !== 'string' || !body[key].trim() || body[key].length > 255) throw new Error(`${key} must be a nonempty string (up to 255 characters)`);
|
||||
}
|
||||
if (!Array.isArray(body.players)) throw new Error('players must be an array');
|
||||
const players = body.players.map((value, index) => {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error(`players[${index}] must be an object`);
|
||||
const player = value as Record<string, unknown>;
|
||||
for (const key of ['name', 'guid'] as const) {
|
||||
if (typeof player[key] !== 'string' || !player[key].trim()) throw new Error(`players[${index}].${key} must be a nonempty string`);
|
||||
}
|
||||
const identity = { name: player.name as string, guid: player.guid as string };
|
||||
if (body.event === 'game_start') return identity;
|
||||
const stats = {} as Omit<TelemetryPlayerStats, keyof TelemetryPlayer>;
|
||||
for (const key of ['kills', 'headshots', 'downs', 'revives', 'score'] as const) {
|
||||
if (!Number.isSafeInteger(player[key]) || (player[key] as number) < 0) throw new Error(`players[${index}].${key} must be a nonnegative integer`);
|
||||
stats[key] = player[key] as number;
|
||||
}
|
||||
return { ...identity, ...stats };
|
||||
});
|
||||
const common = { server: body.server as string, map: body.map as string };
|
||||
if (body.event === 'game_start') return { ...common, event: 'game_start', players };
|
||||
if (!Number.isSafeInteger(body.round) || (body.round as number) < 0) throw new Error('round must be a nonnegative integer');
|
||||
return { ...common, event: body.event as 'round_change' | 'game_end', round: body.round as number, players: players as TelemetryPlayerStats[] };
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
import { onMount } from 'svelte';
|
||||
import Button from '$ui/components/primitives/Button.svelte';
|
||||
import { builtInPlans, requirement, requirementKinds, requirementLabels, parseBetaPlan, type BetaPlan, type RequirementKind } from '$lib/beta-plan.js';
|
||||
import { coverageSummary, needsSessionTelemetry, presetDescriptions, requirementCategory, requirementSummary } from '$lib/beta-plan-summary.js';
|
||||
import { coverageSummary, needsGameTelemetry, presetDescriptions, requirementCategory, requirementSummary } from '$lib/beta-plan-summary.js';
|
||||
let { plan = $bindable<BetaPlan>() }: { plan: BetaPlan } = $props();
|
||||
let presets = $state(builtInPlans);
|
||||
let presetName = $state('');
|
||||
@@ -102,8 +102,8 @@
|
||||
<div><dt>If a check fails</dt><dd>{failureSummary}</dd></div>
|
||||
</dl>
|
||||
<p class="plan-caption">Every requirement below must pass on each qualifying server. You can edit any check.</p>
|
||||
{#if plan.requirements.some(needsSessionTelemetry)}
|
||||
<p class="telemetry-note"><span aria-hidden="true">ⓘ</span> This plan needs session telemetry from the game reporter.</p>
|
||||
{#if plan.requirements.some(needsGameTelemetry)}
|
||||
<p class="telemetry-note"><span aria-hidden="true">ⓘ</span> Gameplay progress comes from game start, round change, and game end reports.</p>
|
||||
{/if}
|
||||
|
||||
<div class="requirements">
|
||||
@@ -120,31 +120,26 @@
|
||||
<div class="fields">
|
||||
<label>Description<input bind:value={r.label} maxlength="200" required /></label>
|
||||
{#if r.kind !== 'manual'}
|
||||
<label>{['failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind) ? 'Maximum allowed' : 'Target'}
|
||||
<input type="number" min={['failure-budget', 'restart-budget', 'clean-logs'].includes(r.kind) ? 0 : ['healthy-minutes', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind) ? 0.01 : 1} step={['healthy-minutes', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind) ? 'any' : 1} bind:value={r.target} required />
|
||||
<label>{r.kind === 'failure-budget' ? 'Maximum allowed' : 'Target'}
|
||||
<input type="number" min={r.kind === 'failure-budget' ? 0 : ['healthy-minutes', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind) ? 0.01 : 1} step={['healthy-minutes', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind) ? 'any' : 1} bind:value={r.target} required />
|
||||
</label>
|
||||
{/if}
|
||||
{#if ['round', 'matches', 'populated-minutes', 'player-hours', 'sustained-population', 'feature'].includes(r.kind)}
|
||||
{#if ['round', 'matches', 'populated-minutes', 'player-hours', 'sustained-population'].includes(r.kind)}
|
||||
<label>Minimum players<input type="number" min="1" step="1" bind:value={r.minPlayers} required /></label>
|
||||
{/if}
|
||||
{#if r.kind === 'matches' || r.kind === 'feature'}
|
||||
<label>Minimum round in the same session<input type="number" min="0" step="1" bind:value={r.minRound} required /></label>
|
||||
{#if r.kind === 'matches'}
|
||||
<label>Minimum round in the same game<input type="number" min="0" step="1" bind:value={r.minRound} required /></label>
|
||||
{/if}
|
||||
{#if r.kind === 'matches'}
|
||||
<label>Minimum match duration (minutes)<input type="number" min="0" step="any" bind:value={r.minMinutes} required /></label>
|
||||
{/if}
|
||||
{#if r.kind === 'feature'}
|
||||
<label>Reported event name<input bind:value={r.event} placeholder="perk.purchased" maxlength="120" required /></label>
|
||||
{/if}
|
||||
</div>
|
||||
{#if r.kind === 'round' || r.kind === 'feature' || r.kind === 'restart-budget' || (r.kind === 'matches' && (r.minRound > 0 || r.minPlayers > 1))}
|
||||
<p>Requires session telemetry from the game reporter. Progress waits for events from this beta’s current run.</p>
|
||||
{#if needsGameTelemetry(r)}
|
||||
<p>Uses game events reported by this server automatically.</p>
|
||||
{:else if r.kind === 'recovery'}
|
||||
<p>After the other requirements pass, restart when empty and repeat them. Every cycle requires fresh evidence and checklist sign-off.</p>
|
||||
{:else if r.kind === 'failure-budget'}
|
||||
<p>Counts transitions from healthy to unhealthy. Missing or stale monitoring resets continuous health and does not earn playtime.</p>
|
||||
{:else if r.kind === 'clean-logs'}
|
||||
<p>Checks new console lines for script errors, script exceptions, unhandled exceptions and fatal errors. An unreadable log cannot pass.</p>
|
||||
<p>Counts transitions from healthy to unhealthy. Missing or stale monitoring resets continuous healthy time.</p>
|
||||
{:else if r.kind === 'manual'}
|
||||
<p>A tester records their name and observation on the beta detail page.</p>
|
||||
{/if}
|
||||
|
||||
@@ -44,10 +44,10 @@
|
||||
{@const requirement = test.plan.requirements.find(r => r.id === progress.id)!}
|
||||
<div class="requirement" class:failed={progress.failed}>
|
||||
<div><strong>{progress.passed ? '✓' : progress.failed ? '✕' : '○'} {progress.label}</strong><span>{progress.detail}</span></div>
|
||||
{#if ['round', 'matches', 'feature', 'populated-minutes', 'player-hours', 'sustained-population'].includes(requirement.kind)}
|
||||
<span>Minimum {requirement.minPlayers} player(s){requirement.minRound ? `, round ${requirement.minRound} in the same session` : ''}{requirement.minMinutes ? `, ${requirement.minMinutes} minutes per match` : ''}{requirement.event ? ` · Event: ${requirement.event}` : ''}</span>
|
||||
{#if ['round', 'matches', 'populated-minutes', 'player-hours', 'sustained-population'].includes(requirement.kind)}
|
||||
<span>Minimum {requirement.minPlayers} player(s){requirement.minRound ? `, round ${requirement.minRound} in the same game` : ''}{requirement.minMinutes ? `, ${requirement.minMinutes} minutes per match` : ''}</span>
|
||||
{/if}
|
||||
{#if !['manual', 'failure-budget', 'restart-budget', 'clean-logs'].includes(requirement.kind)}
|
||||
{#if !['manual', 'failure-budget'].includes(requirement.kind)}
|
||||
<progress aria-label={progress.label} value={Math.min(progress.current, progress.target)} max={progress.target}></progress>
|
||||
{/if}
|
||||
{#if requirement.kind === 'manual' && !progress.passed && !closed}
|
||||
|
||||
@@ -8,10 +8,10 @@
|
||||
&& (!since || Date.parse(report.receivedAt) >= Date.parse(since)));
|
||||
</script>
|
||||
|
||||
{#if report && fresh}
|
||||
{#if report && report.event === 'round_change' && fresh}
|
||||
<div class="round" aria-live="polite">
|
||||
<strong>Round {report.round}</strong>
|
||||
<span title={report.receivedAt}>Last reported {elapsedSince(report.receivedAt, $uiNow)} ago · {report.players} players</span>
|
||||
<span title={report.receivedAt}>Last reported {elapsedSince(report.receivedAt, $uiNow)} ago · {report.players.length} players</span>
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
import { writable, readonly } from 'svelte/store';
|
||||
import { browser } from '$app/environment';
|
||||
import type { RoundTelemetry } from '$lib/telemetry.js';
|
||||
import type { GameTelemetry } from '$lib/telemetry.js';
|
||||
|
||||
export const telemetryByServer = writable<Map<string, RoundTelemetry>>(new Map());
|
||||
export const telemetryByServer = writable<Map<string, GameTelemetry>>(new Map());
|
||||
|
||||
export interface RosterPlayer {
|
||||
num: number | null;
|
||||
@@ -54,9 +54,9 @@ export function startServerStream(): (() => void) | undefined {
|
||||
if (!browser || source) return;
|
||||
source = new EventSource('/api/events/servers');
|
||||
source.addEventListener('server-telemetry', (evt) => {
|
||||
const data = JSON.parse((evt as MessageEvent).data) as { entries: RoundTelemetry[]; snapshot?: boolean };
|
||||
const data = JSON.parse((evt as MessageEvent).data) as { entries: GameTelemetry[]; snapshot?: boolean };
|
||||
telemetryByServer.update((previous) => {
|
||||
const next = data.snapshot ? new Map<string, RoundTelemetry>() : new Map(previous);
|
||||
const next = data.snapshot ? new Map<string, GameTelemetry>() : new Map(previous);
|
||||
for (const entry of data.entries) next.set(entry.server, entry);
|
||||
return next;
|
||||
});
|
||||
|
||||
@@ -33,6 +33,16 @@ export async function POST({ params, request }: { params: { id: string }; reques
|
||||
betaState(id, false);
|
||||
} else if (body.action === 'resume') {
|
||||
if (stored.outcome !== 'paused') throw new Error('Only paused plans can resume; failed or expired plans need a fresh attempt');
|
||||
const runs = db().prepare('SELECT * FROM beta_test_runs WHERE beta_id = ? AND retired_at IS NULL').all(id) as Parameters<typeof readMetrics>[0][];
|
||||
for (const run of runs) {
|
||||
const metrics = readMetrics(run);
|
||||
delete metrics.match;
|
||||
delete metrics.lastSample;
|
||||
delete metrics.wasHealthy;
|
||||
metrics.streaks = {};
|
||||
for (const r of plan.requirements) if (r.kind === 'healthy-minutes') metrics.values[r.id] = 0;
|
||||
saveMetrics(run, metrics);
|
||||
}
|
||||
setOutcome(id, 'active', null);
|
||||
} else if (body.action === 'retry') {
|
||||
// Fresh attempts keep their predecessors as audit history. Preserve the original deadline:
|
||||
|
||||
@@ -1,88 +1,27 @@
|
||||
import { json } from '@sveltejs/kit';
|
||||
import { getServer, listServers } from '$lib/server/fleet/servers.js';
|
||||
import { parseGameEvent, type GameTelemetry } from '$lib/telemetry.js';
|
||||
import { getServer } from '$lib/server/fleet/servers.js';
|
||||
import { saveTelemetry } from '$lib/server/fleet/telemetry.js';
|
||||
import { emit } from '$lib/server/pipeline/bus.js';
|
||||
import { recordGameEvent } from '$lib/server/fleet/beta-tests.js';
|
||||
import { db } from '$lib/server/db/index.js';
|
||||
import { acceptTestSignal, parseTestSignal } from '$lib/server/fleet/beta-tests.js';
|
||||
|
||||
// Reporters fetch this once at process startup and keep the token for that boot.
|
||||
export async function GET({ url }: { url: URL }) {
|
||||
const server = url.searchParams.get('server');
|
||||
if (!server || !getServer(server)) return json({ error: 'Unknown server' }, { status: 404 });
|
||||
const runs = db().prepare(`SELECT r.id AS runId, r.beta_id AS betaId FROM beta_test_runs r
|
||||
JOIN betas b ON b.id = r.beta_id JOIN beta_test_plans p ON p.beta_id = b.id
|
||||
WHERE r.server = ? AND r.retired_at IS NULL AND b.state IN ('running','ready-promote')
|
||||
AND p.outcome IN ('active','passed')`).all(server);
|
||||
return json({ runs }, { headers: { 'Cache-Control': 'no-store' } });
|
||||
}
|
||||
import { emit } from '$lib/server/pipeline/bus.js';
|
||||
|
||||
export async function POST({ request }: { request: Request }) {
|
||||
console.log('[telemetry] received POST /api/telemetry');
|
||||
// Encode payloads as JSON so script-supplied newlines cannot forge log lines.
|
||||
// Bound the preview to keep malformed requests from flooding container logs.
|
||||
const raw = await request.text();
|
||||
const context = {
|
||||
contentType: request.headers.get('content-type'),
|
||||
payloadPreview: raw.slice(0, 4096),
|
||||
payloadTruncated: raw.length > 4096
|
||||
};
|
||||
function reject(status: number, error: string, details: Record<string, unknown> = {}) {
|
||||
console.log(`[telemetry] rejected ${status}: ${JSON.stringify({ error, ...context, ...details })}`);
|
||||
return json({ ok: false, error, details }, { status });
|
||||
}
|
||||
if (request.headers.get('content-type')?.split(';')[0].trim().toLowerCase() !== 'application/json') {
|
||||
return reject(415, 'Content-Type must be application/json', { receivedContentType: context.contentType });
|
||||
return json({ ok: false, error: 'Content-Type must be application/json' }, { status: 415 });
|
||||
}
|
||||
let body;
|
||||
try {
|
||||
body = JSON.parse(raw);
|
||||
} catch (error) {
|
||||
return reject(400, 'Malformed JSON', { parseError: (error as Error).message });
|
||||
}
|
||||
if (!body || typeof body !== 'object' || Array.isArray(body)) {
|
||||
return reject(400, 'Payload must be a JSON object');
|
||||
}
|
||||
const issues: string[] = [];
|
||||
if (body.runId === undefined && body.event !== 'round_change') issues.push('Session telemetry requires runId');
|
||||
if (typeof body.server !== 'string' || !body.server.length || body.server.length > 255) {
|
||||
issues.push('server must be a registered server name (1–255 characters)');
|
||||
}
|
||||
for (const field of ['round', 'players']) {
|
||||
if (!Number.isSafeInteger(body[field]) || body[field] < 0) {
|
||||
issues.push(`${field} must be a nonnegative safe integer (JSON number, not a string)`);
|
||||
}
|
||||
}
|
||||
if (issues.length) {
|
||||
return reject(400, 'Invalid telemetry payload', { issues });
|
||||
}
|
||||
const server = getServer(body.server);
|
||||
if (!server) {
|
||||
const registeredServers = listServers().filter((entry) => !entry.destroyed_at).map((entry) => entry.name);
|
||||
return reject(404, 'Server is not registered', {
|
||||
receivedServer: body.server,
|
||||
hint: 'Send the exact Electric Cherry server name in the server field; matching is case-sensitive.',
|
||||
registeredServers
|
||||
});
|
||||
}
|
||||
if (server.destroyed_at) {
|
||||
return reject(404, 'Server has been destroyed', {
|
||||
receivedServer: body.server, destroyedAt: server.destroyed_at
|
||||
});
|
||||
}
|
||||
let test: { accepted: boolean; reason?: string } | undefined;
|
||||
if (body.runId !== undefined) {
|
||||
try {
|
||||
const signal = parseTestSignal(body);
|
||||
test = acceptTestSignal(body.server, body.runId, body.eventId, signal);
|
||||
} catch (error) { return reject(400, (error as Error).message); }
|
||||
if (body.event !== 'round_change') return json({ ok: true, test });
|
||||
}
|
||||
const telemetry = {
|
||||
server: body.server as string, round: body.round as number,
|
||||
players: body.players as number, receivedAt: new Date().toISOString()
|
||||
};
|
||||
let value: unknown;
|
||||
try { value = await request.json(); }
|
||||
catch { return json({ ok: false, error: 'Malformed JSON' }, { status: 400 }); }
|
||||
let telemetry: GameTelemetry;
|
||||
try { telemetry = { ...parseGameEvent(value), receivedAt: new Date().toISOString() }; }
|
||||
catch (error) { return json({ ok: false, error: (error as Error).message }, { status: 400 }); }
|
||||
const server = getServer(telemetry.server);
|
||||
if (!server || server.destroyed_at) return json({ ok: false, error: 'Server is not registered or has been destroyed' }, { status: 404 });
|
||||
db().transaction(() => {
|
||||
saveTelemetry(telemetry);
|
||||
console.log(`[telemetry] accepted round_change ${JSON.stringify(telemetry)}`);
|
||||
recordGameEvent(telemetry);
|
||||
})();
|
||||
emit({ type: 'server-telemetry', entries: [telemetry] });
|
||||
return json({ ok: true, telemetry, test });
|
||||
return json({ ok: true, telemetry });
|
||||
}
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import type { GameEvent, TelemetryPlayerStats } from '../../src/lib/telemetry.js';
|
||||
|
||||
export const roster = (count = 4): TelemetryPlayerStats[] => Array.from({ length: count }, (_, i) => ({
|
||||
name: `Player ${i + 1}`, guid: `1234567890123456${i}`, kills: 100 + i,
|
||||
headshots: 50, downs: 1, revives: 2, score: 5000
|
||||
}));
|
||||
export function gameEvent(event: GameEvent['event'], round = 25, players = 4, server = 't5_zombie_theater_1', map = 'zombie_theater'): GameEvent {
|
||||
return event === 'game_start'
|
||||
? { event, server, map, players: roster(players).map(({ name, guid }) => ({ name, guid })) }
|
||||
: { event, server, map, round, players: roster(players) };
|
||||
}
|
||||
export async function postTelemetry(body: unknown, contentType = 'application/json') {
|
||||
const { POST } = await import('../../src/routes/api/telemetry/+server.js');
|
||||
return POST({ request: new Request('http://localhost/api/telemetry', {
|
||||
method: 'POST', headers: { 'Content-Type': contentType }, body: JSON.stringify(body)
|
||||
}) });
|
||||
}
|
||||
@@ -1,100 +1,102 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { defaultPlan, parseBetaPlan, requirement, type BetaPlan } from '../../src/lib/beta-plan.js';
|
||||
import { applySample, applySignal, coveragePassed, emptyMetrics, evaluateRequirements, type TestSignal } from '../../src/lib/server/fleet/beta-evaluation.js';
|
||||
import { applySample, applyGameEvent, coveragePassed, emptyMetrics, evaluateRequirements } from '../../src/lib/server/fleet/beta-evaluation.js';
|
||||
import { gameEvent } from '../helpers/telemetry.js';
|
||||
|
||||
const signal = (event: TestSignal['event'], extra: Partial<TestSignal> = {}): TestSignal => ({ event, bootId: 'boot-1', sessionId: 'session-1', sequence: 0, round: 0, players: 4, name: '', ...extra });
|
||||
const planWith = (...kinds: Parameters<typeof requirement>[0][]): BetaPlan => ({ ...defaultPlan(), requirements: kinds.map((k, i) => requirement(k, `r${i}`)) });
|
||||
|
||||
describe('beta requirement evaluation', () => {
|
||||
it('copies a reactive preset without cloning proxies or mutating the saved requirements', () => {
|
||||
it('copies reactive presets without mutating saved requirements', () => {
|
||||
const source = defaultPlan();
|
||||
const proxied = new Proxy(source, { get(target, key, receiver) {
|
||||
return key === 'requirements' ? new Proxy(target.requirements, {}) : Reflect.get(target, key, receiver);
|
||||
} });
|
||||
const selected = parseBetaPlan(proxied);
|
||||
const selected = parseBetaPlan(new Proxy(source, {}));
|
||||
expect(selected).toEqual(source);
|
||||
selected.requirements[0].target = 15;
|
||||
selected.requirements.push(requirement('round', 'round'));
|
||||
expect(source.requirements).toHaveLength(1);
|
||||
expect(source.requirements[0].target).toBe(1);
|
||||
});
|
||||
it('validates plans without accepting unknown kinds, duplicate IDs, fractional counts or unsafe numbers', () => {
|
||||
it('rejects removed checks, duplicate IDs, fractional counts and unsafe numbers', () => {
|
||||
expect(parseBetaPlan(defaultPlan())).toEqual(defaultPlan());
|
||||
expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [] })).toThrow();
|
||||
expect(() => parseBetaPlan({ ...defaultPlan(), timeoutHours: NaN })).toThrow();
|
||||
expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [requirement('round', 'same'), requirement('round', 'same')] })).toThrow();
|
||||
for (const kind of ['feature', 'restart-budget', 'clean-logs']) {
|
||||
expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [{ ...requirement('round', 'r'), kind }] })).toThrow();
|
||||
}
|
||||
expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [{ ...requirement('round', 'r'), target: 1.5 }] })).toThrow();
|
||||
expect(() => parseBetaPlan({ ...defaultPlan(), requirements: [requirement('manual', 'constructor')] })).toThrow();
|
||||
});
|
||||
|
||||
it('counts only qualifying rounds and completed matches from the same session', () => {
|
||||
it('uses qualifying round and population together within the same game, including the final report', () => {
|
||||
const plan = planWith('round', 'matches');
|
||||
plan.requirements[0].minPlayers = 4;
|
||||
plan.requirements[1] = { ...plan.requirements[1], target: 2, minRound: 25, minPlayers: 4, minMinutes: 1 };
|
||||
const m = emptyMetrics();
|
||||
expect(applySignal(plan, m, signal('round_change', { sequence: 1, round: 50 }), 0)).toBe(false);
|
||||
applySignal(plan, m, signal('boot'), 0);
|
||||
applySignal(plan, m, signal('session_start'), 0);
|
||||
applySignal(plan, m, signal('round_change', { sequence: 1, round: 30, players: 1 }), 1000);
|
||||
applySignal(plan, m, signal('match_end', { sequence: 2 }), 61_000);
|
||||
applyGameEvent(plan, m, gameEvent('game_start'), 0);
|
||||
applyGameEvent(plan, m, gameEvent('round_change', 30, 1), 1000);
|
||||
applyGameEvent(plan, m, gameEvent('game_end', 30, 1), 61_000);
|
||||
expect(m.values.r0 ?? 0).toBe(0);
|
||||
expect(m.values.r1 ?? 0).toBe(0);
|
||||
applySignal(plan, m, signal('session_start', { sessionId: 'session-2' }), 62_000);
|
||||
applySignal(plan, m, signal('round_change', { sessionId: 'session-2', sequence: 1, round: 25 }), 63_000);
|
||||
applySignal(plan, m, signal('match_end', { sessionId: 'session-2', sequence: 2 }), 123_000);
|
||||
applyGameEvent(plan, m, gameEvent('game_start'), 62_000);
|
||||
applyGameEvent(plan, m, gameEvent('game_end', 25), 123_000);
|
||||
expect(m.values).toMatchObject({ r0: 25, r1: 1 });
|
||||
expect(applySignal(plan, m, signal('match_end', { sessionId: 'session-2', sequence: 3 }), 124_000)).toBe(false);
|
||||
applyGameEvent(plan, m, gameEvent('game_end', 25), 124_000);
|
||||
expect(m.values.r1).toBe(1);
|
||||
});
|
||||
|
||||
it('rejects out-of-order events and retired boots, while counting actual boot changes once', () => {
|
||||
const plan = planWith('feature', 'restart-budget');
|
||||
plan.requirements[0].event = 'perk.purchased';
|
||||
it('accepts ordinary rounds and game ends without boot events or a discovery handshake', () => {
|
||||
const plan = planWith('round', 'matches');
|
||||
const m = emptyMetrics();
|
||||
applySignal(plan, m, signal('boot'), 0);
|
||||
applySignal(plan, m, signal('session_start'), 0);
|
||||
applySignal(plan, m, signal('feature', { sequence: 2, name: 'perk.purchased' }), 1);
|
||||
expect(applySignal(plan, m, signal('feature', { sequence: 1, name: 'perk.purchased' }), 2)).toBe(false);
|
||||
applySignal(plan, m, signal('boot', { bootId: 'boot-2' }), 3);
|
||||
expect(applySignal(plan, m, signal('boot'), 4)).toBe(false);
|
||||
expect(m.values.r0).toBe(1);
|
||||
expect(evaluateRequirements(plan, m, 0)[1].failed).toBe(true);
|
||||
expect(m.restarts).toBe(1);
|
||||
applyGameEvent(plan, m, gameEvent('round_change'), 0);
|
||||
applyGameEvent(plan, m, gameEvent('game_end'), 1000);
|
||||
expect(m.values).toMatchObject({ r0: 25, r1: 1 });
|
||||
applyGameEvent(plan, m, gameEvent('game_start'), 2000);
|
||||
applyGameEvent(plan, m, gameEvent('game_end', 1), 3000);
|
||||
expect(m.values.r1).toBe(2);
|
||||
});
|
||||
|
||||
it('does not double count unqualified matches across log and telemetry sources', () => {
|
||||
it('does not combine games or maps to satisfy duration and round qualifiers', () => {
|
||||
const plan = planWith('matches');
|
||||
plan.requirements[0] = { ...plan.requirements[0], minRound: 25, minMinutes: 1 };
|
||||
const m = emptyMetrics();
|
||||
applySignal(plan, m, signal('boot'), 0);
|
||||
applySignal(plan, m, signal('session_start'), 0);
|
||||
applySignal(plan, m, signal('match_end', { sequence: 1 }), 1000);
|
||||
expect(m.values.r0).toBeUndefined(); // these requirements use incremental match logs
|
||||
applyGameEvent(plan, m, gameEvent('game_start'), 0);
|
||||
applyGameEvent(plan, m, gameEvent('round_change'), 1000);
|
||||
applyGameEvent(plan, m, gameEvent('game_start'), 60_000);
|
||||
applyGameEvent(plan, m, gameEvent('game_end', 1), 120_000);
|
||||
expect(m.values.r0 ?? 0).toBe(0);
|
||||
applyGameEvent(plan, m, gameEvent('game_end', 30, 4, 't5_zombie_theater_1', 'zombie_moon'), 180_000);
|
||||
expect(m.values.r0 ?? 0).toBe(0);
|
||||
});
|
||||
|
||||
it('integrates populated time and player-hours conservatively and excludes monitoring gaps', () => {
|
||||
const plan = planWith('populated-minutes', 'player-hours', 'healthy-minutes', 'population', 'sustained-population', 'failure-budget');
|
||||
plan.requirements[4].minPlayers = 4;
|
||||
it('derives participation only from game reports, even when rounds are many minutes apart', () => {
|
||||
const plan = planWith('populated-minutes', 'player-hours', 'population', 'sustained-population');
|
||||
plan.requirements[3].minPlayers = 4;
|
||||
const m = emptyMetrics();
|
||||
applySample(plan, m, 0, true, 4);
|
||||
applySample(plan, m, 30_000, true, 4);
|
||||
applySample(plan, m, 60_000, true, 4);
|
||||
expect(m.values).toMatchObject({ r0: 1, r1: 4 / 60, r2: 1, r3: 4, r4: 1 });
|
||||
applySample(plan, m, 300_000, true, 4);
|
||||
applySample(plan, m, 0, true);
|
||||
applySample(plan, m, 30_000, true);
|
||||
expect(m.values).toEqual({});
|
||||
applyGameEvent(plan, m, gameEvent('game_start'), 0);
|
||||
applyGameEvent(plan, m, gameEvent('round_change'), 600_000);
|
||||
expect(m.values).toMatchObject({ r0: 10, r1: 40 / 60, r2: 4, r3: 10 });
|
||||
applyGameEvent(plan, m, gameEvent('game_end', 26, 2), 1_200_000);
|
||||
expect(m.values).toMatchObject({ r0: 20, r1: 1, r3: 10 });
|
||||
applyGameEvent(plan, m, gameEvent('game_start'), 3_000_000);
|
||||
expect(m.values.r0).toBe(20);
|
||||
});
|
||||
|
||||
it('excludes monitoring gaps from healthy time and requires explicit manual sign-off', () => {
|
||||
const plan = planWith('healthy-minutes', 'failure-budget', 'manual');
|
||||
const m = emptyMetrics();
|
||||
applySample(plan, m, 0, true);
|
||||
applySample(plan, m, 60_000, true);
|
||||
expect(m.values.r0).toBe(1);
|
||||
expect(m.values.r2).toBe(0);
|
||||
applySample(plan, m, 330_000, false, 0);
|
||||
applySample(plan, m, 360_000, false, 0);
|
||||
applySample(plan, m, 300_000, true);
|
||||
expect(m.values.r0).toBe(0);
|
||||
applySample(plan, m, 330_000, false);
|
||||
applySample(plan, m, 360_000, false);
|
||||
expect(m.outages).toBe(1);
|
||||
expect(evaluateRequirements(plan, m, 0)[5].failed).toBe(true);
|
||||
});
|
||||
|
||||
it('requires readable logs, boot evidence and explicit manual sign-off', () => {
|
||||
const plan = planWith('clean-logs', 'restart-budget', 'manual');
|
||||
const m = emptyMetrics();
|
||||
expect(evaluateRequirements(plan, m, 0).every(r => !r.passed)).toBe(true);
|
||||
m.logAvailable = true; m.bootId = 'boot'; m.manual.r2 = 'Josh: HUD checked';
|
||||
expect(evaluateRequirements(plan, m, 0).every(r => r.passed)).toBe(true);
|
||||
m.logErrors = 1;
|
||||
expect(evaluateRequirements(plan, m, 0)[0].failed).toBe(true);
|
||||
expect(evaluateRequirements(plan, m, 0)[1].failed).toBe(true);
|
||||
expect(evaluateRequirements(plan, m, 0)[2].passed).toBe(false);
|
||||
m.manual.r2 = 'Josh: HUD checked';
|
||||
expect(evaluateRequirements(plan, m, 0)[2].passed).toBe(true);
|
||||
});
|
||||
|
||||
it('requires each selected coverage group even when many servers on one map pass', () => {
|
||||
|
||||
+119
-54
@@ -3,6 +3,7 @@ import { appendFile, mkdir, readFile, writeFile } from 'node:fs/promises';
|
||||
import { existsSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
import { gameEvent, postTelemetry } from '../helpers/telemetry.js';
|
||||
import { defaultPlan, requirement, type BetaPlan } from '../../src/lib/beta-plan.js';
|
||||
|
||||
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({ komodoRestart: vi.fn(), komodoInspectHealthy: vi.fn() }));
|
||||
@@ -77,56 +78,50 @@ describe('durable beta test plans', () => {
|
||||
expect(getBeta(id)?.state).toBe('ready-promote');
|
||||
});
|
||||
|
||||
it('ignores pre-beta log markers, handles partial lines, and persists cursors across app restarts', async () => {
|
||||
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('matches', 'matches'), requirement('clean-logs', 'logs')] });
|
||||
it('uses script events instead of logs and keeps match progress across app restarts', async () => {
|
||||
await postTelemetry(gameEvent('game_end'));
|
||||
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('matches', 'matches')] });
|
||||
await tick(); await tick();
|
||||
const { getBeta } = await import('../../src/lib/server/fleet/beta.js');
|
||||
const { testView } = await import('../../src/lib/server/fleet/beta-tests.js');
|
||||
expect(getBeta(id)?.state).toBe('running');
|
||||
expect(testView(id)?.servers[0].progress.map(p => p.current)).toEqual([0, 0]);
|
||||
const log = join(fixture.t5, names[0], 'logs/games_sp.log');
|
||||
await appendFile(log, 'InitGame:\n 0:00 J;new;1;Josh\nExitLevel: exe');
|
||||
await appendFile(join(fixture.t5, names[0], 'logs/games_sp.log'), 'InitGame:\n 0:00 J;new;1;Josh\nExitLevel: executed\n');
|
||||
await tick();
|
||||
expect(getBeta(id)?.state).toBe('running');
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(0);
|
||||
await postTelemetry(gameEvent('game_start'));
|
||||
const { closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
closeDb();
|
||||
await appendFile(log, 'cuted\n');
|
||||
await tick(); await tick();
|
||||
expect(getBeta(id)?.state).toBe('ready-promote');
|
||||
await postTelemetry(gameEvent('game_end'));
|
||||
await tick();
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(1);
|
||||
await appendFile(join(fixture.t5, names[0], 'logs/console.log'), 'script runtime error: undefined variable\n');
|
||||
await tick();
|
||||
expect(getBeta(id)?.state).toBe('running');
|
||||
expect(testView(id)?.outcome).toBe('failed');
|
||||
});
|
||||
|
||||
it('accepts durable run-scoped round evidence and rejects stale runs and duplicate events', async () => {
|
||||
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('round', 'round')] });
|
||||
await tick(); await tick();
|
||||
const { acceptTestSignal, currentRun, testView } = await import('../../src/lib/server/fleet/beta-tests.js');
|
||||
const run = currentRun(id, names[0])!;
|
||||
const base = { bootId: 'boot', sessionId: 'session', sequence: 0, round: 0, players: 4, name: '' };
|
||||
expect(acceptTestSignal(names[0], 'stale-run', 'old', { ...base, event: 'boot' }).accepted).toBe(false);
|
||||
expect(acceptTestSignal(names[0], run.id, 'boot', { ...base, event: 'boot' }).accepted).toBe(true);
|
||||
expect(acceptTestSignal(names[0], run.id, 'start', { ...base, event: 'session_start' }).accepted).toBe(true);
|
||||
const round = { ...base, event: 'round_change' as const, sequence: 1, round: 25 };
|
||||
expect(acceptTestSignal(names[0], run.id, 'round', round).accepted).toBe(true);
|
||||
expect(acceptTestSignal(names[0], run.id, 'round', round).accepted).toBe(false);
|
||||
await tick();
|
||||
expect(testView(id)?.outcome).toBe('passed');
|
||||
expect(testView(id)?.evidence.filter(e => e.kind === 'round_change')).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('resets evidence after planned recovery and only passes after testing the new process', async () => {
|
||||
it('automatically credits ordinary rounds only to active deployed beta servers', async () => {
|
||||
const id = await openPlan({ ...defaultPlan(), waveSize: 1, requirements: [requirement('round', 'round')] }, names);
|
||||
await postTelemetry(gameEvent('round_change'));
|
||||
await tick(); await tick();
|
||||
const { testView } = await import('../../src/lib/server/fleet/beta-tests.js');
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(0);
|
||||
await postTelemetry(gameEvent('round_change', 99, 4, names[1]));
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(0);
|
||||
await postTelemetry(gameEvent('round_change'));
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(25);
|
||||
expect(testView(id)?.evidence.filter(e => e.kind === 'round_change')).toHaveLength(1);
|
||||
await tick();
|
||||
expect(existsSync(scriptPath(names[1]))).toBe(true);
|
||||
});
|
||||
|
||||
it('resets evidence after planned recovery and consumes the same simple reports afterward', async () => {
|
||||
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players'), requirement('recovery', 'recovery')] });
|
||||
await tick(); await tick();
|
||||
const { currentRun, testView } = await import('../../src/lib/server/fleet/beta-tests.js');
|
||||
const first = currentRun(id, names[0])!;
|
||||
expect(testView(id)?.outcome).toBe('active');
|
||||
await tick();
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(0);
|
||||
await postTelemetry(gameEvent('game_start'));
|
||||
await tick(); await tick();
|
||||
expect(currentRun(id, names[0])?.id).not.toBe(first.id);
|
||||
expect(currentRun(id, names[0])?.phase).toBe(1);
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(0);
|
||||
await postTelemetry(gameEvent('game_start'));
|
||||
await tick();
|
||||
expect(testView(id)?.outcome).toBe('passed');
|
||||
});
|
||||
@@ -143,6 +138,7 @@ describe('durable beta test plans', () => {
|
||||
it('revokes readiness if health fails while awaiting manual promotion', async () => {
|
||||
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players'), requirement('failure-budget', 'outages')] });
|
||||
await tick(); await tick();
|
||||
await postTelemetry(gameEvent('game_start')); await tick();
|
||||
const { getBeta, promoteBeta } = await import('../../src/lib/server/fleet/beta.js');
|
||||
expect(getBeta(id)?.state).toBe('ready-promote');
|
||||
await tick(15, 0, false);
|
||||
@@ -152,10 +148,9 @@ describe('durable beta test plans', () => {
|
||||
|
||||
it('automatically rolls back only the deployed wave and queues safe rollback restarts', async () => {
|
||||
await writeFile(scriptPath(names[0]), 'original');
|
||||
const id = await openPlan({ ...defaultPlan(), waveSize: 1, failureAction: 'rollback', requirements: [requirement('round', 'round'), requirement('clean-logs', 'logs')] }, names);
|
||||
const id = await openPlan({ ...defaultPlan(), waveSize: 1, failureAction: 'rollback', requirements: [requirement('round', 'round'), requirement('failure-budget', 'outages')] }, names);
|
||||
await tick(); await tick();
|
||||
await appendFile(join(fixture.t5, names[0], 'logs/console.log'), 'fatal error: test\n');
|
||||
await tick();
|
||||
await tick(15, 0, false);
|
||||
const { getBeta } = await import('../../src/lib/server/fleet/beta.js');
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
expect(getBeta(id)?.state).toBe('rolled-back');
|
||||
@@ -176,6 +171,7 @@ describe('durable beta test plans', () => {
|
||||
it('retains artifact ownership while ready for promotion', async () => {
|
||||
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('population', 'players')] });
|
||||
await tick(); await tick();
|
||||
await postTelemetry(gameEvent('game_start')); await tick();
|
||||
const { heldByBeta, ownerOf } = await import('../../src/lib/server/fleet/servers.js');
|
||||
expect(heldByBeta(names[0]).get('test.gsc')).toBe(id);
|
||||
expect(ownerOf(names[0], 'test.gsc')).toBe(id);
|
||||
@@ -185,32 +181,27 @@ describe('durable beta test plans', () => {
|
||||
expect(ownerOf(names[0], 'test.gsc')).toBeNull();
|
||||
});
|
||||
|
||||
it('does not carry joins into a truncated log or deduplicate a genuinely new match after rotation', async () => {
|
||||
it('counts successive games with the same players and ignores repeated game ends', async () => {
|
||||
const id = await openPlan({ ...defaultPlan(), requirements: [{ ...requirement('matches', 'matches'), target: 2 }] });
|
||||
await tick(); await tick();
|
||||
const log = join(fixture.t5, names[0], 'logs/games_sp.log');
|
||||
await appendFile(log, 'InitGame:\n 0:00 J;first;1;Josh\nExitLevel: executed\n');
|
||||
await tick();
|
||||
// Same inode, contents replaced with a larger file: anchor detects truncation/regrowth.
|
||||
await writeFile(log, 'InitGame:\n 0:00 J;second;1;Josh\n' + 'padding\n'.repeat(30) + 'ExitLevel: executed\n');
|
||||
await tick();
|
||||
await postTelemetry(gameEvent('game_start'));
|
||||
await postTelemetry(gameEvent('game_end'));
|
||||
await postTelemetry(gameEvent('game_end'));
|
||||
const { testView } = await import('../../src/lib/server/fleet/beta-tests.js');
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(1);
|
||||
await postTelemetry(gameEvent('game_start'));
|
||||
await postTelemetry(gameEvent('game_end'));
|
||||
await tick();
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(2);
|
||||
expect(testView(id)?.outcome).toBe('passed');
|
||||
});
|
||||
|
||||
it('exposes current run tokens, validates session payloads, and records checklist sign-off through the API', async () => {
|
||||
it('records checklist sign-off and starts a fresh attempt without changing script payloads', async () => {
|
||||
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('round', 'round'), requirement('manual', 'hud')] });
|
||||
await tick(); await tick();
|
||||
const telemetry = await import('../../src/routes/api/telemetry/+server.js');
|
||||
const context = await (await telemetry.GET({ url: new URL(`http://test/api/telemetry?server=${names[0]}`) })).json();
|
||||
expect(context.runs).toHaveLength(1);
|
||||
const body = { server: names[0], runId: context.runs[0].runId, eventId: 'boot', bootId: 'boot', sequence: 0, round: 0, players: 0, event: 'boot' };
|
||||
const post = (payload: unknown) => telemetry.POST({ request: new Request('http://test/api/telemetry', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(payload) }) });
|
||||
expect((await post({ ...body, eventId: null })).status).toBe(400);
|
||||
expect(await (await post(body)).json()).toMatchObject({ test: { accepted: true } });
|
||||
await post({ ...body, eventId: 'start', event: 'session_start', sessionId: 'game', players: 4 });
|
||||
await post({ ...body, eventId: 'round', event: 'round_change', sessionId: 'game', sequence: 1, round: 25, players: 4 });
|
||||
expect(telemetry).not.toHaveProperty('GET');
|
||||
expect(await (await postTelemetry(gameEvent('round_change'))).json()).not.toHaveProperty('test');
|
||||
const controls = await import('../../src/routes/api/betas/[id]/test/+server.js');
|
||||
const result = await controls.POST({ params: { id: String(id) }, request: new Request('http://test', { method: 'POST', body: JSON.stringify({ action: 'sign-off', server: names[0], requirementId: 'hud', note: 'Josh: HUD verified' }) }) });
|
||||
expect(result.status).toBe(200);
|
||||
@@ -219,8 +210,82 @@ describe('durable beta test plans', () => {
|
||||
expect(testView(id)?.outcome).toBe('passed');
|
||||
const retry = await controls.POST({ params: { id: String(id) }, request: new Request('http://test', { method: 'POST', body: JSON.stringify({ action: 'retry' }) }) });
|
||||
expect(retry.status).toBe(200);
|
||||
expect(await (await post({ ...body, eventId: 'late' })).json()).toMatchObject({ test: { accepted: false } });
|
||||
await postTelemetry(gameEvent('round_change'));
|
||||
expect(testView(id)?.servers).toHaveLength(0);
|
||||
await tick(); await tick();
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(0);
|
||||
await postTelemetry(gameEvent('round_change'));
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(25);
|
||||
});
|
||||
|
||||
it('keeps Fleet reporting during pauses without crediting paused gameplay or elapsed time', async () => {
|
||||
const id = await openPlan({ ...defaultPlan(), requirements: [{ ...requirement('populated-minutes', 'time'), target: 60 }] });
|
||||
await tick(); await tick();
|
||||
await postTelemetry(gameEvent('game_start'));
|
||||
vi.setSystemTime(Date.now() + 600_000);
|
||||
await postTelemetry(gameEvent('round_change'));
|
||||
const { testView } = await import('../../src/lib/server/fleet/beta-tests.js');
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(10);
|
||||
const controls = await import('../../src/routes/api/betas/[id]/test/+server.js');
|
||||
const control = (action: string) => controls.POST({ params: { id: String(id) }, request: new Request('http://test', {
|
||||
method: 'POST', body: JSON.stringify({ action })
|
||||
}) });
|
||||
expect((await control('pause')).status).toBe(200);
|
||||
vi.setSystemTime(Date.now() + 3_600_000);
|
||||
await postTelemetry(gameEvent('game_end', 50));
|
||||
const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js');
|
||||
expect(listTelemetry()[0]).toMatchObject({ event: 'game_end', round: 50 });
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(10);
|
||||
expect((await control('resume')).status).toBe(200);
|
||||
await postTelemetry(gameEvent('round_change', 1));
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(10);
|
||||
vi.setSystemTime(Date.now() + 600_000);
|
||||
await postTelemetry(gameEvent('round_change', 2));
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(20);
|
||||
});
|
||||
|
||||
it('migrates existing unsupported checks into review items and resets active progress once', async () => {
|
||||
const id = await openPlan({ ...defaultPlan(), requirements: [requirement('round', 'round')] });
|
||||
await tick(); await tick();
|
||||
await postTelemetry(gameEvent('round_change'));
|
||||
await tick();
|
||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
const oldPlan = { ...defaultPlan(), requirements: [requirement('round', 'round'),
|
||||
...['feature', 'restart-budget', 'clean-logs'].map((kind, i) => ({ ...requirement('manual', `old-${i}`), kind, event: 'perk.purchased' }))] };
|
||||
db().prepare('UPDATE beta_test_plans SET plan = ? WHERE beta_id = ?').run(JSON.stringify(oldPlan), id);
|
||||
db().prepare('INSERT INTO beta_test_presets (name, plan) VALUES (?, ?)').run('Old plan', JSON.stringify(oldPlan));
|
||||
db().prepare("DELETE FROM settings WHERE key = 'game_telemetry_v2'").run();
|
||||
closeDb();
|
||||
const { testView } = await import('../../src/lib/server/fleet/beta-tests.js');
|
||||
const { getBeta } = await import('../../src/lib/server/fleet/beta.js');
|
||||
const view = testView(id)!;
|
||||
expect(view.plan.requirements.map(r => r.kind)).toEqual(['round', 'manual', 'manual', 'manual']);
|
||||
expect(view.plan.requirements.some(r => 'event' in r)).toBe(false);
|
||||
expect(view.servers[0].progress.every(p => !p.passed)).toBe(true);
|
||||
expect(view.evidence.some(e => e.kind === 'round_change')).toBe(true);
|
||||
expect(getBeta(id)?.state).toBe('running');
|
||||
const presets = await import('../../src/routes/api/betas/presets/+server.js');
|
||||
const saved = (await presets.GET().json()).presets.find((p: { name: string }) => p.name === 'Old plan');
|
||||
expect(saved.plan).toEqual(view.plan);
|
||||
await postTelemetry(gameEvent('round_change'));
|
||||
closeDb();
|
||||
expect(testView(id)?.servers[0].progress[0].current).toBe(25);
|
||||
});
|
||||
|
||||
it('uses game_end for the legacy match gate without reading logs', async () => {
|
||||
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
|
||||
const { openBeta, getBeta, listBetaServers } = await import('../../src/lib/server/fleet/beta.js');
|
||||
const { script, version } = createScript({ name: 'Legacy beta', targetRelPath: 'legacy.gsc', content: Buffer.from('main() {}') });
|
||||
const { betaId } = await openBeta({ scripts: [{ scriptId: script.id, scriptVersionId: version.id }], servers: [names[0]], waitForHealth: true, waitForMatch: true });
|
||||
await tick(); await tick();
|
||||
expect(listBetaServers(betaId)[0].stage).toBe('awaiting-match');
|
||||
await appendFile(join(fixture.t5, names[0], 'logs/games_sp.log'), 'ExitLevel: executed\n');
|
||||
await tick();
|
||||
expect(getBeta(betaId)?.state).toBe('running');
|
||||
await postTelemetry(gameEvent('game_end'));
|
||||
await postTelemetry(gameEvent('game_start'));
|
||||
await tick();
|
||||
expect(getBeta(betaId)?.state).toBe('ready-promote');
|
||||
});
|
||||
|
||||
it('stores validated presets as copies and rejects malformed plans', async () => {
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { currentRound, fleetIdentity, matchesFleet, watchSatisfied, type FleetServer, type FleetOperation } from '../../src/lib/fleet-overview.js';
|
||||
import { gameEvent } from '../helpers/telemetry.js';
|
||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
|
||||
const now = Date.parse('2026-09-18T12:00:00Z');
|
||||
@@ -42,7 +43,7 @@ describe('Fleet discovery and match freshness', () => {
|
||||
expect(matchesFleet({ ...server, health: 'starting' }, '', 'attention', '', [], [], now)).toBe(false);
|
||||
});
|
||||
it('only spotlights fresh telemetry from the current occupied match', () => {
|
||||
const report = { server: server.name, round: 38, players: 4, receivedAt: iso(-1000) };
|
||||
const report = { ...gameEvent('round_change', 38, 4, server.name), receivedAt: iso(-1000) };
|
||||
expect(currentRound(server, report, now)).toBe(38);
|
||||
for (const change of [{ receivedAt: iso(-121_000) }, { receivedAt: iso(-61_000) }, { receivedAt: iso(60_000) }]) {
|
||||
expect(currentRound(server, { ...report, ...change }, now)).toBeNull();
|
||||
|
||||
@@ -79,13 +79,12 @@ describe('T6 config lifecycle in a mixed fleet', () => {
|
||||
});
|
||||
|
||||
it('reads the T6 game log and keeps log path traversal checks', async () => {
|
||||
const { resolveLogPath, gameLogKey } = await import('../../src/lib/server/fleet/logs.js');
|
||||
const { scanMatchState } = await import('../../src/lib/server/fleet/match-log.js');
|
||||
const { resolveLogPath, gameLogKey, tailLog } = await import('../../src/lib/server/fleet/logs.js');
|
||||
const logs = join(fixture.t6, town, 'logs');
|
||||
await mkdir(logs, { recursive: true });
|
||||
await writeFile(join(logs, 'games_zm.log'), '0:00 InitGame:\n0:01 J;123;0;Player\n');
|
||||
expect(resolveLogPath(town, gameLogKey(town))).toBe(resolve(logs, 'games_zm.log'));
|
||||
expect(await scanMatchState(town)).toMatchObject({ kind: 'in-progress' });
|
||||
expect((await tailLog(resolveLogPath(town, gameLogKey(town))!, 10))?.lines).toContain('0:01 J;123;0;Player');
|
||||
expect(resolveLogPath('t6_/../../escape', 'games_zm')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
|
||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
import { gameEvent, postTelemetry as post, roster } from '../helpers/telemetry.js';
|
||||
|
||||
let fixture: Fixture;
|
||||
beforeEach(async () => {
|
||||
@@ -10,42 +11,60 @@ beforeEach(async () => {
|
||||
});
|
||||
afterEach(async () => fixture.cleanup());
|
||||
|
||||
const payload = { event: 'round_change', server: 't5_zombie_theater_1', round: 17, players: 4 };
|
||||
async function post(body: unknown, contentType = 'application/json') {
|
||||
const { POST } = await import('../../src/routes/api/telemetry/+server.js');
|
||||
return POST({ request: new Request('http://localhost/api/telemetry', {
|
||||
method: 'POST', headers: { 'Content-Type': contentType }, body: JSON.stringify(body)
|
||||
}) });
|
||||
}
|
||||
const payload = gameEvent('round_change', 17);
|
||||
|
||||
it('persists reports, broadcasts updates, and leaves RCON safety data untouched', async () => {
|
||||
it('persists full player stats, broadcasts updates, and leaves RCON safety data untouched', async () => {
|
||||
const { subscribe } = await import('../../src/lib/server/pipeline/bus.js');
|
||||
const listener = vi.fn();
|
||||
const unsubscribe = subscribe(listener);
|
||||
expect((await post(payload)).status).toBe(200);
|
||||
expect(listener).toHaveBeenCalledWith(expect.objectContaining({ type: 'server-telemetry' }));
|
||||
expect(await (await post(payload)).json()).toEqual({ ok: true, telemetry: { ...payload, receivedAt: expect.any(String) } });
|
||||
expect(listener).toHaveBeenCalledWith({ type: 'server-telemetry', entries: [{ ...payload, receivedAt: expect.any(String) }] });
|
||||
unsubscribe();
|
||||
// A new match can legitimately report a lower round; retries overwrite one row.
|
||||
await post({ ...payload, round: 1 });
|
||||
await post({ ...payload, round: 1 });
|
||||
const { closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
closeDb();
|
||||
const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js');
|
||||
expect(listTelemetry()).toEqual([expect.objectContaining({ round: 1, players: 4 })]);
|
||||
expect(listTelemetry()).toEqual([{ ...payload, round: 1, receivedAt: expect.any(String) }]);
|
||||
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
|
||||
expect(getServer(payload.server)?.last_players).toBeNull();
|
||||
const { GET } = await import('../../src/routes/api/events/servers/+server.js');
|
||||
const response = await GET();
|
||||
const reader = response.body!.getReader();
|
||||
await reader.read(); // greeting
|
||||
const snapshot = await reader.read();
|
||||
expect(new TextDecoder().decode(snapshot.value)).toContain('"round":1');
|
||||
const reader = (await GET()).body!.getReader();
|
||||
await reader.read();
|
||||
const snapshot = new TextDecoder().decode((await reader.read()).value);
|
||||
expect(snapshot).toContain('"round":1');
|
||||
expect(snapshot).toContain('"headshots":50');
|
||||
expect(snapshot).toContain('"guid":"12345678901234560"');
|
||||
await reader.cancel();
|
||||
});
|
||||
|
||||
it.each([null, [], {}, { ...payload, event: 'other' }, { ...payload, round: -1 },
|
||||
{ ...payload, round: 1.5 }, { ...payload, round: '17' }, { ...payload, players: -1 },
|
||||
{ ...payload, players: null }])('rejects invalid telemetry: %j', async (body) => {
|
||||
it('accepts all three events without an active beta and updates match lifecycle', async () => {
|
||||
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
|
||||
const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js');
|
||||
const start = gameEvent('game_start');
|
||||
expect((await post(start)).status).toBe(200);
|
||||
expect(listTelemetry()).toEqual([{ ...start, receivedAt: expect.any(String) }]);
|
||||
expect(getServer(start.server)?.game_started_at).toBeTruthy();
|
||||
expect(getServer(start.server)?.game_ended_at).toBeNull();
|
||||
expect((await post(payload)).status).toBe(200);
|
||||
const end = gameEvent('game_end', 18);
|
||||
expect((await post(end)).status).toBe(200);
|
||||
expect(getServer(start.server)?.game_ended_at).toBeTruthy();
|
||||
expect(listTelemetry()[0]).toMatchObject(end);
|
||||
await post(start);
|
||||
expect(listTelemetry()[0]).not.toHaveProperty('round');
|
||||
expect(getServer(start.server)?.game_ended_at).toBeNull();
|
||||
expect((await post(gameEvent('game_start', 0, 0))).status).toBe(200);
|
||||
expect((await post(gameEvent('game_end', 0, 0))).status).toBe(200);
|
||||
});
|
||||
|
||||
it.each([null, [], {}, ...['boot', 'session_start', 'match_end', 'feature', 'other'].map(event => ({ ...payload, event })),
|
||||
{ ...payload, map: '' }, { ...payload, map: undefined }, { ...payload, round: -1 },
|
||||
{ ...payload, round: 1.5 }, { ...payload, round: '17' }, { ...payload, players: 4 }, { ...payload, players: null },
|
||||
{ ...payload, players: [null] }, { ...payload, players: [{ name: 'Josh', guid: '123' }] },
|
||||
...['name', 'guid', 'kills', 'headshots', 'downs', 'revives', 'score'].map(field => ({ ...payload, players: [{ ...roster(1)[0], [field]: null }] })),
|
||||
{ ...payload, players: [{ ...roster(1)[0], guid: 12345678901234567 }] },
|
||||
{ ...payload, players: [{ ...roster(1)[0], score: -1 }] }
|
||||
])('rejects invalid telemetry: %j', async (body) => {
|
||||
expect((await post(body)).status).toBe(400);
|
||||
});
|
||||
|
||||
@@ -63,3 +82,15 @@ it('rejects malformed JSON', async () => {
|
||||
method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{'
|
||||
}) })).status).toBe(400);
|
||||
});
|
||||
|
||||
it('migrates old round-only storage without fabricating player identities', async () => {
|
||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
db().exec('DROP TABLE server_telemetry; CREATE TABLE server_telemetry (server TEXT PRIMARY KEY, round INTEGER NOT NULL, players INTEGER NOT NULL, received_at TEXT NOT NULL)');
|
||||
db().prepare('INSERT INTO server_telemetry VALUES (?, ?, ?, ?)').run(payload.server, 99, 4, new Date().toISOString());
|
||||
closeDb();
|
||||
const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js');
|
||||
expect(listTelemetry()).toEqual([]);
|
||||
await post(payload);
|
||||
closeDb();
|
||||
expect(listTelemetry()[0]).toMatchObject(payload);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user