Clean up README and organize documentation
This commit is contained in:
@@ -0,0 +1,51 @@
|
||||
# Documentation
|
||||
|
||||
Start with [Getting started](guides/getting-started.md) to set up a controller
|
||||
and launch your first server.
|
||||
|
||||
## Guides
|
||||
|
||||
### Setup and administration
|
||||
|
||||
- [Getting started](guides/getting-started.md) — requirements, installation, and first launch.
|
||||
- [Native hosts](guides/native-hosts.md) — enrollment, game files, capacity, agent updates, and recovery.
|
||||
- [Users and access](guides/authentication.md) — roles, sessions, telemetry credentials, and password recovery.
|
||||
- [Fleet settings](guides/settings.md) — concurrency, retention, and cleanup.
|
||||
|
||||
### Fleet operations
|
||||
|
||||
- [Provisioning](guides/provisioning.md) — lifecycle actions, safety checks, queues, and diagnostics.
|
||||
- [Collections](guides/collections.md) — reusable map lineups and promoted launch files.
|
||||
- [Lobby autoscaling](guides/autoscaling.md) — policies, ownership, pause controls, and the pilot procedure.
|
||||
- [Maintenance](guides/maintenance.md) — health recovery, drift repair, updates, and scheduled restarts.
|
||||
|
||||
### Releases and analytics
|
||||
|
||||
- [Beta test plans](guides/beta-test-plans.md) — game reporters, test requirements, coverage, and outcomes.
|
||||
- [Plugin releases](guides/plugin-releases.md) — GitHub release discovery and imports.
|
||||
- [Metrics](guides/metrics.md) — utilization, capacity pressure, and hosting experiments.
|
||||
- [Leaderboard and matches](guides/leaderboard.md) — player statistics, match history, and the read API.
|
||||
|
||||
## Development
|
||||
|
||||
- [Development guide](development/README.md) — local setup, architecture, checks, and database diagnostics.
|
||||
- [UI patterns](development/ui-patterns.md) — shared components, forms, navigation, and refresh behavior.
|
||||
|
||||
## Plans
|
||||
|
||||
[Lobby autoscaling](plans/lobby-autoscaling-plan.md) records the implemented design
|
||||
and the production dry run, pilot measurements, and rollout still to complete.
|
||||
Current operating instructions live in the guides above.
|
||||
|
||||
## Research
|
||||
|
||||
- [Reducing idle CPU use](research/t5-t6-idle-cpu-research.md) — runtime tuning and lobby availability experiments.
|
||||
- [Leaderboard query benchmark](research/leaderboard-query-benchmark.md) — historical SQL measurements and methodology.
|
||||
|
||||
Research records its baseline and measurement limits; consult the guides for
|
||||
current behavior.
|
||||
|
||||
Keep operating instructions in `guides/`, contributor guidance in `development/`,
|
||||
unfinished delivery or rollout work in `plans/`, and investigations in `research/`.
|
||||
When a plan is complete, fold its lasting guidance into the relevant guide and
|
||||
remove the superseded document. Git history preserves earlier reviews and plans.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 221 KiB |
@@ -0,0 +1,114 @@
|
||||
# Development
|
||||
|
||||
The controller uses SvelteKit, TypeScript and SQLite (`better-sqlite3`), with the
|
||||
Node adapter and server-sent events. The host agent uses Python 3's standard
|
||||
library and runs as a systemd service; hosts do not need Node.js.
|
||||
|
||||
Agents connect outbound to the controller over HTTPS and manage Docker/Plutainer
|
||||
and local RCON on each host. The controller owns fleet metadata and queues; it
|
||||
does not require a Docker socket or direct access to host filesystems.
|
||||
|
||||
## Run locally
|
||||
|
||||
Use Node.js 22 (the container's version), npm and Python 3.
|
||||
|
||||
```sh
|
||||
npm ci
|
||||
npm run agent:package
|
||||
export EC_SETUP_TOKEN="$(openssl rand -hex 32)"
|
||||
export BETA_ARCHIVE_DIR=./data/beta-archive
|
||||
EC_AUTH_ALLOW_HTTP=true npm run dev
|
||||
```
|
||||
|
||||
Use the exported token at `/setup` to create the first administrator, then remove
|
||||
it from the environment. Local metadata defaults to `./data/cherry.db`; use a
|
||||
separate database when testing enrollment. HTTP login is for local development;
|
||||
production agent transport requires HTTPS. See [authentication](../guides/authentication.md)
|
||||
for access and recovery details.
|
||||
|
||||
## Checks and builds
|
||||
|
||||
```sh
|
||||
npm run check
|
||||
npm test
|
||||
npm run test:agent
|
||||
npm run build
|
||||
```
|
||||
|
||||
Builds regenerate the downloadable agent, standalone updater and checksum
|
||||
manifest. They also bundle report workers under `build/reports` and launch
|
||||
preparation under `build/launch`. Development and test commands compile those
|
||||
workers automatically. Restart `npm run dev` after changing worker code.
|
||||
|
||||
Automated tests simulate Docker and game hosts. Real Debian installation and
|
||||
T5/T6 server validation use the [host acceptance checklist](../guides/native-hosts.md#development-and-validation).
|
||||
|
||||
## Code layout
|
||||
|
||||
Paths below are relative to the repository root.
|
||||
|
||||
| Path | Responsibility |
|
||||
| --- | --- |
|
||||
| `src/routes` | Pages and API endpoints |
|
||||
| `src/lib/ui` | Shared components, stores and [UI patterns](ui-patterns.md) |
|
||||
| `src/lib/server/hosts` | Enrollment, readiness and command transport |
|
||||
| `src/lib/server/backends` | Instance identity and remote file access |
|
||||
| `src/lib/server/provisioning/native.ts` | Lifecycle and campaign queues connected to agents |
|
||||
| `src/lib/server/game-files` | Immutable uploads and streaming distribution |
|
||||
| `src/lib/server/db` | SQLite schema, migrations and shared artifact storage |
|
||||
| `src/lib/server/reports` | Metrics, leaderboard and capacity workers |
|
||||
| `agent` | Python service, installer, packaging and tests |
|
||||
|
||||
## Performance diagnostics
|
||||
|
||||
Set `EC_PERF_LOG=true` to log slow requests, report computation/cache timings and
|
||||
event-loop delay. Responses expose `Server-Timing`; browser navigation records an
|
||||
`ec:navigation` performance measure. Report workers use separate read-only SQLite
|
||||
connections and bounded caches. Metrics snapshots can be reused for up to 30
|
||||
seconds when their inputs are unchanged.
|
||||
|
||||
Benchmark the leaderboard against an isolated synthetic database:
|
||||
|
||||
```sh
|
||||
npm run benchmark:leaderboard
|
||||
npm run benchmark:leaderboard -- --scenario all --baseline-ref <commit>
|
||||
```
|
||||
|
||||
The comparison checks response parity and warm query timings.
|
||||
|
||||
Set `EC_SLOW_SQL_MS=20` to log runtime statements taking at least 20 ms. Logging
|
||||
omits bound values and string literals and is disabled by default. Capture timings
|
||||
under the same fleet load before and after database changes. Inspect database
|
||||
size, large tables/indexes, WAL size and free pages inside the controller with:
|
||||
|
||||
```sh
|
||||
npm run diagnostics:database -- /data/cherry.db
|
||||
```
|
||||
|
||||
This opens an existing database read-only without running migrations. Substitute
|
||||
`./data/cherry.db` for a default local installation.
|
||||
|
||||
## Storage and upgrades
|
||||
|
||||
Game-file uploads live in dedicated filesystem storage. Script, plugin and
|
||||
configuration artifact bodies are stored once by SHA-256 in SQLite; versions,
|
||||
staging plans, rollback snapshots and launch commands reference those shared
|
||||
bytes. Startup transactionally backfills older rows. Current agents fetch
|
||||
assigned artifacts through an authenticated endpoint and verify their hashes;
|
||||
older agents receive inline content until they update.
|
||||
|
||||
Back up the persistent data directory before upgrading. Startup logs show
|
||||
migration phases, elapsed time and resident memory. Completed migrations are
|
||||
retained across restarts; interrupted transactions roll back and retry on startup.
|
||||
The one-time vacuum needs disk space for a temporary database copy. **Do not
|
||||
remove `cherry.db-wal` to recover from an interrupted upgrade.**
|
||||
|
||||
The controller uses a 128 MiB page cache; each reader worker uses 32 MiB.
|
||||
Main-database migrations use 16 MiB with memory mapping disabled. Temporary
|
||||
storage stays disk-backed. Containers use `/data/.sqlite-tmp` for scratch files;
|
||||
override `SQLITE_TMPDIR` only with another writable, disk-backed directory.
|
||||
|
||||
For a local data disk, `EC_SQLITE_MMAP_MB=256` enables memory-mapped reads; leave it
|
||||
at zero for network mounts. Query statistics are optimized at startup and hourly.
|
||||
WAL reuse is capped at 64 MiB after checkpoints, though active transactions can
|
||||
temporarily grow it larger.
|
||||
@@ -0,0 +1,40 @@
|
||||
# UI patterns
|
||||
|
||||
Use the shared components in `src/lib/ui/components` to keep pages consistent.
|
||||
Primitives live in its `primitives` directory.
|
||||
|
||||
## Page structure and controls
|
||||
|
||||
- Use `SectionHeader` for page titles and primary actions. The page owns spacing
|
||||
between sections.
|
||||
- Use `Field` with `Input`, `Select` or `Textarea` for labels, help and errors.
|
||||
Give controls distinct `ariaLabel` values when several share one field.
|
||||
- Pair `Tabs` with a labeled panel. Use `ListFooter` for long lists.
|
||||
- Use `StatusBadge` for workflow states and `ServerIdentity` for server names.
|
||||
Keep raw IDs available for copying and troubleshooting.
|
||||
|
||||
## Dialogs and feedback
|
||||
|
||||
Use `Dialog` for modal workflows and `confirmAction` from `src/lib/ui/confirm.ts`
|
||||
for ordinary confirmations. Browser unload protection remains native.
|
||||
|
||||
Keep loaded data visible when a refresh fails, show the error and provide a
|
||||
retry action. Distinguish loading, empty and failed states so operators know
|
||||
whether to wait or act.
|
||||
|
||||
## Live data
|
||||
|
||||
Use `pollVisible` from `src/lib/ui/poll.ts` for scheduled page refreshes. It avoids
|
||||
overlapping requests, pauses while the page is hidden and returns a cleanup
|
||||
function for use when the component unmounts.
|
||||
|
||||
Read live server data through `src/lib/ui/stores/fleet.ts` instead of adding
|
||||
another `/api/servers` polling loop. Server detail views use a scoped Fleet store
|
||||
so their snapshots do not replace the shared fleet.
|
||||
|
||||
## Accessibility and small screens
|
||||
|
||||
Preserve visible keyboard focus, associated field labels and separate navigation
|
||||
and action targets. Avoid nested buttons or placing a button inside a link.
|
||||
Allow long server identities to wrap and tab/filter rows to scroll on narrow
|
||||
screens.
|
||||
@@ -67,4 +67,4 @@ Measurement totals exclude collection gaps. Request-to-verified-readiness summar
|
||||
|
||||
`POST /api/autoscaling` accepts `preview`, `save`, `delete`, or `global-pause`. Delete requires `id` and the current `revision`. Preview/save requests contain `settings`, plus `id` and `revision` when editing. Use `mapSlug: '*'` for every map in the selected game; existing map-specific requests are unchanged. Scope cannot change when editing. Save also requires the preview `token`. Global pause uses a boolean `paused` field. Agent removal claims use a separate host-authenticated endpoint and are bound to the exact command, host, instance generation, policy revision, and expiry.
|
||||
|
||||
Shared contracts live in `src/lib/autoscaling.ts`; the evaluator, snapshots, persistence, ownership, and authorization live in `src/lib/server/autoscaling/`. Provisioning remains the only lifecycle executor. See the [approved implementation plan](lobby-autoscaling-plan.md) for the design and rollout criteria.
|
||||
Shared contracts live in `src/lib/autoscaling.ts`; the evaluator, snapshots, persistence, ownership, and authorization live in `src/lib/server/autoscaling/`. Provisioning remains the only lifecycle executor. See the [approved implementation plan](../plans/lobby-autoscaling-plan.md) for the design and rollout criteria.
|
||||
@@ -20,6 +20,8 @@ and use a different secret from host-agent credentials or account passwords.
|
||||
The JSON event format is unchanged.
|
||||
There are four events. Send them whether or not a beta is active.
|
||||
|
||||
### Event fields
|
||||
|
||||
| Event | Fields |
|
||||
| --- | --- |
|
||||
| `game_start` | `server`, `map`, `players` with each player's `name` and `guid` |
|
||||
@@ -97,6 +99,8 @@ Send the final round and each player's final stats when the game ends.
|
||||
}
|
||||
```
|
||||
|
||||
### Responses and delivery
|
||||
|
||||
Success returns HTTP 200 with `{ "ok": true, "telemetry": { ... } }`.
|
||||
The returned telemetry contains the event and an Electric Cherry `receivedAt`
|
||||
timestamp. A missing/invalid machine token returns 401; an unset or invalid
|
||||
@@ -119,6 +123,8 @@ Reports also feed the persistent [Leaderboard](leaderboard.md), independently
|
||||
of beta activity. That page documents cumulative stat accounting, round credit,
|
||||
and the limits of identifying matches from this protocol.
|
||||
|
||||
### Reporter deployment
|
||||
|
||||
The game scripts are uploaded artifacts, not source maintained in this
|
||||
repository. Their reporter should emit the four payloads above and add the
|
||||
authentication header. Existing anonymous reporters must be updated before
|
||||
@@ -215,12 +221,21 @@ The test window starts when the beta opens or a revision is submitted, including
|
||||
time waiting for empty servers. Missing evidence at the deadline produces **inconclusive**.
|
||||
Health monitoring continues after a pass until promotion.
|
||||
|
||||
### Failure and rollback
|
||||
|
||||
For failures, select hold as failed, pause for investigation, or automatic
|
||||
rollback. Rollback queues restoration only for files touched by the beta.
|
||||
Restoration and restart both wait for an empty server and restart capacity.
|
||||
The manual option holds those changes in a rollback campaign until you select
|
||||
**Resume when empty**. Promotion is always manual.
|
||||
|
||||
### Promotion
|
||||
|
||||
Review the promotion preview before confirming. API clients fetch it with
|
||||
`GET /api/betas/:id/promote` and send its `previewToken` to the same route with
|
||||
`POST`. If the reviewed scope or configuration has changed, the request returns
|
||||
409 and requires a fresh preview.
|
||||
|
||||
Confirming promotion creates a campaign without reading the existing files on
|
||||
additional servers. Each server waits for an empty lobby and restart capacity,
|
||||
then captures rollback snapshots immediately before applying its queued files
|
||||
@@ -233,6 +248,8 @@ Withdrawing changes that have not activated needs no snapshot and leaves those
|
||||
servers' files untouched. If activation was submitted but its snapshot has not
|
||||
been confirmed, reconcile the provisioning operation before rolling back.
|
||||
|
||||
### Pause, resume and retry
|
||||
|
||||
Pause holds evaluation and wave advancement. Resume retains completed progress and the
|
||||
current revision's deadline, but starts new timing intervals. A fresh attempt resets
|
||||
progress and restarts deployed servers when empty. Recovery cycles repeat
|
||||
@@ -1,6 +1,6 @@
|
||||
# Deployment collections
|
||||
|
||||
Open **Operations → Collections** to manage reusable server lineups. Use
|
||||
Open **Fleet → Collections** to manage reusable server lineups. Use
|
||||
**Active launches** to follow progress and **Launch history** to review completed
|
||||
launches and failures. Both views refresh automatically. A
|
||||
collection describes a whole lineup: for example, two Kino lobbies, one Moon lobby,
|
||||
@@ -10,6 +10,8 @@ Collections save game/map selections and lobby counts. Every new server uses the
|
||||
latest promoted scripts, plugins, baseline, and matching configuration overlays
|
||||
when provisioning begins.
|
||||
|
||||
## How lineup counts work
|
||||
|
||||
For example, a collection targeting four Kino and two Der Riese lobbies, with
|
||||
two Kino already registered, creates two Kino and two Der Riese lobbies.
|
||||
The preview shows **Target / Existing or queued / To create** for each map.
|
||||
@@ -25,7 +27,7 @@ It fills deficits without changing existing servers' files.
|
||||
|
||||
## Build and launch a collection
|
||||
|
||||
1. Choose **Create collection** on **Operations → Collections** and name the lineup.
|
||||
1. Choose **Create collection** on **Fleet → Collections** and name the lineup.
|
||||
2. Add maps from Black Ops I or II and set the desired total lobbies for each.
|
||||
3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
|
||||
Choose **Fill missing servers** or **Match collection exactly**, then review
|
||||
@@ -67,6 +69,8 @@ tracks creations and removals, including removal-only plans, and lists the serve
|
||||
that were retained. The existing operation detail page offers removal cancellation
|
||||
and cleanup recovery where applicable.
|
||||
|
||||
### Collection limits and editing
|
||||
|
||||
Collections support up to 40 distinct game/map selections, 1–20 lobbies per map,
|
||||
and 100 servers total. The host's configured port range and existing fleet may
|
||||
allow fewer. A collection can be saved even when there is insufficient capacity
|
||||
@@ -80,6 +84,8 @@ Collections saved by the earlier single-server release load as one-server lineup
|
||||
and can be expanded directly. Any file snapshots in older collections are ignored
|
||||
when preparing new launches.
|
||||
|
||||
### Preview and submission
|
||||
|
||||
A native collection preview reads a shared capacity snapshot, maintained from host
|
||||
heartbeats and fleet changes. It estimates available ports, CPU, memory, and startup
|
||||
slots without running placement for each server. Missing or insufficient capacity
|
||||
@@ -102,11 +108,13 @@ identities change before submission, the request is rejected without changes and
|
||||
requires a fresh preview. Previously saved additive previews are checked against
|
||||
the current shortfall too.
|
||||
|
||||
### Startup queue and progress
|
||||
|
||||
All collections and individual launches share one startup queue, ordered by request
|
||||
time. By default, one server starts at a time. Its startup slot remains occupied
|
||||
until the host agent confirms container health, game status and the selected map
|
||||
are verified, and RCON reports a current Plutonium revision.
|
||||
Only then can the next queued server start. Set **Concurrent server startups** in **Settings → Fleet settings** to
|
||||
Only then can the next queued server start. Set **Concurrent server startups** in **Administration → Settings** to
|
||||
a positive integer to allow more simultaneous startups. Each worker pass admits
|
||||
at most the available slots, even if an attempt fails immediately.
|
||||
|
||||
@@ -124,7 +132,7 @@ Every server links to its Provisioning operation timeline and recovery options,
|
||||
and each child operation links back to its collection launch. Individual operations
|
||||
remain visible in Provisioning activity and history.
|
||||
|
||||
## Create and review a launch
|
||||
## Create and review a single-server launch
|
||||
|
||||
1. Choose **Single server** on Provisioning (or the deploy action on the fleet).
|
||||
2. Choose the current promoted setup and select a game/map, or select a collection.
|
||||
@@ -261,5 +269,4 @@ there while preserving query parameters.
|
||||
`canResumeVerification` for lifecycle operations.
|
||||
- `POST /api/deployments/:id/resume`: resume eligible creation health checks.
|
||||
|
||||
Scheduled game nights and beta-gated expansion remain future layers. Collection
|
||||
launches begin processing immediately after all requests are queued.
|
||||
Collection launches begin processing immediately after all requests are queued.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Getting started
|
||||
|
||||
Electric Cherry runs a web controller and an agent on each game host. The
|
||||
controller stores fleet metadata in SQLite and uploaded game files on disk.
|
||||
Agents run Docker/Plutainer locally and connect outbound to the controller over
|
||||
HTTPS. Running game containers continue through controller or agent outages.
|
||||
|
||||
## Requirements
|
||||
|
||||
- A controller machine with Docker Compose and an HTTPS reverse proxy.
|
||||
- Debian 12 or 13 amd64 game hosts with storage for game files and instances.
|
||||
- Your complete T5 or T6 game folders, including the DLC you intend to host.
|
||||
- Plutonium server keys for the exact lobby identities you will launch.
|
||||
|
||||
The controller does not need access to a Docker socket, game-host mounts, or
|
||||
inbound RCON. Host agents handle local game commands and file operations.
|
||||
|
||||
## Start the controller
|
||||
|
||||
From the repository root:
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
Set these values in `.env`:
|
||||
|
||||
| Variable | Purpose |
|
||||
| --- | --- |
|
||||
| `EC_PUBLIC_URL` | The external HTTPS address of the dashboard. |
|
||||
| `EC_SETUP_TOKEN` | The generated secret, used to create the first administrator. |
|
||||
| `EC_DATA_DIR` | A persistent directory for the database, game files, and beta archives. |
|
||||
| `EC_TELEMETRY_TOKEN` | A separate secret of at least 32 characters for game reporters. Generate it independently with the same command. |
|
||||
|
||||
For an existing installation, retain its original data directory. Compose
|
||||
defaults to `/root/zombies/.electric-cherry`; choose a dedicated directory for a
|
||||
new controller. See [.env.example](../../.env.example) for the other settings.
|
||||
|
||||
```sh
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
Proxy the HTTPS address to controller port 3000. Open the dashboard and follow
|
||||
`/setup` to create the first administrator. Then remove `EC_SETUP_TOKEN` from
|
||||
`.env` and run `docker compose up -d` to recreate the controller. Setup stays
|
||||
closed once an account exists.
|
||||
|
||||
Add other users under **Administration → Users**. The
|
||||
[access guide](authentication.md) explains roles, proxy configuration, and
|
||||
password recovery.
|
||||
|
||||
## Prepare a game host
|
||||
|
||||
1. Open **Library → Game Files** and upload the complete local folder for T5 or
|
||||
T6, including DLC. Uploads can run before any host is enrolled. The latest
|
||||
verified upload becomes that game's default files.
|
||||
2. Open **Fleet → Hosts → Add Host**, configure the host's game-port range and
|
||||
bind address, and run the generated enrollment command on the host.
|
||||
3. Wait for automatic setup to prepare the Plutainer runtime and available game
|
||||
files. The host page reports download, verification, and readiness progress.
|
||||
|
||||
The installer sets up the Python agent as a systemd service and enables automatic
|
||||
agent updates. It installs Docker if needed. Multiple instances share read-only
|
||||
base game files; each instance has its own writable data. See
|
||||
[Native hosts](native-hosts.md) for storage layout, network access, and recovery.
|
||||
|
||||
## Launch your first server
|
||||
|
||||
1. Save the exact Plutonium server keys under **Administration → Keys**.
|
||||
2. Create a server, or build a reusable map lineup in **Fleet → Collections**.
|
||||
3. Review the launch and apply it. Provisioning queues the work, then assigns a
|
||||
host and port when capacity is available.
|
||||
4. Follow the operation until readiness checks pass, then verify that players
|
||||
can find and join the server.
|
||||
|
||||
New launches use the latest verified game files and promoted configuration,
|
||||
scripts, and plugins. Uploading newer game files does not restart existing
|
||||
servers. Disruptive operations normally wait for a fresh empty lobby or a
|
||||
verified stopped container. A rebuild removes the instance's writable files.
|
||||
See [Provisioning](provisioning.md) and [Collections](collections.md) for details.
|
||||
|
||||
## Connect game telemetry
|
||||
|
||||
Configure your game reporter or plugin to send `EC_TELEMETRY_TOKEN` in an
|
||||
authorization header to `POST /api/telemetry`. This credential is separate from
|
||||
browser login, setup, and host enrollment. Ingestion is disabled until the
|
||||
controller has a valid telemetry token.
|
||||
|
||||
The [game reporter protocol](beta-test-plans.md#game-reporter-protocol) defines
|
||||
events and payloads for gameplay statistics and beta-test evidence. Electric
|
||||
Cherry does not include the game reporter scripts themselves.
|
||||
|
||||
## Maintain the installation
|
||||
|
||||
Back up the complete controller data directory, including SQLite metadata and
|
||||
published game files. Preserve that directory across upgrades. Review
|
||||
[Maintenance](maintenance.md) for scheduled work and
|
||||
[Fleet settings](settings.md) for concurrency and retention controls.
|
||||
|
||||
For lobby autoscaling, start with the documented
|
||||
[dry run and pilot](autoscaling.md#dry-run-and-pilot) before enabling live actions.
|
||||
@@ -5,6 +5,9 @@ with one entry per GUID combining Black Ops I (T5) and Black Ops II (T6). Game
|
||||
and map filters limit which matches contribute to each player's statistics and
|
||||
rank. Choose a ranking, search a name or GUID, and share the page URL with its
|
||||
filters intact. Results refresh while the page is visible.
|
||||
|
||||
## Qualifying matches
|
||||
|
||||
Matches only appear in server/global history once they reach round 2.
|
||||
Leaderboards, summaries, map options, profiles and player history include only
|
||||
completed matches with a received `game_end`. Active and interrupted matches do
|
||||
@@ -13,6 +16,10 @@ left before a completed game's end retain their recorded stats.
|
||||
This also excludes existing round 1 records. Early telemetry is retained so
|
||||
qualifying games keep their start time, roster and cumulative counters.
|
||||
|
||||
## Player profiles and match results
|
||||
|
||||
### Profiles and filters
|
||||
|
||||
Click a player's name to open their shareable profile at
|
||||
`/leaderboard/players/[guid]`. Profiles combine Black Ops I (T5) and Black Ops II
|
||||
(T6) records for the same GUID. Existing `/leaderboard/players/[game]/[guid]`
|
||||
@@ -42,6 +49,8 @@ profiles retain the complete tables; both views use the same history filters
|
||||
and pagination. History dates say **First observed**, since a missing start
|
||||
report can make the first recorded observation later than the actual game start.
|
||||
|
||||
### Record evidence and match results
|
||||
|
||||
Highest-round, best-game-kill and highest-score records link to the completed
|
||||
match that earned them on the leaderboard and in lifetime/per-map profile stats.
|
||||
Each source shows its map, game, completion time, server, and whether the match
|
||||
@@ -59,6 +68,8 @@ score are the highest reported values within the match, not necessarily its fina
|
||||
scoreboard. Active and interrupted matches remain listed without results links.
|
||||
Match pages remain available after the original server is removed.
|
||||
|
||||
### Player identity
|
||||
|
||||
The leaderboard uses the existing telemetry events; no reporter change is
|
||||
required. The leaderboard and profiles identify a player by **GUID**, displaying
|
||||
the latest reported name across both games, regardless of game or map filters.
|
||||
@@ -91,6 +102,8 @@ player-rounds: four players progressing from game start to round 8 contribute 28
|
||||
Highest-round records always use the round reached in a completed match,
|
||||
including the final round and players who joined mid-game.
|
||||
|
||||
### Efficiency rankings
|
||||
|
||||
Efficiency rankings are also available for **Headshots / kills**, **Kills per
|
||||
down**, **Average kills**, **Average rounds**, and **Completion rate**. Each
|
||||
requires at least five completed games in the selected game/map scope. Headshot
|
||||
@@ -106,6 +119,8 @@ still describe all players in the game/map selection, including those below the
|
||||
threshold. Profile metrics show their values and whether the player qualifies
|
||||
for the combined lifetime board; a scoped board can have a smaller sample.
|
||||
|
||||
### Participation limits
|
||||
|
||||
Round totals require the player to appear in consecutive observations. If a
|
||||
report omits them, their return starts a new participation interval and the gap
|
||||
is not credited. The payload cannot tell whether a present player was alive,
|
||||
@@ -150,10 +165,10 @@ would require additional reporter identity/ordering information.
|
||||
|
||||
This reference describes the current repository implementation. A deployed
|
||||
instance must run the corresponding code to provide the same behavior. The
|
||||
sources are the [HTTP handler](../src/routes/api/leaderboard/+server.ts),
|
||||
[response types](../src/lib/leaderboard.ts),
|
||||
[query implementation](../src/lib/server/fleet/leaderboard.ts), and
|
||||
[authentication guard](../src/lib/server/auth/guard.ts).
|
||||
sources are the [HTTP handler](../../src/routes/api/leaderboard/+server.ts),
|
||||
[response types](../../src/lib/leaderboard.ts),
|
||||
[query implementation](../../src/lib/server/fleet/leaderboard.ts), and
|
||||
[authentication guard](../../src/lib/server/auth/guard.ts).
|
||||
|
||||
### Authentication and requests
|
||||
|
||||
@@ -322,7 +337,7 @@ summaries as `{ "matches": [...], "total": number }`. These routes still require
|
||||
session authentication. Both accept only `limit` (default 25, range 1–10,000); there is
|
||||
no offset pagination. Results are newest first by first observation, then match
|
||||
ID, and include active/interrupted matches that reached round 2. Their
|
||||
[ServerMatch fields](../src/lib/match-history.ts) differ from leaderboard entries
|
||||
[ServerMatch fields](../../src/lib/match-history.ts) differ from leaderboard entries
|
||||
and do not include player counters or a roster. Unknown server names return an
|
||||
empty list. URL-encode server names in paths.
|
||||
|
||||
@@ -344,47 +359,5 @@ Exact player-rank reads use the same ranked scope, including its summaries and
|
||||
map list. Report work runs in a dedicated worker with revision-keyed response
|
||||
caching as described above.
|
||||
|
||||
The benchmark and measurements below describe the earlier SQL optimization,
|
||||
before the worker and revision caches were added. They are historical results,
|
||||
not measurements of the current HTTP implementation.
|
||||
|
||||
Run the deterministic, isolated benchmark with:
|
||||
|
||||
```sh
|
||||
node scripts/benchmark-leaderboard.mjs --scenario all --baseline-ref eac9ad4 --runs 3 --json /tmp/leaderboard-benchmark.json
|
||||
```
|
||||
|
||||
The script creates temporary SQLite databases, loads the actual query modules,
|
||||
checks complete-response parity against the selected Git revision, and reports
|
||||
timings, source hashes and query plans. It never opens the application database.
|
||||
The small fixture has 2,000 matches and 8,000 participation rows; the large fixture
|
||||
has 50,000 matches and 200,000 participation rows. Both include completed,
|
||||
active, interrupted and nonqualifying records.
|
||||
|
||||
On the local Intel i5-8279U / Node 25.9.0 / SQLite 3.49.2 test environment, three
|
||||
measured runs after warmup gave these median query times on the large fixture:
|
||||
|
||||
| Request | Before | After |
|
||||
| --- | ---: | ---: |
|
||||
| Highest round, all games | 1,717 ms | 829 ms |
|
||||
| Total kills, all games | 1,555 ms | 858 ms |
|
||||
| One game | 899 ms | 491 ms |
|
||||
| One map | 345 ms | 224 ms |
|
||||
| Player search | 1,515 ms | 757 ms |
|
||||
| Deep page | 1,421 ms | 752 ms |
|
||||
| Headshot percentage | 1,444 ms | 760 ms |
|
||||
| Exact player rank | 1,524 ms | 554 ms |
|
||||
|
||||
The small fixture's unfiltered request fell from 41 to 30 ms. All 150 additional
|
||||
small-fixture parity cases and 22 timed small/large request cases matched the
|
||||
baseline's complete responses, including ratios and record evidence. Timings
|
||||
include SQL preparation and evidence lookup, exclude HTTP/browser rendering, and
|
||||
use a warm filesystem cache with alternating implementation order. These are
|
||||
local synthetic measurements, not production latency guarantees. SQL queries
|
||||
run synchronously within the report worker; larger histories still affect
|
||||
report latency and worker capacity.
|
||||
|
||||
Validation for that optimization passed 1,019 unit tests, Svelte checking with no errors
|
||||
or warnings, and a production build. Isolated Firefox walkthroughs at 1440px and
|
||||
390px checked completed-only totals, record links, efficiency qualification,
|
||||
scoped ranks/history, preserved navigation, empty scopes and mobile expansion.
|
||||
See the [historical SQL benchmark](../research/leaderboard-query-benchmark.md)
|
||||
for the earlier optimization measurements and methodology.
|
||||
@@ -3,6 +3,8 @@
|
||||
Metrics is under **Analytics** at `/metrics` (keyboard shortcut: `g m`).
|
||||
Choose the past 24 hours, 7 days, or 30 days. The page refreshes every minute.
|
||||
|
||||
## Overview and filters
|
||||
|
||||
The overview shows **Time with players** (occupied server-hours divided by known
|
||||
server-hours) and **Player-hours**. Known and unknown status time remain visible
|
||||
alongside the totals. **Last collected** is the newest recorded batch, including
|
||||
@@ -18,6 +20,8 @@ Game and map filters apply to the whole report, including its comparisons,
|
||||
recommendations, and heatmap. Map choices preserve the exact recorded game/map
|
||||
pair. Range and scope are shareable through the URL. Table searches remain local.
|
||||
|
||||
## Timeline and tables
|
||||
|
||||
The timeline defaults to **Share of captured time**: each stack includes occupied,
|
||||
idle, and unknown server time, normalized to 100%. This denominator differs from the
|
||||
headline utilization percentage, which excludes unknown time. Switch to **Hours**
|
||||
@@ -35,6 +39,8 @@ name, which may include different instances if a name was reused. **Open current
|
||||
server** appears only when that name is currently registered; its destination
|
||||
represents the current registration rather than every instance in the history.
|
||||
|
||||
## Comparisons and hosting experiments
|
||||
|
||||
**Explore this period** adds experimental decision aids below the main timeline:
|
||||
|
||||
- **Map capacity pressure** pools all-full map-hours divided by known-capacity
|
||||
@@ -72,6 +78,8 @@ occupancy does not establish failed joins or a need for additional capacity.
|
||||
Suggestions use historical observations. Check the current lineup before trying
|
||||
a change; a retained map may no longer have a hosted lobby.
|
||||
|
||||
## Definitions and collection
|
||||
|
||||
- Utilized means a successful status response reported at least one player.
|
||||
- Idle means a successful response reported zero players.
|
||||
- Unknown covers failed checks, provisioning, and unavailable containers; it is excluded from utilization percentages.
|
||||
@@ -84,6 +92,8 @@ Capacity comes from `sv_maxclients` / `sv_maxClients` in the status response. Wh
|
||||
|
||||
Collection begins after this feature is installed; previous occupancy cannot be reconstructed. Samples default to 31 days of retention; Administration → Settings can change this period. Cleanup runs during collection and in the background. They survive removal of a server and contain no player identities. The additive database table is created automatically on startup.
|
||||
|
||||
## API
|
||||
|
||||
`GET /api/metrics` supplies the same report as the page and disables response
|
||||
caching. Query parameters are `days` (1, 7, or 30), `game`, `map` (requires game),
|
||||
and `timeZone` (an IANA zone). Invalid ranges and zones fall back to the defaults;
|
||||
@@ -4,13 +4,6 @@ Electric Cherry controls Docker/Plutainer through an outbound host agent. The co
|
||||
does not need mounted game-server files. It owns desired state, campaigns, collections, safety queues and history;
|
||||
the agent owns local execution and its durable operation journal.
|
||||
|
||||
Fleet health checks query RCON even when container inspection is unavailable. A
|
||||
fresh successful RCON reply can establish game health while container state stays
|
||||
unknown; confirmed stopped, unhealthy, and restarting states still take precedence.
|
||||
Agent 1.4.4 retries transient failures of the read-only `status` query up to three
|
||||
times, with 250 ms and 500 ms delays. Commands that change the game are not retried.
|
||||
Status replies still expire normally, and native timeout diagnostics are retained.
|
||||
|
||||
## Install and launch
|
||||
|
||||
1. Run the controller with persistent `/data` storage. Set `EC_PUBLIC_URL` to its
|
||||
@@ -45,6 +38,8 @@ Status replies still expire normally, and native timeout diagnostics are retaine
|
||||
A fresh destination host downloads centrally stored game files without manual
|
||||
copying or per-container setup. EC does not supply the game content itself.
|
||||
|
||||
### Docker installation
|
||||
|
||||
The installer uses Debian's Docker packages: `docker.io` on Debian 12, plus
|
||||
`docker-cli` on Debian 13. Rerunning it after an earlier Debian 13 install that
|
||||
omitted the client adds the missing client without restarting the existing engine.
|
||||
@@ -52,6 +47,8 @@ An already available Docker installation (including Docker CE) is reused. If the
|
||||
daemon is unavailable, the installer reports the Docker error and stops before
|
||||
enrollment; inspect `systemctl status docker` and `journalctl -u docker`.
|
||||
|
||||
### Automatic host setup
|
||||
|
||||
The controller reconciles automatic setup on agent heartbeats. Command polls only
|
||||
deliver work already queued. Setup first pulls `PLUTAINER_IMAGE` (default
|
||||
`ghcr.io/ayymoss/plutainer:latest`), then prepares the latest verified upload for
|
||||
@@ -67,6 +64,8 @@ source details.
|
||||
Disconnected hosts resume when they
|
||||
reconnect. Draining and disabled hosts do not receive new automatic setup work.
|
||||
|
||||
### Operation history retention
|
||||
|
||||
**Settings → Routine operation history retention (days)** controls cleanup of
|
||||
successful monitoring and file-read records. The default is one day; zero keeps
|
||||
them indefinitely. Actions, failures, negative checks, and unresolved operations
|
||||
@@ -74,6 +73,8 @@ are retained. Agent 1.4.9 also cleans up successful routine journal records afte
|
||||
the controller confirms their final receipt. Older unconfirmed journal entries
|
||||
remain until a replay supplies that confirmation. Cleanup runs in bounded batches.
|
||||
|
||||
### New game files and cached versions
|
||||
|
||||
An automatic download that has not been delivered to the agent can be replaced
|
||||
when a newer upload becomes available. A delivered operation finishes safely
|
||||
before the host prepares the latest files. Failed or incomplete uploads never
|
||||
@@ -86,6 +87,8 @@ cached mutable image tag. Advanced controls on the
|
||||
host and game-file pages can explicitly pull a custom runtime, prepare an older
|
||||
upload or retry preparation immediately.
|
||||
|
||||
### Storage layout
|
||||
|
||||
The default host root is `/var/lib/electric-cherry`; installer `--data-dir` changes
|
||||
it. Use a dedicated, root-owned directory. For a separate disk, use a child of its
|
||||
mount point (for example `/mnt/games/electric-cherry`), with no nested mounts, so
|
||||
@@ -136,6 +139,29 @@ shows **Needs attention**. The updater itself is a stable bootstrap component:
|
||||
it changes only when the installer is rerun. If a future release requires an
|
||||
unsupported updater protocol, rerun that release's installer on the host.
|
||||
|
||||
### Publishing releases and inspecting updates
|
||||
|
||||
When publishing changed agent code, bump its numeric `VERSION` in
|
||||
`agent/ec_agent.py` and rebuild the controller. Packaging derives the manifest
|
||||
version and checksums from the sources; an already installed version is never
|
||||
replaced just because its bytes changed.
|
||||
|
||||
The installer places the updater at
|
||||
`/usr/local/libexec/electric-cherry-update.py` and enables
|
||||
`electric-cherry-update.timer`, which starts the oneshot
|
||||
`electric-cherry-update.service`. Its state is kept under `/etc/electric-cherry`,
|
||||
including `update-status.json`, `update/journal.json`, `agent-health.json` and
|
||||
the shared `update.lock`. Check the timer and worker logs on the host with:
|
||||
|
||||
```sh
|
||||
systemctl status electric-cherry-update.timer
|
||||
journalctl -u electric-cherry-update.service -n 100
|
||||
```
|
||||
|
||||
## Placement and resource capacity
|
||||
|
||||
### Host inventory and eligibility
|
||||
|
||||
The Inventory panel shows CPU usage with agent 1.4.2 and later. It measures
|
||||
host-wide busy time across all cores on a 0–100% scale between inventory samples
|
||||
(normally about 15 seconds). Idle and I/O wait are excluded; stolen CPU time
|
||||
@@ -161,6 +187,8 @@ or on an active host for automatic placement, before a launch is accepted. Admin
|
||||
can prepare them from the host page. Queued launches and migrations never trigger
|
||||
custom image pulls, and container creation cannot implicitly pull a missing image.
|
||||
|
||||
### Resource samples and learned allowances
|
||||
|
||||
Only provisioning reserves a host and port. Agent **1.4.6** fixes container
|
||||
discovery for resource sampling (the
|
||||
earlier collector omitted `label=` from its Docker filter). It samples each
|
||||
@@ -201,6 +229,8 @@ Models expire after 24 hours without measurements. Recalculation happens during
|
||||
heartbeat ingestion, at most once per minute per host/fleet phase; new peaks
|
||||
raise the cache immediately. Launch admission reads only the cached model.
|
||||
|
||||
### Admission and placement
|
||||
|
||||
Scheduling still leaves **5% host CPU and 500 MB RAM (500,000,000 bytes)** for the OS and bursts.
|
||||
These allowances are scheduling estimates, not Docker limits. Collection previews
|
||||
estimate how many new servers fit at their running allowance. Jobs may still be
|
||||
@@ -232,23 +262,6 @@ creation has been submitted, placement stays fixed to prevent duplicate instance
|
||||
Cancelling an unsubmitted launch releases its reservation; uncertain remote work
|
||||
retains capacity until cleanup is confirmed.
|
||||
|
||||
When publishing changed agent code, bump its numeric `VERSION` in
|
||||
`agent/ec_agent.py` and rebuild the controller. Packaging derives the manifest
|
||||
version and checksums from the sources; an already installed version is never
|
||||
replaced just because its bytes changed.
|
||||
|
||||
The installer places the updater at
|
||||
`/usr/local/libexec/electric-cherry-update.py` and enables
|
||||
`electric-cherry-update.timer`, which starts the oneshot
|
||||
`electric-cherry-update.service`. Its state is kept under `/etc/electric-cherry`,
|
||||
including `update-status.json`, `update/journal.json`, `agent-health.json` and
|
||||
the shared `update.lock`. Check the timer and worker logs on the host with:
|
||||
|
||||
```sh
|
||||
systemctl status electric-cherry-update.timer
|
||||
journalctl -u electric-cherry-update.service -n 100
|
||||
```
|
||||
|
||||
## Capacity overview
|
||||
|
||||
Open **Capacity** (`/capacity`) to see the same cached estimates used by collection
|
||||
@@ -298,7 +311,7 @@ retried using the same proposed secret. Revocation affects one host only and doe
|
||||
not stop its running game containers. Re-enrollment requires a fresh token and an
|
||||
explicit installer `--reenroll` when local credentials already exist.
|
||||
|
||||
API outline:
|
||||
## Host and game-file API
|
||||
|
||||
- `GET /api/hosts/:id`: host details and deletion blockers.
|
||||
- `GET /api/hosts/operations?page=1&filter=important`: operation history across
|
||||
@@ -352,6 +365,8 @@ Otherwise the result remains **Unknown** and holds dependent work. Running alone
|
||||
is insufficient evidence that a particular restart happened. EC neither assumes
|
||||
an unreachable server stopped nor launches a duplicate replacement elsewhere.
|
||||
|
||||
### Player checks and activation
|
||||
|
||||
Campaign bytes are staged and verified while players remain connected. Only
|
||||
Provisioning can dispatch activation. It checks source eligibility and shared
|
||||
capacity, and the agent repeats a fresh local RCON occupancy check immediately
|
||||
@@ -377,6 +392,8 @@ restart and create complete only after post-operation health evidence; creation
|
||||
and rebuild also verify the selected map and running Plutonium revision. The
|
||||
base-game library and runtime revision are independent records.
|
||||
|
||||
## Drain and enable hosts
|
||||
|
||||
**Drain host** asks for confirmation, blocks new deployments, and migrates
|
||||
instances to other ready hosts with capacity. Each migration waits for an empty
|
||||
lobby, reserves destination CPU, memory and ports, removes the source instance,
|
||||
@@ -395,6 +412,8 @@ are downloaded as needed and fully verified, including cached copies. Both games
|
||||
must be published. Failed setup or stale inventory keeps the host disabled; only
|
||||
successful preparation and fresh inventory return it to Active.
|
||||
|
||||
## Delete a host
|
||||
|
||||
Administrators can choose **Delete host** on a host card or at the top of its
|
||||
detail page. The confirmation lists anything that must be resolved first:
|
||||
registered servers, reserved ports, or operations already sent to the agent.
|
||||
@@ -421,6 +440,8 @@ the deleted host's copies are removed. Controller operation history is also kept
|
||||
Unrelated containers, files and shared dependencies such as Docker and Python
|
||||
are preserved. Historical entries in shared system logs are not erased.
|
||||
|
||||
### Interrupted cleanup
|
||||
|
||||
Cleanup retries after connection loss or partial file removal. Failures before
|
||||
handoff offer **Retry cleanup** after the installation is repaired; failures in
|
||||
the independent worker retry automatically. The worker stays installed until
|
||||
@@ -429,7 +450,9 @@ retries the completion receipt from memory. A power loss in that final window
|
||||
can leave the controller awaiting confirmation even though removal succeeded;
|
||||
inspect the machine if that status persists. The controller does not guess.
|
||||
|
||||
## Game files and ports
|
||||
## Game files
|
||||
|
||||
### Upload a game folder
|
||||
|
||||
Game Files always shows T5 and T6, each with its own upload button and latest
|
||||
verified files. Successful uploads become latest in publication completion
|
||||
@@ -446,6 +469,8 @@ copy of the game directory, separately from the launcher/runtime. The uploader
|
||||
preserves the supplied files rather than attempting to infer installed maps from
|
||||
a manually maintained DLC list.
|
||||
|
||||
### Transfer and verification
|
||||
|
||||
Folder uploads use chunks of at most 8 MiB, each checked against its SHA-256 before
|
||||
being appended to temporary controller storage. Memory usage does not grow with
|
||||
the size of an individual game file. Pause/retry uses the controller's saved file
|
||||
@@ -473,6 +498,9 @@ timing reports still show byte progress. Agent 1.2.2 provides phase timing;
|
||||
automatic updates install it between host operations.
|
||||
Phase transitions are reported promptly and periodic heartbeats carry progress;
|
||||
controller round trips do not stall every transferred chunk.
|
||||
|
||||
### Cache cleanup and retention
|
||||
|
||||
Corrupt or incomplete content never becomes Ready. Cache inventory is reconciled
|
||||
on heartbeats, including vanished caches. The latest published version of each game
|
||||
is always retained. With agent 1.4.7 or newer, an older host copy is automatically
|
||||
@@ -491,6 +519,8 @@ cannot be downloaded again without a new upload. Cleanup runs on heartbeats,
|
||||
successful publication, and the controller background worker, without needing
|
||||
an open Game Files page.
|
||||
|
||||
### Proxy limits and older uploads
|
||||
|
||||
The controller and reverse proxy must accept 8 MiB upload requests and folder
|
||||
manifests up to 32 MiB. Compose sets `BODY_SIZE_LIMIT=Infinity`; application routes
|
||||
still enforce these limits, declared file sizes and checksums. Set a proxy body
|
||||
@@ -502,65 +532,15 @@ content still in use are preserved; unused older files are cleaned up automatica
|
||||
Legacy source-host imports retain their existing IDs and can complete through
|
||||
their original workflow.
|
||||
|
||||
## Ports and network bindings
|
||||
|
||||
Reservations include host ID, local bind address, protocol and port. Wildcard
|
||||
binds conflict with concrete addresses on the same host. UDP and TCP reservations
|
||||
for one launch commit together; the same number can be used independently on
|
||||
another host. Optional public/NAT address and port are recorded separately and do
|
||||
not change local Docker bindings. EC does not configure external NAT/firewalls.
|
||||
|
||||
## Development and validation
|
||||
|
||||
```sh
|
||||
npm run agent:package
|
||||
npm run check
|
||||
npm test
|
||||
npm run test:agent
|
||||
npm run build
|
||||
```
|
||||
|
||||
The build packages the agent, standalone updater and installer into the static
|
||||
release, generates the agent/updater checksum manifest and serves those same
|
||||
assets in the Node-adapter image. Development serves the freshly packaged static
|
||||
release even when an older production build exists. The Python programs use only
|
||||
the standard library.
|
||||
|
||||
Automated tests cover enrollment/authentication, rotation/revocation, host
|
||||
readiness, scoped ports, immutable library publication, streaming/ranges,
|
||||
checksums, interrupted downloads, instance generation ownership, local occupancy
|
||||
holds, controller restart/reconnect, ambiguous results, occupied campaign staging,
|
||||
activation before images, collections and draining. Host deletion tests cover
|
||||
ownership guards, interrupted cleanup, uninstall receipts, credential revocation
|
||||
and retention of controller uploads/history. Docker/systemd/UDP and host
|
||||
observations are simulated in these tests.
|
||||
|
||||
A real Debian/Docker/Plutonium acceptance run is still required before production
|
||||
rollout. This workspace has no Docker engine or Debian game host, so automated
|
||||
coverage is not evidence that a real T5/T6 server has been deployed here. Run:
|
||||
|
||||
- Installer on Debian 12 and 13 amd64, then rerun without changing identity/data.
|
||||
- Bootstrap an existing host with the current installer, then publish a newer
|
||||
compatible agent release. Verify a timer-driven update, its authenticated
|
||||
heartbeat and unchanged game containers. Exercise an active-operation hold,
|
||||
bad checksum, failed startup/heartbeat rollback, controller downgrade, network
|
||||
loss and host reboot during update. Verify updater protocol mismatch requests
|
||||
an installer rerun.
|
||||
- T5 and T6 deployments and a mixed collection from a fresh destination host.
|
||||
- Start, stop, restart, rebuild, destroy; verify RCON, map and runtime revision.
|
||||
- Read console and game logs, including Plutainer's relative game-log links.
|
||||
- File campaign and rollback, with a player joining between controller check and
|
||||
agent activation, and during file activation before restart; verify that the
|
||||
agent holds disruption.
|
||||
- Drain occupied lobbies and verify shared libraries survive their retirement.
|
||||
- Delete a drained host; verify its game copies, data directory, agent and cleanup
|
||||
services, update timer, configuration and executables are removed, controller uploads/history stay,
|
||||
and unrelated Docker workloads continue. Repeat with connection loss during
|
||||
cleanup and after the final receipt, and a reboot during data removal.
|
||||
- Corrupt/interrupted library downloads, resumed transfer and verified promotion.
|
||||
- Controller restart, agent restart and network loss before execution, during
|
||||
execution and after execution but before acknowledgement. Verify the same
|
||||
operation IDs and absence of duplicate containers/restarts.
|
||||
|
||||
### Provisioning progress and diagnostics
|
||||
## Provisioning progress and diagnostics
|
||||
|
||||
Native launches distinguish initial eligibility evaluation from an actual wait
|
||||
for startup capacity. The operation page records payload preparation, command
|
||||
@@ -575,6 +555,17 @@ The **Controller activity** panel shows current observations and their elapsed
|
||||
time while the operation is active. It is a live snapshot, not historical proof
|
||||
of which task delayed a previous deployment.
|
||||
|
||||
### Health checks
|
||||
|
||||
Fleet health checks query RCON even when container inspection is unavailable. A
|
||||
fresh successful RCON reply can establish game health while container state stays
|
||||
unknown; confirmed stopped, unhealthy, and restarting states still take precedence.
|
||||
Agent 1.4.4 retries transient failures of the read-only `status` query up to three
|
||||
times, with 250 ms and 500 ms delays. Commands that change the game are not retried.
|
||||
Status replies still expire normally, and native timeout diagnostics are retained.
|
||||
|
||||
### Background observations
|
||||
|
||||
Campaign, beta and maintenance observers run independently of provisioning.
|
||||
New requests and completed agent commands wake provisioning; an arrival during a
|
||||
pass schedules another pass. Routine file drift scans run every five minutes;
|
||||
@@ -611,7 +602,27 @@ Older completed operations whose progress snapshots were already discarded
|
||||
cannot recover missing check results from this change. Their surviving events
|
||||
and host command records remain available without inventing verification times.
|
||||
|
||||
### Cancel an unfinished creation
|
||||
### Read operations during downloads
|
||||
|
||||
Agent 1.4.0 separates read-only inspections, file/log reads, and the exact RCON
|
||||
commands `status`, `getstatus`, and `version` from the serial mutation queue.
|
||||
Two bounded read lanes can run during runtime or library downloads. Instance
|
||||
mutations exclude reads of that instance; credential rotation and host cleanup
|
||||
wait for active reads. Both lanes retain durable operation IDs, generation checks,
|
||||
and the updater activity lock. Older agents continue using the combined queue.
|
||||
|
||||
Agents long-poll `GET /api/agent/operations?lane=read|write&wait=20`; requests
|
||||
without those parameters retain immediate combined delivery. The controller wakes
|
||||
pending polls when work is queued and rechecks authentication after waiting.
|
||||
Long polls release the updater lock; execution reacquires it and rechecks update
|
||||
state before accepting work.
|
||||
|
||||
Browser game-folder uploads transfer up to three files concurrently. Chunks of
|
||||
one file remain serial and checksum-verified. Publication waits for admitted file
|
||||
transfers and prevents subsequent writes; pause/resume still uses acknowledged
|
||||
per-file offsets. Host downloads retain their existing bounded concurrency.
|
||||
|
||||
## Cancel an unfinished creation
|
||||
|
||||
On the native creation operation page, **Cancel creation and remove server**
|
||||
stops the launch and queues removal of its exact instance and generation. It
|
||||
@@ -647,22 +658,54 @@ advertises cancellation support, delivery resumes with cancellation cleanup
|
||||
first. This controller-side hold also recovers cancellations already waiting
|
||||
on older agents. Hosts without automatic updates need the current installer.
|
||||
|
||||
## Responsive observations (agent 1.4.0)
|
||||
## Development and validation
|
||||
|
||||
Agent 1.4.0 separates read-only inspections, file/log reads, and the exact RCON
|
||||
commands `status`, `getstatus`, and `version` from the serial mutation queue.
|
||||
Two bounded read lanes can run during runtime or library downloads. Instance
|
||||
mutations exclude reads of that instance; credential rotation and host cleanup
|
||||
wait for active reads. Both lanes retain durable operation IDs, generation checks,
|
||||
and the updater activity lock. Older agents continue using the combined queue.
|
||||
```sh
|
||||
npm run agent:package
|
||||
npm run check
|
||||
npm test
|
||||
npm run test:agent
|
||||
npm run build
|
||||
```
|
||||
|
||||
Agents long-poll `GET /api/agent/operations?lane=read|write&wait=20`; requests
|
||||
without those parameters retain immediate combined delivery. The controller wakes
|
||||
pending polls when work is queued and rechecks authentication after waiting.
|
||||
Long polls release the updater lock; execution reacquires it and rechecks update
|
||||
state before accepting work.
|
||||
The build packages the agent, standalone updater and installer into the static
|
||||
release, generates the agent/updater checksum manifest and serves those same
|
||||
assets in the Node-adapter image. Development serves the freshly packaged static
|
||||
release even when an older production build exists. The Python programs use only
|
||||
the standard library.
|
||||
|
||||
Browser game-folder uploads transfer up to three files concurrently. Chunks of
|
||||
one file remain serial and checksum-verified. Publication waits for admitted file
|
||||
transfers and prevents subsequent writes; pause/resume still uses acknowledged
|
||||
per-file offsets. Host downloads retain their existing bounded concurrency.
|
||||
Automated tests cover enrollment/authentication, rotation/revocation, host
|
||||
readiness, scoped ports, immutable library publication, streaming/ranges,
|
||||
checksums, interrupted downloads, instance generation ownership, local occupancy
|
||||
holds, controller restart/reconnect, ambiguous results, occupied campaign staging,
|
||||
activation before images, collections and draining. Host deletion tests cover
|
||||
ownership guards, interrupted cleanup, uninstall receipts, credential revocation
|
||||
and retention of controller uploads/history. Docker/systemd/UDP and host
|
||||
observations are simulated in these tests.
|
||||
|
||||
A real Debian/Docker/Plutonium acceptance run is still required before production
|
||||
rollout. Automated simulations do not establish successful deployment on real
|
||||
game hosts. Run:
|
||||
|
||||
- Installer on Debian 12 and 13 amd64, then rerun without changing identity/data.
|
||||
- Bootstrap an existing host with the current installer, then publish a newer
|
||||
compatible agent release. Verify a timer-driven update, its authenticated
|
||||
heartbeat and unchanged game containers. Exercise an active-operation hold,
|
||||
bad checksum, failed startup/heartbeat rollback, controller downgrade, network
|
||||
loss and host reboot during update. Verify updater protocol mismatch requests
|
||||
an installer rerun.
|
||||
- T5 and T6 deployments and a mixed collection from a fresh destination host.
|
||||
- Start, stop, restart, rebuild, destroy; verify RCON, map and runtime revision.
|
||||
- Read console and game logs, including Plutainer's relative game-log links.
|
||||
- File campaign and rollback, with a player joining between controller check and
|
||||
agent activation, and during file activation before restart; verify that the
|
||||
agent holds disruption.
|
||||
- Drain occupied lobbies and verify shared libraries survive their retirement.
|
||||
- Delete a drained host; verify its game copies, data directory, agent and cleanup
|
||||
services, update timer, configuration and executables are removed, controller uploads/history stay,
|
||||
and unrelated Docker workloads continue. Repeat with connection loss during
|
||||
cleanup and after the final receipt, and a reboot during data removal.
|
||||
- Corrupt/interrupted library downloads, resumed transfer and verified promotion.
|
||||
- Controller restart, agent restart and network loss before execution, during
|
||||
execution and after execution but before acknowledgement. Verify the same
|
||||
operation IDs and absence of duplicate containers/restarts.
|
||||
@@ -1,7 +1,7 @@
|
||||
# Provisioning
|
||||
|
||||
Open **Operations → Provisioning** for instance lifecycle work and persistent
|
||||
activity. **Operations → Collections** owns reusable lineups and collection launch
|
||||
activity. **Fleet → Collections** owns reusable lineups and collection launch
|
||||
progress; their individual operations remain visible in Provisioning.
|
||||
Provisioning owns the queues, safety checks, execution, and
|
||||
recovery for all six actions. Create a server using **Create server**. For an existing
|
||||
@@ -16,6 +16,10 @@ server, choose **Manage server** or open its **Server actions** menu.
|
||||
| Stop | Wait until empty, stop the container, and verify that the host agent reports it stopped. Keep the instance for a later Start. |
|
||||
| Destroy | Wait until empty, remove the container and its files through the host agent, then retire the instance and registration. |
|
||||
|
||||
## Safety and verification
|
||||
|
||||
### Beta ownership and rebuilds
|
||||
|
||||
Servers linked to a running or ready-to-promote beta cannot be destroyed or
|
||||
rebuilt. Promote or roll back the beta first, including paused or failed test
|
||||
plans. Fleet removal and exact collection launches queue nothing if any server
|
||||
@@ -37,6 +41,8 @@ receipt after reconnecting or restarting. An interrupted fresh startup can resum
|
||||
health verification without repeating the operation. Inspect failures before
|
||||
making another request.
|
||||
|
||||
### Health and runtime checks
|
||||
|
||||
Create and Rebuild finish only after RCON reports a runtime revision at least as
|
||||
new as the current Plutonium production release. The worker checks the production
|
||||
manifest and records the runtime revision for fleet and server version status as
|
||||
@@ -52,6 +58,8 @@ maintenance. A successful agent receipt is followed by actual container and game
|
||||
slot and are not automatically replayed. Stop requires an explicit stopped container state; an
|
||||
unreachable host agent is not evidence that a stop succeeded.
|
||||
|
||||
### Player checks and administrator overrides
|
||||
|
||||
Disruptive operations require a fresh zero-player result from RCON or accepted
|
||||
telemetry, unless the host agent positively reports a stopped container. Fresh occupied
|
||||
telemetry vetoes a conflicting empty RCON reading. Reports must be received within
|
||||
@@ -73,6 +81,8 @@ an update requirement. Submitted commands, host migrations, and completed operat
|
||||
cannot be overridden. Capacity, conflicting work, instance identity, file validation,
|
||||
and post-action health checks still apply.
|
||||
|
||||
## Campaigns, betas and maintenance
|
||||
|
||||
Campaigns, beta runs, and maintenance submit durable restart requests to
|
||||
Provisioning and observe the resulting operation by ID. Their pages link to that
|
||||
operation, and its history links back to the requesting work. Provisioning owns
|
||||
@@ -103,6 +113,8 @@ Related work links back to the requesting campaign, beta, maintenance item, or
|
||||
collection. A completed provisioning job does not imply that its requester has
|
||||
finished its own verification.
|
||||
|
||||
### Verification evidence
|
||||
|
||||
New operations record structured verification evidence alongside their activity:
|
||||
the check result, source, evaluation time, and expected/observed map or Plutonium
|
||||
revision when relevant. Create and Rebuild show **Verify Plutonium revision**, with
|
||||
@@ -121,6 +133,8 @@ succeeds and releases the slot. Capacity and request order are snapshots, not
|
||||
guarantees of the next job admitted or an estimated completion time. Blocked
|
||||
requests can be bypassed by eligible work.
|
||||
|
||||
### Recovery actions and launch files
|
||||
|
||||
Recovery controls use the server's current eligibility checks. **Resume health
|
||||
verification** observes the existing instance without submitting another create or
|
||||
restart. **Resume cleanup** retries local cleanup after confirmed remote deletion,
|
||||
@@ -136,6 +150,8 @@ versions; an attempted selection is not proof that every file was installed.
|
||||
**Launch another server** uses the latest promoted setup with a fresh identity and
|
||||
port, rather than repairing or replaying the recorded job.
|
||||
|
||||
### History and refresh behavior
|
||||
|
||||
Five recent events remain visible. Expand full history to search, change order,
|
||||
and load older pages. A fixed history watermark keeps the viewed pages stable as
|
||||
events arrive; **Jump to latest** incorporates new activity. The detail view
|
||||
@@ -197,7 +213,7 @@ Start and Stop use durable `instance.start` and `instance.stop` host-agent
|
||||
operations. Rebuild uses `instance.rebuild` and replaces the instance files.
|
||||
See [native host management](native-hosts.md) for command delivery and recovery.
|
||||
|
||||
### Additional concurrency limits
|
||||
## Concurrency limits
|
||||
|
||||
Administration → Settings separates Concurrency and Retention, with independent
|
||||
save actions. Existing fleet restart, startup and destruction limits continue to
|
||||
@@ -78,6 +78,8 @@ but keeps compact terminal receipts and request fingerprints. Reusing an idempot
|
||||
key cannot execute an expired command again. Agent mutation journals remain intact;
|
||||
this policy controls controller history, not the agent's execution recovery journal.
|
||||
|
||||
### File bodies and match history
|
||||
|
||||
Expiring campaign file bodies preserves file paths, hashes and summary records but
|
||||
makes file comparisons and rollback unavailable. The detail page identifies this
|
||||
state and rollback requests fail explicitly. Active beta rollback images stay
|
||||
@@ -90,6 +92,8 @@ failures and explicit holds preserve the bytes they need. Shared file bytes are
|
||||
deleted only after their final reference expires; library versions can keep a file
|
||||
available longer. Previously saved retention choices are preserved on upgrade.
|
||||
|
||||
### Database cleanup
|
||||
|
||||
The database upgrade normalizes retention clocks to UTC and enables incremental
|
||||
vacuum. An existing database without a vacuum page map is rebuilt once at startup;
|
||||
allow time for this rebuild and free disk space up to twice the database size.
|
||||
@@ -99,6 +103,8 @@ of holding a temporary copy of the database in RAM. Container scratch files use
|
||||
Normal cleanup releases unused file blobs in bounded batches and incrementally
|
||||
returns free pages to disk after larger prunes.
|
||||
|
||||
### Game files and incomplete uploads
|
||||
|
||||
Game-file minimums apply per game. Grace starts when a version is superseded.
|
||||
Live, stopped, rebuilding and provisioning instances pin their files; unresolved
|
||||
operations and offline host copies also prevent controller deletion. Controller
|
||||
@@ -1,109 +0,0 @@
|
||||
Improve Metrics so an operator can quickly answer how much hosting time had players, which maps receive play, where measured capacity stays full, and whether the recorded history supports a decision.
|
||||
|
||||
This plan combines three UI/UX reviews (information architecture, visual design, and interaction/accessibility) and two software engineering reviews (frontend and metrics data). The reviewers exchanged proposals and challenged the tradeoffs. No product changes are included.
|
||||
|
||||
The review covered the current Svelte page, collector, SQLite aggregation, shared design system, and tests. Desktop (1440px) and mobile (390px) screenshots of the existing build were inspected using an isolated synthetic database. That fixture contained six server names, seven days of samples, an outage, unknown status, and missing capacity. It establishes layout behavior, not production usage patterns.
|
||||
|
||||
| Finding | Consequence | Planned response |
|
||||
| --- | --- | --- |
|
||||
| “Servers utilized” is occupied time divided by known server time. | It can be mistaken for a percentage of server count. | Name the measure “Time with players” and show its denominator. |
|
||||
| Four equal cards include complementary occupied/idle percentages. At 390px, the chart heading starts around 970px down the page. | Repeated summary information delays useful analysis, especially on phones. | Two compact headline metrics and a nearby data-quality strip. |
|
||||
| The chart has endpoint labels but no calibrated vertical ticks. Partial edge periods look like ordinary periods. | A short final bar can look like declining activity. | Add clear axes, explicit partial-period treatment, and inspectable intervals. |
|
||||
| “Updated” is the report calculation time. | A fresh report can contain old samples. | Separate “Report refreshed” from “Last collected.” |
|
||||
| Capacity can be absent even when telemetry supplies valid occupancy. | “Unknown” pressure can be mistaken for a failure or no pressure. | Keep known-capacity duration beside every pressure figure and explain unavailable capacity. |
|
||||
| Tables have fixed ordering and limited comparison affordances. | Operators must scan raw values to identify useful patterns. | Sorting, restrained comparison bars, readable identities, and local search. |
|
||||
| Metrics aggregate historical samples by server name without an instance identifier. | Linking a row to a current server can connect old history to a replacement. | Keep historical row identities unlinked in the first release. |
|
||||
| A synthetic 118,806-row fixture took about 0.99–1.35 seconds to read for the seven-day view. | Synchronous aggregation can block other application requests. | Profile and reduce repeated aggregation work before expanding exploration. |
|
||||
|
||||
The proposed page composition is:
|
||||
|
||||
```text
|
||||
Metrics Refresh
|
||||
24 hours / 7 days / 30 days selected range · timezone
|
||||
|
||||
Time with players Player-hours
|
||||
58.6% 1,350.5
|
||||
555.7 of 948.8 known server-hours Players × observed time
|
||||
|
||||
Known status … · Unknown status … · Last collected …
|
||||
Report refreshed … · Samples retained from …
|
||||
|
||||
Server time over time occupied / idle / unknown
|
||||
[stacked server-hours chart with axes and marked partial periods]
|
||||
[Period selector] [Previous] [Next] [selected interval and values]
|
||||
View exact time data
|
||||
|
||||
Map play time & capacity
|
||||
[sortable ranking; pressure and known-capacity time stay together]
|
||||
|
||||
Server utilization Find a server…
|
||||
[sortable table; friendly names and secondary technical IDs]
|
||||
|
||||
How these estimates work
|
||||
```
|
||||
|
||||
The example values above illustrate the synthetic preview, not a target or recommendation.
|
||||
|
||||
1. Establish a precise, efficient report contract.
|
||||
|
||||
Keep the existing 24-hour, seven-day, and 30-day windows, 45-second observation cap, UTC bucket boundaries, and 31-day retention. Add typed report metadata for the latest recorded collection sample, retained sample bounds, and current registered-server count. Return the actual clipped start/end and partial flag for each bucket. Distinguish report calculation time from sample time. Collection stops when no servers are registered, so old history alone does not imply a failed collector. Prefer factual timestamps; do not apply a ticking 45-second stale threshold to a page refreshed only every minute.
|
||||
|
||||
Keep the formulas explicit: occupied percentage = occupied server-hours / known server-hours; known server-hours = occupied + idle. Unknown samples are separate. Long collection gaps are excluded, so known / (known + unknown) must not be called collection coverage. Earliest retained sample does not establish first-ever collection or uninterrupted history.
|
||||
|
||||
Pressure remains map wall time: “All lobbies full for X during Y with known capacity.” It requires known occupancy and capacity for every tracked lobby in that recorded game/map group. It is not a measure of failed joins or unmet demand. Missing capacity must remain unknown; do not assume four slots or carry old capacity forward without a separate validity policy.
|
||||
|
||||
Profile realistic retained histories, then consolidate repeated sample/bucket aggregation while preserving interval clipping and pressure semantics. The current implementation queries each chart bucket separately in addition to summary and table queries. One profiled seven-day read spent roughly 349ms on summary, 158ms on servers, 441ms on maps, and 109ms across chart buckets: reducing bucket queries alone misses most of the cost. Use parity checks against existing calculations before replacing it; evaluate query plans before adding indexes. Avoid transferring raw samples to the browser. Do not introduce permanent rollups or a background analytics subsystem unless profiling justifies them.
|
||||
|
||||
Primary files: `src/lib/server/fleet/metrics.ts`, `src/routes/metrics/+page.server.ts`, `tests/unit/metrics.test.ts`. Add a shared metrics report type and range parser where both page and API can use them. Schema or collector changes are conditional on measured need.
|
||||
|
||||
2. Make refresh and report states dependable.
|
||||
|
||||
Retain server rendering for the initial page. Add a narrow read-only `/api/metrics?days=…` endpoint using the same report builder and range validation for background refresh. Use the existing `pollVisible` helper to avoid overlapping refreshes and refresh on return to a visible tab. The installed SvelteKit invalidation path can navigate into an error page; catching `invalidateAll()` alone does not guarantee preservation of the last successful report.
|
||||
|
||||
Store the last successful report, update it only after a successful response, and show refreshing, stale/error, and retry feedback inline. Give requests a timeout, abort on navigation/unmount, and use a request/scope identifier so an old response cannot overwrite a newly selected range. Keep range state in the URL. Preserve table search/sort and selected bucket through refresh; reconcile bucket selection when the range changes. Optional shareable table state should use shallow URL updates without rerunning aggregation. Avoid moving keyboard focus or announcing the entire report each minute.
|
||||
|
||||
Distinguish no retained samples, no samples in this selected range, unknown-only observations, unavailable capacity, and a failed report refresh. An empty range should offer a wider range. Do not promise that collection will begin if no servers are registered, or claim “first observations” solely because retention left the table empty.
|
||||
|
||||
Primary files: the metrics route, a new `src/routes/api/metrics/+server.ts`, and existing `src/lib/ui/poll.ts` usage. Keep shared-helper changes minimal.
|
||||
|
||||
3. Rebuild the summary and timeline around readable evidence.
|
||||
|
||||
Replace four equal cards with “Time with players” and “Player-hours.” Put the occupied numerator and known-time denominator beneath the percentage. Keep idle in the chart/table and known/unknown time in a compact strip. Show collection time separately from report refresh; display sparse-history caveats without inventing a confidence score.
|
||||
|
||||
Reuse `Stat` with numeric tweening disabled, existing cards/inputs/empty states, and semantic design tokens. Keep blue for occupied time, a quieter neutral for idle, and patterned unknown. Define chart colors as tokens rather than scattered literals. Use short-duration formatting so a real observation never appears as `0.0 h` just because it is brief. Keep player-hours and server-hours explicitly named.
|
||||
|
||||
Preserve absolute stacked server-hours as the default chart. Add three to five readable vertical ticks, intermediate time labels, marked partial edge buckets, and a persistent selected-period readout. The readout shows the exact interval in the selected timezone, occupied/idle/unknown server time, player-hours, and occupied percentage of known time. UTC-aligned buckets displayed in a local timezone must not be described as local calendar days.
|
||||
|
||||
Use a labelled native period selector and Previous/Next buttons for keyboard and touch access, with pointer selection as a convenience. Keep interactive controls and the readout outside any chart region exposed as `role="img"`. Keep the exact-data table with semantic headers. A bucket with no captured time needs an explicit “No captured time” readout; do not equate it with an idle fleet. Preserve selection by bucket key across minute refreshes.
|
||||
|
||||
On phones, keep the summary compact enough to reach the chart without scrolling through four tall cards. Allow controls to wrap and reduce tick density; do not remove units or denominators to make the view fit. Use a small metrics-specific component set rather than introducing a general dashboard framework or chart dependency.
|
||||
|
||||
4. Make the map and server rankings useful.
|
||||
|
||||
Keep both tables full width. Right-align numeric values, use tabular numerals, add keyboard-operable sort buttons with `aria-sort`, and define deterministic ties and missing-value placement. Keep the map default as player-hours descending. Give the server table an explicit default matching the displayed measure: occupied percentage descending, then known time, with unavailable percentages last.
|
||||
|
||||
Add restrained proportional player-hour bars to map rows. Keep pressure duration, its percentage, and known-capacity duration adjacent. Avoid severity colors or “add servers” recommendations without a defined policy and sufficient evidence. Known-capacity duration is a wall-time quantity, not server-hours.
|
||||
|
||||
Use friendly server names with raw identifiers available as secondary text. Preserve exact recorded game/map grouping and variants. Current collection stores friendly configured variants when resolvable; legacy engine-map strings can be ambiguous, so do not silently relabel them as one specific variant.
|
||||
|
||||
Add clearly labelled table-only search, starting with “Find a server”; it must not imply that the headline metrics or chart were filtered. Keep historical rows unlinked in the first release. A later explicitly labelled “Open current server” action could be added separately, but current registration alone cannot prove historical identity.
|
||||
|
||||
On narrow screens retain identity, the primary measure, and its denominator. Make full details available through a compact row disclosure or a labelled, keyboard-scrollable table region with sticky identity. Test which approach fits actual row lengths before choosing it. Empty searches offer a clear/reset action.
|
||||
|
||||
5. Validate the completed behavior and update usage documentation.
|
||||
|
||||
Extend calculation tests for new metadata and clipped intervals, then compare optimized results to the existing implementation across fixture scenarios. Test refresh errors, retry, visibility changes, overlapping refresh attempts, and responses arriving after a range switch. Prefer behavior checks over tests that duplicate markup.
|
||||
|
||||
Review no samples, a first sample with zero elapsed time, sparse history, all-unknown status, telemetry-only occupancy, missing capacity, nonzero durations under a minute, collection outages, changing fleet membership, partial edge buckets, timezone/DST presentation, removed/reused server names, and T6 variants. Verify sorting with unavailable percentages and equal values.
|
||||
|
||||
Inspect 390px, tablet, and desktop layouts; keyboard-only selection/sorting/table access; touch selection; readable focus states; reduced motion; and screen-reader descriptions. Ensure minute refresh preserves interaction state and background failures keep the last successful report visible. Benchmark 24-hour, seven-day, and 30-day queries against representative retained data, and check impact on concurrent application requests.
|
||||
|
||||
Run the relevant unit tests, `npm run check`, and `npm run build`. Update `docs/metrics.md` to match the labels, states, and counting rules. The review baseline passed the existing four metrics tests and 23 status/observation tests; that does not replace validation of the eventual implementation.
|
||||
|
||||
The team accepted two primary metrics, visible data-quality context, an absolute-hours chart, progressive detail, and small native interaction controls. The engineering review changed two early UX ideas: live historical row links are deferred because names can be reused, and “Last collected” replaces wording that could imply a fresh successful game observation. It also added background fetch isolation and measured query work to the implementation scope.
|
||||
|
||||
Defer a default percentage-only chart, fleet-wide “all full” KPI, period-over-period deltas, heatmaps, automatic hosting recommendations, and global game/map filters. A percentage-only chart can hide tiny denominators; map pressure intervals overlap and cannot be summed into a meaningful fleet total. Comparisons and heatmaps need fair coverage treatment. Global filters should follow later as a complete query scope applied consistently to summary, timeline, and both tables, with URL state.
|
||||
|
||||
Delivery can be split into three reviewable changes: (1) report metadata, measured query improvements, and calculation tests; (2) refresh handling plus summary/timeline and report states; (3) rankings, mobile/accessibility refinement, and documentation. The data contract comes first; frontend layout work can proceed against that agreed contract in parallel. The result should let an operator read utilization correctly, see its evidence limits, inspect a period, and find relevant maps or servers without treating missing data as zero.
|
||||
|
||||
After approval, the base was implemented and committed as `3922e28`. The user also authorized all previously deferred features as experiments. The panel reconvened after that commit; [the review and revised decisions](metrics-review.md) supersede the deferrals above while preserving explicit evidence and metric definitions.
|
||||
@@ -1,36 +0,0 @@
|
||||
The metrics panel reconvened after commit `3922e28` to review the implemented base and revisit the features deferred in the original plan. The user authorized proceeding with all of them, including experiments based on limited evidence.
|
||||
|
||||
The base passed 711 unit tests, Svelte checking with no errors or warnings, and a production build. Firefox checks exercised combined search/sort state, preservation across range changes, pointer and keyboard chart selection, failed background refresh with retained data, successful retry, and 390px layout without page overflow. These checks caught and fixed a shallow-URL reactivity issue before the commit. Calculation parity against the previous implementation held within `1e-7` on a 118,806-row synthetic fixture. Seven-day query time fell from approximately 1,369ms to 367–408ms in the uncontended comparison.
|
||||
|
||||
The post-commit engineering and UX reviews found no blocking issue. Two small follow-ups were accepted: neutral copy for samples with zero elapsed time, and a polite announcement when a user selects a chart period. Background refresh should not repeatedly announce the chart.
|
||||
|
||||
The team revisited the original objections as design constraints:
|
||||
|
||||
| Feature | Original concern | Agreed experiment |
|
||||
| --- | --- | --- |
|
||||
| Percentage chart | Normalization hides sample volume. | Default to share of captured server time, including unknown, with an Hours switch and exact durations in the selected-period readout. |
|
||||
| Fleet pressure score | Map intervals overlap and cannot be added into elapsed fleet time. | Pool full map-hours divided by known-capacity map-hours; name the denominator explicitly and show how many map groups contribute. |
|
||||
| Previous-period changes | Retained and observed history may differ substantially. | Show comparisons from available samples, including both denominators and a partial-history label. Utilization changes are percentage points. |
|
||||
| Activity heatmap | Sparse observations can look like stable recurring demand. | Show duration-weighted occupancy by local weekday/hour, with known/unknown time, contributing dates, and no-data cells distinct from observed zero. |
|
||||
| Hosting recommendations | Occupancy and saturation do not prove unmet demand. | Generate advisory, reversible hosting experiments from visible rules. Include evidence and tentative labels; never automatically create or remove servers. |
|
||||
| Global filters | Inconsistent scopes would make the page misleading. | Apply exact game/map scope to every aggregate, comparison, heatmap, recommendation, and ranking; preserve collection boundaries before filtering. |
|
||||
| Historical server links | A name may have been reused for a new instance. | Label the action “Open current server” and show it only when a current registration exists. Historical totals remain grouped by name. |
|
||||
|
||||
The agreed hierarchy keeps the two headline cards and core timeline. Compact scope controls sit near the time-range control. Pressure, comparisons, advisory experiments, and the secondary heatmap follow the timeline so the added features do not crowd out the primary task on mobile. Use neutral comparison colors, visible units, and concise evidence next to each recommendation.
|
||||
|
||||
The original plan remains as the record of the initial conservative scope. These decisions supersede its deferrals; limited data does not disable a feature when a defined calculation or explicitly tentative experiment is possible. An unavailable denominator still remains unavailable rather than becoming an invented number.
|
||||
|
||||
All seven additions were implemented. The expanded full suite passed 731 tests;
|
||||
subsequent refresh-response hardening passed its focused tests, including a new
|
||||
malformed-response case. Svelte checking reported no errors or warnings and the
|
||||
production build passed. Firefox exercised the percentage/Hours switch with
|
||||
preserved table state, current-registration links, whole-report game/map filters,
|
||||
empty scopes, heatmap keyboard selection and timezone changes, saved preference
|
||||
adoption, and explicit timezone precedence. The final mobile report fits at
|
||||
390px without page overflow.
|
||||
|
||||
The combined 30-day report took 443–446ms on the seven-day, 118,806-row fixture.
|
||||
A separate full-retention fixture with six servers and 535,680 rows took
|
||||
1.96–3.30 seconds for the combined 30-day report. The queries still run
|
||||
synchronously; that measured full-retention cost remains an improvement target.
|
||||
These local synthetic timings are not production guarantees.
|
||||
@@ -1,6 +1,10 @@
|
||||
# Lobby autoscaling implementation plan
|
||||
|
||||
Status: Approved and implemented October 6, 2026. Controller support and agent 1.7.0 are complete; production dry run, pilot measurements, and live rollout remain pending. See [usage and the pilot procedure](autoscaling.md).
|
||||
Status: Approved and implemented October 6, 2026. Controller support and agent 1.7.0 are complete; production dry run, pilot measurements, and live rollout remain pending. See [usage and the pilot procedure](../guides/autoscaling.md).
|
||||
|
||||
The design below preserves the approved scope and subsequent implementation
|
||||
record. Use the linked guide for current policy behavior, including game-wide
|
||||
policies and saved-key requirements added after the initial design.
|
||||
|
||||
Introduce an opt-in policy for each game and map variant that keeps a small number of empty, ready lobbies available. When the last spare fills, request another lobby through Provisioning. Remove surplus lobbies only after sustained, confirmed inactivity. This reduces idle hosting while preserving room for new groups, including when existing Zombies sessions have free player slots but are already well into a game.
|
||||
|
||||
@@ -10,16 +14,16 @@ Start with Create/Destroy on existing native hosts. Run the evaluator in dry run
|
||||
|
||||
| Foundation | Reuse and required work |
|
||||
| --- | --- |
|
||||
| Background worker | Add a requester to [pipeline/worker.ts](../src/lib/server/pipeline/worker.ts), which already reconciles background work every 15 seconds independently of Provisioning. Keep host execution inside Provisioning. |
|
||||
| Occupancy evidence | Reuse [status evidence](../src/lib/server/fleet/status-evidence.ts) and shared freshness rules. Monitoring runs on a 30-second cadence; accepted evidence currently expires after 90 seconds. Positive occupancy vetoes an empty claim. Unknown is never zero. |
|
||||
| Safe creation | Use the durable creation intent and placement path in [provisioning/native.ts](../src/lib/server/provisioning/native.ts). Preserve CPU, memory, port, host eligibility, library, and startup concurrency checks. A blocked create remains queued. |
|
||||
| Background worker | Add a requester to [pipeline/worker.ts](../../src/lib/server/pipeline/worker.ts), which already reconciles background work every 15 seconds independently of Provisioning. Keep host execution inside Provisioning. |
|
||||
| Occupancy evidence | Reuse [status evidence](../../src/lib/server/fleet/status-evidence.ts) and shared freshness rules. Monitoring runs on a 30-second cadence; accepted evidence currently expires after 90 seconds. Positive occupancy vetoes an empty claim. Unknown is never zero. |
|
||||
| Safe creation | Use the durable creation intent and placement path in [provisioning/native.ts](../../src/lib/server/provisioning/native.ts). Preserve CPU, memory, port, host eligibility, library, and startup concurrency checks. A blocked create remains queued. |
|
||||
| Safe removal | Reuse the controller occupancy checks, instance generation checks, host-local check, and cleanup checkpoints. Add autoscaling-specific admission and cancellation behavior before enabling automatic destruction. |
|
||||
| Requester eligibility | [provisioning/worker.ts](../src/lib/server/provisioning/worker.ts) currently gives Create and Destroy separate paths that bypass the runtime requester eligibility check. Autoscaling ownership and policy checks must cover both paths, including delayed execution. A source label alone is insufficient. |
|
||||
| Saved keys | [server-keys.ts](../src/lib/server/fleet/server-keys.ts) stores keys by exact lobby name and accepts lobby numbers 1–20. Allocate only unused identities with saved keys; never copy a key to a different identity. |
|
||||
| Collections | [collections.ts](../src/lib/server/fleet/collections.ts) already counts queued creations and launches promoted recipes. Exact matching can remove lobbies across the fleet, including maps omitted from the collection. Add ownership checks to preview and submission. |
|
||||
| Hosting insights | [metrics-insights.ts](../src/lib/metrics-insights.ts) provides historical suggestions for hosting trials. Use these to help choose policies; the live evaluator needs current empty-lobby evidence and must not execute historical suggestions directly. |
|
||||
| Requester eligibility | [provisioning/worker.ts](../../src/lib/server/provisioning/worker.ts) currently gives Create and Destroy separate paths that bypass the runtime requester eligibility check. Autoscaling ownership and policy checks must cover both paths, including delayed execution. A source label alone is insufficient. |
|
||||
| Saved keys | [server-keys.ts](../../src/lib/server/fleet/server-keys.ts) stores keys by exact lobby name and accepts lobby numbers 1–20. Allocate only unused identities with saved keys; never copy a key to a different identity. |
|
||||
| Collections | [collections.ts](../../src/lib/server/fleet/collections.ts) already counts queued creations and launches promoted recipes. Exact matching can remove lobbies across the fleet, including maps omitted from the collection. Add ownership checks to preview and submission. |
|
||||
| Hosting insights | [metrics-insights.ts](../../src/lib/metrics-insights.ts) provides historical suggestions for hosting trials. Use these to help choose policies; the live evaluator needs current empty-lobby evidence and must not execute historical suggestions directly. |
|
||||
|
||||
Create already uses the promoted launch recipe and capacity-aware placement. Start retains existing files and has recovery concurrency controls, but would need equivalent CPU and memory admission before becoming an autoscaling strategy. The proposed scope follows option 4 in [the idle CPU research](t5-t6-idle-cpu-research.md).
|
||||
Create already uses the promoted launch recipe and capacity-aware placement. Start retains existing files and has recovery concurrency controls, but would need equivalent CPU and memory admission before becoming an autoscaling strategy. The proposed scope follows option 4 in [the idle CPU research](../research/t5-t6-idle-cpu-research.md).
|
||||
|
||||
## Policy and ownership
|
||||
|
||||
@@ -73,7 +77,7 @@ Track continuous emptiness against the exact instance generation, using fresh ob
|
||||
|
||||
Automatic removals always use normal safety gates and never set a safety override. Revalidate policy mode and revision, ownership, membership, remaining spare capacity, conflicts, and occupancy immediately before claiming and dispatching removal. Preserve the agent's fresh local occupancy check immediately before mutation.
|
||||
|
||||
The existing [cancelRemoval](../src/lib/server/provisioning/history.ts) transaction competes with the worker while a removal is waiting. It cannot undo removal after the worker has claimed it. General cancellation also cannot withdraw an already delivered host command. The agent can hold an occupied command and retry later, so cancelling only the controller row would leave stale destructive intent.
|
||||
The existing [cancelRemoval](../../src/lib/server/provisioning/history.ts) transaction competes with the worker while a removal is waiting. It cannot undo removal after the worker has claimed it. General cancellation also cannot withdraw an already delivered host command. The agent can hold an occupied command and retry later, so cancelling only the controller row would leave stale destructive intent.
|
||||
|
||||
Before enabling Create and destroy, extend the controller/agent contract for autoscaling removals: an occupied or unknown host-local check must invalidate that autoscaling removal attempt rather than retain it for eventual execution. A later attempt requires a new controller evaluation and the full empty interval. Carry policy revision and a bounded authorization expiry, and require renewed controller authorization before an agent retry after delay or restart. Pausing or superseding a policy must invalidate authorization that has not begun mutation. Test these races with the real agent journal behavior; do not assume a controller cancellation implies host acknowledgement.
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
# Provisioning detail implementation
|
||||
|
||||
Implemented September 28, 2026, following the [approved review plan](provisioning-improvement-plan.md).
|
||||
|
||||
## Delivered behavior
|
||||
|
||||
The job page now leads with identity, recorded outcome, blocker, and permitted recovery action. Grouped execution phases and timestamped evidence precede supporting metadata and launch files. Mobile follows the same reading order. Operation IDs and links are copyable, and the return link preserves Provisioning filters or leads back to related work.
|
||||
|
||||
The worker records versioned progress snapshots and semantic events at existing decision points. Container, game, map, and occupancy checks identify their source and observation time. Durable milestones preserve remote deletion, local cleanup, command certainty, and operation-owned recovery capacity. A failed restart can retain its failed outcome while later recovery evidence records capacity release. Legacy records retain their original stage and outcome without invented checkpoints.
|
||||
|
||||
Cleanup resumes honor backend capabilities even for deleted identities. Confirmed remote deletion and the existing cleanup-resume stage are saved together before local cleanup, preserving recovery if a later guard or local operation fails. Late observations are rejected after cancellation, deletion, or replacement. Existing occupancy rules, scheduling, command counts, and observation-only resumes are covered by regression tests.
|
||||
|
||||
The page uses server-loaded detail and abortable, visit-scoped refreshes. Action feedback survives background refresh failures; outdated requests cannot replace another job's detail. Terminal records continue visibility-aware polling because recovery, deletion, and capabilities can still change.
|
||||
|
||||
Five recent events are visible. Full history loads in 25-entry pages with literal search, ordering, and a fixed upper event ID. Recent activity stays still while full history is open; new events appear through an explicit Jump to latest control. The legacy detail API still returns full events by default. The new page opts into `?events=recent`.
|
||||
|
||||
Queued recipe captures and files selected for an attempt have separate labels and timestamps. Neither implies successful installation. Related work names resolve when available, while missing source IDs remain visible without broken links.
|
||||
|
||||
## Validation
|
||||
|
||||
- Full suite: **898 tests passed across 72 files**.
|
||||
- Svelte/TypeScript: `npm run check` passed with **0 errors and 0 warnings**.
|
||||
- Production: `npm run build` passed with the Node adapter.
|
||||
- Database tests cover additive migration, historical identity and event preservation, transaction rollback, stable phase timing, semantic outcomes across same-ID resumes, and full versus bounded API compatibility.
|
||||
- Lifecycle tests cover ambiguous and thrown remote replies, telemetry and map evidence, cancelled/replaced identity races, exact recovery ownership, and destroy/rebuild cleanup after confirmed deletion without repeating remote commands.
|
||||
- Refresh/presentation tests cover navigation and action races, persistent acknowledged action results, failed reconciliation, cancellation, deleted cleanup, legacy evidence, and malformed responses.
|
||||
- Firefox checks exercised history pagination, literal search, stable new-event handling, failed-refresh retention and recovery, and layouts at 1440, 390, and 320 pixels without page-wide overflow. Enter and Space operate phase disclosures. A deleted-identity cleanup resume completed through the real local API; retained restart recovery, cancellation, and legacy evidence also rendered correctly.
|
||||
- Reflow also passed at 720 CSS pixels with pixel ratio 2, and long unbroken identifiers fit at 320px. This simulates the available layout width of 200% zoom on a 1440px display; native browser zoom was not measured. A separate CSS-zoom probe kept the detail content in bounds but exposed overflow in the existing global navigation.
|
||||
|
||||
Browser fixtures use a temporary database and disabled external integrations. They exercise the real page and local detail/history APIs; lifecycle tests stub remote systems. This is not a production fleet smoke test or a formal accessibility conformance audit. The build reports the repository's existing `csrf.checkOrigin` deprecation notice.
|
||||
|
||||
## Measurements and screenshots
|
||||
|
||||
For a synthetic 5,001-event operation, full detail serialized to **1,884,119 bytes** and bounded detail to **3,011 bytes**. Thirty local bounded reads averaged **2.93 ms**. These measurements establish payload reduction on the local fixture, not a production latency guarantee; event totals and search still query retained history.
|
||||
|
||||
In the 1440px create/map-wait fixture, the execution heading moved from approximately **1,073px** in the original composition to **675px**. The page keeps the outcome above execution and the recipe in a supporting column. Phone layouts move context below execution and activity.
|
||||
|
||||
Updated screenshots: [desktop create/map wait](provisioning-review-assets/implemented-create-desktop.png), [mobile create/map wait](provisioning-review-assets/implemented-create-mobile.png), [deleted identity with permitted cleanup](provisioning-review-assets/implemented-deleted-cleanup.png), and [failed restart with retained recovery](provisioning-review-assets/implemented-restart-recovery.png). The original review captures remain linked from the approved plan.
|
||||
|
||||
## Compatibility and rollout
|
||||
|
||||
The schema additions are nullable, and old stage text and default API event responses remain supported. Progress is written as new work is evaluated; historical operations without structured evidence use the legacy presentation. Detail reads perform no remote probes.
|
||||
|
||||
The change is implemented in the workspace. Normal deployment should verify one newly instrumented job and one legacy record against the configured integrations. No production deployment was performed during this work.
|
||||
@@ -1,230 +0,0 @@
|
||||
# Provisioning job detail improvement plan
|
||||
|
||||
Review date: September 28, 2026. Source baseline: `5b65520`.
|
||||
|
||||
The recommendation is to make `/provisioning/:id` a complete operational record: identify the job, explain its current outcome and blocker, expose the evidence behind that outcome, and show the next permitted action. Progress and recovery should precede historical launch files.
|
||||
|
||||
Approved and implemented on September 28, 2026. The original review and agreed scope remain below; see the [implementation and validation record](provisioning-implementation.md) for delivered behavior, checks, and updated screenshots.
|
||||
|
||||
## Review method and participants
|
||||
|
||||
The requested panel covered three UI/UX roles, two CI/CD systems engineering roles, and one software engineering role. The platform allowed only three child-agent threads and rejected additional spawns even after those reviews completed. The three threads therefore performed two separate rounds, first UX and then engineering. This provides six role-based reviews, rather than six independent agents. Each round included direct peer discussion; engineering reviewed and challenged the UX proposals.
|
||||
|
||||
| Review role | Proposal and contribution |
|
||||
| --- | --- |
|
||||
| UI/UX: information architecture | Prioritize job identity, outcome, blocker, safe action, progress, activity, and broader work context. Separate historical completion from current server health. |
|
||||
| UI/UX: interaction and accessibility | Use state-specific recovery actions, stable refresh behavior, persistent action feedback, keyboard-accessible progress, and usable history while updates arrive. |
|
||||
| UI/UX: visual design | Replace the overloaded summary and wrapping step chips with a compact outcome card, grouped vertical progress, and supporting metadata. Reuse the existing design system. |
|
||||
| CI/CD systems: lifecycle and queues | Preserve destructive checkpoints, occupancy gates, scheduling, command certainty, identity guards, and recovery-slot semantics. Make queue claims match actual admission behavior. |
|
||||
| CI/CD systems: observability and reliability | Record the source and age of verification evidence, distinguish page freshness from worker checks, preserve recovery evidence, and validate compatibility and failure scenarios. |
|
||||
| Software engineering | Define shared types and component boundaries, reuse server-loaded detail, fix route/poll/action races, and stage additive implementation with regression coverage. |
|
||||
|
||||
The review covered the page, shared history component, lifecycle APIs, worker, queues, persistence, related collection/maintenance/campaign pages, and relevant tests. Root also inspected the current source in Firefox using an isolated temporary database, disabled integration credentials, and synthetic detail responses. Desktop captures used 1440px width; mobile used 390px. These are composed UI scenarios, not executions of real provisioning workflows or measurements of production use.
|
||||
|
||||
Review assets: [create waiting for a map](provisioning-review-assets/create-map-wait-desktop.png), [mobile rebuild composition](provisioning-review-assets/rebuild-mobile.png), and [deleted identity with resumable cleanup](provisioning-review-assets/deleted-cleanup-desktop.png).
|
||||
|
||||
## What feels incomplete today
|
||||
|
||||
The page already has useful recovery buttons, server diagnostics links, a recorded launch recipe, source backlinks, elapsed time, progress steps, and searchable activity. Preserve those capabilities. The gaps concern their priority and the reliability of the story they tell.
|
||||
|
||||
| Finding | Effect on the operator | Evidence |
|
||||
| --- | --- | --- |
|
||||
| One summary mixes status, actions, errors, warnings, diagnostics, map, timing, and provenance. The recipe appears before progress. | Monitoring and recovery compete with secondary information. | [Detail page](../src/routes/provisioning/[id]/+page.svelte), summary and recipe markup. In the synthetic desktop create case, the flow began about 1,073px down. |
|
||||
| Rebuild renders 12 equally weighted wrapping step chips. | Removal, cleanup, and fresh creation do not read as distinct phases. | [Detail page](../src/routes/provisioning/[id]/+page.svelte), `steps`; [OperationHistory](../src/lib/ui/components/OperationHistory.svelte). |
|
||||
| Progress and step duration depend on matching human-readable stage/event text. | New wording can move the highlighted step backward; changes in player count can appear to restart a step timer. | Detail page `currentStep` and `stageStarted`; [creation verification](../src/lib/server/provisioning/instances.ts) now emits “RCON or fresh telemetry” stages. |
|
||||
| Runtime cancellation is handled after runtime-specific explanation logic. | A cancelled start/restart/stop can still claim automatic verification is underway. | Detail page `explanation`. |
|
||||
| The deleted-server flag suppresses every cleanup button and the active-state presentation. | A permitted local cleanup can become inaccessible or appear inactive after resuming. | Detail page recovery controls and `active` versus [history.ts](../src/lib/server/provisioning/history.ts), `canResumeCleanup`. Missing button reproduced with a synthetic permitted response. |
|
||||
| Completion copy asserts RCON health in the present tense. | Telemetry-based verification is misreported, and historical success can look like current health. | Detail page explanations; [status-evidence.ts](../src/lib/server/fleet/status-evidence.ts), `probeServerStatus`. |
|
||||
| One refresh guard and one error string handle reads, mutations, and navigation. | A post-action reload can be dropped, an old request can affect another job, and polling can erase an action error. | Detail page `load`, `resume`, `cancelRemoval`; source-inferred race cases. |
|
||||
| A failed restart can still hold a recovery slot. | A simple failed badge does not explain continued observation or effects on other jobs. | [restarts.ts](../src/lib/server/provisioning/restarts.ts), `refreshRestartRollout`. |
|
||||
| Activity is collapsed, operation ID is absent from the page body, and related-work links are scattered. | Investigation and handoff require avoidable searching and navigation. | Detail page and shared history component. |
|
||||
|
||||
## Proposed page
|
||||
|
||||
Use one reading order across all six actions. At wide widths, metadata and launch inputs can occupy a supporting column. On phones they follow progress and activity in the DOM. Avoid a sticky sidebar.
|
||||
|
||||
```text
|
||||
Provisioning / Restart server
|
||||
Origins · Lobby 1 t6_zombie_origins_1
|
||||
Operation ID [Copy] [Copy link] [View server]
|
||||
|
||||
[Waiting] Waiting for players to leave
|
||||
Last evaluated: 3 players · source and observation time
|
||||
Continues automatically after an eligible empty/stopped check.
|
||||
Relevant permitted action or dependency link
|
||||
Total elapsed · Time in phase · Page updated / Refresh
|
||||
|
||||
EXECUTION AND EVIDENCE OPERATION CONTEXT
|
||||
Eligibility Requested by / requested at
|
||||
Container action Submission attempted / finished
|
||||
Readiness verification Related campaign/beta/maintenance
|
||||
Container Collection launch
|
||||
Game status
|
||||
Selected map, when required Recorded launch files
|
||||
[Launch another server]
|
||||
RECENT ACTIVITY
|
||||
Five latest entries + relevant failure
|
||||
[Full searchable history]
|
||||
```
|
||||
|
||||
The example evidence requires the new contract below; do not populate it from guessed values.
|
||||
|
||||
### Identity, outcome, and next action
|
||||
|
||||
- Show the action, friendly map/lobby title, technical server name, and copyable operation ID/link. Use existing identity and copy components.
|
||||
- Lead with the specific blocker or outcome, followed by what happens next. Place the recorded error and permitted recovery control together. Give warnings a labeled explanation rather than color alone.
|
||||
- Treat deletion as historical identity context. It must not replace the outcome, hide an actionable cleanup, or link an old operation to a replacement server with the same name.
|
||||
- Determine active progress from the operation and recovery state, independently of whether a live server link is available.
|
||||
- Distinguish a recorded outcome from ongoing recovery. A failed request can show “Recovery is still being checked; restart capacity remains reserved” when the backend records that condition.
|
||||
- Use past-tense completion language with its timestamp: “Creation completed. Container readiness, game status, and the selected engine map were verified at …”. Name RCON or telemetry only when recorded. Current health remains on the server page.
|
||||
- Preserve an explicit return link to the originating Provisioning filters or parent work, with a safe local fallback. Group all requesting campaigns, betas, maintenance records, and collection launches under Related work. Keep retained source IDs visible when their originating record no longer exists, without a broken link.
|
||||
- Finishing a restart does not mean its campaign, beta, or maintenance verification has finished. Link back to that work with accurate wording.
|
||||
|
||||
### Progress and evidence
|
||||
|
||||
Use a vertical checklist with grouped phases, icons, explicit text states, and the current/failed phase expanded. Keep completed phases available for inspection. Avoid percentage complete or predicted finish times.
|
||||
|
||||
| Action | Phase sequence and completion evidence |
|
||||
| --- | --- |
|
||||
| Create | Request/startup queue → prepare files and create deployment → start container → container readiness → game status → matching engine map. |
|
||||
| Start | Eligibility and restart capacity → start existing container → container and game verification. Do not imply files were redeployed. |
|
||||
| Restart | Eligibility, occupancy, and capacity → restart submission → container and game verification. Show any continuing recovery separately from request failure. |
|
||||
| Rebuild | Eligibility/capacity → remove old container and deployment → local files/registry cleanup → queue fresh instance → create/start → readiness and map verification. Preserve the reserved identity/port and show when the old instance is gone. |
|
||||
| Stop | Eligibility and occupancy → stop submission → explicit stopped-container confirmation. Explain that files and registration remain. |
|
||||
| Destroy | Eligibility and occupancy → container removal → deployment deletion → local files/registry cleanup. Retain the historical record after removal. |
|
||||
|
||||
Rebuild grouping must retain each irreversible checkpoint. “Remote deployment deleted” and “Local cleanup complete” are different facts; fresh creation must never look complete while old files remain.
|
||||
|
||||
Progress supports waiting, current, completed, failed, cancelled/not run, skipped when actually recorded, and unknown. Do not mark earlier steps complete simply because a string matched a later label. Legacy records retain their raw stage and known outcome, with unrecorded milestones labeled accordingly.
|
||||
|
||||
The blocker panel shows the worker's actual reason: occupied/unknown players, missing configuration, requester hold, conflicting work, startup or restart capacity, removal/port dependency, container readiness, game evidence, or wrong map. Include the blocking operation/source link when known. Keep “Requested by” separate from “Waiting on.”
|
||||
|
||||
Show relevant capacity limits, holders, and this operation's reservation from the same backend rules. If a submission rank is shown, label it as request order: blocked earlier jobs can be bypassed, and source priorities affect admission. Do not say “You are next.”
|
||||
|
||||
Evidence rows show result, evaluation time, source, and expected/observed values where relevant. Engine-map verification does not independently verify a T6 variant/start location. Unknown players are never displayed as zero.
|
||||
|
||||
Separate durable milestones from transient readiness checks. A passed container/game/map observation remains evidence of what passed at its timestamp; a later evaluation may make the current check failed or unknown. Do not leave stale green checks implying current readiness.
|
||||
|
||||
### Actions and terminal states
|
||||
|
||||
| Backend condition | Page behavior |
|
||||
| --- | --- |
|
||||
| `canCancelRemoval` | Offer **Cancel queued removal**, explaining that it is available only before removal begins. Preserve the existing atomic worker/cancellation race protection. |
|
||||
| `canResumeVerification` | Offer **Resume health verification** and explain that it observes the existing instance without creating or restarting it. |
|
||||
| `canResumeCleanup` | Offer **Resume cleanup**, including eligible deleted identities. Explain that instance removal already succeeded. |
|
||||
| `canResumeRebuild` | Offer **Resume rebuild cleanup** and explain that local cleanup precedes creation in the reserved slot. |
|
||||
| Failed/interrupted without a resume capability | Lead with the recorded failure and relevant inspection destination. An uncertain remote response must not produce a generic retry command. |
|
||||
| Cancelled | State that the request is cancelled; distinguish recorded prior work from remaining work that did not run. Do not claim continued execution. |
|
||||
| Warning | Explain the specific remaining issue. Do not assume every warning concerns cleanup. |
|
||||
| Complete | Show the verified historical outcome and appropriate View server / Back to related work / Back to history link. |
|
||||
|
||||
The API revalidates every action. A 409 triggers reconciliation of current capabilities and explains the state change; it never automatically repeats a mutation. Disable duplicate submissions while an action is pending, keep its accessible name, and preserve its result independently of background refresh. If an acknowledged successful action is followed by a failed refresh, retain its success acknowledgment alongside stale-data feedback; a refresh failure does not turn the action into a failure.
|
||||
|
||||
“Launch another server” remains secondary beside the recipe, with its existing explanation that it creates a separate server using current promoted files. It is not a repair or replay of the recorded launch. Create and rebuild already save a recipe while queued, then recapture promoted files when creation starts. Label these **Setup captured when queued** versus **Files selected for this attempt**, with capture times and an explicit recipe status. The queued capture is not a version pin, and selected files do not prove installation succeeded. Legacy records without that distinction retain neutral recorded-recipe wording. Where no recipe exists, explain the absence.
|
||||
|
||||
### Activity, refresh, and accessibility
|
||||
|
||||
- Keep five recent events visible and show a targeted failure excerpt when needed. Retain full-history search, ordering, and Show more. Do not automatically open a large history when a failure arrives.
|
||||
- Load full history on demand in bounded pages rather than retransmitting the entire retained history every five seconds. Preserve the legacy API's default response through an opt-in page query.
|
||||
- Preserve search, disclosure, focus, and reading position across updates. While inspecting older history, show a new-event count and **Jump to latest** instead of moving the content underneath the reader.
|
||||
- Separate initial loading failure, background refresh failure, and action failure. Keep last-known detail visible with an explicit stale message and manual Refresh.
|
||||
- Show **Page updated** separately from **Last evaluated** and evidence receipt time. A successful GET does not prove the worker or server was checked recently. An unchanged event stream does not prove the worker is stalled.
|
||||
- Keep the existing five-second visibility-aware polling initially, including terminal records. Failed jobs may still have recovery activity; another tab may resume them; later deletion can change identity links and capabilities.
|
||||
- Use server-loaded initial detail. On route changes and actions, use abortable reads and request generations so old responses cannot replace new detail or feedback. Force reconciliation after an action even if a poll was already in progress.
|
||||
- Announce meaningful outcome/action changes politely, not every timer tick. Keep textual progress states and `aria-current`; errors remain near their actions. Use a stable loading layout and a clear missing-operation state.
|
||||
- Reuse existing primitives, spacing, colors, typography, and identity components. Verify long identifiers, keyboard access, visible focus, contrast, reduced motion, 320/390px layouts, tablet/desktop layouts, and 200% zoom.
|
||||
|
||||
## Data and implementation design
|
||||
|
||||
The page needs structured facts that the current response does not fully expose. Add a small typed read model and observation metadata; preserve the existing lifecycle state machine and execution predicates.
|
||||
|
||||
| Contract area | Existing foundation | Required addition |
|
||||
| --- | --- | --- |
|
||||
| Identity and outcome | `server_deployments`, historical identity guards | Shared typed detail; recorded result remains distinct from current recovery. |
|
||||
| Phase and timing | Free-text `stage`, request/finish/submission times | Nullable stable phase key/start time; explicit checkpoint states/times; total elapsed remains based on the original request. |
|
||||
| Blocker | Worker decision branches and stage messages | Reason code, explanation, evaluated time, automatic continuation condition, and exact dependency reference where known. |
|
||||
| Command certainty | `submitted_at` is persisted before the network call | Distinguish submission attempted, reply accepted, uncertain outcome, and verified result. Never relabel submission intent as remote acceptance. |
|
||||
| Verification | Existing probes and transient results | Versioned snapshot with container/game/map/occupancy evidence, result/source/evaluation time, source receipt time where available, and expected/actual values. |
|
||||
| Recovery and capacity | `restart_rollout`, creation queue, saved concurrency | Exact operation-owned slot/reservation status, relevant limits/holders, and durable recovery/release evidence. |
|
||||
| Available actions | Existing `canResume*` and `canCancelRemoval` checks | Keep these authoritative; supply contextual explanation without creating new execution capabilities. |
|
||||
| Provenance and recipe | Sources, collection, saved launch recipe | Named/resolvable context links, missing/deleted-source handling, and explicit queued-capture versus attempted-selection status. |
|
||||
| Activity | Ordered event ID/time/message | Optional semantic kind/phase/details while preserving existing event text and order. |
|
||||
|
||||
A suitable small schema extension is nullable `phase_key`, `phase_started_at`, `observed_at`, and versioned `observation_json` on the operation, plus optional event `kind`, `phase_key`, and allowlisted `details_json`. Final naming can follow repository conventions. Snapshot payloads remain bounded and contain only the fields needed by the page, not raw integration responses or credentials.
|
||||
|
||||
Implementation rules:
|
||||
|
||||
1. Record structured blockers and observations where the worker already makes decisions. Do not build a second eligibility engine in the API or browser. The detail GET remains read-only and performs no host-agent/RCON probes.
|
||||
2. Derive current capacity from existing scheduler helpers/tables, with an explicit snapshot time. Persist important milestones against the operation because server-owned recovery rows can later be replaced or deleted. Only attach a live slot when its `operation_id` matches; slot absence alone is not evidence of successful release.
|
||||
3. Update related state, observation, and semantic event atomically around each durable checkpoint, keeping network calls outside database transactions. Dual-write existing stage/event text, especially `REMOVING_SERVER`, which participates in recovery eligibility today.
|
||||
4. Record meaningful phase, blocker, command, and recovery transitions. Update observation timestamps on repeated checks without appending an event every poll. Changing player counts must not reset the phase's start time.
|
||||
5. Preserve prior failure and resume events when an existing operation resumes. New observations must not rewrite the earlier command result or borrow evidence from a newer instance. This does not introduce a new retry engine.
|
||||
6. For historical records, expose available durable facts and unknowns. Do not backfill invented success timestamps, evidence sources, or completed phases from ambiguous prose.
|
||||
7. Keep existing IDs, history, API properties, `/api/deployments` compatibility, and legacy redirects. Use additive migrations and test both fresh and upgraded databases. Preserve the legacy `deployment.warnings` string; expose a validated string array through the additive page projection, with an honest fallback for malformed historical values.
|
||||
|
||||
Suggested boundaries:
|
||||
|
||||
- Shared provisioning detail/progress types near [provisioning.ts](../src/lib/provisioning.ts); a pure presentation mapper for labels, action/state coverage, and legacy fallbacks.
|
||||
- Server serialization and persistence helpers near [history.ts](../src/lib/server/provisioning/history.ts), with instrumentation in [worker.ts](../src/lib/server/provisioning/worker.ts), [instances.ts](../src/lib/server/provisioning/instances.ts), and [restarts.ts](../src/lib/server/provisioning/restarts.ts).
|
||||
- An abortable provisioning refresh/action controller under `src/lib/ui/`, taking lessons from the existing metrics refresh controller without changing `pollVisible` globally.
|
||||
- A thin detail route composed from outcome/actions, execution/evidence, and supporting context. Make activity improvements opt-in or provisioning-specific so campaign/beta history defaults stay intact.
|
||||
- Reuse the already-fetched detail in [the server loader](../src/routes/provisioning/[id]/+page.server.ts), preserving its campaign fallback and 404 behavior.
|
||||
|
||||
The current API returns all events on every poll; the existing Show more limits rendering only. Add an opt-in bounded detail query for this page and its initial server load: five recent entries, event total/latest ID, and the relevant failure excerpt. Fetch full history through `/api/provisioning/:id/events` in 25-entry pages with stable event-ID cursors, literal-text search, and explicit order. Keep existing default detail responses and `/api/deployments` behavior compatible. New events must not shift the active history page; show their count until the reader requests them. Test ordering and cursor stability with concurrent inserts, and measure representative long histories. This is a bounded query change, not a new event-storage system.
|
||||
|
||||
## Discussion decisions
|
||||
|
||||
| Proposal or disagreement | Decision and reason |
|
||||
| --- | --- |
|
||||
| Add a richer dashboard versus reorganize current capabilities | Reorganize around one operation and add only the evidence needed to explain it. Existing recipes, diagnostics, and recovery controls remain useful. |
|
||||
| Automatically expand all history on failure | Show the failure excerpt and five recent events; keep full expansion under operator control to avoid long, moving pages. |
|
||||
| Continue fetching all events because only five are displayed | Use an opt-in bounded detail/history API. A five-entry visual preview alone does not bound network or database work, and provisioning history is retained. Preserve legacy consumers. |
|
||||
| Compress rebuild into a few visual phases | Accepted only if remote deletion, local cleanup, replacement queueing, and verification remain separately inspectable. |
|
||||
| Improve progress using more stage-string matching | Rejected as the primary model. Use typed worker facts with an honest legacy fallback. Preserve old strings for compatibility and safety predicates. |
|
||||
| Show an exact queue position or ETA | Show actual blockers/capacity and optionally request order. Admission skips blocked requests; durations are not predictable from rank. |
|
||||
| Stop refreshing completed/failed jobs | Keep visibility-aware polling initially because recovery, deletion, and permitted actions can still change. |
|
||||
| Treat Deleted as making every action unavailable | Use backend capabilities. A historical server can still have permitted local cleanup without a live-server link. |
|
||||
| Treat failed request and recovery as one status | Keep them separate: the historical failure may remain while recovery continues and releases capacity. |
|
||||
| Add a new event system or scheduler | Rejected. Use additive snapshots and semantic metadata at existing execution boundaries. |
|
||||
|
||||
## Delivery sequence
|
||||
|
||||
These four increments form the proposed implementation scope. The data contract can be agreed first while layout work proceeds against fixtures.
|
||||
|
||||
| Increment | Work | Reviewable result |
|
||||
| --- | --- | --- |
|
||||
| 1. Trustworthy operational facts | Shared types, additive migration, worker checkpoint/evidence writes, typed read model, capability and legacy handling. | All six actions provide accurate progress and unknowns without changing execution, command counts, occupancy rules, or slot release. |
|
||||
| 2. Correct state and refresh behavior | Replace string-derived presentation; fix cancelled/runtime and telemetry copy, deleted cleanup gating, historical outcomes; reuse server-loaded detail; isolate navigation/poll/action requests and feedback. | The existing page remains truthful during cancellation, resume, slow requests, offline refresh, navigation, and continued recovery after failure. |
|
||||
| 3. Complete detail-page composition | Outcome/action card, blocker/evidence panel, grouped vertical phases, five recent events, bounded full-history queries, metadata/provenance/recipe placement, copy links, return context, responsive/accessibility work. | Operators can find the job's outcome, blocking condition, and valid next action without searching the activity log. |
|
||||
| 4. Compatibility and release validation | Exercise the full scenario matrix, migration/API compatibility, shared-component regressions, browser behavior, docs, and long-history measurements. | Implementation is ready for normal deployment with documented semantics and evidence of preserved lifecycle guarantees. |
|
||||
|
||||
Dependencies: evidence-dependent controls wait for increment 1; refresh and visual work can run in parallel against the agreed contract. Increment 4 is a release gate, not a later cleanup task.
|
||||
|
||||
## Acceptance and verification
|
||||
|
||||
The review baseline passed **84 tests across five suites**: provisioning, creation queue, operation history, deleted-server guards, and restart rollout. Those tests establish the existing baseline; they do not validate the proposed implementation. No full-build or accessibility-conformance claim is made for this planning work.
|
||||
|
||||
Implementation acceptance:
|
||||
|
||||
| Scenario | Required result |
|
||||
| --- | --- |
|
||||
| Every action across applicable queued/waiting/running/verifying/success/warning/failure/interruption/cancellation states | Correct headline, phase, evidence, timing, and permitted next action. Impossible combinations have explicit legacy/unknown handling. |
|
||||
| Occupied, unknown, stopped, stale, conflicting RCON/telemetry evidence | UI matches existing eligibility decisions; positive occupancy blocks disruption; stale/pre-submission/future telemetry cannot masquerade as accepted evidence. |
|
||||
| Capacity or parent/dependency hold | Actual blocker and matching links appear; no guaranteed admission order, fabricated check time, or ETA. |
|
||||
| Ambiguous remote reply, then recovery | No duplicate external command; slot stays reserved until the existing rules release/transfer it; historical failure and recovery evidence remain distinguishable. |
|
||||
| Partial destroy/rebuild and deleted identity | Confirmed remote deletion is retained; local cleanup can resume only when permitted; creation waits for full cleanup; old work never targets a replacement instance. |
|
||||
| Cancellation or resume racing a worker claim/newer operation | Backend remains authoritative; 409 reconciles the page; no automatic mutation replay or false success. |
|
||||
| Resume of the same operation ID | Prior failure outcome/time and explicit resume event remain preserved in semantic history while current progress/finish fields update; later completion records its own outcome without a new retry system. |
|
||||
| Telemetry success / wrong map / historical success | Evidence source and timestamps are accurate; expected and observed engine maps are clear; the page does not claim current health from old results. |
|
||||
| Slow poll + action, A→B→A navigation, stale error, unmount | Obsolete responses cannot affect current detail or action feedback; successful mutations invalidate stale reads and trigger reconciliation; pending state resets appropriately. If reconciliation fails, action acknowledgment and stale-data feedback remain distinct. |
|
||||
| Refresh failure and recovery | Last snapshot remains visible with stale labeling; Retry/Refresh works; action errors are not silently erased. |
|
||||
| Long activity, new events while reading | Search/order/disclosures/focus remain stable; new-event indicator works; no forced scrolling. Responses stay bounded; cursor pages have no duplicates/gaps within their documented ordering; legacy defaults remain compatible. Measure payload/query/render cost. |
|
||||
| Promotion occurs while create/rebuild is queued | Queued recipe is labeled as a capture, final attempted selection is recorded at creation, and the page does not imply pinned versions or successful installation. |
|
||||
| Legacy DB, absent recipe/evidence/source, deleted/reused name | Honest missing-data presentation, preserved IDs/events/foreign keys, safe links/capabilities, and unchanged legacy route/API compatibility. |
|
||||
| Keyboard, assistive announcements, phone, zoom, long text | No loss of reachable actions or page-wide overflow; progress is understandable without color; timers do not cause repeated announcements. |
|
||||
|
||||
Extend behavior tests around these risks rather than snapshotting markup or mirroring DTO construction. Verify that detail GETs perform no external probes. Run the affected lifecycle/history/refresh tests, then the full suite, `npm run check`, and `npm run build` for the implementation. Validate desktop/mobile browser fixtures and update [Provisioning documentation](provisioning.md) and relevant README descriptions.
|
||||
|
||||
Preserve the existing execution rules throughout: fresh occupancy gates, startup/restart limits, ambiguous-response handling, historical identity checks, exact cleanup checkpoints, requester ownership, and observation-only resumes. Normal deployment should sample both a newly instrumented operation and a legacy record; presentation rollback must not require deleting the new history.
|
||||
|
||||
Outside this proposal: generic retry/force-run controls, new cancellation semantics, scheduler redesign, real-time transport migration, a general observability platform, broad redesigns of other operation pages, and predicted completion times.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 202 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 130 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 424 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 358 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 295 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 357 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 200 KiB |
@@ -0,0 +1,46 @@
|
||||
# Leaderboard query benchmark
|
||||
|
||||
The benchmark and measurements below describe the earlier SQL optimization,
|
||||
before the worker and revision caches were added. They are historical results,
|
||||
not measurements of the current HTTP implementation.
|
||||
|
||||
Run the deterministic, isolated benchmark with:
|
||||
|
||||
```sh
|
||||
node scripts/benchmark-leaderboard.mjs --scenario all --baseline-ref eac9ad4 --runs 3 --json /tmp/leaderboard-benchmark.json
|
||||
```
|
||||
|
||||
The script creates temporary SQLite databases, loads the actual query modules,
|
||||
checks complete-response parity against the selected Git revision, and reports
|
||||
timings, source hashes and query plans. It never opens the application database.
|
||||
The small fixture has 2,000 matches and 8,000 participation rows; the large fixture
|
||||
has 50,000 matches and 200,000 participation rows. Both include completed,
|
||||
active, interrupted and nonqualifying records.
|
||||
|
||||
On the local Intel i5-8279U / Node 25.9.0 / SQLite 3.49.2 test environment, three
|
||||
measured runs after warmup gave these median query times on the large fixture:
|
||||
|
||||
| Request | Before | After |
|
||||
| --- | ---: | ---: |
|
||||
| Highest round, all games | 1,717 ms | 829 ms |
|
||||
| Total kills, all games | 1,555 ms | 858 ms |
|
||||
| One game | 899 ms | 491 ms |
|
||||
| One map | 345 ms | 224 ms |
|
||||
| Player search | 1,515 ms | 757 ms |
|
||||
| Deep page | 1,421 ms | 752 ms |
|
||||
| Headshot percentage | 1,444 ms | 760 ms |
|
||||
| Exact player rank | 1,524 ms | 554 ms |
|
||||
|
||||
The small fixture's unfiltered request fell from 41 to 30 ms. All 150 additional
|
||||
small-fixture parity cases and 22 timed small/large request cases matched the
|
||||
baseline's complete responses, including ratios and record evidence. Timings
|
||||
include SQL preparation and evidence lookup, exclude HTTP/browser rendering, and
|
||||
use a warm filesystem cache with alternating implementation order. These are
|
||||
local synthetic measurements, not production latency guarantees. SQL queries
|
||||
run synchronously within the report worker; larger histories still affect
|
||||
report latency and worker capacity.
|
||||
|
||||
Validation for that optimization passed 1,019 unit tests, Svelte checking with no errors
|
||||
or warnings, and a production build. Isolated Firefox walkthroughs at 1440px and
|
||||
390px checked completed-only totals, record links, efficiency qualification,
|
||||
scoped ranks/history, preserved navigation, empty scopes and mobile expansion.
|
||||
@@ -2,18 +2,30 @@
|
||||
|
||||
Research date: September 19, 2026. Scope: Plutonium T5 Zombies (`t5sp`) and T6 Zombies (`t6zm`), with particular attention to Electric Cherry's Plutainer/Docker host deployment.
|
||||
|
||||
This investigation predates the implemented lobby autoscaling feature. Repository
|
||||
observations and proposed changes below describe that research baseline. For
|
||||
current behavior, see [Lobby autoscaling](../guides/autoscaling.md) and
|
||||
[Native hosts](../guides/native-hosts.md); the
|
||||
[autoscaling plan](../plans/lobby-autoscaling-plan.md) tracks the remaining pilot.
|
||||
|
||||
## Recommendation and scope
|
||||
|
||||
**Recommendation:** first benchmark Wine's NTSYNC support and audit the effective runtime configuration. If every lobby must remain immediately available, trial CPU limits that apply only while a server is confirmed empty. If spare lobbies can be unavailable temporarily, keep a smaller pool running and start additional servers when demand rises. Stopping unused instances offers the most predictable reduction in their CPU consumption.
|
||||
|
||||
I found no documented, supported switch that makes both T5 and T6 dedicated servers automatically hibernate while preserving normal browser visibility and joins. There are useful alternatives, but they solve different parts of the problem.
|
||||
|
||||
This is research and a proposed evaluation plan. No production measurements were available, and no application code, server configuration, or running servers were changed. Reported savings below are either explicitly identified operator anecdotes or illustrative calculations.
|
||||
|
||||
**What applies to this project.** The repository launches one Plutainer container per lobby, selects `t5sp` or `t6zm`, and defaults new deployments to `ghcr.io/ayymoss/plutainer:latest`. Its creation specification does not explicitly configure CPU quotas or device passthrough. Electric Cherry refreshes server status every 30 seconds, tracks occupancy transitions, and checks container readiness before RCON. These are useful foundations for an idle policy, but they are not an implemented CPU governor. See [deployment creation](../src/lib/server/fleet/create.ts), [configuration](../src/lib/server/config.ts), [status worker](../src/lib/server/pipeline/worker.ts), and [status timing](../src/lib/server-freshness.ts).
|
||||
## Repository baseline
|
||||
|
||||
The repository launches one Plutainer container per lobby, selects `t5sp` or `t6zm`, and defaults new deployments to `ghcr.io/ayymoss/plutainer:latest`. Its creation specification does not explicitly configure CPU quotas or device passthrough. Electric Cherry refreshes server status every 30 seconds, tracks occupancy transitions, and checks container readiness before RCON. These are useful foundations for an idle policy, but they are not an implemented CPU governor. See [deployment creation](../../src/lib/server/fleet/create.ts), [configuration](../../src/lib/server/config.ts), [status worker](../../src/lib/server/pipeline/worker.ts), and [status timing](../../src/lib/server-freshness.ts).
|
||||
|
||||
The live host kernel, Wine version, image digest, installed scripts/plugins, CPU model, and actual idle utilization remain unknown. A repository default of `latest` does not establish which image a running server uses.
|
||||
|
||||
**Why an empty server can still be expensive.** Zero players does not establish that the engine, map scripts, plugin timers, Wine synchronization, or container helpers have stopped working. Those are competing explanations to measure. A map can also behave differently after its final player disconnects than it does immediately after startup. The research supports investigating these layers; it does not establish which one dominates this fleet.
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Expected effect on idle CPU | Availability and gameplay tradeoff | Evidence and priority |
|
||||
| --- | --- | --- | --- |
|
||||
| 1. Enable and verify NTSYNC | Potentially substantial if Wine synchronization dominates | Intended to retain normal joins and gameplay; compatibility testing required | Official Wine/kernel support; Plutonium operator anecdote. First trial |
|
||||
@@ -25,7 +37,9 @@ The live host kernel, Wine version, image digest, installed scripts/plugins, CPU
|
||||
| 7. Wake through a UDP gateway | Could combine stopped backends with automated wakeup | Protocol, listing, session, and timeout problems | Experimental; substantially harder for T5 |
|
||||
| 8. Change the host/runtime or develop engine idling | Potentially substantial, cause dependent | Migration cost or ongoing engine compatibility work | Conditional alternatives after profiling |
|
||||
|
||||
**1. Test NTSYNC before treating Wine overhead as unavoidable.** NTSYNC moves Windows synchronization primitives into a Linux kernel driver. Wine 11.0 documents using it when available and identifies Linux 6.14 as the kernel release providing the required module. The kernel documentation explains that the driver exists because equivalent userspace synchronization has performance limitations and exposes `/dev/ntsync`. These are established mechanisms, not Plutonium benchmark results. [Wine 11.0 release notes](https://github.com/wine-mirror/wine/blob/wine-11.0/ANNOUNCE.md), [Linux NTSYNC documentation](https://docs.kernel.org/userspace-api/ntsync.html).
|
||||
### 1. Test NTSYNC before treating Wine overhead as unavoidable
|
||||
|
||||
NTSYNC moves Windows synchronization primitives into a Linux kernel driver. Wine 11.0 documents using it when available and identifies Linux 6.14 as the kernel release providing the required module. The kernel documentation explains that the driver exists because equivalent userspace synchronization has performance limitations and exposes `/dev/ntsync`. These are established mechanisms, not Plutonium benchmark results. [Wine 11.0 release notes](https://github.com/wine-mirror/wine/blob/wine-11.0/ANNOUNCE.md), [Linux NTSYNC documentation](https://docs.kernel.org/userspace-api/ntsync.html).
|
||||
|
||||
A particularly relevant firsthand report, posted September 9, 2026, describes running Plutonium servers on Linux with NTSYNC and seeing idle usage around 25% of one CPU core. The same operator reports limiting empty servers to 15% of a core and removing that limit when players arrive. The post does not provide a controlled before/after benchmark or establish separate T5 and T6 results, so neither figure should become our target without measurement. The forum was available through search-indexed content; direct retrieval was unreliable during this research. [Operator report on NFOservers](https://forums.nfoservers.com/viewtopic.php?p=85675).
|
||||
|
||||
@@ -38,7 +52,9 @@ For this fleet, the experiment requires all of the following:
|
||||
|
||||
Start with one isolated T5 instance and one T6 instance, retaining the existing image and Wine prefix for rollback. A Wine upgrade may have prefix compatibility consequences; test on copies. Adding an environment variable alone does not establish working NTSYNC. If it is already active, enabling it again will not explain or fix remaining CPU use.
|
||||
|
||||
**2. Audit the work being done while empty.** This is the fastest way to distinguish an inherent runtime cost from an avoidable loop or configuration override.
|
||||
### 2. Audit the work being done while empty
|
||||
|
||||
This is the fastest way to distinguish an inherent runtime cost from an avoidable loop or configuration override.
|
||||
|
||||
First inspect `com_busyWait` in the actual dedicated process. Plutonium's r5162 changelog introduced it in connection with high client frame rates; r5304 changed its default to false for T4/T5/T6. Checking for an old override that enables it is worthwhile. Its effect on dedicated-server CPU is unconfirmed: verify that the setting exists, takes effect, and changes measured behavior before calling it a fix. [Plutonium changelog](https://plutonium.pw/docs/changelog/#r5304).
|
||||
|
||||
@@ -50,7 +66,9 @@ Also identify which processes consume CPU: the game executable, `wineserver`, lo
|
||||
|
||||
Electric Cherry's 30-second status polling is a separate source of work. Measure monitoring overhead before changing it: a long polling interval can make idle-to-active transitions harder to detect promptly. Do not infer that a high game-process CPU reading is caused by the management UI.
|
||||
|
||||
**3. Limit CPU while empty, and restore the normal budget as players arrive.** Docker can change a running Linux container's CPU limits without restarting it. A small controller could apply an idle budget only after repeated fresh zero-player readings and restore the original active budget when activity begins. [Docker container update](https://docs.docker.com/reference/cli/docker/container/update/).
|
||||
### 3. Limit CPU while empty, and restore the normal budget as players arrive
|
||||
|
||||
Docker can change a running Linux container's CPU limits without restarting it. A small controller could apply an idle budget only after repeated fresh zero-player readings and restore the original active budget when activity begins. [Docker container update](https://docs.docker.com/reference/cli/docker/container/update/).
|
||||
|
||||
For example, `--cpus=0.25` means an aggregate allowance equivalent to one-quarter of one logical CPU, not 25% of the entire host. This is an illustration, not a recommended production value. A quota bounds execution time; it does not remove the work the engine wants to perform. A cap below the current demand may therefore reduce the utilization graph while worsening responsiveness. [Docker CPU constraints](https://docs.docker.com/engine/containers/resource_constraints/#cpu).
|
||||
|
||||
@@ -66,11 +84,13 @@ Electric Cherry's current 30-second status cycle can leave a newly joined player
|
||||
|
||||
This is the strongest practical candidate when retaining every online lobby is a firm requirement. The ceiling is dependable; acceptable Plutonium behavior at a particular ceiling must be demonstrated.
|
||||
|
||||
**4. Keep a smaller pool of spare lobbies running.** If many maps have several empty copies, running all copies continuously multiplies the baseline cost. Keep occupied lobbies plus a chosen number of ready empty lobbies per exact map/ruleset, and stop surplus instances after an idle grace period. Start another spare when demand consumes the existing one; scheduled prewarming can cover known busy periods.
|
||||
### 4. Keep a smaller pool of spare lobbies running
|
||||
|
||||
If many maps have several empty copies, running all copies continuously multiplies the baseline cost. Keep occupied lobbies plus a chosen number of ready empty lobbies per exact map/ruleset, and stop surplus instances after an idle grace period. Start another spare when demand consumes the existing one; scheduled prewarming can cover known busy periods.
|
||||
|
||||
This is a proposed capacity policy. Docker stop ends container processes; the resulting elimination of their execution is predictable. Its shutdown behavior depends on the configured stop signal. The upstream Plutainer Dockerfile retrieved during this research specifies `STOPSIGNAL SIGKILL`, so do not assume a graceful shutdown: verify the deployed image and check that logs, stats, and other persistent state finish writing before stopping. [Docker stop behavior](https://docs.docker.com/reference/cli/docker/container/stop/), [Plutainer Dockerfile](https://github.com/Ayymoss/Plutainer/blob/main/Dockerfile).
|
||||
|
||||
Use stop/start semantics that retain the deployment, files, port reservation, and server key. Electric Cherry's destruction workflow removes infrastructure and files and is not a standby mechanism. Collection blueprints currently create shortfalls without removing excess servers, so they do not implement this policy either. [Deployment blueprints](deployment-blueprints.md).
|
||||
Use stop/start semantics that retain the deployment, files, port reservation, and server key. Electric Cherry's destruction workflow removes infrastructure and files and is not a standby mechanism. Collection blueprints currently create shortfalls without removing excess servers, so they do not implement this policy either. [Collections](../guides/collections.md).
|
||||
|
||||
Stopping and pausing need a final occupancy check, protection against a concurrent join, and coordination with deployments, recovery, and monitoring. A zero-player RCON reply alone cannot make a later stop atomic with joining; use a supported admission/drain mechanism if available, or explicitly accept and measure that race in the design. Unknown occupancy must prevent automatic stop or pause.
|
||||
|
||||
@@ -78,19 +98,25 @@ Preserve exact T6 selections: Town, Farm, and other variants cannot be pooled me
|
||||
|
||||
An illustrative calculation: 20 empty instances averaging 0.40 CPU each consume eight logical-CPU equivalents. Keeping five running at the same rate uses two, a 75% reduction in that empty-instance component. These are invented inputs to demonstrate scaling; occupied games, startup bursts, and management services remain additional costs.
|
||||
|
||||
**5. Start infrequent maps on request, or prewarm them on a schedule.** A map reservation page or operator action can start a stopped server, wait for readiness, and then tell players it is available. A schedule can start extra lobbies before a recurring play session and stop them later only if they are confirmed empty. This removes the need to infer intent from game packets.
|
||||
### 5. Start infrequent maps on request, or prewarm them on a schedule
|
||||
|
||||
A map reservation page or operator action can start a stopped server, wait for readiness, and then tell players it is available. A schedule can start extra lobbies before a recurring play session and stop them later only if they are confirmed empty. This removes the need to infer intent from game packets.
|
||||
|
||||
For public discovery, keep popular maps online and make rarely used maps available through this explicit flow. The tradeoff is waiting for startup and reduced browser presence while stopped. Pre-pull images and stage updates outside the wake path; measure startup with realistic caches and map assets.
|
||||
|
||||
T5 needs special treatment. Its official setup documentation says joins use a backend-issued session ID rather than direct IP connections. T6's setup guide documents IP/port connections. Therefore, a T5 wake flow should verify renewed backend registration and browser joinability before telling users it is ready; an RCON response alone does not prove that. The T5 page also contains generic IP-connect advice elsewhere, so use its explicit T5 session caveat and validate actual behavior. [T5 connection FAQ](https://plutonium.pw/docs/server/t5/setting-up-a-server/#why-cant-i-connect-to-my-server-via-the-console), [T6 setup guide](https://plutonium.pw/docs/server/t6/setting-up-a-server/).
|
||||
|
||||
**6. Pause containers as an experiment for faster wakeup.** Docker pause freezes container processes on Linux. This preserves process memory while suspending execution, making it a candidate when cold startup is expensive and RAM is plentiful. [Docker pause](https://docs.docker.com/reference/cli/docker/container/pause/).
|
||||
### 6. Pause containers as an experiment for faster wakeup
|
||||
|
||||
Docker pause freezes container processes on Linux. This preserves process memory while suspending execution, making it a candidate when cold startup is expensive and RAM is plentiful. [Docker pause](https://docs.docker.com/reference/cli/docker/container/pause/).
|
||||
|
||||
The consequence is equally important: the frozen game cannot execute RCON replies, join handling, or backend heartbeats. An external controller must resume it. Treat possible listing expiry, stale sessions, timer jumps, and failed first joins as acceptance tests; successful unpause alone is insufficient. Coordinate monitoring so deliberate suspension is not treated as a fault, and test after both short and overnight pauses.
|
||||
|
||||
I would test stop/start first because it gives a clean startup path. Pause becomes attractive only if measured resume reliability and latency justify it. It retains memory, and it does not make a server remain genuinely responsive while asleep.
|
||||
|
||||
**7. Build a UDP wake gateway only if seamless demand detection is valuable enough.** The concept is a lightweight listener that owns a stable public endpoint, distinguishes a join attempt from a status query, wakes the backend, and forwards game traffic when ready. This is a design proposal, not a verified T5/T6 product or a configuration toggle.
|
||||
### 7. Build a UDP wake gateway only if seamless demand detection is valuable enough
|
||||
|
||||
The concept is a lightweight listener that owns a stable public endpoint, distinguishes a join attempt from a status query, wakes the backend, and forwards game traffic when ready. This is a design proposal, not a verified T5/T6 product or a configuration toggle.
|
||||
|
||||
There is an architectural precedent: Game Server Snooze implements packet-triggered startup, bounded buffering, and idle shutdown for Palworld under Pterodactyl. Its documented game support is Palworld, so it is a reference design, not something established to work with this fleet. [Game Server Snooze upstream](https://github.com/dawidkulpa/game-server-snooze).
|
||||
|
||||
@@ -103,7 +129,9 @@ Its difficult parts are protocol-specific:
|
||||
|
||||
T6 is the more plausible first prototype because direct IP joining is documented, but transparent success still needs testing. For T5, a request-to-start flow is the more credible initial design. Do not assume that answering a generic status query reproduces Plutonium's registration or authentication behavior.
|
||||
|
||||
**8. Conditional alternatives: host changes and engine support.** These merit investigation when the measurements point to them.
|
||||
### 8. Conditional alternatives: host changes and engine support
|
||||
|
||||
These merit investigation when the measurements point to them.
|
||||
|
||||
If the Docker host itself runs inside a VM, audit virtual CPU exposure and host contention. One T4/T5 operator reported resolving excessive CPU consumption by changing a Proxmox Windows VM from `x86-64-v3` to `host`. Another operator in the same thread reported normal behavior with `x86-64-v3`, so this is a conditional clue, not a universal fix or proof of a Plutonium regression. [Firsthand performance discussion](https://forum.plutonium.pw/topic/45013/server-performance-issues/11).
|
||||
|
||||
@@ -115,11 +143,15 @@ The most ambitious route is an upstream engine idle mode or supported plugin-ass
|
||||
|
||||
This route has the best conceptual fit for an always-listed, low-CPU server, but the highest development and compatibility uncertainty. A useful upstream request would include clean-server profiles, per-thread CPU data, exact versions, and measured differences between zero players, connecting players, and active play.
|
||||
|
||||
**Settings and shortcuts I would not prioritize.** Reducing `sv_fps` fleet-wide is not an established solution: Plutonium's changelog records T5 SP `sv_fps` becoming cheat-protected and a historical T6 Zombies dedicated tick-rate increase from 10 to 20 Hz. Neither entry establishes a safe idle-only override. Client FPS advice for `com_maxfps` also does not establish a dedicated-server fix. I found no documented common T5/T6 hibernation setting. [Plutonium changelog](https://plutonium.pw/docs/changelog/).
|
||||
## Lower-priority settings and shortcuts
|
||||
|
||||
Reducing `sv_fps` fleet-wide is not an established solution: Plutonium's changelog records T5 SP `sv_fps` becoming cheat-protected and a historical T6 Zombies dedicated tick-rate increase from 10 to 20 Hz. Neither entry establishes a safe idle-only override. Client FPS advice for `com_maxfps` also does not establish a dedicated-server fix. I found no documented common T5/T6 hibernation setting. [Plutonium changelog](https://plutonium.pw/docs/changelog/).
|
||||
|
||||
CPU shares and affinity can help manage contention, but they should not be mistaken for eliminating idle work. Docker documents that shares act when CPU is constrained and that a CPU set restricts where execution occurs. For actual utilization ceilings, use quotas; for elimination of unnecessary instances' work, stop them. [Docker resource controls](https://docs.docker.com/engine/containers/resource_constraints/#cpu).
|
||||
|
||||
**How to choose with a small, repeatable trial.** First record the actual host/VM details, kernel, image digest, Wine version, Plutonium revision, scripts/plugins, map/ruleset, and CPU limits. Keep those fixed while changing one variable. Pinning an image alone is insufficient if the wrapper updates game binaries at startup; record the resulting binary versions too.
|
||||
## Run a repeatable trial
|
||||
|
||||
First record the actual host/VM details, kernel, image digest, Wine version, Plutonium revision, scripts/plugins, map/ruleset, and CPU limits. Keep those fixed while changing one variable. Pinning an image alone is insufficient if the wrapper updates game binaries at startup; record the resulting binary versions too.
|
||||
|
||||
Use one representative instance from each game, then expand successful changes to other maps. Suggested measurement windows are 10–15 minutes after startup settles, repeated after players join and all leave. These are proposed test windows, not product requirements.
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
# UI refresh
|
||||
|
||||
The interface keeps the existing dark palette and blue accent, with shared controls and task-focused pages.
|
||||
|
||||
The follow-up [UI/UX review implementation](ui-ux-implementation.md) records the approved operational, recovery, accessibility, and mobile improvements, with recommendation coverage and validation.
|
||||
|
||||
1. **Shared design system:** common headers, buttons, fields, dialogs, spacing, radii, and focus styles. Removed unused styles from retired layouts.
|
||||
2. **Libraries:** Plugins opens on the searchable library. Add plugin opens Upload / GitHub tabs. Scripts uses compact status filters; Configs uses a compact summary and Library / Inspect tabs.
|
||||
3. **Beta setup:** `/betas/new` guides Files → Servers → Test plan → Review. Exact versions, selected servers, and the plan survive Back / Continue. Existing beta selection links still redirect correctly.
|
||||
4. **Fleet density:** compact totals and attention summary, with grouping, sorting, and display options behind View options.
|
||||
5. **Server identity:** friendly map names and visible lobby numbers across cards, lists, beta setup, and operations. T6 variants retain their selected map name. Server details provides a copyable raw ID.
|
||||
6. **Deployment organization:** Active / Blueprints / History views. Failures and warnings link to filtered history. Destroy fleet lives in Fleet actions and retains its explicit preview.
|
||||
7. **Navigation:** Fleet / Library / Operations / Betas / Settings, with section navigation and a mobile menu. Existing route URLs remain usable.
|
||||
8. **Status and progress:** shared status labels and tones distinguish ordinary waiting, work in progress, attention, passing tests, and promotion. Deployment, campaign, and beta details share operation history.
|
||||
9. **Server detail hub:** live status, current operations, scoreboard, logs, effective configuration, inventory, diagnostics, and scoped campaign history. The scoreboard shows game-reported names, copyable GUIDs, score, kills, headshots, downs, and revives, with sortable stats, map, round, and update time. Final scores remain visible until the next game starts. Missing or stale player counts are unknown. Inventory identifies scripts, plugins, configs, and other files.
|
||||
10. **Dialogs and feedback:** in-app confirmations for deletion, rollback, reconciliation, and unsaved edits. File comparisons use a native modal dialog with Escape and focus restoration. Refresh errors retain loaded data and provide retry controls.
|
||||
11. **Keyboard and forms:** labeled fields, roving keyboard tabs, visible focus, a skip link, and separate navigation/action targets. Removed nested buttons and buttons inside campaign links.
|
||||
12. **Readability:** brighter secondary metadata, readable small labels, contrasting primary-button text, and full-opacity offline cards.
|
||||
13. **Page states:** compact loading and empty states, actionable failures, and retained snapshots during refresh errors.
|
||||
14. **Mobile:** wrapping identities, scrollable tab/filter rows, compact summaries, responsive forms, and sticky form actions.
|
||||
15. **Loading and rendering:** shared Fleet data, detail tabs loaded on demand, no overlapping scheduled page requests, and polling paused in hidden tabs. Watches can keep Fleet monitoring active. Long libraries, histories, and evidence lists progressively render more items. Server history is filtered before the API limit.
|
||||
|
||||
## Maintaining the patterns
|
||||
|
||||
- Use `SectionHeader` for page titles and primary actions. The containing page owns spacing between sections.
|
||||
- Use `Field` with `Input`, `Select`, or `Textarea` for associated labels, help, and errors. Give controls distinct `ariaLabel` values when several share one field.
|
||||
- Use `Dialog` for new modal workflows and `confirmAction` for ordinary confirmations. Browser unload protection remains native.
|
||||
- Use `StatusBadge` for workflow states and `ServerIdentity` for server names. Preserve raw IDs where operators need to copy or troubleshoot them.
|
||||
- Pair `Tabs` with a labeled panel. Use `ListFooter` for long lists and `pollVisible` for scheduled page refreshes.
|
||||
- Read live server data from the Fleet store instead of starting another `/api/servers` polling loop.
|
||||
@@ -1,59 +0,0 @@
|
||||
# UI/UX review implementation
|
||||
|
||||
Implemented September 19, 2026, following [the approved review](ui-ux-review.md).
|
||||
|
||||
The changes prioritize clear operational outcomes, reviewed action scope, recoverable input, and usable small-screen layouts. Existing server, file, beta, and launch semantics remain visible in the interface rather than being implied by a successful request.
|
||||
|
||||
## Recommendation coverage
|
||||
|
||||
| Review | Implemented behavior |
|
||||
| --- | --- |
|
||||
| 1. Stale Fleet data | Stale cards explicitly show **Players unknown**, retain labeled last-known counts and rosters, and stop advancing live durations. Fleet shows snapshot age and refresh/reconnection status. |
|
||||
| 2. Promotion impact | A promotion review lists exact versions, tested and additional servers, eligibility, staging, and restart policy. The backend checks a review token before writes; changed targets or composed config require a new review. Completion links to the resulting campaign. |
|
||||
| 3. Reconciliation scope | Selected-server and entire-fleet reviews are separate actions. The plan includes scan time, file operations and restart policy. Execution checks target identities and before/after content fingerprints; changed plans return for review. |
|
||||
| 4. Beta reading order | Outcome, blockers, deadline, actions, and server results precede evidence and activity. Shared activity history is collapsed, searchable, ordered, and progressively displayed. |
|
||||
| 5. Beta recovery | Deadlines and pause/retry consequences are explicit. A fresh attempt confirms evidence reset and restarts; expired tests explain rollback and plan reuse. Sign-off availability follows test/run readiness. |
|
||||
| 6. Launch input retention | Refreshing previews retains entered keys only for unchanged exact server IDs. Changed assignments are explained. Keys remain in memory; managing keys in another tab preserves the launch context. |
|
||||
| 7. Provisioning cancellation | Active jobs can be cancelled from operation lists/details, including restarts and health verification. Remaining steps stop; effects already performed are retained. Rollback supersedes conflicting jobs with fresh restoration requests. |
|
||||
| 8. Partial outcomes | Reconciliation reports updated and skipped servers separately, with reasons and follow-up links. All-skipped work is an error outcome. Rollback feedback stays beside its initiating operation. |
|
||||
| 9. Scope clarity | Scope editing previews added/removed eligible servers. Script/plugin unscoping explains retained live copies and tracking detachment; configs explain composition impact. Completion links to reconciliation. Eligibility, promoted versions, and installed inventory have separate labels. |
|
||||
| 10. Accessible controls | Busy buttons retain their accessible names. Field labels focus the associated native control. Consequential errors use alert semantics; completion uses status semantics. Campaign server states have accessible descriptions. |
|
||||
| 11. Server investigation | Offline/stale summaries offer Diagnostics and Logs. URL parameters retain the selected tab, RCON diagnosis precedes telemetry history, and removal is under Server actions. |
|
||||
| 12. Beta drafts | Drafts retain exact versions, label, test plan, selected servers, and wizard step. Resume/discard is explicit, including returns through artifact links. Eligibility is rechecked and compatible selections retained. Unavailable browser storage triggers leave protection. |
|
||||
| 13. Local validation | Blueprints display total capacity and duplicate-map errors. Beta requirements show field-specific errors and units, expand/focus invalid checks, and validate quorum against the selected cohort. |
|
||||
| 14. Navigation context | Fleet views and library/operation filters are represented in URLs. Library loaded counts and appropriate selections survive detail navigation and Back. Server and log tabs have direct links. Explicit Fleet URL choices override saved personal defaults. |
|
||||
| 15. System setup | Settings opens a read-only System status page with configured/unconfigured/unreachable/unverified states, prerequisite instructions, saved-key counts, and first-server guidance. Empty/unconfigured Fleet links directly to setup. |
|
||||
| 16. Stable config preview | The last successful inspection remains visible during edits, with an updating state. Expanded keys survive refresh. Draft comparisons and ignored-content warnings remain near the editing task. |
|
||||
| 17. Config composition | Inspection numbers composition layers, identifies winning sources and superseded assignments, and compares draft values with promoted values. Structured editing supports adding a setting without raw syntax. |
|
||||
| 18. Beta target selection | Files show scope and eligible counts. Conflicts identify incompatible file scopes. Server picking supports search, game/map grouping, health/player context, selected counts and select-visible controls. |
|
||||
| 19. Evidence interpretation | Requirements explain the next qualifying condition using current observations. Relevant-evidence links apply server/event filters; readable summaries precede raw payloads and telemetry diagnosis is directly linked. |
|
||||
| 20. Artifact next action | The latest exact version and its test/active-beta action lead the detail page. Technical hashes/sizes are disclosures, version cards remain readable on phones, unscoped game filtering is consistent, and hidden selections are counted. |
|
||||
| 21. Import continuity | GitHub import shares pending state with its parent dialog, prevents dismissal/tab switches during import, and keeps a completion panel with version/beta links. Script batches show aggregate progress and sticky actions. |
|
||||
| 22. Key management | Game/map/lobby selectors generate the exact ID, with an expert raw-ID option. Rows show saved time, presence-versus-validation wording, deployment implications, and links to existing servers. Launch forms accept the exact missing/replacement key inline. |
|
||||
| 23. Blueprint coverage | Existing/queued inventory is distinguished from healthy playable servers, with linked coverage details. Collections group assignments and show a sticky key count and jump-to-missing-key action. Creation remains limited to the target shortfall. |
|
||||
| 24. Mobile overhead | Fleet avoids redundant headings/zero campaign summaries, empty libraries emphasize their next action, and Leaderboard totals collapse on phones with a compact selected-stat table. Long forms retain reachable actions and focus spacing. |
|
||||
| 25. Consistent hierarchy | Editable controls use stronger dedicated boundaries. Routes have descriptive titles, operational language is clearer, keyboard shortcuts have a help panel, and selected overflow tabs scroll into view without moving the page. |
|
||||
| 26. Investigation/handoffs | Logs support literal search/highlighting, copy, tail download, deep links, explicit follow/refresh state and jump-to-latest. Beta rows include labels, age and blocker context. |
|
||||
|
||||
The review explicitly made overlay-priority reordering and Leaderboard player history/recent-period views conditional on further design or operator demand. Those product extensions remain future discovery work; the composition explanations and mobile Leaderboard changes are implemented.
|
||||
|
||||
Activity and notifications have been removed. Copy results appear next to their controls; GitHub imports retain their inline completion and error states.
|
||||
|
||||
## API and operational boundaries
|
||||
|
||||
- Promotion uses `GET /api/betas/:id/promote` to review impact and `POST` with the returned token as `previewToken` to apply it. A changed plan returns HTTP 409 and needs renewed review.
|
||||
- Reconciliation uses `POST /api/drift` with `dryRun: true` and optional `servers`, then executes with the returned `expectedFingerprint`. Omitted servers explicitly mean the entire fleet. A stale review returns HTTP 409; changes detected during application are reported as skipped files.
|
||||
- `POST /api/deployments/:id/cancel` cancels remaining provisioning work, including restarts and health checks. It does not undo changes already made.
|
||||
- System status exposes no credential values. It reports host-agent readiness and verified game-file caches.
|
||||
- Drafts contain nonsecret beta setup. Launch keys are not stored in browser drafts.
|
||||
- Log search/export covers the loaded tail, not an entire remote file or a full diagnostic archive.
|
||||
|
||||
## Validation
|
||||
|
||||
- Full regression suite: **473 tests passed across 44 files**. Added coverage includes changed promotion scope/config, changed reconciliation bytes/targets, cancellation racing a worker, draft parsing/compatibility, scope previews, config comparison and system-status semantics.
|
||||
- `npm run check`: **0 errors, 0 warnings**.
|
||||
- `npm run build`: **passed**. The existing SvelteKit CSRF configuration deprecation remains a tooling notice.
|
||||
- Local browser walkthroughs use an isolated database and synthetic populated API fixtures, with production integration credentials disabled. Fleet, server diagnostics, logs, Settings, Leaderboard, libraries, beta setup/progress, launches, keys, and reconciliation are exercised at desktop and phone widths.
|
||||
- Browser checks include the loading button accessibility tree, clickable native labels, URL/Back restoration, retained previews/drafts, selected-tab visibility, pending import ownership, and horizontal overflow. These are targeted checks, not a claim of complete accessibility conformance.
|
||||
|
||||
Live host provisioning, real game-server restarts and removals were not executed during verification. Their boundary behavior is covered by regression tests and simulated UI flows.
|
||||
@@ -1,359 +0,0 @@
|
||||
# Electric Cherry UI/UX review
|
||||
|
||||
Reviewed September 19, 2026.
|
||||
|
||||
Implementation follow-up: [completed changes and validation](ui-ux-implementation.md).
|
||||
|
||||
Electric Cherry has a coherent visual foundation and unusually capable operational workflows. The highest-value next investment is making operational decisions easier: what is known, what is waiting, what an action will affect, and how to recover. Preserve the existing dark palette, shared controls, friendly server identities, and guided setup while improving those decisions.
|
||||
|
||||
## Scope and evidence
|
||||
|
||||
This review combines source inspection with a local browser walkthrough. It covers Fleet, server details and diagnostics, Scripts, Configs, Plugins, beta creation and progress, server deployments, blueprints, campaigns, keys, and Leaderboard. Existing design notes and workflow documentation informed the review.
|
||||
|
||||
Browser exploration used an isolated database with external connections disabled. Nine top-level pages were examined at 1440×1000 and 390×844. Populated Fleet, server details, and beta scenarios used synthetic API responses. Beta setup, blueprint validation, and the browser accessibility tree were also exercised. Navigation was checked around its responsive breakpoint. Production integrations, actual restarts, and destructive actions were not executed.
|
||||
|
||||
**Evidence labels:** **Rendered** means reproduced in the local browser, sometimes with an explicitly synthetic populated state. **Source** means verified in UI/API implementation. **Opportunity** means a proposed capability or design direction, rather than a defect. Recommendations are expert judgments; they are not findings from end-user interviews or a complete accessibility conformance audit.
|
||||
|
||||
No application code was changed for this review. The accompanying screenshots contain synthetic review data.
|
||||
|
||||
## What to preserve
|
||||
|
||||
- The shared tokens, restrained blue accent, readable text, friendly map/lobby identities, and separate navigation/action targets.
|
||||
- The existing Fleet search, status filters, pins, card/table views, grouping, live freshness handling in aggregate counts, and lobby watches.
|
||||
- The four-step beta setup, exact version selection, explicit server opt-in, readable test-plan summaries, and review step.
|
||||
- Script batch intake, duplicate recognition, partial-success retry, and direct transfer into beta setup.
|
||||
- Config draft recovery, unsaved-edit protection, effective-setting inspection, and override provenance.
|
||||
- Deployment previews, shared-queue visibility, and separate recovery actions for health verification and filesystem cleanup.
|
||||
- Native modal dialogs, keyboard-operated tabs, visible focus, reduced-motion support, retained snapshots after refresh errors, and progressive list rendering.
|
||||
|
||||
Many generic dashboard recommendations are already implemented. The findings below build on them.
|
||||
|
||||
## Recommended order
|
||||
|
||||
P1 means address first because the issue affects trust, consequential actions, lost work, or basic access. P2 means a meaningful workflow improvement. P3 means refinement or a capability to validate with operators. Relative effort is directional: S is a localized UI change; M crosses several components or state flows; L requires a new backend operation or product capability.
|
||||
|
||||
| First work | Recommendation | Priority | Relative effort |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | Label stale player data consistently | P1 | S |
|
||||
| 2 | Preserve loading-button accessible names | P1 | S |
|
||||
| 3 | Put beta results and decisions above the event log | P1 | S–M |
|
||||
| 4 | Preview promotion targets and link the resulting campaign | P1 | M |
|
||||
| 5 | Make reconciliation scope and partial results explicit | P1 | M |
|
||||
| 6 | Preserve keys when launch previews refresh | P1 | M |
|
||||
| 7 | Explain blueprint and test-plan validation locally | P1/P2 | S–M |
|
||||
| 8 | Surface beta deadlines, reset consequences, and valid recovery actions | P1 | M |
|
||||
|
||||
Cancellation of queued destruction is also important, but needs an explicit backend boundary and should be designed as a separate capability.
|
||||
|
||||
## Recommendations
|
||||
|
||||
### 1. Make stale Fleet cards unmistakable — P1 · Rendered + Source
|
||||
|
||||
Fleet correctly excludes stale servers from live player totals, but passes their old player counts into cards. A stale card can still display a large blue player count, roster, and advancing match duration; its main distinction is an amber status dot and an older “Last seen” timestamp. In the synthetic review, the header reported **3 players online**, while an additional stale card prominently displayed **4**.
|
||||
|
||||
Show **Players unknown** or an em dash as the primary value. Retain historical information under “Last known: 4 players, 3 minutes ago,” with an explicit **Stale** label. Apply the same current-versus-last-known distinction to roster and timing. Add the age of the last successful Fleet refresh and a reconnecting state when appropriate.
|
||||
|
||||
Acceptance: when a fresh server ages into stale status, both card and table stop presenting its previous player count as current. Preserve historical context without requiring interpretation of color.
|
||||
|
||||
Sources: [Fleet page](../src/routes/+page.svelte), lines 56, 62, 208; [ServerCard](../src/lib/ui/components/ServerCard.svelte), lines 72, 114, 150. [Screenshot](ux-review-assets/fleet-stale-count.png).
|
||||
|
||||
### 2. Review the full impact of promotion — P1 · Source
|
||||
|
||||
“Promote” immediately sends a request. The backend expands the tested files to all currently matching scope targets and queues restarts for additional servers. The UI does not show that expansion at the moment of approval. Afterward, it ignores the returned campaign ID.
|
||||
|
||||
Add a concise promotion review with exact versions, tested servers, additional eligible servers, and restart behavior. Use a final label such as **Promote to 12 servers**. Explain eligibility, file staging, and activation separately. After submission, show **Promotion created — 9 servers awaiting activation** with **View campaign**. Keep “Promoted version” distinct from “Running everywhere.”
|
||||
|
||||
Acceptance: an operator who tested one lobby can identify the entire affected set and follow remaining restarts without searching another section.
|
||||
|
||||
Sources: [Beta detail](../src/routes/betas/[id]/+page.svelte), lines 72–79; [promotion implementation](../src/lib/server/fleet/beta.ts), lines 778, 1017, 1060.
|
||||
|
||||
### 3. Make drift reconciliation match the reviewed scope — P1 · Source
|
||||
|
||||
The drift list can be filtered by server, but Reconcile posts an empty body and affects the full fleet. The confirmation includes counts but omits automatic restarts. Execution rescans, so the action can also differ from the displayed scan.
|
||||
|
||||
Separate **Reconcile selected servers** from **Reconcile entire fleet**. Show a file-level plan, scan time, affected servers, and activation policy. If the reviewed plan changes materially, show the updated plan before execution. The API already accepts `servers` and `dryRun`, providing a starting point.
|
||||
|
||||
Acceptance: filtering or selecting one server never leaves the action scope ambiguous. An operator can explain which files will be restored or removed and when servers will restart.
|
||||
|
||||
Sources: [Campaigns page](../src/routes/campaigns/+page.svelte), lines 118, 124, 203, 268; [drift API](../src/routes/api/drift/+server.ts), line 28; [drift implementation](../src/lib/server/fleet/drift.ts), line 265.
|
||||
|
||||
### 4. Put beta outcome and next action above activity — P1 · Rendered + Source
|
||||
|
||||
Beta detail renders up to 25 expanded activity events before files, requirements, per-server progress, and actions. In a synthetic three-server beta with three requirements per server and 25 events:
|
||||
|
||||
| Content starts | Desktop 1440×1000 | Mobile 390×844 |
|
||||
| --- | ---: | ---: |
|
||||
| Activity | 436 px | 720 px |
|
||||
| Test plan/results | 2,623 px | 3,590 px |
|
||||
| Pending sign-off | 3,528 px | 4,846 px |
|
||||
| Promote / Rollback | 4,011 px | 5,462 px |
|
||||
|
||||
Place a compact outcome summary first: passed servers, outstanding conditions, deadline, and primary action. Follow with a failed/waiting-first server table. Put activity under a disclosure or separate tab. Keep the useful operation-flow steps, but reduce the oversized started/finished block.
|
||||
|
||||
Acceptance: without scrolling through historical events, an operator can answer “What is blocking this beta, and what can I do?”
|
||||
|
||||
Sources: [Beta detail](../src/routes/betas/[id]/+page.svelte), line 137; [OperationHistory](../src/lib/ui/components/OperationHistory.svelte), lines 21, 46. [Screenshot](ux-review-assets/beta-activity-first.png).
|
||||
|
||||
### 5. Make beta time and recovery rules visible — P1 · Source
|
||||
|
||||
Progress describes a testing window but does not show the actual expiry or time remaining. Waiting for empty servers consumes the window; pausing and retrying retain the original deadline. Starting a fresh attempt resets evidence and causes new restarts. Sign-off controls can appear in states where the API rejects them.
|
||||
|
||||
Show **Ends tomorrow at 14:30 · 6h remaining**, plus **Pausing does not extend this deadline**. Explain evidence reset and restart consequences before a fresh attempt. Offer only valid actions, with a reason for unavailable sign-off. For deadline expiry, explain the required rollback/new-beta sequence and offer a way to reuse the plan.
|
||||
|
||||
Acceptance: paused, failed, inconclusive, and passed tests each have an understandable next step; a retry never silently implies extra testing time.
|
||||
|
||||
Sources: [BetaTestProgress](../src/lib/ui/components/BetaTestProgress.svelte), lines 29, 35, 53; [test action API](../src/routes/api/betas/[id]/test/+server.ts), lines 21, 48–53; [test-plan documentation](beta-test-plans.md).
|
||||
|
||||
### 6. Preserve launch inputs through preview refresh — P1 · Source
|
||||
|
||||
Refreshing a single-server preview clears its entered key. Refreshing a collection clears all entered keys. Preview expiry and capacity changes are normal recovery paths, so users can lose substantial input while doing what the UI requires.
|
||||
|
||||
Keep entered keys in memory, indexed by exact server ID. Reuse a key only when that ID survives the refreshed allocation; flag changed assignments explicitly. Preserve nonsecret launch choices across a visit to Manage keys. Do not persist raw keys into local browser drafts.
|
||||
|
||||
Acceptance: after entering five keys, an unchanged preview refresh retains all five. A new lobby assignment never inherits another server’s key.
|
||||
|
||||
Sources: [DeployServerDialog](../src/lib/ui/components/DeployServerDialog.svelte), lines 41–45, 66, 123; [DeployCollectionDialog](../src/lib/ui/components/DeployCollectionDialog.svelte), line 27.
|
||||
|
||||
### 7. Give queued destructive work a withdrawal window — P1 · Opportunity
|
||||
|
||||
Server removal can wait for occupied or unreachable lobbies. Current detail pages provide resume actions but no cancellation before destruction starts. A delayed command can therefore outlive the operator’s original intention.
|
||||
|
||||
Add **Cancel queued removal** while the operation is still waiting, with a precise backend transition after which cancellation is unavailable. For fleet removal, show per-server waiting/removing/completed states and cancellation eligibility. Clearly distinguish cancelling a request from restoring a server already removed.
|
||||
|
||||
Acceptance: an operator can withdraw a waiting removal, and a race with worker execution cannot leave the UI claiming cancellation after removal has begun.
|
||||
|
||||
Sources: [DestroyServerModal](../src/lib/ui/components/DestroyServerModal.svelte), lines 36–40; [HousePartyDialog](../src/lib/ui/components/HousePartyDialog.svelte), line 51; [deployment detail](../src/routes/provisioning/[id]/+page.svelte), lines 110–123. Existing consequence confirmations should remain.
|
||||
|
||||
### 8. Report partial success as a distinct outcome — P1 · Source
|
||||
|
||||
Reconciliation returns per-server skipped reasons, but the UI reads aggregate changed-file counts. Errors and success messages also share a blue-toned flash container. Rollback feedback can appear in the drift card far from the row that initiated it.
|
||||
|
||||
Use structured completion feedback: **7 servers updated, 2 skipped**, with reasons, relevant retry actions, and a campaign link. Place feedback beside the affected task. Keep success, partial success, and failure visually and programmatically distinct.
|
||||
|
||||
Acceptance: a request that skipped every target cannot look like an uneventful successful run; the operator can find the unresolved servers immediately.
|
||||
|
||||
Sources: [Campaigns page](../src/routes/campaigns/+page.svelte), lines 128–138, 181–184, 299, 508; [drift implementation](../src/lib/server/fleet/drift.ts), lines 349, 371, 383–410.
|
||||
|
||||
### 9. Explain scope changes and distinguish eligibility from installation — P1 · Source
|
||||
|
||||
The artifact scope dialog offers selectors and Save. Unscoping scripts/plugins detaches inventory tracking while leaving live copies; unscoping configs changes the intended composed configuration. Both can create reconciliation work. The UI discards the returned detach result for scripts/plugins. The detail label “Applies to” also represents matching scope, not verified installed versions.
|
||||
|
||||
Preview added/removed eligible servers. Explain retained live files when unscoping, show the affected count afterward, and link to the relevant reconciliation view. Determine config impact from composition rather than the script/plugin detach count. Use separate labels for **Eligible servers**, **Promoted version**, and **Installed version**.
|
||||
|
||||
Acceptance: changing eligibility never suggests that running servers have already changed. An operator knows when follow-up reconciliation is required.
|
||||
|
||||
Sources: [ArtifactDetail](../src/lib/ui/components/ArtifactDetail.svelte), lines 183, 298, 380; [scope API](../src/routes/api/scripts/[id]/scope/+server.ts), lines 12, 30; [unscoping behavior](../src/lib/server/fleet/beta.ts), lines 1081–1108.
|
||||
|
||||
### 10. Keep controls named and feedback accessible during actions — P1 · Rendered + Source
|
||||
|
||||
The shared Button hides its label with `visibility: hidden` while its spinner is hidden from assistive technology. Browser accessibility inspection confirmed that **Refresh** becomes an unnamed button while busy. It retains `disabled` and `busy`, but loses its name.
|
||||
|
||||
Preserve the accessible label during loading and announce meaningful progress separately. Audit action failures for alert semantics, and success messages for status semantics. Replace per-server status dots whose explanation exists only in hover text with visible or accessible text. Ensure Field labels focus their associated controls; current label context provides accessible naming, but many visible labels are divs.
|
||||
|
||||
Acceptance: the same action has a stable accessible name before, during, and after execution; keyboard and touch users can identify which server failed without hovering.
|
||||
|
||||
Sources: [Button](../src/lib/ui/components/primitives/Button.svelte), lines 63–71, 155; [Field](../src/lib/ui/components/primitives/Field.svelte), line 25; [campaign mini-pipelines](../src/routes/campaigns/+page.svelte), line 380. These recommendations follow W3C guidance on [Name, Role, Value](https://www.w3.org/WAI/WCAG22/Understanding/name-role-value.html) and [Status Messages](https://www.w3.org/WAI/WCAG22/Understanding/status-messages.html).
|
||||
|
||||
### 11. Make server detail lead toward the likely next task — P2 · Rendered + Source
|
||||
|
||||
An offline server still opens on Scoreboard. In the reviewed state this shows “Waiting for game data,” while the only header action is Destroy. The diagnostic view places telemetry request history before the RCON diagnostic, even when connectivity is the immediate issue.
|
||||
|
||||
Keep a stable tab choice, but add an issue summary above it: **RCON has not answered for 1 hour — Diagnose connection · Open logs**. Put removal under a secondary Server actions menu. Allow direct URLs to Logs, Config, and Diagnostics so operation failures can link to the relevant place. Prioritize the diagnostic relevant to the reported failure.
|
||||
|
||||
Acceptance: from an offline Fleet card, the operator can reach the relevant diagnostic in one clear action. Do not automatically switch tabs while someone is reading.
|
||||
|
||||
Sources: [server detail](../src/routes/servers/[name]/+page.svelte), lines 103, 226, 331–337; [PlayerScoreboard](../src/lib/ui/components/PlayerScoreboard.svelte), line 37. [Screenshot](ux-review-assets/offline-server.png).
|
||||
|
||||
### 12. Preserve beta drafts and compatible server selections — P2 · Source
|
||||
|
||||
The wizard preserves Back/Continue state, but navigation or reload loses the draft. Changing or removing a file clears every selected server without explanation.
|
||||
|
||||
Extend the Config editor’s existing draft pattern to beta setup: resume/discard an unfinished draft, retain exact version IDs, and revalidate availability when reopening. Preserve selected servers that still satisfy the new scope intersection; explain any removals. Warn before leaving a dirty draft when recovery is unavailable.
|
||||
|
||||
Acceptance: visiting a file’s detail to check a version does not require rebuilding the test plan and server selection.
|
||||
|
||||
Sources: [beta setup](../src/routes/betas/new/+page.svelte), lines 34, 114, 130, 139; [existing ConfigEditor pattern](../src/lib/ui/components/ConfigEditor.svelte), lines 27–79.
|
||||
|
||||
### 13. Explain why a form cannot proceed — P1/P2 · Rendered + Source
|
||||
|
||||
With six distinct blueprint maps set to 20 lobbies each, both Save actions are disabled with no explanation of the 100-server limit. This was reproduced in the browser. Duplicate maps can also pass client validation and fail on submission. Beta-plan errors can refer to internal names such as `timeoutHours` or an unspecified requirement target.
|
||||
|
||||
Show **120 / 100 target lobbies — reduce the total by 20**. Prevent or merge duplicate maps. Associate errors with the relevant controls, expand a collapsed invalid requirement, and move focus to it. Use target units such as rounds, matches, minutes, or players. Validate quorum against selected servers before final submission.
|
||||
|
||||
Acceptance: every disabled forward/save action has a discoverable reason, and a rejected submission points directly to the field needing correction.
|
||||
|
||||
Sources: [BlueprintBuilder](../src/lib/ui/components/BlueprintBuilder.svelte), lines 18, 81–90; [blueprint validation](../src/lib/server/fleet/launch-recipes.ts), lines 57–63; [BetaPlanBuilder](../src/lib/ui/components/BetaPlanBuilder.svelte), line 123; [beta setup validation](../src/routes/betas/new/+page.svelte), line 68.
|
||||
|
||||
### 14. Preserve task context in URLs and navigation — P2 · Source
|
||||
|
||||
Library filters, sort, loaded count, and selection mostly live in component state. Server detail tabs also default locally. Moving among details and returning can force the operator to rebuild context; another person cannot open the same filtered working view.
|
||||
|
||||
Put filter/sort/view/tab state in URL parameters. Preserve scroll and appropriate selection state across detail navigation. Keep Fleet display preferences in local storage, but distinguish personal defaults from an explicitly shared view. Reuse the Leaderboard’s existing URL-filter pattern.
|
||||
|
||||
Acceptance: browser Back restores the operator’s working set, and a copied Diagnostics or filtered library URL opens the intended view.
|
||||
|
||||
Sources: [Scripts](../src/routes/scripts/+page.svelte), lines 13–23; [Plugins](../src/routes/plugins/+page.svelte), lines 44–49; [ConfigLibrary](../src/lib/ui/components/ConfigLibrary.svelte), lines 24–29; [Leaderboard](../src/routes/leaderboard/+page.svelte).
|
||||
|
||||
### 15. Add a setup and integration-health home — P2 · Opportunity
|
||||
|
||||
Settings currently leads only to server keys. Integration information is available deep in server diagnostics; the empty Fleet encourages creation even when external services are not configured. The first-run user needs a short path from prerequisites to a functioning lobby.
|
||||
|
||||
Add a read-only System status page and a contextual first-run checklist: host agents enrolled/connected, local RCON responding, verified game files available, keys available, and first server ready. Link directly to relevant setup instructions. Distinguish **not configured**, **unreachable**, and **server offline**. Avoid repeatedly displaying the full checklist after setup succeeds.
|
||||
|
||||
Acceptance: someone with an empty installation can identify the next missing prerequisite without opening an unrelated server or reading the full README.
|
||||
|
||||
Sources: [navigation](../src/routes/+layout.svelte), lines 80–96; [Fleet empty state](../src/routes/+page.svelte), line 177; [integration store](../src/lib/ui/stores/fleet.ts), line 13; [README setup](../README.md).
|
||||
|
||||
### 16. Keep config previews stable while typing — P2 · Source
|
||||
|
||||
ConfigInspector clears its current inspection immediately on an edit, then waits for its debounce and request. The effective settings disappear while typing, which can disrupt comparison and expanded setting details.
|
||||
|
||||
Retain the last result with **Updating preview…**, replace it when the current request completes, and preserve expanded keys and scroll position. Identify values changed by the draft. Show ignored-content warnings near Save when they affect the submitted config.
|
||||
|
||||
Acceptance: typing several characters does not repeatedly collapse the working preview or shift the editor layout.
|
||||
|
||||
Sources: [ConfigInspector](../src/lib/ui/components/ConfigInspector.svelte), lines 35–54, 75–79; [ConfigTextEditor](../src/lib/ui/components/ConfigTextEditor.svelte).
|
||||
|
||||
### 17. Make config composition easier to reason about — P2 · Opportunity
|
||||
|
||||
The inspector already provides useful provenance, but overlay precedence depends on creation order. Understanding why a setting wins can require understanding historical ordering.
|
||||
|
||||
Extend the existing final-value and winning-source summaries with a numbered composition sequence and per-layer overridden-setting counts. Consider explicit overlay priorities only with a before/after effective-value preview. Add an “Add setting” key/value row to the structured editor, while preserving raw text mode.
|
||||
|
||||
Acceptance: an operator can explain a setting’s final value and add a basic assignment without reverse-engineering creation history or switching to raw syntax unnecessarily.
|
||||
|
||||
Sources: [ConfigInspector](../src/lib/ui/components/ConfigInspector.svelte), lines 79–97; [ConfigTextEditor](../src/lib/ui/components/ConfigTextEditor.svelte), lines 19–35.
|
||||
|
||||
### 18. Make beta compatibility and target selection visible — P2 · Source
|
||||
|
||||
File picking exposes little scope context, while target eligibility is the intersection of all file scopes. An empty intersection produces a generic instruction to choose different scripts. Candidate servers then appear as a flat checkbox collection.
|
||||
|
||||
Show scope and eligible count beside each file/version. Explain concrete conflicts, such as **This Origins script and Town config have no shared servers**. Add target search, map/game grouping, selected count, and current player/health context. Include select-visible for larger fleets while preserving explicit opt-in.
|
||||
|
||||
Acceptance: users can resolve an incompatible bundle without trial and error, and can select a sensible test cohort without scanning every raw entry.
|
||||
|
||||
Sources: [beta setup](../src/routes/betas/new/+page.svelte), lines 159, 237, 276–278.
|
||||
|
||||
### 19. Explain what stalled requirements are waiting for — P2 · Source
|
||||
|
||||
Counters such as `current / target` do not sufficiently distinguish missing telemetry, too few players, a pending game end, or the next report needed to credit gameplay time. Raw evidence JSON leaves that interpretation to the operator.
|
||||
|
||||
Add a next-condition explanation per requirement: **Waiting for a completed match with at least 2 players; last report 4m ago**. Link to the relevant telemetry diagnostic. Summarize evidence in ordinary language, then offer raw JSON. Add server/event filters and “Show evidence for this check.”
|
||||
|
||||
Acceptance: a requirement stuck at zero has a visible explanation and a direct investigation path.
|
||||
|
||||
Sources: [BetaTestProgress](../src/lib/ui/components/BetaTestProgress.svelte), lines 46, 67; [evaluation](../src/lib/server/fleet/beta-evaluation.ts), line 79; [telemetry contract](beta-test-plans.md).
|
||||
|
||||
### 20. Put the next release action at the top of artifact details — P2 · Source
|
||||
|
||||
The header emphasizes Edit scope and Upload new version. Testing is lower in version history, and the “open a beta” link goes to the generic beta list. Version rows emphasize checksum, size, and timestamps; their grid also has fewer explicit columns than some rows have children.
|
||||
|
||||
Add a latest-version summary even before first promotion, with **Test latest version**, **View active beta**, or **Review passed test** as appropriate. Put version, state, change notes, and action first. Move hash and byte size into technical details. Use deliberately labeled mobile version cards.
|
||||
|
||||
Also standardize game-filter behavior for unscoped files across Scripts and Plugins, and show **3 selected · 1 hidden by filters** when selections are concealed.
|
||||
|
||||
Acceptance: after uploading a first version, the next useful action is immediately visible and carries that exact version into setup.
|
||||
|
||||
Sources: [ArtifactDetail](../src/lib/ui/components/ArtifactDetail.svelte), lines 269–278, 312–327, 347–359, 500; [Scripts filters](../src/routes/scripts/+page.svelte), lines 37, 135; [Plugins filters](../src/routes/plugins/+page.svelte), line 51.
|
||||
|
||||
### 21. Own import progress at the dialog level — P2 · Source
|
||||
|
||||
GitHub import tracks busy state inside its child component, while the parent dialog and source tabs only observe file-upload state. The import can continue after dismissal or a tab change, then complete with a navigation or modal-close callback. Import-only success content is immediately hidden when the parent closes.
|
||||
|
||||
Share the pending operation state with the parent. Either keep the import visibly in progress, or allow dismissal with an explicit persistent completion notification. Avoid surprise navigation after the user has moved on. Show **Plugin imported — View version · Start beta** after success.
|
||||
|
||||
For long script batches, add aggregate inspection/upload progress and a sticky action footer while preserving current partial-success retry.
|
||||
|
||||
Acceptance: closing or switching import UI never makes the request’s outcome disappear or causes unexplained later navigation.
|
||||
|
||||
Sources: [Plugins dialog](../src/routes/plugins/+page.svelte), lines 217–220; [PluginGithubImport](../src/lib/ui/components/PluginGithubImport.svelte), lines 18, 60–75, 141; [ScriptUploadDrawer](../src/lib/ui/components/ScriptUploadDrawer.svelte), lines 59, 101, 185.
|
||||
|
||||
### 22. Complete key management within the launch journey — P2 · Source + Opportunity
|
||||
|
||||
The standalone key form asks users to type a machine ID despite existing game/map catalogs. Key rows omit their available update timestamp. “Deployed” indicates that the server exists, not that it uses the latest saved key.
|
||||
|
||||
Generate the exact ID from game/map/lobby selectors, retain a raw-ID option for experts, and link deployed rows to their server. Show **Saved on… · Used on next deployment**. Let the launch flow add or replace its exact missing key without losing launch context. Distinguish a stored key’s presence from credential validation.
|
||||
|
||||
Acceptance: users can add the right key without manually constructing an identifier and can understand whether a running server is affected by an update.
|
||||
|
||||
Sources: [server keys](../src/routes/server-keys/+page.svelte), lines 14, 81–105; [launch dialog](../src/lib/ui/components/DeployServerDialog.svelte), line 123.
|
||||
|
||||
### 23. Separate blueprint coverage from playable capacity — P2 · Source
|
||||
|
||||
A blueprint can be satisfied by existing stopped servers. Its “Existing / queued” identities are exposed through hover text, without an obvious path to inspect them. Meeting an inventory target does not establish that those lobbies are playable.
|
||||
|
||||
Show **3 existing: 1 healthy, 1 stopped, 1 queued** with an expandable linked list. When no creation is needed, offer **Review existing servers**. Retain the current create-only-shortfall semantics, and say explicitly that excess servers remain and existing servers are not updated.
|
||||
|
||||
For collections needing many keys, add a sticky summary, **12 of 20 keys ready**, grouping, and jump-to-next-missing-key.
|
||||
|
||||
Acceptance: “Target satisfied” cannot be mistaken for “All lobbies ready to play.”
|
||||
|
||||
Sources: [DeployCollectionDialog](../src/lib/ui/components/DeployCollectionDialog.svelte), lines 57–59, 72–84.
|
||||
|
||||
### 24. Reduce mobile overhead before useful content — P2 · Rendered + Opportunity
|
||||
|
||||
The inspected mobile pages generally fit the viewport, but some put substantial control and summary space ahead of the task. On the reviewed 390px Fleet, the first server card began around 614px down. The empty Scripts screen still shows zero-count filters, search, sorting, and select-visible controls. Leaderboard rankings begin below its filters and five summary metrics.
|
||||
|
||||
On phones, prioritize one compact health summary, search, and the most useful status filter; keep display options in a small secondary control. Simplify true empty states to the relevant next action. Let Leaderboard summary metrics collapse and offer a compact player/selected-stat presentation while retaining the full scrollable table. Add sticky summaries/actions to long collection and batch forms, checking that they do not cover focused fields.
|
||||
|
||||
Acceptance: a phone user reaches a real server, record, or first-run action quickly, without losing access to advanced controls. Validate sticky behavior with keyboard navigation and zoom, following [W3C focus visibility guidance](https://www.w3.org/WAI/WCAG22/Understanding/focus-not-obscured-minimum).
|
||||
|
||||
Sources: [Fleet layout](../src/routes/+page.svelte), lines 180–202, 334; [Scripts](../src/routes/scripts/+page.svelte); [Leaderboard layout](../src/routes/leaderboard/+page.svelte), lines 126–187.
|
||||
|
||||
### 25. Sharpen visual hierarchy and vocabulary consistently — P2/P3 · Rendered + Source
|
||||
|
||||
The visual language is coherent, but small controls, nested cards, repeated section headings, and technical metadata can compete with primary decisions. Input borders use the same very faint token family as passive surfaces. Some routes also retain the generic browser title “Electric Cherry,” making multiple open tabs hard to distinguish.
|
||||
|
||||
Use a dedicated editable-control boundary token, reserve strong surfaces for actionable sections, and reduce repeated title/subtitle layers in builders. Keep body text and important form instructions comfortably readable; use compact metadata selectively. Add descriptive route titles, especially server name and lobby number.
|
||||
|
||||
Use “server” consistently for the managed unit; introduce “lobby” as its player-facing meaning. Prefer **Preview missing servers** over “Conform fleet.” Use “Files” consistently for scripts/configs/plugins where a collective noun is needed. Keep raw IDs, RCON, host-agent status, and hashes accessible in technical details. Expose the existing keyboard shortcuts through a small help panel.
|
||||
|
||||
Acceptance: primary tasks are identifiable by placement and wording, and multiple browser tabs are distinguishable. Contrast should be measured before claiming a conformance result.
|
||||
|
||||
Sources: [tokens](../src/lib/ui/styles/tokens.css), lines 57–71, 86–87; [BetaPlanBuilder](../src/lib/ui/components/BetaPlanBuilder.svelte); [navigation and shortcuts](../src/routes/+layout.svelte), lines 37–65; [deployment blueprint actions](../src/routes/provisioning/+page.svelte), lines 125–132.
|
||||
|
||||
### 26. Make investigation and handoffs easier over time — P3 · Opportunity
|
||||
|
||||
Logs provide refresh, auto-follow behavior, and a useful tail, but no in-panel search or downloadable investigation bundle. Lobby watches work across navigation, while completed notifications are transient application state and are not a reviewable activity inbox. Beta list rows could also provide more timing and blocker context.
|
||||
|
||||
Add log search/highlighting, copy/download, and a deep link preserving the selected server/log. Make the existing auto-refresh/follow state explicit and offer **Jump to latest**. Consider a small recent-notifications panel for completed watches and actions, without implying background push support. Add beta age, an operator label, and a one-line blocker to list rows. For Leaderboard, investigate player detail/history and recent-period views only after confirming operator demand; existing all-time statistics and counting explanations are a solid base.
|
||||
|
||||
Acceptance: operators can return to a completed notification or share a precise diagnostic view without reconstructing the investigation.
|
||||
|
||||
Sources: [LogPanel](../src/lib/ui/components/LogPanel.svelte); [watch handling](../src/lib/ui/stores/fleet.ts); [beta list](../src/routes/betas/+page.svelte); [Leaderboard](../src/routes/leaderboard/+page.svelte).
|
||||
|
||||
## Suggested layout for operational detail pages
|
||||
|
||||
Use the same reading order across beta, campaign, and server-deployment details:
|
||||
|
||||
1. **Outcome and next action:** current state, elapsed/remaining time, blocker, primary recovery or completion action.
|
||||
2. **Affected servers:** progress counts and a failed/waiting-first list with links to relevant diagnostics.
|
||||
3. **Change summary:** exact files/versions or deployment recipe, including what is staged versus active.
|
||||
4. **Evidence and activity:** searchable or collapsible history, with the latest meaningful event summarized near the top.
|
||||
|
||||
This gives the existing shared components a consistent purpose without forcing every operation into identical controls.
|
||||
|
||||
## Practical delivery sequence
|
||||
|
||||
**First pass — clarity and trust:** address stale card values, unnamed loading buttons, silent blueprint limits, beta results placement, and misleading/remote action feedback. These are bounded changes with clear reproduction cases.
|
||||
|
||||
**Second pass — complete action flows:** add promotion and reconciliation reviews, explicit beta timing/retry semantics, retained launch keys, beta drafts, import completion state, and links from outcomes to their campaign or recovery destination.
|
||||
|
||||
**Third pass — scale and polish:** improve target pickers, library context restoration, mobile density, config composition, setup status, and evidence navigation. Design queued-operation cancellation separately with its backend guarantees.
|
||||
|
||||
## Validation plan
|
||||
|
||||
Use a fixture with online, occupied, empty, starting, stale, and unreachable servers; a library large enough to require filtering; an expired preview; an active beta with a long history; and partial deployment failures. Ask representative operators to perform these tasks without narration or hints:
|
||||
|
||||
| Task | Observable success |
|
||||
| --- | --- |
|
||||
| Identify a safe empty lobby | User distinguishes confirmed empty from stale/unknown counts. |
|
||||
| Promote a beta tested on one server | User correctly predicts the entire affected set and restart behavior. |
|
||||
| Reconcile one selected server | User can name the scope and identify skipped or failed work afterward. |
|
||||
| Recover a stalled beta | User finds the blocker, deadline, and valid next action without reading the event log. |
|
||||
| Refresh a five-key launch preview | Unchanged server IDs retain their entered keys; changed IDs are clearly called out. |
|
||||
| Fix an invalid blueprint | User finds the total/duplicate issue without guessing why Save is disabled. |
|
||||
| Investigate an offline server on a phone | User reaches the relevant diagnostic promptly and keeps server context. |
|
||||
| Complete a flow using keyboard/assistive technology | Actions keep names while loading; focus remains visible; errors and completion are announced. |
|
||||
|
||||
Measure completion, errors, unnecessary backtracking, lost input, and time to identify the next action. Establish a baseline before setting numerical improvement targets. Source review and synthetic browser scenarios establish the issues above, but observed operator behavior should guide the final interaction details.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 115 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 120 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 60 KiB |
@@ -1,7 +0,0 @@
|
||||
# Watchdog has moved to Maintenance
|
||||
|
||||
Open **Operations → Maintenance** for health recovery and drift repair, together
|
||||
with Plutonium updates and scheduled restarts. Existing Watchdog URLs redirect
|
||||
to Maintenance; incidents, operation IDs, and history are preserved.
|
||||
|
||||
See [Maintenance](maintenance.md) for policies, recovery behavior, and the API.
|
||||
Reference in New Issue
Block a user