Clean up README and organize documentation
This commit is contained in:
@@ -0,0 +1,141 @@
|
||||
# Users and access
|
||||
|
||||
Electric Cherry requires a user session for operator pages and management APIs.
|
||||
HTTPS protects the connection; server-side role checks protect each operation.
|
||||
There is no anonymous management mode or public registration.
|
||||
|
||||
Leaderboard API reads (`GET` and `HEAD /api/leaderboard`) are public and need no
|
||||
session or credentials. This exception covers only that exact API route;
|
||||
leaderboard/profile pages, match-history routes and management operations retain
|
||||
their access checks. HTTPS requirements still apply. See the
|
||||
[leaderboard API reference](leaderboard.md#read-api) for requests and response fields.
|
||||
|
||||
## First administrator and upgrades
|
||||
|
||||
1. Generate a setup secret: `openssl rand -hex 32`.
|
||||
2. Set `EC_SETUP_TOKEN` to that value in the controller environment (or Compose
|
||||
`.env`), set `EC_PUBLIC_URL` to the external HTTPS URL, and start EC. For a
|
||||
direct Node deployment, set `ORIGIN` to that same URL.
|
||||
3. Visit EC. `/setup` asks for the setup token, a username, display name and
|
||||
password. Only the holder of the token can create the first administrator.
|
||||
4. Remove `EC_SETUP_TOKEN` after setup and restart/recreate the controller.
|
||||
Setup remains closed once any user exists, even if the token is still set.
|
||||
5. Use **Administration → Users** to create additional accounts. Share temporary
|
||||
passwords privately; each user must change theirs on first login.
|
||||
|
||||
Existing installations become protected on upgrade. Existing game servers and
|
||||
agent enrollment credentials continue working independently of browser login.
|
||||
Preserve the current database when upgrading: accounts and sessions are stored
|
||||
there alongside the fleet metadata. Never publish a blank setup page with a
|
||||
guessable token. An unset setup token holds setup closed.
|
||||
|
||||
## Roles
|
||||
|
||||
| Capability | Viewer | Operator | Administrator |
|
||||
| --- | --- | --- | --- |
|
||||
| Fleet, metrics, matches and operation summaries | Yes | Yes | Yes |
|
||||
| Raw configs, file diffs, logs and telemetry diagnostics | No | Yes | Yes |
|
||||
| Deploy/start/stop/restart/rebuild/remove servers | No | Yes | Yes |
|
||||
| Scripts, plugins, configs, campaigns, betas and collections | Read summaries | Manage | Manage |
|
||||
| Host setup, deletion, enrollment, credentials, runtime preparation and game-file upload/cache | Read status | Read status | Manage |
|
||||
| Server keys and global settings/maintenance policies | No | No | Manage |
|
||||
| Create users, change roles, disable accounts and reset passwords | No | No | Yes |
|
||||
| Own password and sign out | Yes | Yes | Yes |
|
||||
|
||||
Operators can launch using server keys already saved by an administrator. They
|
||||
cannot save keys indirectly through a launch request. An operator's content
|
||||
permissions include executable scripts and plugins; assign that role only to
|
||||
people trusted to operate the game fleet.
|
||||
|
||||
Permissions are checked on the server for API calls, normal pages and Svelte
|
||||
page-data requests. Hiding UI controls is a convenience, not the access boundary.
|
||||
The controller prevents disabling or demoting the last enabled administrator.
|
||||
Accounts are disabled rather than deleted so history retains their identity.
|
||||
|
||||
## Passwords, sessions and recovery
|
||||
|
||||
Passwords are 8–128 characters and stored using salted scrypt hashes. Temporary
|
||||
passwords created or reset by administrators must be changed before any fleet
|
||||
access. Password changes require the current password.
|
||||
|
||||
Sessions use random tokens in HTTP-only cookies, with only token hashes stored in
|
||||
SQLite. HTTPS uses a Secure `__Host-` cookie. Sessions expire after 12 hours, or
|
||||
30 minutes without authenticated requests; background dashboard polling counts
|
||||
as activity. Login replaces the existing browser session. Logout revokes it.
|
||||
Changing a password, changing a role, disabling an account, or revoking that
|
||||
user's sessions invalidates their existing sessions. Live event streams check
|
||||
revocation every 15 seconds and then close.
|
||||
|
||||
Login, initial setup and password verification are rate-limited with persistent
|
||||
15-minute windows. The database stores audit entries for login results, account
|
||||
changes and successful management requests without recording passwords, tokens
|
||||
or request bodies. Administrators can review recent entries in Users.
|
||||
|
||||
An administrator can reset another user's password from Users. If the only
|
||||
administrator forgets their password, an operator with local controller/container
|
||||
access can run the recovery command in an interactive terminal:
|
||||
|
||||
```sh
|
||||
docker compose exec electric-cherry node scripts/reset-admin.mjs admin
|
||||
# Direct Node deployment, using the correct database:
|
||||
DB_PATH=./data/cherry.db node scripts/reset-admin.mjs admin
|
||||
```
|
||||
|
||||
Replace `admin` with the existing administrator's username. The command prompts
|
||||
without echoing the password, resets that enabled administrator only, revokes
|
||||
their sessions and records the recovery. It does not expose a remote recovery
|
||||
endpoint or accept passwords as command arguments.
|
||||
|
||||
## Agent and telemetry connections
|
||||
|
||||
Host agents retain enrollment/host-token authentication under `/api/agent/`.
|
||||
Installer and release files remain public so a new machine can enroll. A browser
|
||||
login never substitutes for a host credential, and host tokens cannot manage EC.
|
||||
The agent and standalone updater source contain no credentials. Automatic
|
||||
updates fetch the controller's public HTTPS release manifest and verify its
|
||||
checksums and compatibility. The installed host retains its own credential;
|
||||
the updater accepts an upgrade only after a fresh authenticated heartbeat from
|
||||
the new agent process. Updating agent code does not change browser sessions or
|
||||
user roles. See [automatic agent updates](native-hosts.md#automatic-agent-updates)
|
||||
for the one-time installer upgrade required by existing hosts.
|
||||
Host deletion uses a separate operation-scoped receipt token so the uninstall
|
||||
worker can confirm removal and retry its acknowledgment after host credentials
|
||||
are revoked. That token authorizes only the corresponding cleanup receipt.
|
||||
|
||||
Game reporters must send a separate secret of at least 32 characters configured
|
||||
as `EC_TELEMETRY_TOKEN`:
|
||||
|
||||
```http
|
||||
POST /api/telemetry
|
||||
Authorization: Bearer <telemetry-token>
|
||||
Content-Type: application/json
|
||||
```
|
||||
|
||||
`X-EC-Telemetry-Token` is also supported. Cookies, query parameters and JSON fields
|
||||
cannot supply it. A missing server token disables ingestion; a missing or wrong
|
||||
request token is rejected before parsing or recording the payload. Existing JSON
|
||||
event fields are unchanged. Configure your uploaded reporter/plugin, or a trusted
|
||||
relay if that reporter cannot set headers, before upgrading. Use HTTPS for the
|
||||
reporter-facing endpoint. Rotating this shared token requires updating reporters.
|
||||
|
||||
## Proxy and local development
|
||||
|
||||
Keep Node's internal port behind your TLS proxy and configure `ORIGIN` correctly.
|
||||
Management writes require an exact matching `Origin` header, including login,
|
||||
JSON requests and multipart uploads. Do not configure a proxy to replace arbitrary
|
||||
incoming origins with your trusted origin. Machine endpoints use their own token
|
||||
checks and do not require a browser Origin.
|
||||
|
||||
Use the adapter's real client-address configuration only with a trusted proxy
|
||||
that strips spoofed forwarding headers; login throttling otherwise uses the direct
|
||||
peer address. Forwarded host or user headers do not authenticate users.
|
||||
|
||||
Local HTTP browser development requires `EC_AUTH_ALLOW_HTTP=true` and a loopback
|
||||
hostname (`localhost`, `127.0.0.1` or `::1`). This uses a separate development
|
||||
cookie. It never enables HTTP login for a LAN/public hostname. Agent HTTP
|
||||
development remains separately controlled by `EC_AGENT_ALLOW_HTTP` and the agent's
|
||||
loopback-only option.
|
||||
|
||||
The implementation follows OWASP's guidance for [authorization on every request](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html#validate-the-permissions-on-every-request),
|
||||
[password hashing](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#scrypt),
|
||||
and [session management](https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html).
|
||||
@@ -0,0 +1,70 @@
|
||||
# Lobby autoscaling
|
||||
|
||||
Open **Fleet → Autoscaling** to keep empty lobbies available for new groups. Policies apply to one map variant or every map in a game. A game-wide policy applies the same minimum, maximum, and spare targets independently to each map, including all variants. A partly occupied Zombies lobby is not an empty spare, even when it still has player slots.
|
||||
|
||||
Autoscaling is opt-in. Installing the feature creates no policies and changes no lobby counts. Use a two-week dry run before enabling live actions.
|
||||
|
||||
## Configure a policy
|
||||
|
||||
Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.
|
||||
|
||||
Choose **Preview changes** to inspect the existing lobbies, saved key coverage, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving.
|
||||
|
||||
| Mode | Behavior |
|
||||
| --- | --- |
|
||||
| Off | No new actions. Unsubmitted work is withdrawn and delivered operations remain tracked until settled. |
|
||||
| Dry run | Records hypothetical decisions without reserving identities, hosts, or ports. Collections retain control of the lineup. |
|
||||
| Create only | Adds a lobby when eligible. Surplus removal remains advisory. |
|
||||
| Create and destroy | Adds lobbies and removes surplus lobbies after continuous empty evidence and all removal checks pass. |
|
||||
|
||||
Creating a policy in any mode, including Dry run or Off, requires saved Plutonium keys for enough exact lobby identities to support the configured maximum on every selected map. Preview lists each shortage, and Save rechecks coverage transactionally. Non-Off edits also require coverage; an existing policy can always be turned Off if keys have been removed. Only supported lobby identities numbered 1–20 with nonblank saved keys count. Keys may be nonconsecutive: if only lobby 3 and lobby 8 are keyed, those identities can be used. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
|
||||
|
||||
The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies share one cooldown and one outstanding operation across their maps, prioritizing additions before removals. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.
|
||||
|
||||
## Ownership and blockers
|
||||
|
||||
A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and require key coverage for the maximum on each map. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish.
|
||||
|
||||
Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and missing-key waits.
|
||||
|
||||
Current positive occupancy from either telemetry or RCON vetoes an empty claim. Empty intervals reset when players arrive, evidence becomes unknown, the instance or lifecycle changes, or observation continuity breaks. Controller downtime does not count toward the empty duration. Failed or uncertain automatic operations pause further actions until an operator inspects their Provisioning recovery and reviews the policy again.
|
||||
|
||||
## Pause and cancellation
|
||||
|
||||
Use **Pause all** to stop new decisions globally, or **Edit or pause → Off** for one policy. A global pause retains each live policy's ownership; setting it Off releases ownership once its outstanding operation settles. Resuming the global switch preserves each policy's selected mode.
|
||||
|
||||
Unsubmitted creates and queued removals are withdrawn where possible. Cancelling an autoscaling operation from Provisioning also turns its policy Off so it cannot immediately request a replacement.
|
||||
|
||||
An automatic removal requires fresh controller eligibility, a short-lived authorization, and a host-local occupancy check. The agent claims authorization once, repeats its occupancy check, and must begin mutation within five seconds of requesting the claim. Occupied, unknown, expired, interrupted-before-mutation, or revoked attempts end without waiting to destroy the lobby later. A new attempt needs a new decision and another complete empty interval. Automatic removals cannot use the administrator safety override.
|
||||
|
||||
The authorization claim is the cancellation boundary. A pause after that point cannot guarantee withdrawal; the page retains the outstanding operation until its receipt is reconciled. Do not treat cancellation as proof that a delivered host command had no effect. Uncertain commands retain their identity and resource reservations.
|
||||
|
||||
Turning a policy Off does not recreate lobbies it removed or remove lobbies it created. Once operations settle, use Collections to restore a fixed lineup through the normal capacity and empty-server gates.
|
||||
|
||||
## Delete a policy
|
||||
|
||||
Choose **Delete policy** and confirm to remove its settings, decisions, and measurements. Existing lobbies, saved keys, and Provisioning history are kept. Deleting a live policy releases ownership without changing the lineup, and a new policy can reuse the same scope.
|
||||
|
||||
Deletion requires the current policy revision and no unsettled autoscaling operation. If work is outstanding, turn the policy Off and wait for its operation to settle before deleting. Delivered or uncertain host commands retain their policy and admission evidence until reconciliation finishes.
|
||||
|
||||
## Dry run and pilot
|
||||
|
||||
The production observation period and live pilot have not been run as part of implementation. Begin them on the operator-selected maps after deploying the controller and compatible agents.
|
||||
|
||||
1. Select one or two representative maps and save policies in Dry run for two weeks. Record the baseline lineup, keys, versions, host CPU and memory, player-hours, and comparable days of the week.
|
||||
2. Review recent hypothetical actions, empty lobby-hours, minutes below the spare target, unknown-status minutes, capacity/key blockers, and projected action frequency. Dry run observes the actual lineup; it cannot predict how players would redistribute after a removal.
|
||||
3. Enable Create only on the pilot maps after resolving blockers. Measure request-to-ready time from Provisioning, browser visibility, and successful external joins separately. Increase the spare target if creation latency leaves new groups without a suitable lobby.
|
||||
4. Agree the acceptable spare-deficit duration for the pilot, verify agent capability, and enable Create and destroy. Keep normal gameplay and joins under observation while comparing actual host CPU and empty lobby-hours with the baseline.
|
||||
5. Expand one map at a time only when savings are measurable and spare availability and gameplay meet the agreed tolerance. Pause immediately if those conditions regress.
|
||||
|
||||
The page retains up to 31 days of decision evidence and observations, with the 30 most recent decision changes shown. Repeated unchanged evaluations update the decision's last-seen time instead of adding a row. Unresolved intent survives retention. Provisioning history retains the operation and policy backlink separately.
|
||||
|
||||
Measurement totals exclude collection gaps. Request-to-verified-readiness summaries include queue time and only completed creates; they do not establish browser visibility or successful joins. “Time with players” in Metrics can rise simply because idle lobbies leave its denominator. Use host CPU and memory measurements alongside player-hours and spare availability when assessing savings.
|
||||
|
||||
## API and implementation
|
||||
|
||||
`GET /api/autoscaling` returns policies, current evidence, recent decisions, measurements, and global pause state. Viewers may read it; mutations require an operator or administrator, using the existing authentication and origin checks.
|
||||
|
||||
`POST /api/autoscaling` accepts `preview`, `save`, `delete`, or `global-pause`. Delete requires `id` and the current `revision`. Preview/save requests contain `settings`, plus `id` and `revision` when editing. Use `mapSlug: '*'` for every map in the selected game; existing map-specific requests are unchanged. Scope cannot change when editing. Save also requires the preview `token`. Global pause uses a boolean `paused` field. Agent removal claims use a separate host-authenticated endpoint and are bound to the exact command, host, instance generation, policy revision, and expiry.
|
||||
|
||||
Shared contracts live in `src/lib/autoscaling.ts`; the evaluator, snapshots, persistence, ownership, and authorization live in `src/lib/server/autoscaling/`. Provisioning remains the only lifecycle executor. See the [approved implementation plan](../plans/lobby-autoscaling-plan.md) for the design and rollout criteria.
|
||||
@@ -0,0 +1,308 @@
|
||||
# Beta test plans
|
||||
|
||||
Create a plan in Betas, choose its servers, and deploy. The server's game script
|
||||
reports gameplay to Electric Cherry. The same reports update Fleet and any active
|
||||
beta on that server.
|
||||
|
||||
## Game reporter protocol
|
||||
|
||||
Set `EC_TELEMETRY_TOKEN` on the Electric Cherry controller to a dedicated random
|
||||
secret of at least 32 characters (for example, generate 64 hexadecimal characters
|
||||
with `openssl rand -hex 32`). Configure the same secret in each game reporter or
|
||||
its trusted local relay. Use a production HTTPS URL or a reverse proxy that
|
||||
terminates TLS; browser login cookies do not authorize game telemetry.
|
||||
|
||||
Send JSON to `POST /api/telemetry` with `Content-Type: application/json` and either
|
||||
`Authorization: Bearer <EC_TELEMETRY_TOKEN>` or
|
||||
`X-EC-Telemetry-Token: <EC_TELEMETRY_TOKEN>`. If both headers are present, they must
|
||||
contain the same secret. Keep the secret out of URLs, payloads and browser code,
|
||||
and use a different secret from host-agent credentials or account passwords.
|
||||
The JSON event format is unchanged.
|
||||
There are four events. Send them whether or not a beta is active.
|
||||
|
||||
### Event fields
|
||||
|
||||
| Event | Fields |
|
||||
| --- | --- |
|
||||
| `game_start` | `server`, `map`, `players` with each player's `name` and `guid` |
|
||||
| `round_change` | `server`, `map`, `round`, `players` with each player's `name`, `guid`, `kills`, `headshots`, `downs`, `revives`, and `score` |
|
||||
| `heartbeat` | Same fields as `round_change`; sent periodically during a round to refresh the current round and player stats |
|
||||
| `game_end` | `server`, `map`, `round`, `players` with each player's `name`, `guid`, `kills`, `headshots`, `downs`, `revives`, and `score` |
|
||||
|
||||
`server` is the exact registered Electric Cherry server name. `map` is the
|
||||
current map. `players` is the full player array, including an empty array when
|
||||
there are no players. Names and GUIDs are strings; keep GUIDs as strings to
|
||||
preserve their precision. Round and stat values are nonnegative JSON integers.
|
||||
Each GUID must appear only once within a report's player array.
|
||||
|
||||
Every event also accepts an optional per-player `ping`: the game server's current
|
||||
latency measurement in milliseconds, as a nonnegative JSON integer (for example,
|
||||
`"ping": 42`). Send `null` or omit the field when unavailable; do not substitute
|
||||
zero for a missing measurement. Older reporters without this field remain valid.
|
||||
Fleet prefers telemetry ping and fills missing readings from fresh RCON status,
|
||||
matching players only by GUID. For T6, RCON's
|
||||
hexadecimal GUIDs are compared with telemetry's decimal GUIDs using exact integer
|
||||
conversion; stored GUIDs remain unchanged. T5 GUIDs are compared as reported.
|
||||
Occupied lobbies continue RCON polling every 30 seconds.
|
||||
RCON supplements only ping; telemetry still supplies roster membership and scores.
|
||||
RCON pings expire after 90 seconds and must match the current map and server
|
||||
lifecycle. Missing or ambiguous matches display `—`. Telemetry ping is retained
|
||||
and streamed with the latest report; it does not contribute to leaderboard statistics.
|
||||
|
||||
### Game start
|
||||
|
||||
Send when the game starts. Round and stats are not required.
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "game_start",
|
||||
"server": "t6_zombie_town_1",
|
||||
"map": "zm_transit",
|
||||
"players": [
|
||||
{ "name": "Josh", "guid": "12345678901234567" },
|
||||
{ "name": "Alex", "guid": "23456789012345678" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Round change
|
||||
|
||||
Send the new round and each player's current stats.
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "round_change",
|
||||
"server": "t6_zombie_town_1",
|
||||
"map": "zm_transit",
|
||||
"round": 25,
|
||||
"players": [
|
||||
{ "name": "Josh", "guid": "12345678901234567", "kills": 420, "headshots": 160, "downs": 1, "revives": 3, "score": 32000 },
|
||||
{ "name": "Alex", "guid": "23456789012345678", "kills": 380, "headshots": 140, "downs": 3, "revives": 1, "score": 28000 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Game end
|
||||
|
||||
Send the final round and each player's final stats when the game ends.
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "game_end",
|
||||
"server": "t6_zombie_town_1",
|
||||
"map": "zm_transit",
|
||||
"round": 26,
|
||||
"players": [
|
||||
{ "name": "Josh", "guid": "12345678901234567", "kills": 445, "headshots": 170, "downs": 2, "revives": 3, "score": 33500 },
|
||||
{ "name": "Alex", "guid": "23456789012345678", "kills": 401, "headshots": 146, "downs": 4, "revives": 1, "score": 29750 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Responses and delivery
|
||||
|
||||
Success returns HTTP 200 with `{ "ok": true, "telemetry": { ... } }`.
|
||||
The returned telemetry contains the event and an Electric Cherry `receivedAt`
|
||||
timestamp. A missing/invalid machine token returns 401; an unset or invalid
|
||||
controller `EC_TELEMETRY_TOKEN` returns 503. Authentication happens before payload
|
||||
validation. Authenticated invalid JSON or fields return 400, an unknown/destroyed
|
||||
server 404, and a non-JSON content type 415.
|
||||
|
||||
The script sends each event once, in gameplay order, directly to Electric Cherry.
|
||||
There is no discovery request, beta token, boot/session ID, sequence number,
|
||||
custom feature event, external helper, or separate beta acknowledgement.
|
||||
Basic payload checks happen in the receiver.
|
||||
|
||||
Electric Cherry saves the latest complete report, including player stats, and
|
||||
streams it to connected clients. A new start clears the previous round display;
|
||||
an end closes the match. Receipt order determines the latest report. This small
|
||||
protocol does not attempt to identify delayed events from earlier games or
|
||||
provide exactly-once delivery across games.
|
||||
|
||||
Reports also feed the persistent [Leaderboard](leaderboard.md), independently
|
||||
of beta activity. That page documents cumulative stat accounting, round credit,
|
||||
and the limits of identifying matches from this protocol.
|
||||
|
||||
### Reporter deployment
|
||||
|
||||
The game scripts are uploaded artifacts, not source maintained in this
|
||||
repository. Their reporter should emit the four payloads above and add the
|
||||
authentication header. Existing anonymous reporters must be updated before
|
||||
telemetry is accepted; if a reporter cannot set HTTP headers, configure a trusted
|
||||
host-local relay to add the header. This repository does not bundle that relay or
|
||||
assume a particular GSC HTTP implementation. Rotating the shared secret requires
|
||||
updating the controller environment and all reporters/relays together.
|
||||
|
||||
### Telemetry diagnostics
|
||||
|
||||
Open a server's **Diagnostics → Telemetry API requests** to inspect incoming
|
||||
`POST /api/telemetry` calls whose top-level `server` exactly matches its registered
|
||||
name. Each entry includes the Electric Cherry receipt time, HTTP result, error
|
||||
reason, event, map, round, player count, content type, and submitted JSON. Rejected
|
||||
reports are included, even when the content type is wrong but the body is readable
|
||||
JSON after authentication. Unauthorized requests are rejected before their bodies
|
||||
are read and are not retained in per-server diagnostics. Authentication headers
|
||||
and their secrets are never stored. Processing failures return 500 and record the internal error when diagnostic
|
||||
storage is available; failed database transactions do not update the scoreboard.
|
||||
|
||||
The panel refreshes every five seconds while visible and has a manual refresh
|
||||
button. History is stored in SQLite starting with this version, with up to 200
|
||||
requests per server from the last seven days and a fleet-wide cap of 10,000
|
||||
requests. Old entries are pruned when new requests arrive, and expired entries
|
||||
are excluded from reads. Payload previews are limited to 16,384 characters and
|
||||
marked when truncated. Request headers other than content type are not stored.
|
||||
`GET /api/servers/{name}/telemetry-requests` returns the same recent history,
|
||||
newest first, with caching disabled.
|
||||
|
||||
- **No entries:** no attributable requests are retained for this exact name.
|
||||
Check the reporter's destination and server field. Calls that never reach
|
||||
Electric Cherry, malformed JSON, and bodies without a usable server name
|
||||
cannot appear in this server's log. Earlier requests are not backfilled.
|
||||
Missing/invalid telemetry credentials also produce no retained request; inspect
|
||||
the reporter's HTTP result for 401 or an unconfigured-controller 503.
|
||||
- **400 / 415:** inspect the error and payload. One invalid player's stats or a
|
||||
duplicate GUID rejects the entire report. These requests do not update the
|
||||
scoreboard.
|
||||
- **404:** the reported server was not registered or had been destroyed at the
|
||||
time of the request.
|
||||
- **200:** Electric Cherry processed the report. An empty `players` array
|
||||
produces an empty scoreboard, including when sent with `game_end`. If a
|
||||
populated accepted report is missing from the scoreboard, investigate the
|
||||
live update/browser path.
|
||||
- **500:** Electric Cherry failed to process the request. Inspect the recorded
|
||||
error and the application log.
|
||||
|
||||
RCON's online status and player count are separate from gameplay telemetry, so
|
||||
an occupied server can still have no scoreboard report.
|
||||
|
||||
## Requirements
|
||||
|
||||
Requirements on a qualifying server are ANDed:
|
||||
|
||||
| Requirement | Evidence |
|
||||
| --- | --- |
|
||||
| Reach round X | A round change, heartbeat, or game end reports round >= X with the required player count |
|
||||
| Complete qualifying matches | Game end at round 2 or later, optionally qualified by a higher round, player count, and duration |
|
||||
| Populated minutes | Time between consecutive game reports with enough players at both ends |
|
||||
| Player-hours | Populated minutes multiplied by the lower reported player count, divided by 60 |
|
||||
| Reach player count | Player array length in a game report |
|
||||
| Sustained population | Consecutive report intervals above the player threshold within one game |
|
||||
| Continuous healthy minutes | Consecutive fresh container/RCON observations |
|
||||
| Maximum health outages | Transitions from observed healthy to unhealthy |
|
||||
| Successful recovery cycles | Repeat the other requirements after each additional empty-server restart |
|
||||
| Tester checklist | Tester name and observation entered on the beta detail page |
|
||||
|
||||
All gameplay requirements use the script reports. Match counts do not read game
|
||||
logs. The reported round and required player count must occur together in the
|
||||
same game; observations from different games cannot combine to qualify a match.
|
||||
The final game-end report can supply that qualifying round. Consecutive duplicate
|
||||
game ends do not add another completion.
|
||||
|
||||
Match duration requires a game start and uses receipt times. Population time
|
||||
advances only when another report arrives in the same game; it uses the lower
|
||||
player count at the interval's endpoints. There is no heartbeat requirement or
|
||||
short interval limit between rounds. These are estimates between reported
|
||||
snapshots, not continuous measurements of player presence.
|
||||
|
||||
Health checks and safe restart gates use the existing container/RCON monitoring.
|
||||
Health intervals spanning more than 60 seconds earn no healthy time.
|
||||
|
||||
## Coverage, waves and outcomes
|
||||
|
||||
Coverage can require all selected servers, a quorum, one per selected map,
|
||||
one per selected game, or one per selected game/map combination. Coverage
|
||||
metadata is pinned when the beta opens.
|
||||
|
||||
Set wave size to 1 for a canary, a larger number for batches, or 0 for all
|
||||
selected servers. Later waves receive files after the previous waves pass.
|
||||
Health-outage budget violations on any tested server block promotion.
|
||||
|
||||
The test window starts when the beta opens or a revision is submitted, including
|
||||
time waiting for empty servers. Missing evidence at the deadline produces **inconclusive**.
|
||||
Health monitoring continues after a pass until promotion.
|
||||
|
||||
### Failure and rollback
|
||||
|
||||
For failures, select hold as failed, pause for investigation, or automatic
|
||||
rollback. Rollback queues restoration only for files touched by the beta.
|
||||
Restoration and restart both wait for an empty server and restart capacity.
|
||||
The manual option holds those changes in a rollback campaign until you select
|
||||
**Resume when empty**. Promotion is always manual.
|
||||
|
||||
### Promotion
|
||||
|
||||
Review the promotion preview before confirming. API clients fetch it with
|
||||
`GET /api/betas/:id/promote` and send its `previewToken` to the same route with
|
||||
`POST`. If the reviewed scope or configuration has changed, the request returns
|
||||
409 and requires a fresh preview.
|
||||
|
||||
Confirming promotion creates a campaign without reading the existing files on
|
||||
additional servers. Each server waits for an empty lobby and restart capacity,
|
||||
then captures rollback snapshots immediately before applying its queued files
|
||||
and restarting. Campaign history records snapshot capture, and file comparisons
|
||||
become available once those snapshots are confirmed. Tested servers retain their
|
||||
original pre-beta snapshots; their composed configuration is still checked at
|
||||
promotion to determine whether another activation is needed.
|
||||
|
||||
Withdrawing changes that have not activated needs no snapshot and leaves those
|
||||
servers' files untouched. If activation was submitted but its snapshot has not
|
||||
been confirmed, reconcile the provisioning operation before rolling back.
|
||||
|
||||
### Pause, resume and retry
|
||||
|
||||
Pause holds evaluation and wave advancement. Resume retains completed progress and the
|
||||
current revision's deadline, but starts new timing intervals. A fresh attempt resets
|
||||
progress and restarts deployed servers when empty. Recovery cycles repeat
|
||||
gameplay milestones and checklist sign-offs while carrying health outages forward.
|
||||
|
||||
Plans, progress, and evidence persist in SQLite. The detail page shows the most
|
||||
recent 200 evidence entries. On upgrading from the old reporter protocol,
|
||||
active progress resets. Unsupported checks in existing plans and saved presets
|
||||
become manual review items; they cannot silently pass.
|
||||
|
||||
## Updating an active beta
|
||||
|
||||
Choose **Update beta** on the beta detail page, upload replacements for existing
|
||||
bundle files, and review the changes and selected servers before choosing
|
||||
**Deploy revision**. Text files show a diff when both files are small enough;
|
||||
binary and large files show their checksums. Files must actually change.
|
||||
|
||||
The beta keeps its ID, servers, scope and test policy. Each revision creates new
|
||||
file versions, clears current test progress and checklist sign-offs, repeats the
|
||||
configured waves, and starts a fresh testing window at submission. Files install
|
||||
and servers restart through the normal empty-server and capacity gates. Promotion
|
||||
requires the new revision to pass. Pause and retry retain the current revision's
|
||||
deadline; they do not create a revision.
|
||||
|
||||
Revision history retains downloadable bundle files and previous test results,
|
||||
including the last 200 evidence entries shown at replacement. All underlying
|
||||
attempts and evidence remain in SQLite, separated by revision. Rejecting the beta
|
||||
restores the original pre-beta files, even after multiple revisions or while a
|
||||
new revision is waiting to deploy. Files originally absent are removed.
|
||||
|
||||
Updates are available until promotion or rollback, including paused, failed and
|
||||
expired tests. An activation already in progress must settle first; interrupted
|
||||
file writes and deleted participants must be resolved before an update. Pending,
|
||||
unsubmitted restarts are superseded. Adding/removing files, changing targets or
|
||||
changing the policy still requires a new beta.
|
||||
|
||||
## API and storage
|
||||
|
||||
`POST /api/betas` accepts `testPlan`.
|
||||
`GET /api/betas/:id` includes the plan, outcome, per-server progress, and
|
||||
recent evidence under `test`.
|
||||
|
||||
`POST /api/betas/:id/test` accepts `pause`, `resume`, `retry`, or
|
||||
`sign-off` (with `server`, `requirementId`, and `note`).
|
||||
`GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are
|
||||
copied into a beta; later edits do not change an active plan.
|
||||
|
||||
`POST /api/betas/:id/revisions` accepts multipart `expectedRevision`, optional
|
||||
`notes`, and replacement uploads named `file:<scriptId>`. A stale revision is
|
||||
rejected. `GET /api/betas/:id` includes the current `revision` and `revisions`
|
||||
history; `GET /api/betas/:id/revisions/:revision/files/:scriptId` downloads an
|
||||
archived file (or the original pinned file before its first update).
|
||||
|
||||
Internal progress records support attempts and recovery cycles. The reporter
|
||||
does not know about or supply these records. Legacy betas without a test plan
|
||||
also use `game_end` at round 2 or later for their match-completion gate.
|
||||
@@ -0,0 +1,272 @@
|
||||
# Deployment collections
|
||||
|
||||
Open **Fleet → Collections** to manage reusable server lineups. Use
|
||||
**Active launches** to follow progress and **Launch history** to review completed
|
||||
launches and failures. Both views refresh automatically. A
|
||||
collection describes a whole lineup: for example, two Kino lobbies, one Moon lobby,
|
||||
and two Town lobbies. Counts are desired fleet totals: existing and queued
|
||||
servers count toward each target, and only missing servers are created.
|
||||
Collections save game/map selections and lobby counts. Every new server uses the
|
||||
latest promoted scripts, plugins, baseline, and matching configuration overlays
|
||||
when provisioning begins.
|
||||
|
||||
## How lineup counts work
|
||||
|
||||
For example, a collection targeting four Kino and two Der Riese lobbies, with
|
||||
two Kino already registered, creates two Kino and two Der Riese lobbies.
|
||||
The preview shows **Target / Existing or queued / To create** for each map.
|
||||
Matching is by game and exact map variant; T6 Town and Farm are distinct even
|
||||
though they share an engine map. Provisioning identities are counted once when
|
||||
they appear in both the server registry and active creation queue.
|
||||
|
||||
Registered servers count even if stopped or unhealthy; conforming does not
|
||||
restart or reconfigure them. Failed creation jobs without a registered server,
|
||||
destroyed records, and servers awaiting removal do not count. The default
|
||||
**Fill missing servers** mode keeps excess servers and maps outside the collection.
|
||||
It fills deficits without changing existing servers' files.
|
||||
|
||||
## Build and launch a collection
|
||||
|
||||
1. Choose **Create collection** on **Fleet → Collections** and name the lineup.
|
||||
2. Add maps from Black Ops I or II and set the desired total lobbies for each.
|
||||
3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
|
||||
Choose **Fill missing servers** or **Match collection exactly**, then review
|
||||
the server IDs and estimated capacity. Native host ports are assigned later.
|
||||
4. Lobbies with a saved key show **Saved key ready** and reuse it automatically.
|
||||
Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**.
|
||||
5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact
|
||||
mode) and follow the collection progress page.
|
||||
If every target is already met, choose **Done**; no operation is created.
|
||||
|
||||
### Match collection exactly
|
||||
|
||||
Exact mode applies to the entire registered and queued fleet across both games.
|
||||
The preview lists servers to keep, create, and remove. It keeps deployed lobbies
|
||||
before unstarted creations, then the lowest lobby numbers within each game/map
|
||||
variant. It removes excess lobbies and maps outside the collection. Retained
|
||||
servers keep their containers, files, ports, and configuration without a restart.
|
||||
The option is chosen per launch; opening another preview defaults to fill mode.
|
||||
|
||||
Review the removal list and check its confirmation before applying. Removed
|
||||
servers lose their containers, files, and registrations; saved keys remain.
|
||||
Unstarted creations are cancelled without booting a container. Running servers
|
||||
use the normal removal queue and require a fresh zero-player RCON response.
|
||||
Occupied or unreachable servers wait. Already pending removals are reused.
|
||||
|
||||
Creation intents and removals are queued in one database transaction. A
|
||||
missing key or changed fleet rejects the entire request. Any fleet identity,
|
||||
map, port, or pending-removal change since an exact preview requires a new review.
|
||||
Servers added after submission are not included. Applying the same preview again
|
||||
returns the same collection without repeating operations.
|
||||
|
||||
Native launches can be queued even when the current port range is full. Removed
|
||||
servers retain their ports until cleanup completes; provisioning assigns newly
|
||||
available ports as each creation starts. A failed or cancelled removal can leave
|
||||
creations waiting for capacity, while other eligible hosts can still be used.
|
||||
Legacy launches retain their reviewed port dependencies. Queued work and any
|
||||
assigned reservations survive app restarts. The progress page
|
||||
tracks creations and removals, including removal-only plans, and lists the servers
|
||||
that were retained. The existing operation detail page offers removal cancellation
|
||||
and cleanup recovery where applicable.
|
||||
|
||||
### Collection limits and editing
|
||||
|
||||
Collections support up to 40 distinct game/map selections, 1–20 lobbies per map,
|
||||
and 100 servers total. The host's configured port range and existing fleet may
|
||||
allow fewer. A collection can be saved even when there is insufficient capacity
|
||||
to launch it. Saving a collection does not capture or pin managed file versions.
|
||||
|
||||
Use **Edit** on a collection card to change its name, description, maps, or lobby
|
||||
counts. Already reviewed launches keep their original lineup, including after
|
||||
collection edits or removal; their managed files still follow the latest promotions.
|
||||
Previously saved collections automatically use this policy without being resaved.
|
||||
Collections saved by the earlier single-server release load as one-server lineups
|
||||
and can be expanded directly. Any file snapshots in older collections are ignored
|
||||
when preparing new launches.
|
||||
|
||||
### Preview and submission
|
||||
|
||||
A native collection preview reads a shared capacity snapshot, maintained from host
|
||||
heartbeats and fleet changes. It estimates available ports, CPU, memory, and startup
|
||||
slots without running placement for each server. Missing or insufficient capacity
|
||||
is a warning: provisioning can wait for it. The preview does not reserve resources.
|
||||
|
||||
Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in
|
||||
one database transaction, and returns the progress page ID. Native jobs initially
|
||||
have no host or port; each job performs its capacity checks asynchronously during
|
||||
provisioning. If a server lacks a valid key or its reviewed identity changed,
|
||||
no jobs or key changes from that submission are committed. Legacy submissions also
|
||||
revalidate their reviewed ports. Refreshing
|
||||
the preview retains typed keys only for unchanged server IDs still missing a saved
|
||||
key, and resets removal confirmation.
|
||||
|
||||
Submitting the same preview again returns the same collection, including after
|
||||
a lost response or an application restart. Reusing the collection with a new
|
||||
preview recomputes the shortfall, including servers queued by earlier launches.
|
||||
It does not add another full copy of the lineup. If the shortfall or reviewed
|
||||
identities change before submission, the request is rejected without changes and
|
||||
requires a fresh preview. Previously saved additive previews are checked against
|
||||
the current shortfall too.
|
||||
|
||||
### Startup queue and progress
|
||||
|
||||
All collections and individual launches share one startup queue, ordered by request
|
||||
time. By default, one server starts at a time. Its startup slot remains occupied
|
||||
until the host agent confirms container health, game status and the selected map
|
||||
are verified, and RCON reports a current Plutonium revision.
|
||||
Only then can the next queued server start. Set **Concurrent server startups** in **Administration → Settings** to
|
||||
a positive integer to allow more simultaneous startups. Each worker pass admits
|
||||
at most the available slots, even if an attempt fails immediately.
|
||||
|
||||
Names, any assigned ports, and queue order survive application restarts. A server still booting
|
||||
or waiting for health keeps its slot until it passes verification or is removed.
|
||||
A lost startup response also keeps the slot because the container may already be
|
||||
running. A confirmed creation failure is recorded for review, allowing later
|
||||
queued servers to proceed on subsequent passes without undoing successful launches.
|
||||
|
||||
The collection launch page at `/collections/launches/:id` separates ready,
|
||||
launching, queued, and failed servers. Its breadcrumb returns to Collections. It
|
||||
highlights the current startup, shows each waiting server's queue position, and
|
||||
links to earlier launches when another collection occupies the available capacity.
|
||||
Every server links to its Provisioning operation timeline and recovery options,
|
||||
and each child operation links back to its collection launch. Individual operations
|
||||
remain visible in Provisioning activity and history.
|
||||
|
||||
## Create and review a single-server launch
|
||||
|
||||
1. Choose **Single server** on Provisioning (or the deploy action on the fleet).
|
||||
2. Choose the current promoted setup and select a game/map, or select a collection.
|
||||
3. Choose **Preview launch**. Review the image reference, configuration sources,
|
||||
managed files and versions, selected map, lobby, port, and prerequisite checks.
|
||||
4. Enter the Plutonium key for the displayed server ID, or reuse its saved key.
|
||||
5. Choose **Launch server** and follow the operation to verified health.
|
||||
|
||||
A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If
|
||||
another launch takes the lobby, submission stops before saving the key;
|
||||
refresh the preview and review the newly assigned ID. Refreshing a preview clears
|
||||
the typed key so it cannot accidentally be saved against a different lobby.
|
||||
|
||||
Host selection, game-file versions, runtime configuration and lobby availability
|
||||
are checked before queueing. Provisioning assigns host capacity and physical
|
||||
ports when dispatching. The host agent performs container and RCON verification
|
||||
locally after launch. See [native hosts](native-hosts.md) for readiness checks.
|
||||
T6 variants retain their exact catalog slug and launch ruleset; RCON health checks
|
||||
verify the engine map, not the variant's start location.
|
||||
|
||||
## Save and reuse a collection
|
||||
|
||||
Expand **Make this a reusable collection** in a launch preview and give it a name
|
||||
and optional description. You can save the map even when host configuration or
|
||||
available capacity blocks launching. Names are unique without regard to case.
|
||||
|
||||
Each map in a collection retains the game, exact map variant, and desired lobby
|
||||
count. Image references, managed files, and composed configuration are resolved
|
||||
from the current setup for each new launch. Credentials, lobby IDs, and ports are
|
||||
not copied from the source server.
|
||||
|
||||
Use **Conform fleet** on a collection card to review missing servers. Remove a
|
||||
collection from the library when it is no longer needed; its existing servers,
|
||||
reviewed previews, and deployment history remain intact.
|
||||
|
||||
On a creation detail page, **Launch again** selects the original game and map
|
||||
with the next available lobby and port, using the latest promoted setup. The
|
||||
preview also lets you save that map as a collection with a target of one lobby.
|
||||
The recorded files remain visible as deployment history. Older deployments
|
||||
without a recorded recipe retain their history but cannot offer this action.
|
||||
|
||||
## Promotions and deployment history
|
||||
|
||||
A preview shows the currently promoted managed files, configuration, and image
|
||||
reference for the selected map. It does not pin those versions. New launches
|
||||
resolve the promoted setup again when provisioning begins, so
|
||||
promotions made after a preview or while a job is queued apply to that server.
|
||||
This applies to collections, single-server launches, and **Launch again**.
|
||||
|
||||
Previews and queued jobs store file metadata without copying script or plugin
|
||||
contents for every lobby. Provisioning captures the bytes when each job starts.
|
||||
Native host ranking, file reads, and command encoding run in a worker thread.
|
||||
The controller makes a short atomic reservation check after ranking, then
|
||||
rechecks capacity, promotions, credentials, and safety
|
||||
before submitting those commands. Agent delivery streams the stored payloads in
|
||||
small chunks, while progress polling reads metadata without loading file bytes.
|
||||
|
||||
The files selected at provisioning, their hashes and version metadata, composed
|
||||
configuration, selected map, and image reference are recorded in SQLite for
|
||||
deployment history. These records do not override promotions on future launches.
|
||||
Existing API clients that omit a preview and jobs queued before this update
|
||||
also use the promoted setup at provisioning time.
|
||||
|
||||
A submitted preview is bound to one deployment: retrying the same submission
|
||||
returns the original operation, including after an application restart or a lost
|
||||
HTTP response. The reviewed game/map, lobby ID, and key checks are preserved;
|
||||
native ports are reserved only during provisioning.
|
||||
Already running servers continue to receive updates through campaigns, betas,
|
||||
and drift reconciliation using the server's game/map scopes.
|
||||
Container tags can resolve to newer images; use a digest for a fixed image.
|
||||
Without a promoted baseline, Plutainer seeds its own defaults on first boot,
|
||||
which are not captured. Host, network, game-file volumes, destination directories,
|
||||
and RCON/Plutonium credentials use the environment and key store at launch time.
|
||||
The current main config filename and launch settings apply to new deployments.
|
||||
|
||||
Recipes do not include the separately stored Plutonium or integration credentials.
|
||||
Managed files can themselves contain sensitive configuration, so protect database
|
||||
backups as you would the existing artifact library and saved-key database. Public
|
||||
recipe responses contain file metadata and hashes, not file contents.
|
||||
|
||||
## Progress and interrupted operations
|
||||
|
||||
Creation verifies container health, then RCON, then the selected engine map.
|
||||
The detail page explains the current condition, displays elapsed time, and keeps
|
||||
its activity history. A completed container start is not reported as a completed
|
||||
server launch until the health checks pass.
|
||||
|
||||
For failed or interrupted creations, **Resume health verification** appears only
|
||||
when the original server is still registered on its assigned port and no later
|
||||
lifecycle operation has targeted that ID. It resumes observation only; it never
|
||||
recreates or restarts a container. Missing servers and later destruction attempts
|
||||
require inspection rather than replaying provisioning automatically.
|
||||
After an application restart, the worker automatically resumes observation of
|
||||
eligible interrupted startups so their occupied slots are retained. It never
|
||||
repeats container creation or deployment as part of this recovery.
|
||||
|
||||
## API
|
||||
|
||||
Collections retain the existing `/api/blueprints` endpoints and `blueprintId` /
|
||||
`blueprintName` fields for compatibility. Saved data needs no migration. Old
|
||||
`/provisioning?view=blueprints` links redirect to `/collections`.
|
||||
Collection launch details live at `/collections/launches/:id`; old
|
||||
`/provisioning/collections/:id` and `/deployments/collections/:id` links redirect
|
||||
there while preserving query parameters.
|
||||
|
||||
- `GET /api/blueprints`: list lineup summaries with game/map selections and counts.
|
||||
- `POST /api/blueprints`: save `{ name, description?, previewId }` or
|
||||
`{ name, description?, deploymentId }` for a single-map target of one lobby, or
|
||||
`{ name, description?, entries: [{ game, mapSlug, count }] }` for a collection.
|
||||
- `PUT /api/blueprints/:id`: edit `{ name, description?, entries }`. All entries
|
||||
use the latest promoted setup when launched.
|
||||
- `DELETE /api/blueprints/:id`: remove a library entry.
|
||||
- `POST /api/deployment-collections/preview`: preview `{ blueprintId, mode? }`,
|
||||
where mode is `fill` (default) or `exact`. Returns `mode`, `retainedServers`,
|
||||
`removals` (name/reason), creation `servers`, and configured `launchConcurrency`.
|
||||
- `POST /api/deployment-collections`: atomically queue `{ previewId, keys? }`,
|
||||
where `keys` maps exact preview server IDs to optional replacement keys.
|
||||
A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200,
|
||||
without creating a collection, saving keys, or kicking the worker.
|
||||
- `GET /api/deployment-collections`: list collection progress summaries with
|
||||
separate `queued`, `provisioning`, and `verifying` counts.
|
||||
- `GET /api/deployment-collections/:id`: collection mode, retained server IDs,
|
||||
per-operation progress with `action: create | destroy`,
|
||||
each waiting server's global `queuePosition`, and the shared queue's concurrency,
|
||||
active startups, and queued count.
|
||||
- `POST /api/deployments/preview`: preview `{ game, mapSlug }` or `{ deploymentId }`
|
||||
for a new single server using the latest promoted setup. All collections, including one-server
|
||||
targets, must use the collection endpoint so existing servers count correctly.
|
||||
Returns an ID, expiry, recipe summary, capacity, key
|
||||
availability (never its value), and prerequisite checks.
|
||||
- `POST /api/servers`: existing request plus optional `previewId`; the game and
|
||||
map must match that preview. `expectedName` remains supported.
|
||||
- `GET /api/deployments/:id`: includes `recipe`, `blueprintName`, and
|
||||
`canResumeVerification` for lifecycle operations.
|
||||
- `POST /api/deployments/:id/resume`: resume eligible creation health checks.
|
||||
|
||||
Collection launches begin processing immediately after all requests are queued.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Getting started
|
||||
|
||||
Electric Cherry runs a web controller and an agent on each game host. The
|
||||
controller stores fleet metadata in SQLite and uploaded game files on disk.
|
||||
Agents run Docker/Plutainer locally and connect outbound to the controller over
|
||||
HTTPS. Running game containers continue through controller or agent outages.
|
||||
|
||||
## Requirements
|
||||
|
||||
- A controller machine with Docker Compose and an HTTPS reverse proxy.
|
||||
- Debian 12 or 13 amd64 game hosts with storage for game files and instances.
|
||||
- Your complete T5 or T6 game folders, including the DLC you intend to host.
|
||||
- Plutonium server keys for the exact lobby identities you will launch.
|
||||
|
||||
The controller does not need access to a Docker socket, game-host mounts, or
|
||||
inbound RCON. Host agents handle local game commands and file operations.
|
||||
|
||||
## Start the controller
|
||||
|
||||
From the repository root:
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
Set these values in `.env`:
|
||||
|
||||
| Variable | Purpose |
|
||||
| --- | --- |
|
||||
| `EC_PUBLIC_URL` | The external HTTPS address of the dashboard. |
|
||||
| `EC_SETUP_TOKEN` | The generated secret, used to create the first administrator. |
|
||||
| `EC_DATA_DIR` | A persistent directory for the database, game files, and beta archives. |
|
||||
| `EC_TELEMETRY_TOKEN` | A separate secret of at least 32 characters for game reporters. Generate it independently with the same command. |
|
||||
|
||||
For an existing installation, retain its original data directory. Compose
|
||||
defaults to `/root/zombies/.electric-cherry`; choose a dedicated directory for a
|
||||
new controller. See [.env.example](../../.env.example) for the other settings.
|
||||
|
||||
```sh
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
Proxy the HTTPS address to controller port 3000. Open the dashboard and follow
|
||||
`/setup` to create the first administrator. Then remove `EC_SETUP_TOKEN` from
|
||||
`.env` and run `docker compose up -d` to recreate the controller. Setup stays
|
||||
closed once an account exists.
|
||||
|
||||
Add other users under **Administration → Users**. The
|
||||
[access guide](authentication.md) explains roles, proxy configuration, and
|
||||
password recovery.
|
||||
|
||||
## Prepare a game host
|
||||
|
||||
1. Open **Library → Game Files** and upload the complete local folder for T5 or
|
||||
T6, including DLC. Uploads can run before any host is enrolled. The latest
|
||||
verified upload becomes that game's default files.
|
||||
2. Open **Fleet → Hosts → Add Host**, configure the host's game-port range and
|
||||
bind address, and run the generated enrollment command on the host.
|
||||
3. Wait for automatic setup to prepare the Plutainer runtime and available game
|
||||
files. The host page reports download, verification, and readiness progress.
|
||||
|
||||
The installer sets up the Python agent as a systemd service and enables automatic
|
||||
agent updates. It installs Docker if needed. Multiple instances share read-only
|
||||
base game files; each instance has its own writable data. See
|
||||
[Native hosts](native-hosts.md) for storage layout, network access, and recovery.
|
||||
|
||||
## Launch your first server
|
||||
|
||||
1. Save the exact Plutonium server keys under **Administration → Keys**.
|
||||
2. Create a server, or build a reusable map lineup in **Fleet → Collections**.
|
||||
3. Review the launch and apply it. Provisioning queues the work, then assigns a
|
||||
host and port when capacity is available.
|
||||
4. Follow the operation until readiness checks pass, then verify that players
|
||||
can find and join the server.
|
||||
|
||||
New launches use the latest verified game files and promoted configuration,
|
||||
scripts, and plugins. Uploading newer game files does not restart existing
|
||||
servers. Disruptive operations normally wait for a fresh empty lobby or a
|
||||
verified stopped container. A rebuild removes the instance's writable files.
|
||||
See [Provisioning](provisioning.md) and [Collections](collections.md) for details.
|
||||
|
||||
## Connect game telemetry
|
||||
|
||||
Configure your game reporter or plugin to send `EC_TELEMETRY_TOKEN` in an
|
||||
authorization header to `POST /api/telemetry`. This credential is separate from
|
||||
browser login, setup, and host enrollment. Ingestion is disabled until the
|
||||
controller has a valid telemetry token.
|
||||
|
||||
The [game reporter protocol](beta-test-plans.md#game-reporter-protocol) defines
|
||||
events and payloads for gameplay statistics and beta-test evidence. Electric
|
||||
Cherry does not include the game reporter scripts themselves.
|
||||
|
||||
## Maintain the installation
|
||||
|
||||
Back up the complete controller data directory, including SQLite metadata and
|
||||
published game files. Preserve that directory across upgrades. Review
|
||||
[Maintenance](maintenance.md) for scheduled work and
|
||||
[Fleet settings](settings.md) for concurrency and retention controls.
|
||||
|
||||
For lobby autoscaling, start with the documented
|
||||
[dry run and pilot](autoscaling.md#dry-run-and-pilot) before enabling live actions.
|
||||
@@ -0,0 +1,363 @@
|
||||
# Leaderboard
|
||||
|
||||
Open **Leaderboard** in the main navigation to compare players across the fleet,
|
||||
with one entry per GUID combining Black Ops I (T5) and Black Ops II (T6). Game
|
||||
and map filters limit which matches contribute to each player's statistics and
|
||||
rank. Choose a ranking, search a name or GUID, and share the page URL with its
|
||||
filters intact. Results refresh while the page is visible.
|
||||
|
||||
## Qualifying matches
|
||||
|
||||
Matches only appear in server/global history once they reach round 2.
|
||||
Leaderboards, summaries, map options, profiles and player history include only
|
||||
completed matches with a received `game_end`. Active and interrupted matches do
|
||||
not contribute. A player must also be observed at round 2 or later; players who
|
||||
left before a completed game's end retain their recorded stats.
|
||||
This also excludes existing round 1 records. Early telemetry is retained so
|
||||
qualifying games keep their start time, roster and cumulative counters.
|
||||
|
||||
## Player profiles and match results
|
||||
|
||||
### Profiles and filters
|
||||
|
||||
Click a player's name to open their shareable profile at
|
||||
`/leaderboard/players/[guid]`. Profiles combine Black Ops I (T5) and Black Ops II
|
||||
(T6) records for the same GUID. Existing `/leaderboard/players/[game]/[guid]`
|
||||
links redirect to the combined profile. Profiles show all ten lifetime statistics,
|
||||
headshots as a percentage of kills, kills per down, average kills and observed
|
||||
rounds per game, and completion rate. Each map has its own complete set of
|
||||
records; map records and match history identify their game. Match history shows
|
||||
individual counters, server, date and completion credit in pages of 25 matches
|
||||
across both games. Map names link back to that map's leaderboard in its game.
|
||||
Use **Refresh stats** to load the latest reports. Ratios with a zero denominator
|
||||
display a dash. Lifetime statistics and per-map records always cover all recorded
|
||||
maps across both games.
|
||||
|
||||
Opening a player from the leaderboard preserves the originating game, map,
|
||||
metric, search and page in the **Leaderboard** return link. The profile's
|
||||
**Selected leaderboard** controls initialize from that game/map/metric and show
|
||||
the player's rank within that competition, independent of the original search.
|
||||
These controls also filter match history; lifetime totals stay unchanged.
|
||||
History pagination retains the selection and return link. Map links preserve
|
||||
the selected ranking metric. A player without qualifying matches in a selection
|
||||
can still view their lifetime profile and choose another scope.
|
||||
|
||||
On phones, per-map records and match history use compact cards with the map,
|
||||
game, highest round and kills up front. Expand **All stats and record runs**
|
||||
or **More match stats** for the remaining counters and source evidence. Desktop
|
||||
profiles retain the complete tables; both views use the same history filters
|
||||
and pagination. History dates say **First observed**, since a missing start
|
||||
report can make the first recorded observation later than the actual game start.
|
||||
|
||||
### Record evidence and match results
|
||||
|
||||
Highest-round, best-game-kill and highest-score records link to the completed
|
||||
match that earned them on the leaderboard and in lifetime/per-map profile stats.
|
||||
Each source shows its map, game, completion time, server, and whether the match
|
||||
start was recorded. **Run details** shows when the player was first observed:
|
||||
at the recorded start, at a reported round, or unknown for older stored records.
|
||||
Presence is not proof of continuous survival. Equal personal bests link to the
|
||||
earliest recorded completion, then the lowest match ID when times tie.
|
||||
|
||||
Completed matches link to their own results page at `/matches/[id]` from global
|
||||
Match History, server history, and player profiles. The page shows the recorded
|
||||
map and server, final round, start/end times, duration when the start was observed,
|
||||
player counts, and the full saved roster. Players missing from the final report
|
||||
retain their earlier counters; players with no stats display dashes. Counters and
|
||||
score are the highest reported values within the match, not necessarily its final
|
||||
scoreboard. Active and interrupted matches remain listed without results links.
|
||||
Match pages remain available after the original server is removed.
|
||||
|
||||
### Player identity
|
||||
|
||||
The leaderboard uses the existing telemetry events; no reporter change is
|
||||
required. The leaderboard and profiles identify a player by **GUID**, displaying
|
||||
the latest reported name across both games, regardless of game or map filters.
|
||||
Different names for the same GUID do not split their leaderboard entry or
|
||||
profile; identical names with different GUIDs remain separate. Stored player
|
||||
records remain keyed by **game + GUID** for telemetry ingestion; combining their
|
||||
identity requires no rekeying. The record-source upgrade automatically adds
|
||||
nullable participation-start fields; existing rows keep an unknown start instead
|
||||
of inferring one. GUIDs stay strings.
|
||||
|
||||
## Recorded statistics
|
||||
|
||||
| Statistic | Meaning |
|
||||
| --- | --- |
|
||||
| Highest round | Highest round reported with that player present, including players who joined mid-game. |
|
||||
| Total kills, headshots, revives, downs | Sum of each player's highest reported counter in each game. Round snapshots are cumulative, so they are never added together within a game. |
|
||||
| Total rounds | Round progression during observed participation, added across games. A player observed at game start and then round 8 receives seven rounds. A player first seen at round 8 gets a highest-round record of 8, but zero total rounds; reaching round 9 then adds one. |
|
||||
| Games played | Completed games in which a stats report at round 2 or later contains the player, even if they left before the end. Start-only and round 1 participation do not count. |
|
||||
| Games completed | Games whose `game_end` roster contains the player. Disconnected players retain earlier stats without receiving completion credit. |
|
||||
| Best game kills | Highest kill count recorded in any one game. |
|
||||
| Highest reported score | Highest observed score balance. Spending points can lower the score, so this is not cumulative points earned. |
|
||||
|
||||
Every ranking sorts highest first, including downs (a count, not an efficiency
|
||||
rating). Equal values share a rank; name and GUID provide a stable display order.
|
||||
Searching preserves the player's rank for the selected game/map filters, or
|
||||
across both games when unfiltered. Summary cards describe the same selection,
|
||||
regardless of the search text. Player counts include each qualifying GUID once,
|
||||
including when they have records in both games. Summary rounds are
|
||||
player-rounds: four players progressing from game start to round 8 contribute 28.
|
||||
Highest-round records always use the round reached in a completed match,
|
||||
including the final round and players who joined mid-game.
|
||||
|
||||
### Efficiency rankings
|
||||
|
||||
Efficiency rankings are also available for **Headshots / kills**, **Kills per
|
||||
down**, **Average kills**, **Average rounds**, and **Completion rate**. Each
|
||||
requires at least five completed games in the selected game/map scope. Headshot
|
||||
rate also requires 100 kills; kills per down requires five downs. The board shows
|
||||
the numerator, denominator and game count beside each ratio. Ratios are computed
|
||||
from aggregate counters, not an average of individual match percentages.
|
||||
Completion rate is the share of those ended games whose final report contained
|
||||
the player; it is not a win rate. Active and interrupted games do not enter any
|
||||
denominator. Zero-denominator ratios remain unknown and are not ranked.
|
||||
|
||||
Qualification is applied before ranking, search and pagination. Summary cards
|
||||
still describe all players in the game/map selection, including those below the
|
||||
threshold. Profile metrics show their values and whether the player qualifies
|
||||
for the combined lifetime board; a scoped board can have a smaller sample.
|
||||
|
||||
### Participation limits
|
||||
|
||||
Round totals require the player to appear in consecutive observations. If a
|
||||
report omits them, their return starts a new participation interval and the gap
|
||||
is not credited. The payload cannot tell whether a present player was alive,
|
||||
spectating, or absent between reports. Mid-game snapshots still contribute their
|
||||
reported kills and other counters, even when earlier participation is unknown.
|
||||
|
||||
## Persistence and match boundaries
|
||||
|
||||
Match and player records live in SQLite alongside the rest of Electric Cherry.
|
||||
They persist across app restarts and server deletion, and are not subject to
|
||||
campaign history retention. The latest scoreboard and leaderboard update in the
|
||||
same transaction. Each match keeps player counters even after someone leaves.
|
||||
|
||||
A `game_start` opens a match; `game_end` completes it. A changed map, a lower round,
|
||||
or a round report after a completed game can also establish a new match when the
|
||||
start was missed. Superseded active matches remain in history as interrupted.
|
||||
Repeated starts before any stats belong to the same match. Identical consecutive
|
||||
payloads (ignoring receipt time and roster order) do not add records. Repeated
|
||||
snapshots in a match cannot increase counters or rounds twice.
|
||||
|
||||
The first upgrade imports the latest stored full scoreboard from each server
|
||||
once. That provides the available counters and highest round, but not discarded
|
||||
earlier games or unobserved round progression. Old round-only snapshots have no
|
||||
player identity and cannot be imported. Tracking continues from future reports.
|
||||
|
||||
T6 selections sharing an engine map, such as Town Survival and TranZit, remain
|
||||
separate using the server's registered map variant. Unknown variants remain
|
||||
explicitly identified instead of being labeled as a known selection. A match
|
||||
retains the game/map identity recorded at ingestion even if the server is later
|
||||
reconfigured or removed.
|
||||
|
||||
The current protocol sends events in gameplay order and has no match ID, event
|
||||
ID, or source sequence. A late report from an older game cannot be reliably
|
||||
distinguished from a new game. Likewise, identical starts before any stats are
|
||||
indistinguishable from retries. Send reports once in order. Within a match,
|
||||
counter decreases use the previous high-water mark; if a reconnect resets a
|
||||
player's counters, subsequent activity may be undercounted until those counters
|
||||
pass their earlier maxima. Exact replay handling and counter-reset accounting
|
||||
would require additional reporter identity/ordering information.
|
||||
|
||||
## Read API
|
||||
|
||||
This reference describes the current repository implementation. A deployed
|
||||
instance must run the corresponding code to provide the same behavior. The
|
||||
sources are the [HTTP handler](../../src/routes/api/leaderboard/+server.ts),
|
||||
[response types](../../src/lib/leaderboard.ts),
|
||||
[query implementation](../../src/lib/server/fleet/leaderboard.ts), and
|
||||
[authentication guard](../../src/lib/server/auth/guard.ts).
|
||||
|
||||
### Authentication and requests
|
||||
|
||||
`GET /api/leaderboard` is a public JSON endpoint; `HEAD` is public too. A tool
|
||||
does not need an account, session cookie, API key or authorization header to
|
||||
read it. Reads also work before administrator setup and with an expired session
|
||||
or a session awaiting a password change. This exception applies only to reads
|
||||
of this exact API route. Leaderboard/profile HTML pages, match-history APIs and
|
||||
management routes still require a session; see [Users and access](authentication.md).
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
EC_BASE_URL='https://ec.example.com'
|
||||
curl --fail-with-body --silent --show-error \
|
||||
--get "$EC_BASE_URL/api/leaderboard" \
|
||||
--data-urlencode 'game=t6' \
|
||||
--data-urlencode 'metric=kills' \
|
||||
--data-urlencode 'limit=100' \
|
||||
--data-urlencode 'offset=0'
|
||||
```
|
||||
|
||||
HTTPS is required. Local HTTP development requires `EC_AUTH_ALLOW_HTTP=true`
|
||||
and a loopback hostname. The read request does not require an `Origin` header.
|
||||
This endpoint does not add cross-origin browser access (CORS) headers; the
|
||||
example above works for command-line or server-side clients. Successful responses use
|
||||
`Content-Type: application/json`, `Cache-Control: no-store` and `Vary: Cookie`.
|
||||
|
||||
### Query parameters
|
||||
|
||||
All parameters are optional. Names and enumerated values are case-sensitive.
|
||||
Use URL encoding for map keys and search text.
|
||||
|
||||
| Parameter | Default | Accepted values and behavior |
|
||||
| --- | --- | --- |
|
||||
| `game` | Both games | Exactly `t5` or `t6`. An empty value is invalid. |
|
||||
| `map` | All maps | Opaque `maps[].key`, at most 1024 characters. Empty means no map filter. An unknown key or a game/map mismatch returns an empty selection, not an error. |
|
||||
| `metric` | `highestRound` | One of the metric keys below. An empty or unknown value is invalid. |
|
||||
| `q` | No search | At most 100 characters before trimming. Matches a literal substring of the current name or GUID, ignoring ASCII letter case. Surrounding whitespace is trimmed; whitespace-only input means no search. `%` and `_` are literal characters, not wildcards. |
|
||||
| `limit` | `50` | Integer from 1 to 100. |
|
||||
| `offset` | `0` | Integer from 0 to 1,000,000; counts matching entries to skip, not ranks or pages. |
|
||||
|
||||
`limit` and `offset` accept decimal digits only. Empty strings, signs, spaces,
|
||||
fractions and exponent notation are rejected rather than clamped. String
|
||||
length limits use JavaScript string length (UTF-16 code units). Unrecognized
|
||||
query parameters are ignored; if a parameter is repeated, its first value wins.
|
||||
|
||||
Metric keys are `highestRound`, `kills`, `roundsSurvived`, `headshots`, `revives`,
|
||||
`downs`, `gamesPlayed`, `gamesCompleted`, `bestKills`, `bestScore`, `headshotRate`,
|
||||
`killsPerDown`, `averageKills`, `averageRounds`, and `completionRate`.
|
||||
|
||||
### Success response
|
||||
|
||||
HTTP `200` returns the following top-level fields directly, with no `ok` or
|
||||
`data` wrapper:
|
||||
|
||||
| Field | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `entries` | Entry array | Requested page, ordered by the selected metric descending. |
|
||||
| `maps` | Map array | Available completed-match map options, filtered only by `game`, independent of `map`, `q`, metric qualification and pagination. |
|
||||
| `total` | number | Number of matching GUIDs after metric qualification and search, before pagination. |
|
||||
| `limit`, `offset` | number | Accepted pagination values. |
|
||||
| `metric` | string | Selected metric key, including the default when omitted. |
|
||||
| `summary` | object | Totals for all qualifying completed participation in the game/map scope, independent of `q`, metric qualification and pagination. |
|
||||
|
||||
Each entry combines one GUID's qualifying participation across the selected
|
||||
games/maps. It contains all of these fields regardless of the selected metric:
|
||||
|
||||
| Entry field | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `guid` | string | Player identity. Preserve it as a string, including leading zeros. |
|
||||
| `name` | string | Latest reported name across both games, even if that report is outside the selected scope or from an unfinished match. |
|
||||
| `rank` | number | Rank within the scope after metric qualification, before search/pagination. Ties share rank and leave gaps: 1, 1, 3. |
|
||||
| `highestRound`, `kills`, `roundsSurvived`, `headshots`, `revives`, `downs`, `gamesPlayed`, `gamesCompleted`, `bestKills`, `bestScore` | number | The ten counters defined under [Recorded statistics](#recorded-statistics). |
|
||||
| `headshotRate`, `killsPerDown`, `averageKills`, `averageRounds`, `completionRate` | number or null | Ratios defined below; `null` if the denominator is zero. |
|
||||
| `lastSeen` | string | Latest player observation among the qualifying participation in this scope, not the latest global identity update. |
|
||||
| `records` | object | Personal-best evidence keyed by `highestRound`, `bestKills` and `bestScore`. |
|
||||
|
||||
Entries have no `game` field. Equal metric values are displayed in name order
|
||||
(ASCII case-insensitive, then exact name), then GUID order. `q` is a substring
|
||||
search, not an exact-GUID lookup; a tool looking up one GUID must check equality
|
||||
on returned `guid` values and account for pagination.
|
||||
|
||||
| Ratio | Calculation | Requirement to rank by this metric |
|
||||
| --- | --- | --- |
|
||||
| `headshotRate` | `headshots / kills` | `gamesPlayed >= 5` and `kills >= 100` |
|
||||
| `killsPerDown` | `kills / downs` | `gamesPlayed >= 5` and `downs >= 5` |
|
||||
| `averageKills` | `kills / gamesPlayed` | `gamesPlayed >= 5` |
|
||||
| `averageRounds` | `roundsSurvived / gamesPlayed` | `gamesPlayed >= 5` |
|
||||
| `completionRate` | `gamesCompleted / gamesPlayed` | `gamesPlayed >= 5` |
|
||||
|
||||
These are unrounded ratios, not formatted strings. For `headshotRate` and
|
||||
`completionRate`, `0.5` means 50%. Requirements use the selected game/map scope
|
||||
and apply only to the metric being ranked. Entries on a counter leaderboard
|
||||
still expose ratio values even when they would not qualify for a ratio ranking.
|
||||
`gamesPlayed` counts ended matches with qualifying participation, including
|
||||
matches the player left early; it does not require completion credit.
|
||||
|
||||
Each map option has string fields `key`, `game`, `map` and `name`. `map` is the
|
||||
engine map identifier; `name` is the display name. Pass `key` as the request's
|
||||
`map` parameter, preferably together with the option's `game`. Options come from
|
||||
completed matches reaching round 2, so an option can exist without any qualifying
|
||||
players. Removed servers' maps remain available.
|
||||
|
||||
| Summary field | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `players` | number | Unique qualifying GUIDs, combined across games. |
|
||||
| `matches` | number | Distinct completed matches with qualifying player participation; not the sum of `gamesPlayed`. |
|
||||
| `kills`, `roundsSurvived` | number | Sums across qualifying players; rounds are player-rounds. |
|
||||
| `highestRound` | number | Highest qualifying player's recorded round. |
|
||||
| `firstTrackedAt` | string or null | Earliest match `startedAt` among those matches. |
|
||||
| `updatedAt` | string or null | Latest match report receipt time among those matches; not a response-generation time or a complete data-change token. |
|
||||
|
||||
For an empty scope, numeric summary fields are zero and both timestamps are
|
||||
`null`. A search with no results or an offset beyond `total` returns
|
||||
`entries: []` while preserving the scope's summary and available maps. An
|
||||
efficiency board can likewise have no entries while `summary.players > 0`.
|
||||
|
||||
Each `records` value has the following fields. Evidence uses the same game/map
|
||||
scope as the entry, and ties choose the earliest completion, then lowest match
|
||||
ID. The three records can refer to different matches.
|
||||
|
||||
| Evidence field | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `matchId`, `value` | number | Completed match ID and that personal-best value. Link to `/matches/{matchId}` for its results page. |
|
||||
| `game`, `mapKey`, `mapName`, `server` | string | Recorded game, opaque map key, map display name and server name. |
|
||||
| `startedAt` | string | First recorded match observation; only the game start when `startObserved` is true. |
|
||||
| `endedAt` | string or null | Recorded completion time; can be unknown in older stored records. |
|
||||
| `startObserved` | boolean | Whether a `game_start` was received for the match. |
|
||||
| `presentAtStart` | boolean or null | Whether the player was first observed at the recorded start; `null` for unknown older participation. |
|
||||
| `firstObservedRound` | number or null | Round of the player's first observation, or unknown. This does not prove continuous survival. |
|
||||
|
||||
Timestamps reflect controller receipt/observation times, normally ISO 8601 UTC
|
||||
strings such as `2026-10-05T12:00:00.000Z`, not reporter gameplay-clock times.
|
||||
|
||||
### Errors, pagination and freshness
|
||||
|
||||
Invalid query values return HTTP `400` with
|
||||
`{ "ok": false, "kind": "bad-request", "error": "..." }`. Public reads do not
|
||||
return `401` for missing/expired sessions or `403` for a required password change.
|
||||
Sending host/enrollment bearer credentials still returns `403` because those
|
||||
credentials are restricted to the host-agent API; omit them. Disallowed HTTP transport
|
||||
returns `426` with an `error` string. Report-service failures can produce `5xx`;
|
||||
check status and content type before assuming the success schema or a uniform
|
||||
error body.
|
||||
|
||||
For successive pages, retain the same filters and increase `offset` by `limit`
|
||||
until `offset + entries.length >= total` or the page is empty. There is no cursor,
|
||||
snapshot token, date-range filter or incremental `since` parameter. Each report
|
||||
reads a consistent database snapshot, but separate requests may see newly ended
|
||||
matches or name changes. A multi-page crawl can therefore miss or repeat players
|
||||
as their ordering changes; deduplicate by GUID and refresh as needed. The offset
|
||||
cap also limits how far a tool can page through a very large selection.
|
||||
|
||||
Reports run in a worker and use caches keyed by database revision, invalidated
|
||||
by writes to match, participation and player-identity tables. There is no fixed
|
||||
TTL wait before newly committed leaderboard data becomes eligible for a fresh
|
||||
report. Identical requests already in flight may share a response. Neither
|
||||
`summary.updatedAt` nor an entry's `lastSeen` should be used as a cache revision:
|
||||
an identity rename can change results without changing those timestamps.
|
||||
|
||||
### Related routes and current limits
|
||||
|
||||
`GET /api/matches` and `GET /api/servers/{name}/matches` return JSON match
|
||||
summaries as `{ "matches": [...], "total": number }`. These routes still require
|
||||
session authentication. Both accept only `limit` (default 25, range 1–10,000); there is
|
||||
no offset pagination. Results are newest first by first observation, then match
|
||||
ID, and include active/interrupted matches that reached round 2. Their
|
||||
[ServerMatch fields](../../src/lib/match-history.ts) differ from leaderboard entries
|
||||
and do not include player counters or a roster. Unknown server names return an
|
||||
empty list. URL-encode server names in paths.
|
||||
|
||||
`/leaderboard/players/{guid}` and `/matches/{id}` are HTML pages backed by server
|
||||
loaders. There is currently no dedicated JSON endpoint at
|
||||
`/api/leaderboard/players/{guid}` or `/api/matches/{id}` for complete profiles,
|
||||
player match history or match rosters. A tool needing those records will need
|
||||
an API extension; the leaderboard endpoint supplies aggregate stats and
|
||||
personal-best evidence only.
|
||||
|
||||
## Query performance
|
||||
|
||||
On a cache miss, leaderboard reads aggregate the selected completed participation
|
||||
for rankings and summary totals. The ranked scope and map options are reused
|
||||
across search/pagination requests at the same database revision; filtering and
|
||||
pagination operate on that ranked list. Empty and out-of-range pages still
|
||||
return accurate totals. Record evidence is fetched only for requested entries.
|
||||
Exact player-rank reads use the same ranked scope, including its summaries and
|
||||
map list. Report work runs in a dedicated worker with revision-keyed response
|
||||
caching as described above.
|
||||
|
||||
See the [historical SQL benchmark](../research/leaderboard-query-benchmark.md)
|
||||
for the earlier optimization measurements and methodology.
|
||||
@@ -0,0 +1,178 @@
|
||||
# Maintenance
|
||||
|
||||
Open **Operations → Maintenance** to keep existing servers healthy and current.
|
||||
Maintenance owns observations, policies, schedules, requests, and verification of
|
||||
maintenance outcomes. Provisioning owns lifecycle execution, player checks,
|
||||
conflicts, rolling capacity, container recovery, and persistent operation history.
|
||||
Campaigns retain managed-file deployment evidence; Collections define lineups.
|
||||
|
||||
## Views
|
||||
|
||||
- **Overview** shows active work, attention, upcoming restarts, and Plutonium version
|
||||
coverage. Filter by capability or server. **Check now** refreshes health, drift,
|
||||
the production release, and running revisions.
|
||||
- **Policies & schedules** controls health recovery, configuration drift, automatic
|
||||
Plutonium updates, and the optional daily restart schedule independently.
|
||||
- **Activity** combines recovery findings, update results, and scheduled restarts.
|
||||
Follow a finding for evidence and attempts, or its Provisioning operation for
|
||||
execution progress. Cancel unsubmitted runtime restarts here.
|
||||
|
||||
Fleet and server details link their revision indicators to Maintenance. Failed
|
||||
runtime maintenance appears in Fleet attention, even when the underlying restart
|
||||
succeeded. Provisioning and repair Campaigns link back to the requesting work.
|
||||
|
||||
## Policies and pause
|
||||
|
||||
Health recovery and drift repair inherit the previous Watchdog enabled setting.
|
||||
Automatic Plutonium updates are **disabled by default**. Existing daily schedules
|
||||
retain their enabled state, timezone, and next occurrence.
|
||||
|
||||
**Pause all maintenance** holds new remediation, scheduled runs, and unsubmitted
|
||||
maintenance operations. Scans and result verification continue; submitted
|
||||
operations finish recovery. Campaigns, betas, and manually requested Provisioning
|
||||
operations continue independently. Resuming keeps each policy's saved setting.
|
||||
|
||||
Disabling health recovery or drift repair holds queued work for that capability
|
||||
and prevents further automatic attempts, including health escalation. Disabling
|
||||
automatic updates holds queued automatic update requests. Explicitly requested
|
||||
updates remain operator-authorized work. Disabling a daily schedule prevents
|
||||
future runs but does not cancel requests it already created.
|
||||
|
||||
All capabilities respect intentionally stopped servers. Start them explicitly
|
||||
through Provisioning before expecting maintenance to act on them.
|
||||
|
||||
While Provisioning owns a server, Maintenance skips new health/drift observations,
|
||||
version probes, automatic updates, and scheduled restart requests for that server.
|
||||
Queued, waiting, running, and recovering operations count, including failed
|
||||
operations that retain recovery capacity. Other servers continue normally. Existing
|
||||
maintenance tracks its own operation to completion; revision verification waits
|
||||
until any later operation releases the server.
|
||||
|
||||
## Plutonium updates
|
||||
|
||||
The published revision comes from the official production updater manifest.
|
||||
Running revisions come from RCON `version` after checking container readiness.
|
||||
Checks run hourly while the application is running. Failed release checks retain
|
||||
the cached release with an error. Unsupported replies, stale checks, unhealthy
|
||||
containers, and evidence from before a lifecycle submission remain unverified.
|
||||
A missing revision is never treated as current or as an automatic update target.
|
||||
|
||||
**Update on restart** requests ordinary Provisioning restarts for verified outdated
|
||||
servers. **Update outdated servers on restart** applies to the filtered list.
|
||||
Automatic updates, when enabled, apply to verified outdated servers across the
|
||||
fleet, with one attempt per server and published revision. Existing pending
|
||||
maintenance is reused. Occupied servers and servers with conflicting beta or
|
||||
campaign work wait without consuming a restart slot.
|
||||
|
||||
Plutainer's startup updater must be enabled (`PLUTAINER_AUTO_UPDATE`). Electric
|
||||
Cherry does not download binaries, pull a container image, or pin the runtime to
|
||||
a particular revision. The observed published revision is recorded as the minimum
|
||||
expected result; the actual revision is queried after recovery.
|
||||
|
||||
A healthy restart and a successful update are separate outcomes. If the server
|
||||
recovers on an older or unknown revision, the restart is successful in Provisioning
|
||||
and the update needs attention in Maintenance. Update verification never triggers
|
||||
a rebuild or an automatic restart loop. Inspect the updater configuration and
|
||||
server logs, then use **Retry after inspection** to create a new attempt. Ambiguous
|
||||
or failed lifecycle requests require inspection of the existing Provisioning
|
||||
operation and its retained recovery slot first.
|
||||
|
||||
Update attempts persist as suppression evidence across app restarts and history
|
||||
retention. Cancelling an automatic attempt suppresses another automatic attempt for
|
||||
that same revision; an operator can request an update explicitly.
|
||||
|
||||
## Scheduled restarts
|
||||
|
||||
Daily restarts are independent of update availability. At the saved queue time,
|
||||
Maintenance requests a restart for each server whose desired state is running.
|
||||
These requests have no revision requirement. Servers owned by Provisioning are
|
||||
skipped for that occurrence; the schedule advances normally. Requests already
|
||||
queued can wait for players, campaign work, or beta work to finish.
|
||||
|
||||
Saving a schedule uses the selected timezone and follows daylight saving changes.
|
||||
Skipped times run at the first valid minute after the gap; repeated times run once
|
||||
on their first occurrence. Legacy UTC schedules keep their timing until saved.
|
||||
Changing the browser display timezone does not change the saved schedule.
|
||||
|
||||
Schedules survive application restarts. Missed days, including while globally
|
||||
paused, become one run on resume. Pending work is deduplicated per server. The
|
||||
worker starts on the first application request, so keep the app running and
|
||||
reachable. Finished scheduled and legacy maintenance activity is retained for
|
||||
30 days by default (configurable in Administration → Settings); update attempt records remain durable.
|
||||
|
||||
## Health recovery and configuration drift
|
||||
|
||||
Fleet keeps successful game and running-container reports fresh for 90 seconds.
|
||||
Two consecutive failed monitoring checks mark RCON health stale; a first timeout
|
||||
or send failure remains visible in diagnostics without changing the health badge.
|
||||
Configuration and invalid-reply errors, including authentication failures, take
|
||||
effect immediately, as do failed manual probes. Successful RCON checks reset the
|
||||
failure count; skipped checks do not count. Without fresh telemetry, RCON evidence
|
||||
older than 120 seconds is offline. Explicit stopped/unhealthy container states
|
||||
and lifecycle changes still take precedence.
|
||||
|
||||
The status monitor collects container and game health every 30 seconds. Once per
|
||||
minute, the watchdog evaluates that completed evidence without issuing its own
|
||||
routine health probes. Samples expire after 90 seconds; missing or stale evidence
|
||||
leaves health unknown and cannot justify recovery. Samples retain their original
|
||||
timestamps and instance identity, and each failure sample counts at most once.
|
||||
A monitoring gap or changed instance resets the sustained failure window. After
|
||||
an application restart, routine watchdog checks wait for a new status-monitor pass.
|
||||
**Check now** and recovery verification still request fresh evidence, and
|
||||
Provisioning retains its checks before executing recovery actions.
|
||||
|
||||
Three observations over at least two minutes are required before an automatic
|
||||
health restart; starting or restarting containers get five minutes. Unknown host-agent responses are observations,
|
||||
not proof of a stopped container. Repeated manual checks do not accelerate this
|
||||
window. A successful routine or manual game-status probe between scans resets
|
||||
the failure streak. Missing RCON configuration, unresolved ports, and invalid
|
||||
replies remain diagnostic findings and cannot authorize health recovery.
|
||||
|
||||
A health finding gets one restart, then one rebuild if recovery fails after five
|
||||
minutes and fresh health checks still justify recovery. If current container and
|
||||
game checks succeed before an automatic health restart or rebuild is submitted,
|
||||
the obsolete operation is cancelled and the finding is resolved. Unknown health
|
||||
holds recovery for another check. Drift repairs and requested lifecycle operations
|
||||
retain their own eligibility rules.
|
||||
|
||||
Escalation transfers only that finding's restart slot. A failed rebuild
|
||||
requires operator attention. **Rebuild deletes all instance files** and uses the
|
||||
latest promoted setup, the same server identity and port, and the saved server key.
|
||||
Missing keys block teardown. Submitted rebuild verification continues while paused.
|
||||
|
||||
Drift includes missing promoted files, changed bytes, and extra files in managed
|
||||
script, config, and plugin directories. Config comparison includes the composed
|
||||
baseline, overlays, and the selected map. Unmanaged default config is exempt until
|
||||
a baseline is promoted. Missing mounts are scan errors, not empty directories.
|
||||
Routine drift scans run every five minutes per server, reducing file traffic
|
||||
while health observations continue every minute. **Check now**, a replacement
|
||||
registration, or a new lifecycle submission bypasses the drift interval. Failed
|
||||
file scans retry on the next observer pass; post-rebuild verification still reads
|
||||
fresh files.
|
||||
|
||||
Drift repair creates a Campaign capturing observed differences and requests a
|
||||
Provisioning rebuild. The Campaign observes that rebuild without requesting
|
||||
another restart. Its detection evidence and the actual launch recipe remain
|
||||
available even if promotions change while the work waits. Histories survive
|
||||
instance removal. Active beta and campaign work defers new recovery.
|
||||
|
||||
**Dismiss** suppresses health/drift attempts until the condition clears. **Retry
|
||||
recovery** begins a fresh bounded attempt after inspection. Existing operations
|
||||
must finish or be cancelled first. Fresh verification resolves the finding.
|
||||
|
||||
## Compatibility and API
|
||||
|
||||
Existing job, incident, and operation IDs are retained. Upgrade migration adds
|
||||
request metadata without replaying commands; existing pending requests keep their
|
||||
original verification requirements. `/watchdog` and `/watchdog/:id` redirect to
|
||||
Maintenance, preserving query parameters. Old Watchdog and Plutonium APIs remain
|
||||
compatible. The previous Watchdog enabled switch controls health and drift only.
|
||||
|
||||
- `GET /api/maintenance`: policies, recovery findings, release/version coverage,
|
||||
schedule, and recent runtime activity with linked operation stages.
|
||||
- `POST /api/maintenance`: actions `scan`, `pause` (`paused` boolean), `policy`
|
||||
(`policy`: `health`, `drift`, or `updates`; `enabled` boolean), `update` (optional
|
||||
`servers` array), `schedule` (`enabled`, `time`, `timeZone`), `cancel` or `retry`
|
||||
(`id`, the numeric runtime maintenance activity ID).
|
||||
- `/api/maintenance/:id`: GET recovery evidence and attempts; POST `retry` or
|
||||
`dismiss` for a health/drift finding.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Metrics
|
||||
|
||||
Metrics is under **Analytics** at `/metrics` (keyboard shortcut: `g m`).
|
||||
Choose the past 24 hours, 7 days, or 30 days. The page refreshes every minute.
|
||||
|
||||
## Overview and filters
|
||||
|
||||
The overview shows **Time with players** (occupied server-hours divided by known
|
||||
server-hours) and **Player-hours**. Known and unknown status time remain visible
|
||||
alongside the totals. **Last collected** is the newest recorded batch, including
|
||||
unknown checks; **Report refreshed** is when this report was calculated. Collection
|
||||
stops when no servers are registered. Earliest retained observations do not prove
|
||||
continuous history or the original installation date.
|
||||
|
||||
Background refresh keeps the last successful report visible on failure and offers
|
||||
Retry. Returning to a visible tab refreshes immediately. The selected range is
|
||||
stored in the URL, and refreshes preserve table controls and chart selection.
|
||||
|
||||
Game and map filters apply to the whole report, including its comparisons,
|
||||
recommendations, and heatmap. Map choices preserve the exact recorded game/map
|
||||
pair. Range and scope are shareable through the URL. Table searches remain local.
|
||||
|
||||
## Timeline and tables
|
||||
|
||||
The timeline defaults to **Share of captured time**: each stack includes occupied,
|
||||
idle, and unknown server time, normalized to 100%. This denominator differs from the
|
||||
headline utilization percentage, which excludes unknown time. Switch to **Hours**
|
||||
to compare absolute server time. Periods without captured time remain gaps.
|
||||
The timeline has labelled axes and marks partial edge periods. Select a period,
|
||||
use Previous/Next, or select a bar to inspect its exact interval and values. The
|
||||
time table supplies the same figures for comparison and assistive technology.
|
||||
Zero captured time is distinct from an observed empty server. Short durations use
|
||||
seconds or minutes so a real observation is not rounded to zero.
|
||||
|
||||
Map and server tables support numeric sorting and local search. Their searches
|
||||
apply only to the named table. Known-capacity duration stays next to all-full time.
|
||||
Server identities retain the technical name; rows describe history grouped by
|
||||
name, which may include different instances if a name was reused. **Open current
|
||||
server** appears only when that name is currently registered; its destination
|
||||
represents the current registration rather than every instance in the history.
|
||||
|
||||
## Comparisons and hosting experiments
|
||||
|
||||
**Explore this period** adds experimental decision aids below the main timeline:
|
||||
|
||||
- **Map capacity pressure** pools all-full map-hours divided by known-capacity
|
||||
map-hours across the selected map groups. Overlapping map intervals are counted
|
||||
separately. The score is not elapsed fleet time or a fleet health score; its
|
||||
denominator and contributing map count remain visible.
|
||||
- **Previous-period comparisons** use the preceding equally long window with
|
||||
identical filters. Utilization changes are percentage points; player-hours
|
||||
changes are relative percentages. A previous observed zero can show **New**.
|
||||
Missing evidence remains unavailable. Partial retained history is labelled;
|
||||
a 30-day comparison usually has little previous history because default retention is
|
||||
31 days. Both periods' known time is shown. Bounds cannot establish that
|
||||
collection was uninterrupted, and changes are not adjusted for fleet size.
|
||||
- **Activity heatmap** groups captured time by weekday and hour in its displayed
|
||||
timezone. Color represents occupied / known server time. Exact cell values
|
||||
include unknown time and contributing dates. No-data cells differ from observed
|
||||
zero utilization; repeated DST hours contribute their actual elapsed time.
|
||||
- **Hosting experiments** are generated automatically from the selected map
|
||||
observations. They are advisory and never create, restart, or destroy a server.
|
||||
|
||||
Hosting Rules v1 deliberately offer experiments even with small samples:
|
||||
|
||||
| Recorded condition | Suggested experiment |
|
||||
| --- | --- |
|
||||
| At least 20% of known capacity time had every lobby full | Trial one additional lobby during busy hours. |
|
||||
| Capacity unavailable and players present for at least 80% of known server time | Tentatively trial an extra lobby using occupancy as a proxy. |
|
||||
| Players present for at most 10% of known time, with under 1% measured pressure or unavailable capacity | Review one idle lobby or a shorter hosting window while preserving access to the map. |
|
||||
| Occupancy unavailable | Check telemetry/RCON reporting and watch for known player counts. |
|
||||
| Other observations | Check missing capacity or keep the lineup and watch peaks. |
|
||||
|
||||
Each suggestion includes evidence and an outcome to watch. Unknown capacity,
|
||||
less than one hour of relevant known evidence, or substantial unknown captured
|
||||
status makes the suggestion tentative. These thresholds are product hypotheses;
|
||||
occupancy does not establish failed joins or a need for additional capacity.
|
||||
Suggestions use historical observations. Check the current lineup before trying
|
||||
a change; a retained map may no longer have a hosted lobby.
|
||||
|
||||
## Definitions and collection
|
||||
|
||||
- Utilized means a successful status response reported at least one player.
|
||||
- Idle means a successful response reported zero players.
|
||||
- Unknown covers failed checks, provisioning, and unavailable containers; it is excluded from utilization percentages.
|
||||
- Player-hours integrate the reported player count over observed time. Bots are included if the server counts them as players.
|
||||
- All lobbies full measures map wall time when every tracked lobby in that game/map group has a known capacity and is full. Missing capacity or status makes that interval unknown. This is a capacity-pressure indicator, not a count of rejected joins or unmet demand.
|
||||
|
||||
Observations are recorded by the existing status worker. Each remains valid until the next collection, up to the 90-second freshness limit. Longer manager outages are excluded. Charts split time at UTC hour/day boundaries; partial periods remain partial. Map variants use the configured variant when it matches the reported map.
|
||||
|
||||
Capacity comes from `sv_maxclients` / `sv_maxClients` in the status response. When unavailable, the UI displays unknown rather than assuming a four-player limit. Capacity coverage is the map time for which every tracked lobby reported both occupancy and capacity.
|
||||
|
||||
Collection begins after this feature is installed; previous occupancy cannot be reconstructed. Samples default to 31 days of retention; Administration → Settings can change this period. Cleanup runs during collection and in the background. They survive removal of a server and contain no player identities. The additive database table is created automatically on startup.
|
||||
|
||||
## API
|
||||
|
||||
`GET /api/metrics` supplies the same report as the page and disables response
|
||||
caching. Query parameters are `days` (1, 7, or 30), `game`, `map` (requires game),
|
||||
and `timeZone` (an IANA zone). Invalid ranges and zones fall back to the defaults;
|
||||
invalid scope syntax is normalized. Fleet collection timestamps remain separate
|
||||
from selected-scope evidence timestamps. Collection boundaries are determined
|
||||
before scope filtering, so samples from other groups still end a batch interval.
|
||||
@@ -0,0 +1,711 @@
|
||||
# Native host management
|
||||
|
||||
Electric Cherry controls Docker/Plutainer through an outbound host agent. The controller
|
||||
does not need mounted game-server files. It owns desired state, campaigns, collections, safety queues and history;
|
||||
the agent owns local execution and its durable operation journal.
|
||||
|
||||
## Install and launch
|
||||
|
||||
1. Run the controller with persistent `/data` storage. Set `EC_PUBLIC_URL` to its
|
||||
HTTPS origin and proxy TLS to port 3000. Set a random `EC_SETUP_TOKEN` and create
|
||||
the first administrator at `/setup`. Management pages/APIs require user login
|
||||
and roles; agent endpoints authenticate their own host credentials. See
|
||||
[Users and access](authentication.md) for setup and password recovery.
|
||||
2. Open **Hosts → Add Host**, choose a name, local game-port range and bind address,
|
||||
and generate an enrollment. Run the displayed command on Debian 12/13 amd64.
|
||||
3. The installer validates the platform, installs Python 3 and an absent Docker
|
||||
installation, verifies the downloaded agent and updater SHA-256 over HTTPS,
|
||||
enrolls it and enables `electric-cherry-agent.service` and its automatic update
|
||||
timer. It needs no Git, Node.js or local build.
|
||||
4. In **Game Files**, upload your complete local folder under T5 or T6. Include
|
||||
the DLC files in that folder; there is no separate DLC declaration, library
|
||||
name or version label. The upload becomes the latest files for that game after
|
||||
verification. Uploading also works before enrolling any host.
|
||||
5. Hosts automatically prepare the configured Plutainer runtime and the latest
|
||||
verified T5 and T6 files whenever available. **Automatic host setup** reports
|
||||
progress and errors. Wait for verification before launching. Hosts report
|
||||
connection and readiness separately; deployment also checks the particular
|
||||
game files, image, capacity and ports required for the launch.
|
||||
6. Launch a server or collection. The latest game files and default runtime
|
||||
are selected automatically. An advanced runtime override is
|
||||
available for images an admin has already prepared on the destination host.
|
||||
If the latest files change after previewing, refresh the preview
|
||||
before launching.
|
||||
Apply saves the requested lobby IDs. Provisioning selects a host and atomically
|
||||
reserves UDP/TCP ports when capacity is available.
|
||||
Saved Plutonium server keys remain associated with their exact server names.
|
||||
|
||||
A fresh destination host downloads centrally stored game files without manual
|
||||
copying or per-container setup. EC does not supply the game content itself.
|
||||
|
||||
### Docker installation
|
||||
|
||||
The installer uses Debian's Docker packages: `docker.io` on Debian 12, plus
|
||||
`docker-cli` on Debian 13. Rerunning it after an earlier Debian 13 install that
|
||||
omitted the client adds the missing client without restarting the existing engine.
|
||||
An already available Docker installation (including Docker CE) is reused. If the
|
||||
daemon is unavailable, the installer reports the Docker error and stops before
|
||||
enrollment; inspect `systemctl status docker` and `journalctl -u docker`.
|
||||
|
||||
### Automatic host setup
|
||||
|
||||
The controller reconciles automatic setup on agent heartbeats. Command polls only
|
||||
deliver work already queued. Setup first pulls `PLUTAINER_IMAGE` (default
|
||||
`ghcr.io/ayymoss/plutainer:latest`), then prepares the latest verified upload for
|
||||
T5 and for T6 when available, one setup operation at a time. Uploading new files
|
||||
or enrolling another host needs no manual propagation. Setup waits for an active, connected,
|
||||
healthy host with sufficient storage. Failed operations retry automatically with
|
||||
backoff from one minute to fifteen minutes; their errors remain visible in the
|
||||
host's setup panel and **Hosts → Operation history**. This shared history lists
|
||||
operations across hosts, newest first, with 25 rows per page and host, target,
|
||||
and source details. It defaults to **Actions & errors**; **Routine checks** and
|
||||
**All operations** include successful monitoring reads. Older records may lack
|
||||
source details.
|
||||
Disconnected hosts resume when they
|
||||
reconnect. Draining and disabled hosts do not receive new automatic setup work.
|
||||
|
||||
### Operation history retention
|
||||
|
||||
**Settings → Routine operation history retention (days)** controls cleanup of
|
||||
successful monitoring and file-read records. The default is one day; zero keeps
|
||||
them indefinitely. Actions, failures, negative checks, and unresolved operations
|
||||
are retained. Agent 1.4.9 also cleans up successful routine journal records after
|
||||
the controller confirms their final receipt. Older unconfirmed journal entries
|
||||
remain until a replay supplies that confirmation. Cleanup runs in bounded batches.
|
||||
|
||||
### New game files and cached versions
|
||||
|
||||
An automatic download that has not been delivered to the agent can be replaced
|
||||
when a newer upload becomes available. A delivered operation finishes safely
|
||||
before the host prepares the latest files. Failed or incomplete uploads never
|
||||
replace the latest verified files.
|
||||
|
||||
Verified previous uploads remain cached and running servers keep their existing
|
||||
files and runtime; uploading new files does not restart servers. Automatic setup
|
||||
ensures that the configured image is present; it does not refresh an already
|
||||
cached mutable image tag. Advanced controls on the
|
||||
host and game-file pages can explicitly pull a custom runtime, prepare an older
|
||||
upload or retry preparation immediately.
|
||||
|
||||
### Storage layout
|
||||
|
||||
The default host root is `/var/lib/electric-cherry`; installer `--data-dir` changes
|
||||
it. Use a dedicated, root-owned directory. For a separate disk, use a child of its
|
||||
mount point (for example `/mnt/games/electric-cherry`), with no nested mounts, so
|
||||
host deletion can remove the entire directory. The installer records ownership
|
||||
and refuses to adopt unrelated files. Data is separated into `libraries/<game>/<version>/content`,
|
||||
`instances/<UUID>/<generation>/app`, and `agent/operations.sqlite3`. Base libraries
|
||||
are mounted read-only and instance deletion removes only that instance's files.
|
||||
The controller stores large files under `GAME_FILES_ROOT` (default beside the
|
||||
SQLite database), with manifests/checksums and operation metadata in SQLite.
|
||||
|
||||
The agent uses Plutainer's documented [game-file and app mounts](https://github.com/Ayymoss/Plutainer/blob/main/docs/volumes-and-configs.md).
|
||||
Its log reader supports relative log links that resolve inside the instance app
|
||||
directory, while file writes reject symlinks and paths outside that directory.
|
||||
|
||||
## Automatic agent updates
|
||||
|
||||
New installations enable automatic agent updates by default. Each host checks
|
||||
its controller's release over HTTPS about every five minutes, beginning about
|
||||
two minutes after boot, with up to 30 seconds of jitter. Updating the controller
|
||||
therefore makes its compatible, newer agent release available to enrolled hosts.
|
||||
There is no controller command to schedule and no update button to press.
|
||||
|
||||
**Existing hosts need the current installer once.** From the host's **Agent
|
||||
access**, generate an installer command and rerun it on the same machine, with
|
||||
the same controller and data directory. Do not use `--reenroll` for this upgrade.
|
||||
The installer preserves host identity, credentials, game files and running
|
||||
containers. Agent version 1.2.0 includes automatic update support; the host page
|
||||
shows **Setup required** until a fresh heartbeat confirms the updater is installed.
|
||||
|
||||
The **Automatic agent updates** panel reports the running version, target
|
||||
version, last check, progress and any error. Offline hosts show their last
|
||||
reported status. Deferred updates report why they are waiting: for example an
|
||||
active agent operation, an unavailable controller or a newer updater bootstrap.
|
||||
Checks also defer while the agent service is stopped; an update does not start a
|
||||
service that was stopped outside the update workflow.
|
||||
The independent updater holds the same execution lock, verifies the release
|
||||
checksum, version and supported protocol, replaces agent code, and restarts the
|
||||
agent service. It never restarts Docker or game containers. It confirms an
|
||||
authenticated heartbeat from the new agent process before accepting the update;
|
||||
failure restores the previous agent code. Its durable journal permits recovery
|
||||
after a host reboot or interrupted update.
|
||||
|
||||
The updater only installs newer compatible versions; rolling the controller back
|
||||
does not downgrade agents. A rejected version/checksum pair stays blocked until
|
||||
the controller publishes a different pair that is still newer than the installed
|
||||
agent. Inspect the reported error when a host
|
||||
shows **Needs attention**. The updater itself is a stable bootstrap component:
|
||||
it changes only when the installer is rerun. If a future release requires an
|
||||
unsupported updater protocol, rerun that release's installer on the host.
|
||||
|
||||
### Publishing releases and inspecting updates
|
||||
|
||||
When publishing changed agent code, bump its numeric `VERSION` in
|
||||
`agent/ec_agent.py` and rebuild the controller. Packaging derives the manifest
|
||||
version and checksums from the sources; an already installed version is never
|
||||
replaced just because its bytes changed.
|
||||
|
||||
The installer places the updater at
|
||||
`/usr/local/libexec/electric-cherry-update.py` and enables
|
||||
`electric-cherry-update.timer`, which starts the oneshot
|
||||
`electric-cherry-update.service`. Its state is kept under `/etc/electric-cherry`,
|
||||
including `update-status.json`, `update/journal.json`, `agent-health.json` and
|
||||
the shared `update.lock`. Check the timer and worker logs on the host with:
|
||||
|
||||
```sh
|
||||
systemctl status electric-cherry-update.timer
|
||||
journalctl -u electric-cherry-update.service -n 100
|
||||
```
|
||||
|
||||
## Placement and resource capacity
|
||||
|
||||
### Host inventory and eligibility
|
||||
|
||||
The Inventory panel shows CPU usage with agent 1.4.2 and later. It measures
|
||||
host-wide busy time across all cores on a 0–100% scale between inventory samples
|
||||
(normally about 15 seconds). Idle and I/O wait are excluded; stolen CPU time
|
||||
counts as unavailable capacity. The first sample after startup, unavailable
|
||||
counters, and older agents show **Not reported**. Offline hosts retain their
|
||||
last reported value. The controller defaults to retaining CPU and memory samples for seven days (missing measurements
|
||||
remain unknown). Scheduling checks capacity against the latest heartbeat and uses
|
||||
the last five minutes to rank eligible hosts; the Inventory panel shows the latest
|
||||
reading.
|
||||
|
||||
New servers are placed automatically. Collection previews use cached availability
|
||||
estimates and submission saves unplaced jobs without running capacity checks.
|
||||
Provisioning checks each job asynchronously: a connected, active host must have
|
||||
verified game files, its runtime, storage, a free UDP/TCP port, and valid CPU and
|
||||
memory measurements. Hosts with stale or missing measurements cannot accept new
|
||||
servers. Existing API clients may still explicitly constrain a launch to a host.
|
||||
|
||||
Queued jobs retain their accepted game-file version. Automatic host preparation
|
||||
also downloads the game files needed by unplaced jobs; a newer upload does not
|
||||
remove files still needed by the queue. Only the configured default runtime is
|
||||
pulled automatically. Custom images must already be cached on the selected host,
|
||||
or on an active host for automatic placement, before a launch is accepted. Admins
|
||||
can prepare them from the host page. Queued launches and migrations never trigger
|
||||
custom image pulls, and container creation cannot implicitly pull a missing image.
|
||||
|
||||
### Resource samples and learned allowances
|
||||
|
||||
Only provisioning reserves a host and port. Agent **1.4.6** fixes container
|
||||
discovery for resource sampling (the
|
||||
earlier collector omitted `label=` from its Docker filter). It samples each
|
||||
managed running container with a bounded
|
||||
[`docker stats`](https://docs.docker.com/reference/cli/docker/container/stats/)
|
||||
batch during inventory collection. CPU is normalized to cores (100% Docker CPU
|
||||
is one core); memory is the Linux working usage reported by Docker, excluding
|
||||
reclaimable cache. Missing or failed samples stay unknown instead of becoming
|
||||
zero usage. Samples must match the host's live instance and generation.
|
||||
Measurements arrive through `/api/agent/heartbeat`, normally every 15 seconds
|
||||
plus collection and delivery time, independently of RCON or inspection operations.
|
||||
Sampling failures appear on the host's Learned server capacity panel. Existing
|
||||
containers can begin contributing running samples as soon as the agent updates;
|
||||
startup samples require observing the first five minutes of a container's run.
|
||||
|
||||
The controller defaults to retaining per-instance minute peaks for 24 hours and caches learned
|
||||
CPU/memory allowances. The first five minutes after a container starts count as
|
||||
startup; later samples train the running allowance. Startup uses the observed
|
||||
maximum and running uses the 95th percentile of minute peaks, with **no per-server
|
||||
headroom**. Each calculation considers up to the latest 10,000 minute peaks.
|
||||
Host-local evidence is preferred to account for different hardware; new hosts
|
||||
can borrow the fleet's learned allowance. Startup allowances are never lower
|
||||
than running allowances. The host page shows both allowances and their source.
|
||||
Expand **View samples** in Learned server capacity to inspect retained minute
|
||||
peaks for the last 24 hours. Filter by startup/running phase or switch from this
|
||||
host to the entire fleet. Rows show time, server and generation, CPU cores, and
|
||||
memory in MiB; CPU and memory peaks may come from different
|
||||
heartbeats. Results are paginated, and **Refresh samples** loads the latest stored
|
||||
measurements without contacting agents. The allowance count may lag the table
|
||||
until the next calculation or reflect a different source.
|
||||
|
||||
Lower allowances require at least three startup minute samples spanning two
|
||||
minutes, or twenty running minute samples spanning five minutes. Until then,
|
||||
**0.5 core and 1 GiB** are temporary cold-start allowances, raised immediately
|
||||
if observations exceed them. They are replaced by measured allowances once the
|
||||
evidence threshold is met; a tiny 0.01-core floor prevents zero-CPU admission.
|
||||
Models expire after 24 hours without measurements. Recalculation happens during
|
||||
heartbeat ingestion, at most once per minute per host/fleet phase; new peaks
|
||||
raise the cache immediately. Launch admission reads only the cached model.
|
||||
|
||||
### Admission and placement
|
||||
|
||||
Scheduling still leaves **5% host CPU and 500 MB RAM (500,000,000 bytes)** for the OS and bursts.
|
||||
These allowances are scheduling estimates, not Docker limits. Collection previews
|
||||
estimate how many new servers fit at their running allowance. Jobs may still be
|
||||
queued when that estimate is insufficient. Current host CPU and
|
||||
available memory account for existing containers and other processes. Observed
|
||||
containers are not topped up to historical startup or running peaks, even while
|
||||
awaiting health verification. Established instances are covered by host usage
|
||||
when individual container measurements are unavailable.
|
||||
|
||||
Dispatch checks the next launch's startup allowance against live host usage.
|
||||
Unplaced queued launches wait their turn without holding physical resources.
|
||||
Any previously assigned reservations remain protected. Submitted or uncertain launches missing a matching-generation
|
||||
measurement still consume a full startup allowance until telemetry accounts for
|
||||
them. This lets collections start as capacity
|
||||
becomes available, without stacking every queued server's startup peak.
|
||||
Capacity errors report projected usage, usable capacity and reservations.
|
||||
|
||||
When startup capacity is available, provisioning checks each eligible host’s
|
||||
current CPU and available memory against reservations and headroom.
|
||||
Among hosts that fit, it chooses the lowest projected CPU/memory utilization,
|
||||
using the busier of the latest reading and the five-minute average for ranking.
|
||||
A historical resource dip cannot block a host whose current capacity has recovered.
|
||||
Historical ranking runs in a worker thread. The controller rechecks current
|
||||
capacity and reserves the selected host and port atomically, then records them
|
||||
in the launch recipe. Host network defaults
|
||||
are resolved again for the final host; explicit overrides are preserved. If no
|
||||
host qualifies, the request waits and is checked again. Once remote staging or
|
||||
creation has been submitted, placement stays fixed to prevent duplicate instances.
|
||||
Cancelling an unsubmitted launch releases its reservation; uncertain remote work
|
||||
retains capacity until cleanup is confirmed.
|
||||
|
||||
## Capacity overview
|
||||
|
||||
Open **Capacity** (`/capacity`) to see the same cached estimates used by collection
|
||||
previews. Available totals include hosts ready for the default runtime and latest
|
||||
files for the selected game. **All games** counts shared resources once, including
|
||||
hosts ready for either game; it is not a guarantee that every game fits every host.
|
||||
The host breakdown shows readiness, missing measurements, available ports, learned
|
||||
running/startup allowances, and the resource limiting additional servers.
|
||||
|
||||
Startup slots and queued jobs are fleet-wide, even with a host or game selected.
|
||||
Server estimates use running allowances; provisioning still checks the larger
|
||||
startup allowance before starting each job. Port availability uses each host's
|
||||
configured bind address. Queued unplaced jobs do not reserve physical resources.
|
||||
|
||||
The history graph offers the last hour, 24 hours, or seven days of measured CPU
|
||||
and memory headroom, after the usual OS reserve. It averages each host's reports
|
||||
within each period, then sums hosts for fleet views. Missing reports appear as gaps.
|
||||
Historical measurements cover the selected currently registered hosts and do not
|
||||
reconstruct past readiness, port reservations, or startup slots. Reports refresh
|
||||
automatically while the page is visible, and historical queries run in a separate
|
||||
reader worker. `GET /api/capacity` supports `range=1h|24h|7d`, `host=<host ID>`, and
|
||||
`game=all|t5|t6`; host/game selections also remain in the page URL.
|
||||
|
||||
## Identity, transport and credentials
|
||||
|
||||
Enrollment tokens are random, hashed at rest, single-use and expire after 15
|
||||
minutes. Consumption and credential issuance are transactional. Enrollment only
|
||||
works at the enrollment endpoint. Each host receives a distinct bearer credential
|
||||
and the controller only stores its hash. Host and enrollment bearer credentials
|
||||
are rejected outside `/api/agent/`.
|
||||
|
||||
The agent uses HTTPS, operating-system certificate validation, outbound heartbeats
|
||||
and polling; Docker has no exposed remote API. Agent endpoints reject HTTP unless
|
||||
`EC_AGENT_ALLOW_HTTP=true` is explicitly enabled for development. The agent's
|
||||
insecure development option is limited to loopback. Public release assets contain
|
||||
no credentials. The release checksum is authenticated by the controller's HTTPS
|
||||
origin; there is no separate offline release-signing infrastructure.
|
||||
|
||||
Browser sessions do not authorize agents. Similarly, host credentials never grant
|
||||
access to operator pages. Game telemetry uses a separate `EC_TELEMETRY_TOKEN`,
|
||||
configured in each reporter's authorization header; see the
|
||||
[telemetry contract](beta-test-plans.md#game-reporter-protocol).
|
||||
|
||||
Hosts offers credential rotation and revocation. Rotation persists the new secret
|
||||
on the agent before switching controller credentials, so a lost response can be
|
||||
retried using the same proposed secret. Revocation affects one host only and does
|
||||
not stop its running game containers. Re-enrollment requires a fresh token and an
|
||||
explicit installer `--reenroll` when local credentials already exist.
|
||||
|
||||
## Host and game-file API
|
||||
|
||||
- `GET /api/hosts/:id`: host details and deletion blockers.
|
||||
- `GET /api/hosts/operations?page=1&filter=important`: operation history across
|
||||
hosts, newest first, with 25 summaries per page and filtered total/page counts.
|
||||
Filters are `important` (default), `routine`, and `all`. Summaries include the
|
||||
server, safe target path or status command, and initiating source when recorded.
|
||||
Payloads and results are omitted.
|
||||
- `DELETE /api/hosts/:id`: administrator requests cleanup of an unused host;
|
||||
returns `202` while remote removal is pending, `200` for an unenrolled host,
|
||||
or `409` when the agent needs updating or unresolved ownership remains.
|
||||
- `POST /api/agent/enroll`: consume enrollment token and return host credentials.
|
||||
- `GET /api/agent/release/manifest.json`: public agent version, protocol and
|
||||
checksum plus `updaterProtocolVersion` and `updaterSha256` for installer bootstrap.
|
||||
- `GET /api/agent/release/ec-agent.py` and `ec-agent-update.py`: public release
|
||||
source, verified against that HTTPS manifest before installation.
|
||||
- `POST /api/agent/heartbeat`: authenticated inventory, current operation and caches.
|
||||
- `GET /api/agent/operations`: deliver/reconcile durable operation IDs.
|
||||
- `POST /api/agent/operations/:id`: authenticated acceptance, progress or result.
|
||||
- `POST /api/agent/decommission/:id`: cleanup worker reports file removal and
|
||||
agent uninstall using its own operation-scoped receipt credential.
|
||||
- `GET /api/game-files`: returns `{libraries, hosts}`. `libraries` always contains
|
||||
two game slots, T5 and T6, including empty slots. Each slot has a nullable
|
||||
`latestVersionId` identifying its latest verified upload, plus upload history.
|
||||
- `POST /api/game-files`: administrator creates a folder upload with
|
||||
`{game: "t5" | "t6", files: [{path, size, lastModified}]}`. Upload IDs and
|
||||
internal revision metadata are generated automatically; no library name,
|
||||
version label or description is required.
|
||||
- `GET /api/game-files/:id/upload`: durable per-file offsets and publication status.
|
||||
- `PUT /api/game-files/:id/upload/files/:path`: administrator uploads a raw chunk
|
||||
with `X-Upload-Offset`, `X-Upload-Length` and `X-Content-SHA256` headers.
|
||||
- `POST /api/game-files/:id/upload/complete`: administrator starts asynchronous
|
||||
verification/publication; poll the upload endpoint for completion.
|
||||
- `/api/agent/libraries/...`: authenticated game-file downloads. Legacy
|
||||
`/api/agent/imports/...` endpoints remain compatible with already queued imports.
|
||||
|
||||
The operation envelope is independent of polling and can be carried over a
|
||||
persistent transport later. No unrestricted shell operation is available.
|
||||
|
||||
## Safety and durable operations
|
||||
|
||||
Each remote request has an immutable ID and payload, plus an instance UUID and
|
||||
generation where applicable. Reusing a key with a different request is rejected.
|
||||
The controller records remote IDs before dispatch. The agent persists accepted,
|
||||
running, completed, failed and unknown states in SQLite. Lost acknowledgments
|
||||
return the recorded result; restarting the controller never creates a new
|
||||
mutation simply because the previous reply was lost.
|
||||
|
||||
The agent reconciles ambiguous results using owned Docker labels, generations and
|
||||
persisted execution checkpoints when those observations prove the outcome.
|
||||
Otherwise the result remains **Unknown** and holds dependent work. Running alone
|
||||
is insufficient evidence that a particular restart happened. EC neither assumes
|
||||
an unreachable server stopped nor launches a duplicate replacement elsewhere.
|
||||
|
||||
### Player checks and activation
|
||||
|
||||
Campaign bytes are staged and verified while players remain connected. Only
|
||||
Provisioning can dispatch activation. It checks source eligibility and shared
|
||||
capacity, and the agent repeats a fresh local RCON occupancy check immediately
|
||||
before activation/restart, stop, removal or rebuild. Busy or unverifiable lobbies
|
||||
wait indefinitely, including multi-hour Zombies sessions. Before images returned
|
||||
by the agent populate existing campaign/beta rollback history. A beta cannot be
|
||||
closed while its remote activation receipt is still unresolved.
|
||||
|
||||
An empty Fleet card does not replace this final host-local check: the card may
|
||||
use telemetry, and the controller's RCON parser is more permissive than the
|
||||
agent's disruption guard. Agent 1.4.1 includes the local RCON failure reason in
|
||||
the operation's hold message (configuration, timeout, unreachable endpoint,
|
||||
authentication, invalid packet, or an unrecognized player count). For a parser
|
||||
hold, inspect the raw `status` reply in **Diagnose connection**. Unknown counts
|
||||
remain held; RCON availability alone does not prove the lobby is empty.
|
||||
It also recognizes T6's `num score bot ping` header, which earlier agents
|
||||
rejected even when the complete player table was empty.
|
||||
|
||||
Rebuild uses a new generation, fresh instance storage and the latest promoted
|
||||
configuration/scripts/plugins. It retains the selected game-file version and
|
||||
runtime image. Old generation commands cannot target the replacement. Start,
|
||||
restart and create complete only after post-operation health evidence; creation
|
||||
and rebuild also verify the selected map and running Plutonium revision. The
|
||||
base-game library and runtime revision are independent records.
|
||||
|
||||
## Drain and enable hosts
|
||||
|
||||
**Drain host** asks for confirmation, blocks new deployments, and migrates
|
||||
instances to other ready hosts with capacity. Each migration waits for an empty
|
||||
lobby, reserves destination CPU, memory and ports, removes the source instance,
|
||||
then rebuilds the same server identity and map on the destination using its saved
|
||||
key and current promoted configuration/scripts/plugins. The address or port may
|
||||
change. Active beta ownership and other operations still gate migration, and the
|
||||
agent checks occupancy again before removal. Unavailable capacity leaves the
|
||||
source running; unknown removal results retain their reservation until the
|
||||
original operation reconciles. Migrations survive controller restarts.
|
||||
|
||||
Once all migrations pass health verification and the source is empty, it becomes
|
||||
**Disabled** automatically. There is no manual Disable action. **Enable** starts
|
||||
automatic setup while the host remains outside the capacity pool: the configured
|
||||
runtime is pulled again and both latest published BO1 (T5) and BO2 (T6) libraries
|
||||
are downloaded as needed and fully verified, including cached copies. Both games
|
||||
must be published. Failed setup or stale inventory keeps the host disabled; only
|
||||
successful preparation and fresh inventory return it to Active.
|
||||
|
||||
## Delete a host
|
||||
|
||||
Administrators can choose **Delete host** on a host card or at the top of its
|
||||
detail page. The confirmation lists anything that must be resolved first:
|
||||
registered servers, reserved ports, or operations already sent to the agent.
|
||||
Drain the host or remove its servers, then let outstanding operations reconcile.
|
||||
An unenrolled host is removed immediately. An enrolled host stays visible as
|
||||
**Deleting** until its agent confirms cleanup. Offline hosts wait for their next
|
||||
connection; their absence is never treated as proof of removal. Existing agents
|
||||
need the current installer: generate a fresh command under **Agent access**, rerun it
|
||||
on the host, and wait for a heartbeat advertising uninstall support.
|
||||
|
||||
Deletion blocks new operations, cancels queued commands that were never delivered,
|
||||
and invalidates unused installer links. A separate cleanup service removes the
|
||||
host's downloaded game libraries, instance files, agent journal and configuration,
|
||||
owned containers, and recorded runtime images unused by other containers. It
|
||||
stops and removes the agent service and executable, automatic update timer,
|
||||
update service, updater executable and update state, then removes its own files.
|
||||
The controller hides the host and revokes its ordinary credentials only after
|
||||
receiving the final uninstall receipt. Adding the machine again requires a new
|
||||
host enrollment.
|
||||
|
||||
Game folders uploaded through Electric Cherry belong to the controller. Those
|
||||
original uploads remain available for other hosts and future deployments; only
|
||||
the deleted host's copies are removed. Controller operation history is also kept.
|
||||
Unrelated containers, files and shared dependencies such as Docker and Python
|
||||
are preserved. Historical entries in shared system logs are not erased.
|
||||
|
||||
### Interrupted cleanup
|
||||
|
||||
Cleanup retries after connection loss or partial file removal. Failures before
|
||||
handoff offer **Retry cleanup** after the installation is repaired; failures in
|
||||
the independent worker retry automatically. The worker stays installed until
|
||||
the controller acknowledges data cleanup. After removing its final files it
|
||||
retries the completion receipt from memory. A power loss in that final window
|
||||
can leave the controller awaiting confirmation even though removal succeeded;
|
||||
inspect the machine if that status persists. The controller does not guess.
|
||||
|
||||
## Game files
|
||||
|
||||
### Upload a game folder
|
||||
|
||||
Game Files always shows T5 and T6, each with its own upload button and latest
|
||||
verified files. Successful uploads become latest in publication completion
|
||||
order, not by their start date or an entered label. A newer upload still being
|
||||
transferred or verified leaves the current files available. Previous uploads
|
||||
appear in history, and interrupted uploads can be resumed separately.
|
||||
|
||||
Choose the actual game folder, not its parent, a ZIP archive or the Plutonium
|
||||
launcher data folder. The selected folder's name is removed once; nested paths
|
||||
such as `zone/english/...` and `main/...` stay intact. Plutonium's
|
||||
[T5 server guide](https://plutonium.pw/docs/server/t5/setting-up-a-server/) and
|
||||
[T6 server guide](https://plutonium.pw/docs/server/t6/setting-up-a-server/) use a
|
||||
copy of the game directory, separately from the launcher/runtime. The uploader
|
||||
preserves the supplied files rather than attempting to infer installed maps from
|
||||
a manually maintained DLC list.
|
||||
|
||||
### Transfer and verification
|
||||
|
||||
Folder uploads use chunks of at most 8 MiB, each checked against its SHA-256 before
|
||||
being appended to temporary controller storage. Memory usage does not grow with
|
||||
the size of an individual game file. Pause/retry uses the controller's saved file
|
||||
offsets. After a page reload, choose **Resume upload** and reselect the original,
|
||||
unchanged folder; relative paths, sizes and modification times must match. Keep
|
||||
the page open while transferring. Once publication starts, the controller hashes
|
||||
every complete file and publishes the directory atomically; closing the browser
|
||||
does not stop that verification. An interrupted controller can retry publication.
|
||||
Verified uploads are immutable; changing files means uploading another folder
|
||||
for the same game. Hosts first download every file with up to four concurrent
|
||||
transfers and HTTP Range resumption. After all transfers finish, a separate
|
||||
verification pass reads every file from disk and checks its complete SHA-256,
|
||||
including any resumed prefix. Only then is the complete cache promoted atomically.
|
||||
Corrupt staged files are removed so the next attempt downloads them again.
|
||||
Separating the phases removes checksum work from the transfer loop and exposes
|
||||
disk-verification time separately; it does not guarantee a shorter total duration.
|
||||
|
||||
The Game Files and host setup pages show **Download** and **Verify** as separate
|
||||
steps, each with its own progress. Elapsed time records active work across
|
||||
resumptions and stops when work finishes or fails. Remaining time and speed use
|
||||
the current step's measured progress; a download estimate does not include the
|
||||
verification pass that follows. Estimates appear after enough progress is
|
||||
measured and are hidden for stale or offline hosts. Existing agents without
|
||||
timing reports still show byte progress. Agent 1.2.2 provides phase timing;
|
||||
automatic updates install it between host operations.
|
||||
Phase transitions are reported promptly and periodic heartbeats carry progress;
|
||||
controller round trips do not stall every transferred chunk.
|
||||
|
||||
### Cache cleanup and retention
|
||||
|
||||
Corrupt or incomplete content never becomes Ready. Cache inventory is reconciled
|
||||
on heartbeats, including vanished caches. The latest published version of each game
|
||||
is always retained. With agent 1.4.7 or newer, an older host copy is automatically
|
||||
eligible for removal once no live instance reservation on that host references it.
|
||||
The game-file minimum count, supersession grace period and explicit protections in
|
||||
Administration → Settings also apply. Stopped,
|
||||
provisioning, and rebuilding instances still pin their version. The agent also
|
||||
refuses deletion if any Docker container (including stopped containers) mounts
|
||||
those files. Cleanup includes partial downloads and retries after interruption.
|
||||
|
||||
The controller removes an older version's files after every host has reported its
|
||||
copy absent and no instance or unresolved operation references that version.
|
||||
Offline hosts keep their last reported copies pinned until they reconnect.
|
||||
Upload metadata, manifests, and operation history remain visible; removed files
|
||||
cannot be downloaded again without a new upload. Cleanup runs on heartbeats,
|
||||
successful publication, and the controller background worker, without needing
|
||||
an open Game Files page.
|
||||
|
||||
### Proxy limits and older uploads
|
||||
|
||||
The controller and reverse proxy must accept 8 MiB upload requests and folder
|
||||
manifests up to 32 MiB. Compose sets `BODY_SIZE_LIMIT=Infinity`; application routes
|
||||
still enforce these limits, declared file sizes and checksums. Set a proxy body
|
||||
limit of at least 32 MiB (for example `client_max_body_size 32m` in nginx).
|
||||
Existing uploads, host caches and server references remain usable. Previously
|
||||
named libraries are grouped under T5 or T6, and the most recently published
|
||||
upload for each game becomes its latest files. Their immutable IDs and stored
|
||||
content still in use are preserved; unused older files are cleaned up automatically.
|
||||
Legacy source-host imports retain their existing IDs and can complete through
|
||||
their original workflow.
|
||||
|
||||
## Ports and network bindings
|
||||
|
||||
Reservations include host ID, local bind address, protocol and port. Wildcard
|
||||
binds conflict with concrete addresses on the same host. UDP and TCP reservations
|
||||
for one launch commit together; the same number can be used independently on
|
||||
another host. Optional public/NAT address and port are recorded separately and do
|
||||
not change local Docker bindings. EC does not configure external NAT/firewalls.
|
||||
|
||||
## Provisioning progress and diagnostics
|
||||
|
||||
Native launches distinguish initial eligibility evaluation from an actual wait
|
||||
for startup capacity. The operation page records payload preparation, command
|
||||
submission, agent acknowledgement and execution, and separate container, game,
|
||||
map and Plutonium revision checks. Evidence remains available after completion.
|
||||
|
||||
The **Host agent execution** panel reads the durable command state directly,
|
||||
including its queue time and latest report time. Agent receipts are recorded in
|
||||
the provisioning history when received; they do not wait for a fleet scan.
|
||||
An agent command marked completed still requires EC readiness verification.
|
||||
The **Controller activity** panel shows current observations and their elapsed
|
||||
time while the operation is active. It is a live snapshot, not historical proof
|
||||
of which task delayed a previous deployment.
|
||||
|
||||
### Health checks
|
||||
|
||||
Fleet health checks query RCON even when container inspection is unavailable. A
|
||||
fresh successful RCON reply can establish game health while container state stays
|
||||
unknown; confirmed stopped, unhealthy, and restarting states still take precedence.
|
||||
Agent 1.4.4 retries transient failures of the read-only `status` query up to three
|
||||
times, with 250 ms and 500 ms delays. Commands that change the game are not retried.
|
||||
Status replies still expire normally, and native timeout diagnostics are retained.
|
||||
|
||||
### Background observations
|
||||
|
||||
Campaign, beta and maintenance observers run independently of provisioning.
|
||||
New requests and completed agent commands wake provisioning; an arrival during a
|
||||
pass schedules another pass. Routine file drift scans run every five minutes;
|
||||
manual checks bypass that interval. Agent 1.4.9 supports one hash manifest per
|
||||
drift scan, avoiding separate list/stat/read operations for unchanged files.
|
||||
Older agents use the existing file checks. Overlapping identical native
|
||||
observations share one operation until it completes. Routine status monitoring
|
||||
and maintenance additionally share successful health results for up to 30 seconds
|
||||
from the request start, preserving the original observation time. Changed server
|
||||
identity or lifecycle intent invalidates reuse. Manual checks and disruption
|
||||
safety checks continue to request fresh evidence.
|
||||
Status scans allow two checks per host, so a slow host cannot stall the fleet.
|
||||
Accepted or running health operations are withheld from the other health lane
|
||||
for 30 seconds; an expired replay lease allows recovery with the original ID.
|
||||
Agent 1.4.10 includes generation-tagged container states in its 15-second
|
||||
heartbeat, using the same Docker inspection as resource sampling (including
|
||||
stopped containers). Routine monitoring uses samples younger than 30 seconds,
|
||||
preserving their sample time and falling back to inspection when evidence is
|
||||
missing, stale, or predates lifecycle changes. RCON status queries share one
|
||||
operation per host, with two concurrent local probes and a 20-second batch
|
||||
deadline. Individual failures remain visible per server. Both additions are
|
||||
capability-gated; older agents retain individual inspection and RCON operations.
|
||||
Agent 1.4.11 caches parsed library manifests during inventory collection until
|
||||
their modification time, size, or file identity changes. Deleted manifests leave
|
||||
the cache, while transfer progress and partial-file sizes are sampled each time.
|
||||
Launch and download integrity verification continue to read from disk.
|
||||
Agent 1.4.5 reserves two observation workers for instance health and read-only RCON,
|
||||
and one for files, logs, and host inventory. Background reads arrive in batches of
|
||||
up to four. File receipt uploads release the instance lock before contacting the
|
||||
controller. Older agents retain the shared read lane until upgraded. Startup and
|
||||
disruption safety gates remain enforced at command admission.
|
||||
|
||||
Older completed operations whose progress snapshots were already discarded
|
||||
cannot recover missing check results from this change. Their surviving events
|
||||
and host command records remain available without inventing verification times.
|
||||
|
||||
### Read operations during downloads
|
||||
|
||||
Agent 1.4.0 separates read-only inspections, file/log reads, and the exact RCON
|
||||
commands `status`, `getstatus`, and `version` from the serial mutation queue.
|
||||
Two bounded read lanes can run during runtime or library downloads. Instance
|
||||
mutations exclude reads of that instance; credential rotation and host cleanup
|
||||
wait for active reads. Both lanes retain durable operation IDs, generation checks,
|
||||
and the updater activity lock. Older agents continue using the combined queue.
|
||||
|
||||
Agents long-poll `GET /api/agent/operations?lane=read|write&wait=20`; requests
|
||||
without those parameters retain immediate combined delivery. The controller wakes
|
||||
pending polls when work is queued and rechecks authentication after waiting.
|
||||
Long polls release the updater lock; execution reacquires it and rechecks update
|
||||
state before accepting work.
|
||||
|
||||
Browser game-folder uploads transfer up to three files concurrently. Chunks of
|
||||
one file remain serial and checksum-verified. Publication waits for admitted file
|
||||
transfers and prevents subsequent writes; pause/resume still uses acknowledged
|
||||
per-file offsets. Host downloads retain their existing bounded concurrency.
|
||||
|
||||
## Cancel an unfinished creation
|
||||
|
||||
On the native creation operation page, **Cancel creation and remove server**
|
||||
stops the launch and queues removal of its exact instance and generation. It
|
||||
also works when readiness checks are stuck or the creation failed. This explicit
|
||||
cancellation skips player-count checks on both the controller and agent, even
|
||||
if players have joined. Completed creations use the normal safe removal flow.
|
||||
|
||||
The cancelled operation links to a durable cleanup operation with agent receipts
|
||||
and progress. An unsubmitted launch releases its reservations immediately. Once
|
||||
an agent command was queued, ports and the instance identity remain reserved
|
||||
until the agent confirms container and instance-file removal. Shared game
|
||||
libraries, runtime images, saved server keys, and operation history are retained.
|
||||
Repeated cancellation requests return the same cleanup operation. A queued
|
||||
normal removal waiting for occupancy is superseded by this cleanup.
|
||||
|
||||
Remote cancellation requires agent 1.3.0 or later (`instance.cancel-create`
|
||||
capability). An older agent is never sent the unsupported cleanup command;
|
||||
the cleanup page explains that it is waiting for an update. Offline or disabled
|
||||
hosts retain the cleanup request until they reconnect or are enabled. Heartbeats
|
||||
notify an executing create to stop between Docker commands; an already executing
|
||||
Docker command must return or time out before cleanup can run. Cancellation
|
||||
does not require the original creation to acknowledge completion. The agent
|
||||
records a removal tombstone before cleanup so a delayed creation command cannot
|
||||
recreate the instance, and resumes interrupted cleanup under the same operation ID.
|
||||
|
||||
While cancellation requires a newer agent, the controller pauses all command
|
||||
delivery to that host, including background observations and command replays.
|
||||
Older agents hold their update lock across a whole command batch; continuing to
|
||||
feed them work could otherwise starve the updater. Already delivered work can
|
||||
finish and report normally, and heartbeats continue. The next automatic update
|
||||
check can install the release once the batch has drained. When a heartbeat
|
||||
advertises cancellation support, delivery resumes with cancellation cleanup
|
||||
first. This controller-side hold also recovers cancellations already waiting
|
||||
on older agents. Hosts without automatic updates need the current installer.
|
||||
|
||||
## Development and validation
|
||||
|
||||
```sh
|
||||
npm run agent:package
|
||||
npm run check
|
||||
npm test
|
||||
npm run test:agent
|
||||
npm run build
|
||||
```
|
||||
|
||||
The build packages the agent, standalone updater and installer into the static
|
||||
release, generates the agent/updater checksum manifest and serves those same
|
||||
assets in the Node-adapter image. Development serves the freshly packaged static
|
||||
release even when an older production build exists. The Python programs use only
|
||||
the standard library.
|
||||
|
||||
Automated tests cover enrollment/authentication, rotation/revocation, host
|
||||
readiness, scoped ports, immutable library publication, streaming/ranges,
|
||||
checksums, interrupted downloads, instance generation ownership, local occupancy
|
||||
holds, controller restart/reconnect, ambiguous results, occupied campaign staging,
|
||||
activation before images, collections and draining. Host deletion tests cover
|
||||
ownership guards, interrupted cleanup, uninstall receipts, credential revocation
|
||||
and retention of controller uploads/history. Docker/systemd/UDP and host
|
||||
observations are simulated in these tests.
|
||||
|
||||
A real Debian/Docker/Plutonium acceptance run is still required before production
|
||||
rollout. Automated simulations do not establish successful deployment on real
|
||||
game hosts. Run:
|
||||
|
||||
- Installer on Debian 12 and 13 amd64, then rerun without changing identity/data.
|
||||
- Bootstrap an existing host with the current installer, then publish a newer
|
||||
compatible agent release. Verify a timer-driven update, its authenticated
|
||||
heartbeat and unchanged game containers. Exercise an active-operation hold,
|
||||
bad checksum, failed startup/heartbeat rollback, controller downgrade, network
|
||||
loss and host reboot during update. Verify updater protocol mismatch requests
|
||||
an installer rerun.
|
||||
- T5 and T6 deployments and a mixed collection from a fresh destination host.
|
||||
- Start, stop, restart, rebuild, destroy; verify RCON, map and runtime revision.
|
||||
- Read console and game logs, including Plutainer's relative game-log links.
|
||||
- File campaign and rollback, with a player joining between controller check and
|
||||
agent activation, and during file activation before restart; verify that the
|
||||
agent holds disruption.
|
||||
- Drain occupied lobbies and verify shared libraries survive their retirement.
|
||||
- Delete a drained host; verify its game copies, data directory, agent and cleanup
|
||||
services, update timer, configuration and executables are removed, controller uploads/history stay,
|
||||
and unrelated Docker workloads continue. Repeat with connection loss during
|
||||
cleanup and after the final receipt, and a reboot during data removal.
|
||||
- Corrupt/interrupted library downloads, resumed transfer and verified promotion.
|
||||
- Controller restart, agent restart and network loss before execution, during
|
||||
execution and after execution but before acknowledgement. Verify the same
|
||||
operation IDs and absence of duplicate containers/restarts.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Plugin releases from GitHub
|
||||
|
||||
On **Plugins**, paste a public GitHub repository or release URL into **Import
|
||||
from GitHub**. Repository and `/releases/latest` URLs preview the latest stable
|
||||
release. Specific `/releases/tag/v3.1.0` links and
|
||||
`/releases#release-v3.1.0` links select that exact release, including prereleases.
|
||||
|
||||
Review the notes, select a DLL, give the plugin a name, and choose an optional
|
||||
T5/T6 game scope. Map scopes can be set on the plugin detail page. **Import only**
|
||||
saves the binary; **Import and test** opens the beta builder with that exact
|
||||
plugin version selected. Select servers and a test plan to start testing.
|
||||
Imports alone never change server files or promote versions.
|
||||
|
||||
Each import stores the publisher's tag, release notes, GitHub repository,
|
||||
release URL, release and asset IDs, asset filename, publication date and a
|
||||
SHA-256 checksum. If GitHub supplies a SHA-256 asset digest, the download must
|
||||
match it. The existing upload number still distinguishes different builds.
|
||||
Version history links back to each imported release.
|
||||
|
||||
## Discovering updates
|
||||
|
||||
The plugins page and each plugin's GitHub panel check for the latest stable
|
||||
release. Lookups are cached per repository for five minutes, including when
|
||||
using **Check for updates**. No background upgrades or automatic promotions
|
||||
occur. Failed checks are shown per plugin so one unavailable repository does
|
||||
not hide other updates.
|
||||
|
||||
**Review update** opens release notes and a DLL picker. The previously imported
|
||||
asset filename is selected when available. Multiple unfamiliar DLLs require a
|
||||
selection. Importing into an existing plugin keeps its scope and deployment
|
||||
filename. An active beta must be completed before another release is imported.
|
||||
|
||||
Update tracking follows the most recently imported version's repository and
|
||||
asset name. An imported release is distinguished from a promoted release;
|
||||
importing the same release asset and binary into that plugin again reuses the
|
||||
existing version. An older stable release is not offered as an upgrade over a
|
||||
newer imported prerelease. A plugin uploaded manually starts tracking once a
|
||||
GitHub release is imported from its detail page.
|
||||
|
||||
## Limits
|
||||
|
||||
- Public repositories on github.com only; no GitHub credentials are needed.
|
||||
- Direct `.dll` release assets up to 64 MB; archives and source-code downloads
|
||||
are not extracted. Other formats can still use the manual upload workflow.
|
||||
- Release tags have the same 80-character limit as manually entered versions.
|
||||
- GitHub rate limits, missing releases, download failures and checksum failures
|
||||
produce errors without saving a partial import.
|
||||
- Downloads follow only HTTPS redirects to GitHub's release asset hosts and
|
||||
have a size limit and timeout.
|
||||
|
||||
## API
|
||||
|
||||
- `GET /api/plugins/github/release?url=...` previews a release and its DLL assets.
|
||||
- `POST /api/plugins/github/import` accepts JSON `{ url, assetId, name,
|
||||
scopeType?, scopeValue? }` for a new plugin, or `{ url, assetId, pluginId }`
|
||||
for an existing plugin. Use the preview's exact release URL, not a mutable
|
||||
`latest` URL. Returns plugin/version IDs, checksum and the beta builder URL.
|
||||
- `GET /api/plugins/updates` checks all plugins; `?pluginId=123` checks one.
|
||||
|
||||
The server resolves release metadata again during import. Client-supplied notes,
|
||||
tags and download URLs are not used as provenance or download instructions.
|
||||
@@ -0,0 +1,230 @@
|
||||
# Provisioning
|
||||
|
||||
Open **Operations → Provisioning** for instance lifecycle work and persistent
|
||||
activity. **Fleet → Collections** owns reusable lineups and collection launch
|
||||
progress; their individual operations remain visible in Provisioning.
|
||||
Provisioning owns the queues, safety checks, execution, and
|
||||
recovery for all six actions. Create a server using **Create server**. For an existing
|
||||
server, choose **Manage server** or open its **Server actions** menu.
|
||||
|
||||
| Action | Behavior |
|
||||
| --- | --- |
|
||||
| Create | Create a new instance using the latest promoted files and verify container health, game status, the selected map, and the Plutonium revision. |
|
||||
| Start | Start the existing container and verify container health and RCON. |
|
||||
| Restart | Wait until empty, restart the existing container, and verify container health and RCON. |
|
||||
| Rebuild | Wait until empty, replace the instance through the host agent, preserving its server identity and port while advancing its generation. Verify container health, game status, map, and Plutonium revision. |
|
||||
| Stop | Wait until empty, stop the container, and verify that the host agent reports it stopped. Keep the instance for a later Start. |
|
||||
| Destroy | Wait until empty, remove the container and its files through the host agent, then retire the instance and registration. |
|
||||
|
||||
## Safety and verification
|
||||
|
||||
### Beta ownership and rebuilds
|
||||
|
||||
Servers linked to a running or ready-to-promote beta cannot be destroyed or
|
||||
rebuilt. Promote or roll back the beta first, including paused or failed test
|
||||
plans. Fleet removal and exact collection launches queue nothing if any server
|
||||
selected for removal belongs to an active beta. Provisioning checks again before
|
||||
teardown; already queued removals wait while their server belongs to an active
|
||||
beta. Opening a beta also requires cancelling or finishing any pending destruction
|
||||
or rebuild on its selected servers.
|
||||
|
||||
Rebuild requires confirmation because **no instance data is preserved**. Its fresh
|
||||
files come from the latest promoted setup when creation begins, including changes
|
||||
promoted while waiting. The independently saved key in **Keys** must exist before
|
||||
teardown. Blueprint definitions, promoted artifacts, saved keys, and historical
|
||||
activity belong to Electric Cherry, and are not instance files.
|
||||
|
||||
Rebuild reserves the original ID and port throughout teardown and creation, even
|
||||
when a lower lobby is available. It shares the existing creation queue and startup
|
||||
limit. The agent records each operation durably, and the controller reconciles its
|
||||
receipt after reconnecting or restarting. An interrupted fresh startup can resume
|
||||
health verification without repeating the operation. Inspect failures before
|
||||
making another request.
|
||||
|
||||
### Health and runtime checks
|
||||
|
||||
Create and Rebuild finish only after RCON reports a runtime revision at least as
|
||||
new as the current Plutonium production release. The worker checks the production
|
||||
manifest and records the runtime revision for fleet and server version status as
|
||||
part of provisioning. An unavailable release or unknown runtime revision keeps the
|
||||
job awaiting verification and holds its startup slot while the worker retries.
|
||||
A confirmed older runtime fails the operation with its expected and observed
|
||||
revisions; it does not trigger an automatic restart. Check Plutainer's startup
|
||||
updater configuration and server logs before retrying. Telemetry can verify game
|
||||
status and map, but cannot verify the Plutonium revision.
|
||||
|
||||
Start and Restart share the rolling restart limit with campaigns, betas, and
|
||||
maintenance. A successful agent receipt is followed by actual container and game-status checks. Failed or ambiguous restart requests retain their recovery
|
||||
slot and are not automatically replayed. Stop requires an explicit stopped container state; an
|
||||
unreachable host agent is not evidence that a stop succeeded.
|
||||
|
||||
### Player checks and administrator overrides
|
||||
|
||||
Disruptive operations require a fresh zero-player result from RCON or accepted
|
||||
telemetry, unless the host agent positively reports a stopped container. Fresh occupied
|
||||
telemetry vetoes a conflicting empty RCON reading. Reports must be received within
|
||||
90 seconds; recovery verification also excludes reports from before the submitted
|
||||
operation. Telemetry can verify the game's activity and selected map after the host agent
|
||||
reports healthy. Runtime actions wait for conflicting
|
||||
campaign, beta, maintenance, or provisioning work. Duplicate pending requests for
|
||||
the same action return the existing operation; a different action returns a
|
||||
conflict. Requests and events survive application restarts.
|
||||
|
||||
For a failed container stuck waiting for player status, an administrator can open
|
||||
its queued or waiting **Restart**, **Stop**, **Rebuild**, or **Destroy** operation
|
||||
and choose **Admin override…**. The confirmation modal explains the interruption
|
||||
and deletion risks. Confirming bypasses player-count and empty-server checks in
|
||||
both the controller and host agent for that operation only. The approval persists
|
||||
across restarts and records the administrator and time in the operation history.
|
||||
The host agent must support overrides (version 1.4.15 or newer); older agents show
|
||||
an update requirement. Submitted commands, host migrations, and completed operations
|
||||
cannot be overridden. Capacity, conflicting work, instance identity, file validation,
|
||||
and post-action health checks still apply.
|
||||
|
||||
## Campaigns, betas and maintenance
|
||||
|
||||
Campaigns, beta runs, and maintenance submit durable restart requests to
|
||||
Provisioning and observe the resulting operation by ID. Their pages link to that
|
||||
operation, and its history links back to the requesting work. Provisioning owns
|
||||
the empty-server check, conflict handling, restart capacity, remote command, and
|
||||
health verification. The callers retain file deployment, beta evidence, and
|
||||
maintenance outcome verification. Jobs remain visible while waiting for players,
|
||||
other work, or integration configuration.
|
||||
|
||||
[Maintenance](maintenance.md) also submits recovery operations and links each attempt
|
||||
to its incident. A failed Maintenance restart can hand its reserved slot to a rebuild
|
||||
after five minutes; unrelated recovery slots remain held. Drift campaigns follow
|
||||
their linked rebuild without requesting another restart. Their file evidence and
|
||||
history remain available after the instance is removed.
|
||||
|
||||
Each campaign activation has its own durable operation and file selection. A failed or interrupted request keeps
|
||||
its original ID and never automatically replays. An explicit beta retry starts a
|
||||
new attempt, preserving prior history; paused or closed beta work cannot start a
|
||||
queued restart. Recovery of an already submitted restart continues even if its
|
||||
requester closes. Upgrades adopt existing restart intent for observation only.
|
||||
|
||||
## Operation details
|
||||
|
||||
Each job page puts its recorded outcome, blocking condition, and available next
|
||||
action before launch files. Execution is grouped into phases, with separate
|
||||
milestones for remote deletion, local cleanup, and replacement creation during a
|
||||
rebuild. Copy the operation ID or link when investigating or handing off a job.
|
||||
Related work links back to the requesting campaign, beta, maintenance item, or
|
||||
collection. A completed provisioning job does not imply that its requester has
|
||||
finished its own verification.
|
||||
|
||||
### Verification evidence
|
||||
|
||||
New operations record structured verification evidence alongside their activity:
|
||||
the check result, source, evaluation time, and expected/observed map or Plutonium
|
||||
revision when relevant. Create and Rebuild show **Verify Plutonium revision**, with
|
||||
separate evidence for the production manifest and the running revision from RCON.
|
||||
Game status can be verified by RCON or accepted fresh telemetry. Engine-map checks
|
||||
do not independently verify a T6 start location. Passed checks describe their
|
||||
recorded observation; later checks can fail or become unknown. Completion describes
|
||||
what was verified at that time, while **View server** opens the current instance.
|
||||
Older history retains its recorded stages and outcomes; unavailable evidence is
|
||||
not inferred from stage wording.
|
||||
|
||||
Submission attempted, command accepted, and verified completion are separate
|
||||
milestones. A failed restart may still have ongoing recovery observation and hold
|
||||
restart capacity. Its original failure remains in history even if recovery later
|
||||
succeeds and releases the slot. Capacity and request order are snapshots, not
|
||||
guarantees of the next job admitted or an estimated completion time. Blocked
|
||||
requests can be bypassed by eligible work.
|
||||
|
||||
### Recovery actions and launch files
|
||||
|
||||
Recovery controls use the server's current eligibility checks. **Resume health
|
||||
verification** observes the existing instance without submitting another create or
|
||||
restart. **Resume cleanup** retries local cleanup after confirmed remote deletion,
|
||||
including eligible historical deleted identities; replacement or newer-operation
|
||||
guards still apply. **Resume rebuild cleanup** completes cleanup before creating
|
||||
the replacement in its reserved slot. Resuming the same operation preserves its
|
||||
earlier failure and resume events. Uncertain commands have no generic Retry action.
|
||||
|
||||
Create and rebuild capture a setup when queued and select the latest promoted
|
||||
files again when creation begins. The page distinguishes **Setup captured when
|
||||
queued** from **Files selected for this attempt**. A queued capture does not pin
|
||||
versions; an attempted selection is not proof that every file was installed.
|
||||
**Launch another server** uses the latest promoted setup with a fresh identity and
|
||||
port, rather than repairing or replaying the recorded job.
|
||||
|
||||
### History and refresh behavior
|
||||
|
||||
Five recent events remain visible. Expand full history to search, change order,
|
||||
and load older pages. A fixed history watermark keeps the viewed pages stable as
|
||||
events arrive; **Jump to latest** incorporates new activity. The detail view
|
||||
fetches a bounded recent-event window, while full history loads on demand.
|
||||
|
||||
The page refreshes every five seconds while visible, including terminal jobs whose
|
||||
recovery or available actions may still change. **Page updated** describes the
|
||||
last successful detail fetch; it is separate from the worker's evaluation and
|
||||
telemetry receipt times. A refresh failure keeps the last snapshot visible and
|
||||
does not erase an acknowledged action result. These timestamps are not worker
|
||||
heartbeat or completion-time promises.
|
||||
|
||||
## Routes and compatibility
|
||||
|
||||
- `/provisioning`: active operations and history filtered by action.
|
||||
- `/provisioning/:id`: operation stages, errors, timestamps, and activity.
|
||||
- `/collections`: saved lineups, active launches, and launch history.
|
||||
- `/collections/launches/:id`: collection progress, with links to each operation.
|
||||
- `GET /api/provisioning`: lifecycle history (`deployments` response property).
|
||||
- `POST /api/provisioning`: queue `{ "server": "t5_zombie_theater_1", "action": "start" }`.
|
||||
Supported actions are `start`, `restart`, `rebuild`, and `stop`. Returns HTTP 202
|
||||
with `deploymentId`. Create continues through `POST /api/servers`; Destroy uses
|
||||
`DELETE /api/servers/:name`.
|
||||
- `/api/provisioning/:id`, `/:id/resume`, `/:id/cancel`, `/preview`, and
|
||||
`/house-party` preserve the existing operation, recovery, removal cancellation,
|
||||
launch preview, and fleet destruction APIs.
|
||||
- `POST /api/provisioning/:id/cancel` cancels any active create, start, restart,
|
||||
rebuild, stop, or destroy job, including health verification. It releases that
|
||||
job's recovery capacity and prevents further steps; completed effects are not
|
||||
undone. Undelivered host commands are withdrawn. Commands already delivered
|
||||
retain their receipts, and replacements wait for those mutations to settle.
|
||||
Cancelling is idempotent. Lists and details expose `canCancel`.
|
||||
- `POST /api/provisioning/:id/cancel-creation` retains the explicit native creation
|
||||
cleanup action: cancel the launch and remove its partial instance. Cancelling
|
||||
a job alone retains its existing instance/files for inspection or later cleanup.
|
||||
- Beta rollback cancels conflicting provisioning jobs on its original servers
|
||||
before queuing independent restoration restarts. It does not wait for the
|
||||
superseded restart's health verification. Native activation receipts must still
|
||||
provide durable before images; rollback reconciles completed receipts itself.
|
||||
Manual rollback keeps the replacement jobs held until the campaign is resumed.
|
||||
- `GET /api/provisioning/:id?events=recent` returns the same detail with five recent
|
||||
events plus activity totals/latest ID and a failure excerpt. Omitting the option
|
||||
preserves the full legacy event response. Progress, capacity, named related work,
|
||||
and normalized warning messages are additive fields; the original warning string
|
||||
and lifecycle fields remain available.
|
||||
- `GET /api/provisioning/:id/events` reads history with `order=newest|oldest`,
|
||||
literal-text `query`, `limit` (default 25, maximum 100), an exclusive event-ID
|
||||
`cursor`, and an upper-ID `watermark`. Keep the returned watermark when loading
|
||||
more and use `nextCursor` until it is null. Begin without a watermark to include
|
||||
newly arrived events. These reads do not contact host agents or probe RCON.
|
||||
|
||||
Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving
|
||||
query parameters. Collection launch links at `/provisioning/collections/:id` or
|
||||
`/deployments/collections/:id` redirect to `/collections/launches/:id` and preserve
|
||||
query parameters. `/api/deployments` remains compatible, including old campaign
|
||||
detail endpoints. No history or IDs are renamed or discarded.
|
||||
|
||||
Start and Stop use durable `instance.start` and `instance.stop` host-agent
|
||||
operations. Rebuild uses `instance.rebuild` and replaces the instance files.
|
||||
See [native host management](native-hosts.md) for command delivery and recovery.
|
||||
|
||||
## Concurrency limits
|
||||
|
||||
Administration → Settings separates Concurrency and Retention, with independent
|
||||
save actions. Existing fleet restart, startup and destruction limits continue to
|
||||
apply. Additional per-host restart, startup and destruction limits, a combined
|
||||
lifecycle limit, a rebuild limit, and a limit on hosts preparing game files default
|
||||
to 0 (no additional cap), preserving existing throughput on upgrade.
|
||||
|
||||
Lifecycle work counts once toward the combined limit, including launch preparation
|
||||
and unresolved delivered commands. Rebuilds include drain migrations and retain
|
||||
capacity between source removal and destination creation. Per-host recovery slots
|
||||
remain occupied until health verification. Lowering limits does not cancel work.
|
||||
Game-file preparation is limited when commands are delivered; downloads and imports
|
||||
share the cap, including manual requests. Replayed or interrupted commands retain
|
||||
the same host slot. Runtime image pulls and per-host file transfer counts are unchanged.
|
||||
@@ -0,0 +1,137 @@
|
||||
# Fleet settings
|
||||
|
||||
Administration → Settings has independent **Concurrency** and **Retention**
|
||||
sections. Settings persist in SQLite, apply without restarting the controller,
|
||||
and are writable only by administrators. Saving one section preserves the other
|
||||
section's saved values. Upgrades preserve existing values and add new defaults.
|
||||
|
||||
## Concurrency
|
||||
|
||||
The existing fleet restart, startup and destruction limits default to 1. Additional
|
||||
limits default to 0 (no additional cap):
|
||||
|
||||
- Restarts per host, including starts and rebuild recovery.
|
||||
- Startups per host, including creation and rebuilds.
|
||||
- Destructions per host, including migration source removal.
|
||||
- Combined lifecycle work across create/start/stop/restart/rebuild/destroy.
|
||||
- Rebuilds across the fleet, including host drain migrations.
|
||||
- Hosts downloading or verifying game files simultaneously, including manual imports.
|
||||
|
||||
Limits are checked again before command submission after asynchronous preflight.
|
||||
Launch-file preparation reserves capacity. A lifecycle workflow counts once in the
|
||||
combined pool; unresolved commands and recovery keep capacity occupied. Migrations
|
||||
keep rebuild capacity through source removal and destination creation. Lower limits
|
||||
do not cancel admitted work. Game-file preparation slots are assigned on agent
|
||||
command delivery and survive interrupted transfers. Runtime pulls, the four file
|
||||
transfers per host and the two health probes per host remain unchanged.
|
||||
|
||||
## Retention
|
||||
|
||||
For age settings, 0 means no expiry. An explicit version-count policy can still
|
||||
expire unused artifact versions when their age policy is disabled. Game-file grace
|
||||
is a delay: 0 adds no delay. A positive age and version count together preserve the
|
||||
minimum count as well as versions younger than the age threshold.
|
||||
|
||||
| History or storage | Default |
|
||||
| --- | --- |
|
||||
| Campaigns | 7 days |
|
||||
| Successful routine operations | 1 day |
|
||||
| Provisioning workflows | Indefinite |
|
||||
| Agent actions | Indefinite |
|
||||
| Explicitly resolved failures | Indefinite |
|
||||
| Non-update maintenance jobs | 30 days |
|
||||
| Resolved recovery incidents | Indefinite |
|
||||
| Host resource history | 7 days |
|
||||
| Instance resource samples | 1 day |
|
||||
| Fleet occupancy metrics | 31 days |
|
||||
| Telemetry diagnostics | 7 days; 200 per server; 10,000 overall |
|
||||
| Completed/interrupted matches | Indefinite |
|
||||
| Closed betas and test evidence | Indefinite |
|
||||
| Campaign file bodies and closed beta snapshots | 30 days |
|
||||
| Completed launch and command file bodies | 30 days |
|
||||
| Unused artifact versions | No automatic expiry; no minimum count |
|
||||
| Unused game files | No grace period; keep latest 1 version per game |
|
||||
| Incomplete browser uploads | Indefinite |
|
||||
| Authentication audit events | Indefinite |
|
||||
| Minimum history count | 0 additional records |
|
||||
|
||||
Telemetry age and count policies are independent; 0 disables each individual cap.
|
||||
Diagnostic responses remain bounded to 10,000 rows when the per-server retention
|
||||
cap is disabled. Metric and capacity chart time ranges are unchanged; increasing
|
||||
storage retention does not automatically add new chart ranges. Capacity learning
|
||||
continues to use its 24-hour window independently of longer sample retention.
|
||||
|
||||
### Cleanup and dependencies
|
||||
|
||||
Background cleanup runs in bounded batches. New age policies for previously
|
||||
permanent history default to indefinite. Pending work, unresolved failures,
|
||||
recovery dependencies and explicitly protected records are never removed merely
|
||||
because they are old. Referenced artifact versions are retained, including versions
|
||||
referenced by historical campaigns and betas. Collection workflows with unfinished
|
||||
members, cancellation links and recovery slots can keep provisioning history longer
|
||||
than its configured age.
|
||||
|
||||
Update attempts and dismissed watchdog incidents remain durable suppression evidence.
|
||||
Host setup, credential/decommission operations and referenced agent commands are
|
||||
protected. Expired unreferenced action history drops payloads, results and events
|
||||
but keeps compact terminal receipts and request fingerprints. Reusing an idempotency
|
||||
key cannot execute an expired command again. Agent mutation journals remain intact;
|
||||
this policy controls controller history, not the agent's execution recovery journal.
|
||||
|
||||
### File bodies and match history
|
||||
|
||||
Expiring campaign file bodies preserves file paths, hashes and summary records but
|
||||
makes file comparisons and rollback unavailable. The detail page identifies this
|
||||
state and rollback requests fail explicitly. Active beta rollback images stay
|
||||
protected. Expiring matches removes their player participation rows and changes
|
||||
leaderboard/player totals; lifetime aggregates are not retained separately.
|
||||
|
||||
Completed launch recipes and command receipts also retain their metadata when
|
||||
their file bodies expire. Unfinished workflows, unreconciled commands, unresolved
|
||||
failures and explicit holds preserve the bytes they need. Shared file bytes are
|
||||
deleted only after their final reference expires; library versions can keep a file
|
||||
available longer. Previously saved retention choices are preserved on upgrade.
|
||||
|
||||
### Database cleanup
|
||||
|
||||
The database upgrade normalizes retention clocks to UTC and enables incremental
|
||||
vacuum. An existing database without a vacuum page map is rebuilt once at startup;
|
||||
allow time for this rebuild and free disk space up to twice the database size.
|
||||
The rebuild uses disk-backed temporary storage and a 16 MiB page cache instead
|
||||
of holding a temporary copy of the database in RAM. Container scratch files use
|
||||
`/data/.sqlite-tmp`; `[database]` startup logs identify the current migration.
|
||||
Normal cleanup releases unused file blobs in bounded batches and incrementally
|
||||
returns free pages to disk after larger prunes.
|
||||
|
||||
### Game files and incomplete uploads
|
||||
|
||||
Game-file minimums apply per game. Grace starts when a version is superseded.
|
||||
Live, stopped, rebuilding and provisioning instances pin their files; unresolved
|
||||
operations and offline host copies also prevent controller deletion. Controller
|
||||
cleanup still waits for every host to report its copy absent. Incomplete browser
|
||||
uploads expire based on their last activity, including resume checks; active chunk
|
||||
writes and publication hold locks against cleanup. Metadata remains after files
|
||||
expire. Raising retention cannot restore previously deleted bytes.
|
||||
|
||||
### Preview and record protection
|
||||
|
||||
Expand **Cleanup preview and protected records** to preview the values currently
|
||||
entered without saving them. Preview lists eligible records and known payload
|
||||
sizes; SQLite file space may be reused rather than immediately returned to the OS.
|
||||
Policies can overlap, so payload sizes should not be added together. Unknown row,
|
||||
index, upload and agent-disk overhead is not included.
|
||||
|
||||
Use a record's ID from its detail URL or the preview to protect it. Protection can
|
||||
be removed in the same panel. Protecting a campaign or beta also protects its file
|
||||
bodies; protecting those bodies prevents deletion of their parent history.
|
||||
Game-file and incomplete-upload protections apply to storage cleanup.
|
||||
|
||||
The minimum history count preserves recent records per history category. It does
|
||||
not apply to telemetry caps, metric/resource samples or game-file storage, which
|
||||
have their own policies. Explicit holds protect individual history/storage records,
|
||||
not individual resource samples or telemetry diagnostics.
|
||||
|
||||
Failed campaigns, provisioning workflows and agent commands require explicit
|
||||
resolution before the resolved-failure period applies. Resolution does not bypass
|
||||
recovery/dependency checks. Resolving a failed campaign stops automatic campaign
|
||||
retry requests; it does not cancel commands already submitted.
|
||||
Reference in New Issue
Block a user