Clean up README and organize documentation

docs/research/leaderboard-query-benchmark.md

@@ -0,0 +1,46 @@
+# Leaderboard query benchmark
+
+The benchmark and measurements below describe the earlier SQL optimization,
+before the worker and revision caches were added. They are historical results,
+not measurements of the current HTTP implementation.
+
+Run the deterministic, isolated benchmark with:
+
+```sh
+node scripts/benchmark-leaderboard.mjs --scenario all --baseline-ref eac9ad4 --runs 3 --json /tmp/leaderboard-benchmark.json
+```
+
+The script creates temporary SQLite databases, loads the actual query modules,
+checks complete-response parity against the selected Git revision, and reports
+timings, source hashes and query plans. It never opens the application database.
+The small fixture has 2,000 matches and 8,000 participation rows; the large fixture
+has 50,000 matches and 200,000 participation rows. Both include completed,
+active, interrupted and nonqualifying records.
+
+On the local Intel i5-8279U / Node 25.9.0 / SQLite 3.49.2 test environment, three
+measured runs after warmup gave these median query times on the large fixture:
+
+| Request | Before | After |
+| --- | ---: | ---: |
+| Highest round, all games | 1,717 ms | 829 ms |
+| Total kills, all games | 1,555 ms | 858 ms |
+| One game | 899 ms | 491 ms |
+| One map | 345 ms | 224 ms |
+| Player search | 1,515 ms | 757 ms |
+| Deep page | 1,421 ms | 752 ms |
+| Headshot percentage | 1,444 ms | 760 ms |
+| Exact player rank | 1,524 ms | 554 ms |
+
+The small fixture's unfiltered request fell from 41 to 30 ms. All 150 additional
+small-fixture parity cases and 22 timed small/large request cases matched the
+baseline's complete responses, including ratios and record evidence. Timings
+include SQL preparation and evidence lookup, exclude HTTP/browser rendering, and
+use a warm filesystem cache with alternating implementation order. These are
+local synthetic measurements, not production latency guarantees. SQL queries
+run synchronously within the report worker; larger histories still affect
+report latency and worker capacity.
+
+Validation for that optimization passed 1,019 unit tests, Svelte checking with no errors
+or warnings, and a production build. Isolated Firefox walkthroughs at 1440px and
+390px checked completed-only totals, record links, efficiency qualification,
+scoped ranks/history, preserved navigation, empty scopes and mobile expansion.

docs/research/t5-t6-idle-cpu-research.md

@@ -0,0 +1,174 @@
+# Reducing idle CPU use in T5 and T6 Zombies servers
+
+Research date: September 19, 2026. Scope: Plutonium T5 Zombies (`t5sp`) and T6 Zombies (`t6zm`), with particular attention to Electric Cherry's Plutainer/Docker host deployment.
+
+This investigation predates the implemented lobby autoscaling feature. Repository
+observations and proposed changes below describe that research baseline. For
+current behavior, see [Lobby autoscaling](../guides/autoscaling.md) and
+[Native hosts](../guides/native-hosts.md); the
+[autoscaling plan](../plans/lobby-autoscaling-plan.md) tracks the remaining pilot.
+
+## Recommendation and scope
+
+**Recommendation:** first benchmark Wine's NTSYNC support and audit the effective runtime configuration. If every lobby must remain immediately available, trial CPU limits that apply only while a server is confirmed empty. If spare lobbies can be unavailable temporarily, keep a smaller pool running and start additional servers when demand rises. Stopping unused instances offers the most predictable reduction in their CPU consumption.
+
+I found no documented, supported switch that makes both T5 and T6 dedicated servers automatically hibernate while preserving normal browser visibility and joins. There are useful alternatives, but they solve different parts of the problem.
+
+This is research and a proposed evaluation plan. No production measurements were available, and no application code, server configuration, or running servers were changed. Reported savings below are either explicitly identified operator anecdotes or illustrative calculations.
+
+## Repository baseline
+
+The repository launches one Plutainer container per lobby, selects `t5sp` or `t6zm`, and defaults new deployments to `ghcr.io/ayymoss/plutainer:latest`. Its creation specification does not explicitly configure CPU quotas or device passthrough. Electric Cherry refreshes server status every 30 seconds, tracks occupancy transitions, and checks container readiness before RCON. These are useful foundations for an idle policy, but they are not an implemented CPU governor. See [deployment creation](../../src/lib/server/fleet/create.ts), [configuration](../../src/lib/server/config.ts), [status worker](../../src/lib/server/pipeline/worker.ts), and [status timing](../../src/lib/server-freshness.ts).
+
+The live host kernel, Wine version, image digest, installed scripts/plugins, CPU model, and actual idle utilization remain unknown. A repository default of `latest` does not establish which image a running server uses.
+
+**Why an empty server can still be expensive.** Zero players does not establish that the engine, map scripts, plugin timers, Wine synchronization, or container helpers have stopped working. Those are competing explanations to measure. A map can also behave differently after its final player disconnects than it does immediately after startup. The research supports investigating these layers; it does not establish which one dominates this fleet.
+
+## Options
+
+| Option | Expected effect on idle CPU | Availability and gameplay tradeoff | Evidence and priority |
+| --- | --- | --- | --- |
+| 1. Enable and verify NTSYNC | Potentially substantial if Wine synchronization dominates | Intended to retain normal joins and gameplay; compatibility testing required | Official Wine/kernel support; Plutonium operator anecdote. First trial |
+| 2. Audit busy-wait settings, scripts, and helpers | Small to substantial, depending on the actual cause | Usually compatible with keeping lobbies online | Documented settings plus diagnostic experiments. First trial |
+| 3. Apply CPU quotas only when empty | Enforceable ceiling on each empty container's CPU time | Excessive throttling can delay joins, RCON, and heartbeats | Docker-supported mechanism; automatic policy needs development |
+| 4. Run fewer spare lobbies | Removes the stopped instances' process CPU use | Fewer immediately available lobbies | Predictable mechanism; best larger saving if acceptable |
+| 5. Start on request or on a schedule | Removes process CPU use during stopped periods | Players wait for startup; stopped servers cannot provide live listings | Practical alternative for infrequent maps or private sessions |
+| 6. Pause and resume containers | Suspends their process execution while retaining memory | Server cannot answer while paused; resume behavior needs validation | Docker supports pause; Plutonium behavior unproven |
+| 7. Wake through a UDP gateway | Could combine stopped backends with automated wakeup | Protocol, listing, session, and timeout problems | Experimental; substantially harder for T5 |
+| 8. Change the host/runtime or develop engine idling | Potentially substantial, cause dependent | Migration cost or ongoing engine compatibility work | Conditional alternatives after profiling |
+
+### 1. Test NTSYNC before treating Wine overhead as unavoidable
+
+NTSYNC moves Windows synchronization primitives into a Linux kernel driver. Wine 11.0 documents using it when available and identifies Linux 6.14 as the kernel release providing the required module. The kernel documentation explains that the driver exists because equivalent userspace synchronization has performance limitations and exposes `/dev/ntsync`. These are established mechanisms, not Plutonium benchmark results. [Wine 11.0 release notes](https://github.com/wine-mirror/wine/blob/wine-11.0/ANNOUNCE.md), [Linux NTSYNC documentation](https://docs.kernel.org/userspace-api/ntsync.html).
+
+A particularly relevant firsthand report, posted September 9, 2026, describes running Plutonium servers on Linux with NTSYNC and seeing idle usage around 25% of one CPU core. The same operator reports limiting empty servers to 15% of a core and removing that limit when players arrive. The post does not provide a controlled before/after benchmark or establish separate T5 and T6 results, so neither figure should become our target without measurement. The forum was available through search-indexed content; direct retrieval was unreliable during this research. [Operator report on NFOservers](https://forums.nfoservers.com/viewtopic.php?p=85675).
+
+For this fleet, the experiment requires all of the following:
+
+- A host kernel with the complete NTSYNC implementation enabled and loaded; check the distribution build or backport, not just a version number.
+- A compatible Wine build inside the actual game container.
+- Access to `/dev/ntsync` inside that container, including permissions for the user running Wine. Docker supports targeted device mapping; whole-container privileged mode is unnecessary for this purpose. [Docker device access](https://docs.docker.com/engine/containers/run/#runtime-privilege-and-linux-capabilities).
+- Verification that the Wine runtime actually uses the device, through runtime diagnostics or process/device inspection, followed by measurements on both games.
+
+Start with one isolated T5 instance and one T6 instance, retaining the existing image and Wine prefix for rollback. A Wine upgrade may have prefix compatibility consequences; test on copies. Adding an environment variable alone does not establish working NTSYNC. If it is already active, enabling it again will not explain or fix remaining CPU use.
+
+### 2. Audit the work being done while empty
+
+This is the fastest way to distinguish an inherent runtime cost from an avoidable loop or configuration override.
+
+First inspect `com_busyWait` in the actual dedicated process. Plutonium's r5162 changelog introduced it in connection with high client frame rates; r5304 changed its default to false for T4/T5/T6. Checking for an old override that enables it is worthwhile. Its effect on dedicated-server CPU is unconfirmed: verify that the setting exists, takes effect, and changes measured behavior before calling it a fix. [Plutonium changelog](https://plutonium.pw/docs/changelog/#r5304).
+
+Next compare an isolated server with the normal scripts/plugins against the same map with only the minimum required configuration. If CPU falls materially, restore additions in groups to locate the cause. Candidates include frequent entity scans, polling with very short waits, repeated HTTP requests, excessive logging, and timers that continue after the last player leaves. These are diagnostic hypotheses, not findings about this project's deployed scripts. Event-based work or longer waits can help a particular script; they do not put the whole engine to sleep.
+
+Check that custom scripts preserve the normal wait for players before starting gameplay. T6's reference configuration documents `zombies_minplayers` with a default of one. This is a match-start gate, with no documented promise of engine hibernation or lower CPU. Do not raise it arbitrarily and prevent solo players from starting. [T6 reference configuration](https://github.com/xerxes-at/T6ServerConfigs/blob/master/localappdata/Plutonium/storage/t6/dedicated_zm.cfg).
+
+Also identify which processes consume CPU: the game executable, `wineserver`, log helpers, or another service. Plutainer documents a log symlink watcher with a default two-second polling interval and allows its interval to be changed or the watcher disabled. Testing a longer interval is reasonable if that helper is measurable across many containers; expect delayed discovery of new logs, and retain needed logging. The wrapper also has startup update controls, which matter to repeatable startup tests. [Plutainer configuration](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md).
+
+Electric Cherry's 30-second status polling is a separate source of work. Measure monitoring overhead before changing it: a long polling interval can make idle-to-active transitions harder to detect promptly. Do not infer that a high game-process CPU reading is caused by the management UI.
+
+### 3. Limit CPU while empty, and restore the normal budget as players arrive
+
+Docker can change a running Linux container's CPU limits without restarting it. A small controller could apply an idle budget only after repeated fresh zero-player readings and restore the original active budget when activity begins. [Docker container update](https://docs.docker.com/reference/cli/docker/container/update/).
+
+For example, `--cpus=0.25` means an aggregate allowance equivalent to one-quarter of one logical CPU, not 25% of the entire host. This is an illustration, not a recommended production value. A quota bounds execution time; it does not remove the work the engine wants to perform. A cap below the current demand may therefore reduce the utilization graph while worsening responsiveness. [Docker CPU constraints](https://docs.docker.com/engine/containers/resource_constraints/#cpu).
+
+The policy should handle several cases explicitly:
+
+- Require fresh, repeated empty readings and an idle grace period. Include spectators and connecting players where those states are observable. A timeout or stale zero-player snapshot is unknown occupancy.
+- Give startup, map changes, and recovery the normal CPU budget.
+- Restore that budget on a connection attempt if a reliable signal exists. Otherwise, use a conservative idle limit that still supports joining and a detection interval demonstrated to be fast enough.
+- Restore the active budget when occupancy becomes unknown. A separate watchdog or expiring idle-policy lease should restore it if the controller stops running.
+- Coordinate quota ownership with the host agent so container recreation does not silently lose the policy or reinstate an obsolete limit.
+
+Electric Cherry's current 30-second status cycle can leave a newly joined player under the idle cap until the next successful poll. An aggressive cap should not depend on that cycle alone. Quotas can also allow brief bursts followed by throttling within their scheduling period, so evaluate join time and responsiveness, not just average CPU. The kernel exposes `nr_throttled` and `throttled_usec` in cgroup v2 `cpu.stat` to help identify this behavior. [Linux CPU controller](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu).
+
+This is the strongest practical candidate when retaining every online lobby is a firm requirement. The ceiling is dependable; acceptable Plutonium behavior at a particular ceiling must be demonstrated.
+
+### 4. Keep a smaller pool of spare lobbies running
+
+If many maps have several empty copies, running all copies continuously multiplies the baseline cost. Keep occupied lobbies plus a chosen number of ready empty lobbies per exact map/ruleset, and stop surplus instances after an idle grace period. Start another spare when demand consumes the existing one; scheduled prewarming can cover known busy periods.
+
+This is a proposed capacity policy. Docker stop ends container processes; the resulting elimination of their execution is predictable. Its shutdown behavior depends on the configured stop signal. The upstream Plutainer Dockerfile retrieved during this research specifies `STOPSIGNAL SIGKILL`, so do not assume a graceful shutdown: verify the deployed image and check that logs, stats, and other persistent state finish writing before stopping. [Docker stop behavior](https://docs.docker.com/reference/cli/docker/container/stop/), [Plutainer Dockerfile](https://github.com/Ayymoss/Plutainer/blob/main/Dockerfile).
+
+Use stop/start semantics that retain the deployment, files, port reservation, and server key. Electric Cherry's destruction workflow removes infrastructure and files and is not a standby mechanism. Collection blueprints currently create shortfalls without removing excess servers, so they do not implement this policy either. [Collections](../guides/collections.md).
+
+Stopping and pausing need a final occupancy check, protection against a concurrent join, and coordination with deployments, recovery, and monitoring. A zero-player RCON reply alone cannot make a later stop atomic with joining; use a supported admission/drain mechanism if available, or explicitly accept and measure that race in the design. Unknown occupancy must prevent automatic stop or pause.
+
+Preserve exact T6 selections: Town, Farm, and other variants cannot be pooled merely because they share an engine map. Also account for Zombies sessions that are technically joinable but unattractive to new players because of the current round or server rules. Spare capacity should follow practical joinability, not only a non-full player count.
+
+An illustrative calculation: 20 empty instances averaging 0.40 CPU each consume eight logical-CPU equivalents. Keeping five running at the same rate uses two, a 75% reduction in that empty-instance component. These are invented inputs to demonstrate scaling; occupied games, startup bursts, and management services remain additional costs.
+
+### 5. Start infrequent maps on request, or prewarm them on a schedule
+
+A map reservation page or operator action can start a stopped server, wait for readiness, and then tell players it is available. A schedule can start extra lobbies before a recurring play session and stop them later only if they are confirmed empty. This removes the need to infer intent from game packets.
+
+For public discovery, keep popular maps online and make rarely used maps available through this explicit flow. The tradeoff is waiting for startup and reduced browser presence while stopped. Pre-pull images and stage updates outside the wake path; measure startup with realistic caches and map assets.
+
+T5 needs special treatment. Its official setup documentation says joins use a backend-issued session ID rather than direct IP connections. T6's setup guide documents IP/port connections. Therefore, a T5 wake flow should verify renewed backend registration and browser joinability before telling users it is ready; an RCON response alone does not prove that. The T5 page also contains generic IP-connect advice elsewhere, so use its explicit T5 session caveat and validate actual behavior. [T5 connection FAQ](https://plutonium.pw/docs/server/t5/setting-up-a-server/#why-cant-i-connect-to-my-server-via-the-console), [T6 setup guide](https://plutonium.pw/docs/server/t6/setting-up-a-server/).
+
+### 6. Pause containers as an experiment for faster wakeup
+
+Docker pause freezes container processes on Linux. This preserves process memory while suspending execution, making it a candidate when cold startup is expensive and RAM is plentiful. [Docker pause](https://docs.docker.com/reference/cli/docker/container/pause/).
+
+The consequence is equally important: the frozen game cannot execute RCON replies, join handling, or backend heartbeats. An external controller must resume it. Treat possible listing expiry, stale sessions, timer jumps, and failed first joins as acceptance tests; successful unpause alone is insufficient. Coordinate monitoring so deliberate suspension is not treated as a fault, and test after both short and overnight pauses.
+
+I would test stop/start first because it gives a clean startup path. Pause becomes attractive only if measured resume reliability and latency justify it. It retains memory, and it does not make a server remain genuinely responsive while asleep.
+
+### 7. Build a UDP wake gateway only if seamless demand detection is valuable enough
+
+The concept is a lightweight listener that owns a stable public endpoint, distinguishes a join attempt from a status query, wakes the backend, and forwards game traffic when ready. This is a design proposal, not a verified T5/T6 product or a configuration toggle.
+
+There is an architectural precedent: Game Server Snooze implements packet-triggered startup, bounded buffering, and idle shutdown for Palworld under Pterodactyl. Its documented game support is Palworld, so it is a reference design, not something established to work with this fleet. [Game Server Snooze upstream](https://github.com/dawidkulpa/game-server-snooze).
+
+Its difficult parts are protocol-specific:
+
+- Browser refreshes, monitoring, scans, and genuine joins must not all wake a server; otherwise idle savings disappear.
+- Client retries, challenge exchanges, authentication, packet buffering limits, and startup time must work together.
+- The proxy and backend must have deliberate port ownership and source-address handling; two processes cannot simply bind the same endpoint.
+- Stopped backends cannot refresh their own listings or sessions. A UDP listener alone does not solve discovery, particularly T5's session-based joins.
+
+T6 is the more plausible first prototype because direct IP joining is documented, but transparent success still needs testing. For T5, a request-to-start flow is the more credible initial design. Do not assume that answering a generic status query reproduces Plutonium's registration or authentication behavior.
+
+### 8. Conditional alternatives: host changes and engine support
+
+These merit investigation when the measurements point to them.
+
+If the Docker host itself runs inside a VM, audit virtual CPU exposure and host contention. One T4/T5 operator reported resolving excessive CPU consumption by changing a Proxmox Windows VM from `x86-64-v3` to `host`. Another operator in the same thread reported normal behavior with `x86-64-v3`, so this is a conditional clue, not a universal fix or proof of a Plutonium regression. [Firsthand performance discussion](https://forum.plutonium.pw/topic/45013/server-performance-issues/11).
+
+Proxmox documents the performance rationale for exposing host CPU features and the tradeoff for migration between unlike hosts. Test a clone if this matches the infrastructure; do not change VM compatibility blindly. [Proxmox administration guide, CPU type discussion](https://pve.proxmox.com/pve-docs-6/pve-admin-guide.pdf). This older official reference supports the CPU-feature tradeoff, not current Proxmox defaults.
+
+A controlled native-Windows comparison can also isolate Wine's contribution. Plutonium publishes Windows hosting instructions, but that does not establish how much CPU a migration would save. Compare total host cost, including the Windows guest if virtualized, and expect substantial operational changes to this Docker-based fleet. Native Windows is a benchmark candidate before it is a migration recommendation. [Official T5 hosting instructions](https://plutonium.pw/docs/server/t5/setting-up-a-server/).
+
+The most ambitious route is an upstream engine idle mode or supported plugin-assisted governor that reduces simulation work while keeping networking and session maintenance responsive. The official SDK has player connection callbacks and frame/timer scheduling facilities, but documents no turnkey hibernation API. Those callbacks could provide better activity signals; they do not prove that safely changing engine sleep/tick behavior is possible through the SDK. Inserting an arbitrary sleep into a game callback could block networking or gameplay. [Plutonium plugin SDK](https://plutonium.pw/docs/modding/plugin-sdk/v1-api/).
+
+This route has the best conceptual fit for an always-listed, low-CPU server, but the highest development and compatibility uncertainty. A useful upstream request would include clean-server profiles, per-thread CPU data, exact versions, and measured differences between zero players, connecting players, and active play.
+
+## Lower-priority settings and shortcuts
+
+Reducing `sv_fps` fleet-wide is not an established solution: Plutonium's changelog records T5 SP `sv_fps` becoming cheat-protected and a historical T6 Zombies dedicated tick-rate increase from 10 to 20 Hz. Neither entry establishes a safe idle-only override. Client FPS advice for `com_maxfps` also does not establish a dedicated-server fix. I found no documented common T5/T6 hibernation setting. [Plutonium changelog](https://plutonium.pw/docs/changelog/).
+
+CPU shares and affinity can help manage contention, but they should not be mistaken for eliminating idle work. Docker documents that shares act when CPU is constrained and that a CPU set restricts where execution occurs. For actual utilization ceilings, use quotas; for elimination of unnecessary instances' work, stop them. [Docker resource controls](https://docs.docker.com/engine/containers/resource_constraints/#cpu).
+
+## Run a repeatable trial
+
+First record the actual host/VM details, kernel, image digest, Wine version, Plutonium revision, scripts/plugins, map/ruleset, and CPU limits. Keep those fixed while changing one variable. Pinning an image alone is insufficient if the wrapper updates game binaries at startup; record the resulting binary versions too.
+
+Use one representative instance from each game, then expand successful changes to other maps. Suggested measurement windows are 10–15 minutes after startup settles, repeated after players join and all leave. These are proposed test windows, not product requirements.
+
+| Test | What it resolves |
+| --- | --- |
+| Normal deployment, empty immediately after startup | Initial baseline |
+| Same instance after the last player disconnects | Whether work accumulates or persists after play |
+| Minimal scripts/plugins on the same map | Contribution from custom behavior |
+| Verified NTSYNC versus current synchronization path | Wine synchronization contribution |
+| One idle quota at a time, with repeated fresh joins | Lowest useful ceiling without degraded access |
+| Stop/start or pause/resume after brief and long inactivity | Real wake latency, listing recovery, and join reliability |
+| Occupied session with ordinary and high-round gameplay | Whether the change preserves active performance |
+
+Collect container CPU averages and peaks, process/thread CPU, memory, quota throttling, RCON response times/failures, external browser visibility, first-join success, and time until players can actually play. Measure the whole host as well to catch work shifted into a proxy or management process. Use unprofiled runs for final CPU comparisons because tracing itself adds overhead.
+
+Normalize CPU units before comparing reports. Docker's Linux calculation expresses utilization relative to logical CPUs: 100% corresponds approximately to one fully used logical CPU. Thus 40% is approximately 0.4 CPU, not necessarily 40% of the machine. Dashboard and Windows figures may use different denominators. [Docker's CPU calculation](https://github.com/docker/cli/blob/master/cli/command/container/stats_helpers.go).
+
+For a trial to pass, require repeatable idle savings, successful fresh joins from outside the host, normal browser/session recovery, and no material degradation during play. Reject a lower CPU reading accompanied by join failures, delayed quota restoration, or persistent health errors. Roll back the single experimental change and retain its measurements.
+
+My proposed order is NTSYNC verification and configuration/script isolation, then conservative idle quotas if all lobbies must stay online. If that leaves too much baseline consumption, reduce running spare capacity and add scheduled or explicit startup. Pursue pause, transparent packet wakeup, or engine changes only when their additional complexity solves a measured limitation of those simpler options.