Automatically place servers using learned resource capacity

This commit is contained in:
2026-10-04 21:04:09 -04:00
parent a1b957392f
commit e6cdf9eb7f
25 changed files with 927 additions and 85 deletions
+53 -3
View File
@@ -27,8 +27,8 @@ the agent owns local execution and its durable operation journal.
progress and errors. Wait for verification before launching. Hosts report
connection and readiness separately; deployment also checks the particular
game files, image, capacity and ports required for the launch.
6. Launch a server or collection. Select the host; the latest game files and
default runtime are selected automatically. An advanced runtime override is
6. Launch a server or collection. The host, latest game files and default
runtime are selected automatically after checking capacity. An advanced runtime override is
available. If the latest files change after previewing, refresh the preview
before launching.
EC selects lobby IDs and reserves UDP/TCP ports atomically.
@@ -121,7 +121,57 @@ host-wide busy time across all cores on a 0–100% scale between inventory sampl
(normally about 15 seconds). Idle and I/O wait are excluded; stolen CPU time
counts as unavailable capacity. The first sample after startup, unavailable
counters, and older agents show **Not reported**. Offline hosts retain their
last reported value. This is current inventory, not historical metrics.
last reported value. The controller retains CPU and memory samples for seven days (missing measurements
remain unknown). Scheduling uses the last five minutes as well as the latest
heartbeat; the Inventory panel continues to show the latest reading.
New servers are placed automatically. Preview and submission perform local
capacity checks without contacting agents: a connected, active host must have
verified game files, its runtime, storage, a free UDP/TCP port, and valid CPU and
memory measurements. Hosts with stale or missing measurements cannot accept new
servers. Existing API clients may still explicitly constrain a launch to a host.
Admission reserves a provisional host and port atomically, including collection
launches. Agent **1.4.3** samples each managed running container with a bounded
[`docker stats`](https://docs.docker.com/reference/cli/docker/container/stats/)
batch during inventory collection. CPU is normalized to cores (100% Docker CPU
is one core); memory is the Linux working usage reported by Docker, excluding
reclaimable cache. Missing or failed samples stay unknown instead of becoming
zero usage. Samples must match the host's live instance and generation.
The controller retains per-instance minute peaks for 24 hours and caches learned
CPU/memory allowances. The first five minutes after a container starts count as
startup; later samples train the running allowance. Startup uses the observed
maximum and running uses the 95th percentile of minute peaks, both with **25%
headroom**. Each calculation considers up to the latest 10,000 minute peaks.
Host-local evidence is preferred to account for different hardware; new hosts
can borrow the fleet's learned allowance. Startup allowances are never lower
than running allowances. The host page shows both allowances and their source.
Lower allowances require at least three startup minute samples spanning two
minutes, or twenty running minute samples spanning five minutes. Until then,
**0.5 core and 1 GiB** are temporary cold-start allowances, raised immediately
if observations exceed them. They are replaced by measured allowances once the
evidence threshold is met; a tiny 0.01-core floor prevents zero-CPU admission.
Models expire after 24 hours without measurements. Recalculation happens during
heartbeat ingestion, at most once per minute per host/fleet phase; new peaks
raise the cache immediately. Launch admission reads only the cached model.
Scheduling still leaves **10% host CPU and 512 MiB RAM** for the OS and bursts.
These allowances are scheduling reservations, not Docker limits. Queued and
unresolved creations consume the startup allowance; established instances keep
the running allowance even while idle or stopped. Observed host load also counts,
so other processes and previously unseen load increases reduce available capacity.
When startup capacity is available, provisioning rechecks all eligible hosts and
chooses the lowest projected CPU/memory utilization, using the busier of the
latest reading and the five-minute average. It moves the provisional reservation
atomically and records the final host in the launch recipe. Host network defaults
are resolved again for the final host; explicit overrides are preserved. If no
host qualifies, the request waits and is checked again. Once remote staging or
creation has been submitted, placement stays fixed to prevent duplicate instances.
Cancelling an unsubmitted launch releases its reservation; uncertain remote work
retains capacity until cleanup is confirmed.
When publishing changed agent code, bump its numeric `VERSION` in
`agent/ec_agent.py` and rebuild the controller. Packaging derives the manifest