Add configurable history and storage retention policies

docs/maintenance.md

@@ -98,7 +98,7 @@ Schedules survive application restarts. Missed days, including while globally
 paused, become one run on resume. Pending work is deduplicated per server. The
 worker starts on the first application request, so keep the app running and
 reachable. Finished scheduled and legacy maintenance activity is retained for
-30 days; update attempt records remain durable.
+30 days by default (configurable in Administration → Settings); update attempt records remain durable.
 
 ## Health recovery and configuration drift
 

docs/metrics.md

@@ -45,8 +45,8 @@ represents the current registration rather than every instance in the history.
   identical filters. Utilization changes are percentage points; player-hours
   changes are relative percentages. A previous observed zero can show **New**.
   Missing evidence remains unavailable. Partial retained history is labelled;
-  a 30-day comparison usually has little previous history because retention is
-  only 31 days. Both periods' known time is shown. Bounds cannot establish that
+  a 30-day comparison usually has little previous history because default retention is
+  31 days. Both periods' known time is shown. Bounds cannot establish that
   collection was uninterrupted, and changes are not adjusted for fleet size.
 - **Activity heatmap** groups captured time by weekday and hour in its displayed
   timezone. Color represents occupied / known server time. Exact cell values
@@ -82,7 +82,7 @@ Observations are recorded by the existing status worker. Each remains valid unti
 
 Capacity comes from `sv_maxclients` / `sv_maxClients` in the status response. When unavailable, the UI displays unknown rather than assuming a four-player limit. Capacity coverage is the map time for which every tracked lobby reported both occupancy and capacity.
 
-Collection begins after this feature is installed; previous occupancy cannot be reconstructed. Samples are retained for 31 days and pruned during collection. They survive removal of a server and contain no player identities. The additive database table is created automatically on startup.
+Collection begins after this feature is installed; previous occupancy cannot be reconstructed. Samples default to 31 days of retention; Administration → Settings can change this period. Cleanup runs during collection and in the background. They survive removal of a server and contain no player identities. The additive database table is created automatically on startup.
 
 `GET /api/metrics` supplies the same report as the page and disables response
 caching. Query parameters are `days` (1, 7, or 30), `game`, `map` (requires game),

docs/native-hosts.md

@@ -140,7 +140,7 @@ host-wide busy time across all cores on a 0–100% scale between inventory sampl
 (normally about 15 seconds). Idle and I/O wait are excluded; stolen CPU time
 counts as unavailable capacity. The first sample after startup, unavailable
 counters, and older agents show **Not reported**. Offline hosts retain their
-last reported value. The controller retains CPU and memory samples for seven days (missing measurements
+last reported value. The controller defaults to retaining CPU and memory samples for seven days (missing measurements
 remain unknown). Scheduling checks capacity against the latest heartbeat and uses
 the last five minutes to rank eligible hosts; the Inventory panel shows the latest
 reading.
@@ -171,7 +171,7 @@ Sampling failures appear on the host's Learned server capacity panel. Existing
 containers can begin contributing running samples as soon as the agent updates;
 startup samples require observing the first five minutes of a container's run.
 
-The controller retains per-instance minute peaks for 24 hours and caches learned
+The controller defaults to retaining per-instance minute peaks for 24 hours and caches learned
 CPU/memory allowances. The first five minutes after a container starts count as
 startup; later samples train the running allowance. Startup uses the observed
 maximum and running uses the 95th percentile of minute peaks, with **no per-server
@@ -471,7 +471,9 @@ controller round trips do not stall every transferred chunk.
 Corrupt or incomplete content never becomes Ready. Cache inventory is reconciled
 on heartbeats, including vanished caches. The latest published version of each game
 is always retained. With agent 1.4.7 or newer, an older host copy is automatically
-removed once no live instance reservation on that host references it. Stopped,
+eligible for removal once no live instance reservation on that host references it.
+The game-file minimum count, supersession grace period and explicit protections in
+Administration → Settings also apply. Stopped,
 provisioning, and rebuilding instances still pin their version. The agent also
 refuses deletion if any Docker container (including stopped containers) mounts
 those files. Cleanup includes partial downloads and retries after interruption.

docs/settings.md

@@ -0,0 +1,115 @@
+# Fleet settings
+
+Administration → Settings has independent **Concurrency** and **Retention**
+sections. Settings persist in SQLite, apply without restarting the controller,
+and are writable only by administrators. Saving one section preserves the other
+section's saved values. Upgrades preserve existing values and add new defaults.
+
+## Concurrency
+
+The existing fleet restart, startup and destruction limits default to 1. Additional
+limits default to 0 (no additional cap):
+
+- Restarts per host, including starts and rebuild recovery.
+- Startups per host, including creation and rebuilds.
+- Destructions per host, including migration source removal.
+- Combined lifecycle work across create/start/stop/restart/rebuild/destroy.
+- Rebuilds across the fleet, including host drain migrations.
+- Hosts downloading or verifying game files simultaneously, including manual imports.
+
+Limits are checked again before command submission after asynchronous preflight.
+Launch-file preparation reserves capacity. A lifecycle workflow counts once in the
+combined pool; unresolved commands and recovery keep capacity occupied. Migrations
+keep rebuild capacity through source removal and destination creation. Lower limits
+do not cancel admitted work. Game-file preparation slots are assigned on agent
+command delivery and survive interrupted transfers. Runtime pulls, the four file
+transfers per host and the two health probes per host remain unchanged.
+
+## Retention
+
+For age settings, 0 means no expiry. An explicit version-count policy can still
+expire unused artifact versions when their age policy is disabled. Game-file grace
+is a delay: 0 adds no delay. A positive age and version count together preserve the
+minimum count as well as versions younger than the age threshold.
+
+| History or storage | Default |
+| --- | --- |
+| Campaigns | 7 days |
+| Successful routine operations | 1 day |
+| Provisioning workflows | Indefinite |
+| Agent actions | Indefinite |
+| Explicitly resolved failures | Indefinite |
+| Non-update maintenance jobs | 30 days |
+| Resolved recovery incidents | Indefinite |
+| Host resource history | 7 days |
+| Instance resource samples | 1 day |
+| Fleet occupancy metrics | 31 days |
+| Telemetry diagnostics | 7 days; 200 per server; 10,000 overall |
+| Completed/interrupted matches | Indefinite |
+| Closed betas and test evidence | Indefinite |
+| Campaign file bodies and closed beta snapshots | No separate expiry |
+| Unused artifact versions | No automatic expiry; no minimum count |
+| Unused game files | No grace period; keep latest 1 version per game |
+| Incomplete browser uploads | Indefinite |
+| Authentication audit events | Indefinite |
+| Minimum history count | 0 additional records |
+
+Telemetry age and count policies are independent; 0 disables each individual cap.
+Diagnostic responses remain bounded to 10,000 rows when the per-server retention
+cap is disabled. Metric and capacity chart time ranges are unchanged; increasing
+storage retention does not automatically add new chart ranges. Capacity learning
+continues to use its 24-hour window independently of longer sample retention.
+
+### Cleanup and dependencies
+
+Background cleanup runs in bounded batches. New age policies for previously
+permanent history default to indefinite. Pending work, unresolved failures,
+recovery dependencies and explicitly protected records are never removed merely
+because they are old. Referenced artifact versions are retained, including versions
+referenced by historical campaigns and betas. Collection workflows with unfinished
+members, cancellation links and recovery slots can keep provisioning history longer
+than its configured age.
+
+Update attempts and dismissed watchdog incidents remain durable suppression evidence.
+Host setup, credential/decommission operations and referenced agent commands are
+protected. Expired unreferenced action history drops payloads, results and events
+but keeps compact terminal receipts and request fingerprints. Reusing an idempotency
+key cannot execute an expired command again. Agent mutation journals remain intact;
+this policy controls controller history, not the agent's execution recovery journal.
+
+Expiring campaign file bodies preserves file paths, hashes and summary records but
+makes file comparisons and rollback unavailable. The detail page identifies this
+state and rollback requests fail explicitly. Active beta rollback images stay
+protected. Expiring matches removes their player participation rows and changes
+leaderboard/player totals; lifetime aggregates are not retained separately.
+
+Game-file minimums apply per game. Grace starts when a version is superseded.
+Live, stopped, rebuilding and provisioning instances pin their files; unresolved
+operations and offline host copies also prevent controller deletion. Controller
+cleanup still waits for every host to report its copy absent. Incomplete browser
+uploads expire based on their last activity, including resume checks; active chunk
+writes and publication hold locks against cleanup. Metadata remains after files
+expire. Raising retention cannot restore previously deleted bytes.
+
+### Preview and record protection
+
+Expand **Cleanup preview and protected records** to preview the values currently
+entered without saving them. Preview lists eligible records and known payload
+sizes; SQLite file space may be reused rather than immediately returned to the OS.
+Policies can overlap, so payload sizes should not be added together. Unknown row,
+index, upload and agent-disk overhead is not included.
+
+Use a record's ID from its detail URL or the preview to protect it. Protection can
+be removed in the same panel. Protecting a campaign or beta also protects its file
+bodies; protecting those bodies prevents deletion of their parent history.
+Game-file and incomplete-upload protections apply to storage cleanup.
+
+The minimum history count preserves recent records per history category. It does
+not apply to telemetry caps, metric/resource samples or game-file storage, which
+have their own policies. Explicit holds protect individual history/storage records,
+not individual resource samples or telemetry diagnostics.
+
+Failed campaigns, provisioning workflows and agent commands require explicit
+resolution before the resolved-failure period applies. Resolution does not bypass
+recovery/dependency checks. Resolving a failed campaign stops automatic campaign
+retry requests; it does not cancel commands already submitted.