Move fleet retention and concurrency controls into Settings
Persist settings in SQLite, import legacy environment values once, and apply changes to cleanup and deployment queues without restarting. Closes #51
This commit is contained in:
+1
-10
@@ -9,16 +9,7 @@ BETA_ARCHIVE_DIR=/root/zombies/beta-archive
|
||||
# SQLite location
|
||||
DB_PATH=/data/cherry.db
|
||||
|
||||
# Retention (days) for deployment rows and blob diffs
|
||||
DEPLOY_KEEP_DAYS=7
|
||||
|
||||
# Fleet-wide rolling restart limit, shared by deployments and betas.
|
||||
# A slot stays occupied until Komodo and RCON confirm recovery.
|
||||
DEPLOYMENT_RESTART_CONCURRENCY=1
|
||||
|
||||
# New-server startup limit, shared by all collections and single-server launches.
|
||||
# A slot stays occupied through Komodo, RCON, and selected-map verification.
|
||||
DEPLOYMENT_CREATE_CONCURRENCY=1
|
||||
# Configure campaign retention and restart/startup concurrency in Settings → Fleet settings.
|
||||
|
||||
# Komodo
|
||||
KOMODO_ADDRESS=https://komodo.example.internal
|
||||
|
||||
@@ -98,6 +98,30 @@ submitted recovery continue. Each capability retains its own policy. Automatic
|
||||
updates remain disabled until explicitly enabled. See [Maintenance](docs/maintenance.md)
|
||||
for verification, retry, scheduling, and compatibility details.
|
||||
|
||||
## Fleet settings
|
||||
|
||||
**Settings → Fleet settings** controls campaign history retention (default 7 days),
|
||||
concurrent restarts (default 1), and concurrent server startups (default 1).
|
||||
Values are positive whole numbers; retention supports up to 36500 days. Settings
|
||||
are saved in SQLite for all users and apply at the next queue or cleanup check
|
||||
without restarting the app. Lowering a limit leaves active operations running
|
||||
and holds queued work until capacity is available. Collection previews and
|
||||
progress pages display the current startup limit.
|
||||
|
||||
Retention removes eligible old campaign records and file diffs. Pending, failed,
|
||||
held, and health-recovery-linked campaigns remain protected; Provisioning history
|
||||
is retained. Shortening retention can remove older history on the next sweep.
|
||||
|
||||
On the first startup after upgrading, existing `DEPLOY_KEEP_DAYS`,
|
||||
`DEPLOYMENT_RESTART_CONCURRENCY`, and `DEPLOYMENT_CREATE_CONCURRENCY` environment
|
||||
values initialize the saved settings. Keep custom values for that first startup;
|
||||
afterward, the database values take precedence and those variables can be removed.
|
||||
The supplied Compose and Komodo examples use the built-in defaults.
|
||||
|
||||
`GET /api/settings` reads the three values (`deployKeepDays`, `restartConcurrency`,
|
||||
`createConcurrency`); `PUT /api/settings` saves all three together as JSON.
|
||||
Invalid or incomplete requests return HTTP 400 without changing saved settings.
|
||||
|
||||
## How restarts stay safe
|
||||
|
||||
Betas support configurable gameplay, participation, stability and manual-review
|
||||
@@ -107,7 +131,7 @@ requirements, saved presets, coverage rules and staged waves. See
|
||||
File changes are staged sequentially. Their activation uses rolling restarts:
|
||||
by default only **one server across the fleet** can restart or recover at a
|
||||
time. Promotions, rollbacks, drift reconciliation, and beta restarts share this
|
||||
limit. Set `DEPLOYMENT_RESTART_CONCURRENCY` to a positive integer to raise it.
|
||||
limit. Adjust **Concurrent restarts** in **Settings → Fleet settings** to raise it.
|
||||
|
||||
The Provisioning worker runs every 15s. Campaigns, betas, and maintenance submit
|
||||
durable restart requests and read their results. Provisioning checks recovering
|
||||
@@ -164,8 +188,8 @@ launches. A startup slot stays occupied until Komodo is healthy and RCON or
|
||||
fresh telemetry reports the selected map; an unready server holds the queue until it recovers
|
||||
or is removed. The queue survives application restarts. The collection preview
|
||||
shows the startup limit, and its progress page shows active startups and each
|
||||
server's place in the shared queue. Set `DEPLOYMENT_CREATE_CONCURRENCY` to a
|
||||
positive integer to increase the limit. Each worker pass starts at most the
|
||||
server's place in the shared queue. Adjust **Concurrent server startups** in
|
||||
**Settings → Fleet settings** to increase the limit. Each worker pass starts at most the
|
||||
available number of slots; immediate failures do not drain the remaining queue.
|
||||
Existing single-server collections can be expanded with **Edit**. Single-server
|
||||
launches and launching again from recorded recipes remain available.
|
||||
|
||||
@@ -31,9 +31,6 @@ services:
|
||||
REL_DEST: runtime/plutonium/storage/t5/raw/scripts/sp
|
||||
BETA_ARCHIVE_DIR: /root/zombies/beta-archive
|
||||
DB_PATH: /data/cherry.db
|
||||
DEPLOY_KEEP_DAYS: "7"
|
||||
DEPLOYMENT_RESTART_CONCURRENCY: ${DEPLOYMENT_RESTART_CONCURRENCY:-1}
|
||||
DEPLOYMENT_CREATE_CONCURRENCY: ${DEPLOYMENT_CREATE_CONCURRENCY:-1}
|
||||
KOMODO_ADDRESS: ${KOMODO_ADDRESS}
|
||||
KOMODO_API_KEY: ${KOMODO_API_KEY}
|
||||
KOMODO_API_SECRET: ${KOMODO_API_SECRET}
|
||||
|
||||
@@ -95,7 +95,7 @@ the current shortfall too.
|
||||
All collections and individual launches share one startup queue, ordered by request
|
||||
time. By default, one server starts at a time. Its startup slot remains occupied
|
||||
until Komodo is healthy, RCON answers, and the reported map matches the selection.
|
||||
Only then can the next queued server start. Set `DEPLOYMENT_CREATE_CONCURRENCY` to
|
||||
Only then can the next queued server start. Set **Concurrent server startups** in **Settings → Fleet settings** to
|
||||
a positive integer to allow more simultaneous startups. Each worker pass admits
|
||||
at most the available slots, even if an attempt fails immediately.
|
||||
|
||||
|
||||
@@ -27,8 +27,6 @@ PLUTAINER_T6_GAMEFILES_VOLUME = "t6_game_files"
|
||||
REL_DEST = "runtime/plutonium/storage/t5/raw/scripts/sp"
|
||||
BETA_ARCHIVE_DIR = "/root/zombies/beta-archive"
|
||||
DB_PATH = "/data/cherry.db"
|
||||
DEPLOY_KEEP_DAYS = "7"
|
||||
DEPLOYMENT_RESTART_CONCURRENCY = "1"
|
||||
|
||||
# secrets: bind via Komodo's env-file or secret store
|
||||
KOMODO_ADDRESS = "https://komodo.example.internal"
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
export interface FleetSettings {
|
||||
deployKeepDays: number;
|
||||
restartConcurrency: number;
|
||||
createConcurrency: number;
|
||||
}
|
||||
|
||||
export const defaultFleetSettings: FleetSettings = {
|
||||
deployKeepDays: 7,
|
||||
restartConcurrency: 1,
|
||||
createConcurrency: 1
|
||||
};
|
||||
|
||||
export const fleetSettingLabels: Record<keyof FleetSettings, string> = {
|
||||
deployKeepDays: 'Campaign history retention',
|
||||
restartConcurrency: 'Concurrent restarts',
|
||||
createConcurrency: 'Concurrent server startups'
|
||||
};
|
||||
|
||||
export function validateFleetSettings(input: unknown): FleetSettings {
|
||||
if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('Provide fleet settings.');
|
||||
const values = input as Record<string, unknown>;
|
||||
for (const key of Object.keys(values)) {
|
||||
if (!Object.hasOwn(defaultFleetSettings, key)) throw new Error(`Unknown setting: ${key}`);
|
||||
}
|
||||
for (const key of Object.keys(defaultFleetSettings) as (keyof FleetSettings)[]) {
|
||||
const value = values[key];
|
||||
if (typeof value !== 'number' || !Number.isSafeInteger(value) || value < 1) {
|
||||
throw new Error(`${fleetSettingLabels[key]} must be a positive whole number.`);
|
||||
}
|
||||
if (key === 'deployKeepDays' && value > 36500) throw new Error('Campaign history retention must be at most 36500 days.');
|
||||
}
|
||||
return { deployKeepDays: values.deployKeepDays as number, restartConcurrency: values.restartConcurrency as number,
|
||||
createConcurrency: values.createConcurrency as number };
|
||||
}
|
||||
@@ -15,12 +15,6 @@ function num(name: string, fallback: number): number {
|
||||
return n;
|
||||
}
|
||||
|
||||
function positiveInt(name: string, fallback: number): number {
|
||||
const value = num(name, fallback);
|
||||
if (!Number.isSafeInteger(value) || value < 1) throw new Error(`env ${name} must be a positive integer`);
|
||||
return value;
|
||||
}
|
||||
|
||||
export const config = {
|
||||
serversRoot: req('SERVERS_ROOT', '/root/zombies/t5'),
|
||||
t6ServersRoot: env.T6_SERVERS_ROOT ?? join(dirname(req('SERVERS_ROOT', '/root/zombies/t5')), 't6'),
|
||||
@@ -38,9 +32,6 @@ export const config = {
|
||||
// Cleared per-beta on promote/rollback.
|
||||
betaArchiveDir: req('BETA_ARCHIVE_DIR', '/root/zombies/beta-archive'),
|
||||
dbPath: req('DB_PATH', './data/cherry.db'),
|
||||
deployKeepDays: num('DEPLOY_KEEP_DAYS', 7),
|
||||
restartConcurrency: positiveInt('DEPLOYMENT_RESTART_CONCURRENCY', 1),
|
||||
createConcurrency: positiveInt('DEPLOYMENT_CREATE_CONCURRENCY', 1),
|
||||
httpTimeoutMs: num('HTTP_TIMEOUT_MS', 10_000),
|
||||
komodo: {
|
||||
address: env.KOMODO_ADDRESS ?? '',
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
import type Database from 'better-sqlite3';
|
||||
import { defaultFleetSettings, validateFleetSettings, type FleetSettings } from '../../fleet-settings.js';
|
||||
|
||||
export const FLEET_SETTINGS_KEY = 'fleet_settings';
|
||||
|
||||
/** Import legacy environment values once; persisted settings win on every later boot. */
|
||||
export function initializeFleetSettings(handle: Database.Database): void {
|
||||
if (handle.prepare('SELECT 1 FROM settings WHERE key = ?').get(FLEET_SETTINGS_KEY)) return;
|
||||
const values = { ...defaultFleetSettings };
|
||||
const legacy: Record<keyof FleetSettings, string> = {
|
||||
deployKeepDays: 'DEPLOY_KEEP_DAYS', restartConcurrency: 'DEPLOYMENT_RESTART_CONCURRENCY',
|
||||
createConcurrency: 'DEPLOYMENT_CREATE_CONCURRENCY'
|
||||
};
|
||||
for (const key of Object.keys(legacy) as (keyof FleetSettings)[]) {
|
||||
const raw = process.env[legacy[key]];
|
||||
if (raw) values[key] = Number(raw);
|
||||
}
|
||||
handle.prepare('INSERT OR IGNORE INTO settings (key, value) VALUES (?, ?)')
|
||||
.run(FLEET_SETTINGS_KEY, JSON.stringify(validateFleetSettings(values)));
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import schemaSql from './schema.sql?raw';
|
||||
import { requirement, type BetaPlan, type BetaRequirement } from '../../beta-plan.js';
|
||||
import { emptyMetrics } from '../fleet/beta-evaluation.js';
|
||||
import { backfillLeaderboard } from '../fleet/leaderboard-ingest.js';
|
||||
import { initializeFleetSettings } from './fleet-settings.js';
|
||||
|
||||
let _db: Database.Database | null = null;
|
||||
|
||||
@@ -345,6 +346,7 @@ export function db(): Database.Database {
|
||||
migrateGameTelemetry(handle);
|
||||
migrateTable(handle, 'leaderboard_matches', [{ name: 'start_players', ddl: 'ALTER TABLE leaderboard_matches ADD COLUMN start_players INTEGER' }]);
|
||||
backfillLeaderboard(handle);
|
||||
initializeFleetSettings(handle);
|
||||
} catch (err) {
|
||||
handle.close();
|
||||
throw err;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { fleetSettings } from '../settings.js';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import type { CollectionDetail, CollectionMode, CollectionPreview, CollectionSummary } from '$lib/deployment-plan.js';
|
||||
import { db } from '../db/index.js';
|
||||
@@ -106,7 +107,7 @@ export function previewCollection(blueprintId: unknown, mode: unknown = 'fill'):
|
||||
db().prepare('DELETE FROM server_collection_previews WHERE expires_at < ? AND collection_id IS NULL').run(new Date().toISOString());
|
||||
db().prepare('INSERT INTO server_collection_previews (id, expires_at, blueprint_name, entries, servers, mode, fleet_snapshot) VALUES (?, ?, ?, ?, ?, ?, ?)')
|
||||
.run(id, expiresAt, blueprintName, JSON.stringify(entries), JSON.stringify(servers), mode, mode === 'exact' ? JSON.stringify(fleetSnapshot()) : null);
|
||||
return { id, expiresAt, blueprintName, mode, retainedServers, removals, launchConcurrency: config.createConcurrency, servers, blockers, ready: blockers.length === 0,
|
||||
return { id, expiresAt, blueprintName, mode, retainedServers, removals, launchConcurrency: fleetSettings().createConcurrency, servers, blockers, ready: blockers.length === 0,
|
||||
entries: entries.map((entry, index) => ({ count: entry.count, ...counts[index], recipe: summarizeRecipe(entry.recipe) })) };
|
||||
})();
|
||||
}
|
||||
@@ -216,6 +217,6 @@ export function getCollection(id: string): CollectionDetail | null {
|
||||
const queue = creationQueue();
|
||||
const positions = new Map(queue.queued.map((job, index) => [job.id, index + 1]));
|
||||
const stored = db().prepare('SELECT retained_servers FROM server_launch_collections WHERE id = ?').get(id) as { retained_servers: string };
|
||||
return { collection, retainedServers: JSON.parse(stored.retained_servers), queue: { concurrency: config.createConcurrency, active: queue.active, queued: queue.queued.length },
|
||||
return { collection, retainedServers: JSON.parse(stored.retained_servers), queue: { concurrency: fleetSettings().createConcurrency, active: queue.active, queued: queue.queued.length },
|
||||
servers: servers.map(server => ({ ...server, queuePosition: positions.get(server.id) ?? null })) };
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { config } from '../config.js';
|
||||
import { fleetSettings } from '../settings.js';
|
||||
import { db } from '../db/index.js';
|
||||
import { emit } from './bus.js';
|
||||
import { requestRestart } from '../provisioning/index.js';
|
||||
@@ -44,7 +44,7 @@ async function runSweep(): Promise<SweepResult> {
|
||||
db().prepare(`DELETE FROM deployments WHERE started_at < ?
|
||||
AND NOT EXISTS (SELECT 1 FROM watchdog_actions WHERE campaign_id = deployments.id) AND NOT EXISTS (
|
||||
SELECT 1 FROM deployment_servers WHERE deployment_id = deployments.id AND restart_state IN ('pending','failed','held'))`)
|
||||
.run(new Date(Date.now() - config.deployKeepDays * 86_400_000).toISOString());
|
||||
.run(new Date(Date.now() - fleetSettings().deployKeepDays * 86_400_000).toISOString());
|
||||
emit({ type: 'sweep-tick', frozen: false });
|
||||
return result;
|
||||
}
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { fleetSettings } from '../settings.js';
|
||||
import { probePlayerCount, probeServerStatus } from '../fleet/status-evidence.js';
|
||||
import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEvent, REMOVING_SERVER, resumeCleanup, resumeVerification, claimRemoval } from './history.js';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
@@ -258,7 +259,7 @@ export async function rebuildServer(id: string): Promise<void> {
|
||||
const job = db().prepare(`SELECT j.*, d.stage FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id
|
||||
WHERE d.id = ? AND d.action = 'rebuild' AND d.state = 'waiting'`).get(id) as (CreationJob & { stage: string }) | undefined;
|
||||
if (!job) return;
|
||||
if (creating || destroying.size || creationQueue().active.length >= config.createConcurrency) {
|
||||
if (creating || destroying.size || creationQueue().active.length >= fleetSettings().createConcurrency) {
|
||||
// The cleanup checkpoint must survive capacity waits and app restarts.
|
||||
if (job.stage !== REMOVING_SERVER) deploymentStage(id, 'Waiting for a server creation slot before rebuilding');
|
||||
return;
|
||||
@@ -775,7 +776,7 @@ async function runCreations(): Promise<CreationOutcomes> {
|
||||
// Snapshot the budget once. Fast success/failure must not turn a single
|
||||
// worker pass into an unbounded stream of expensive container startups.
|
||||
const queue = creationQueue();
|
||||
const available = Math.max(0, config.createConcurrency - queue.active.length);
|
||||
const available = Math.max(0, fleetSettings().createConcurrency - queue.active.length);
|
||||
const eligible = queue.queued.filter(({ id }) => {
|
||||
const blocker = creationBlocker(id);
|
||||
if (blocker) deploymentStage(id, blocker);
|
||||
@@ -785,7 +786,7 @@ async function runCreations(): Promise<CreationOutcomes> {
|
||||
if (creating || destroying.size > 0) break;
|
||||
// API calls can cancel a selected job or resume an older startup while
|
||||
// a previous provision awaits Komodo. Recheck without expanding this pass.
|
||||
if (creationQueue().active.length >= config.createConcurrency) break;
|
||||
if (creationQueue().active.length >= fleetSettings().createConcurrency) break;
|
||||
if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND action IN ('create', 'rebuild') AND state = 'queued'").get(id)) continue;
|
||||
const blocker = creationBlocker(id);
|
||||
if (blocker) {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { config } from '../config.js';
|
||||
import { fleetSettings } from '../settings.js';
|
||||
import { db } from '../db/index.js';
|
||||
import { komodoInspectHealthy, komodoRestart, komodoRuntimeAction } from '../integrations/komodo.js';
|
||||
import { probeServerStatus } from '../fleet/status-evidence.js';
|
||||
@@ -49,7 +49,7 @@ export function rolloutWaitReason(server: string, operationId = ''): string | nu
|
||||
return `Waiting for the previous restart of this server to recover${slot.error ? `: ${slot.error}` : ''}`;
|
||||
}
|
||||
const { n } = db().prepare('SELECT COUNT(*) AS n FROM restart_rollout WHERE healthy_at IS NULL').get() as { n: number };
|
||||
return n - (ownHandoff && slot?.healthy_at === null ? 1 : 0) >= config.restartConcurrency ? 'Queued for a rolling restart; waiting for another server to recover' : null;
|
||||
return n - (ownHandoff && slot?.healthy_at === null ? 1 : 0) >= fleetSettings().restartConcurrency ? 'Queued for a rolling restart; waiting for another server to recover' : null;
|
||||
}
|
||||
|
||||
export type RolloutRestartResult =
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
import { validateFleetSettings, type FleetSettings } from '../fleet-settings.js';
|
||||
import { db } from './db/index.js';
|
||||
import { FLEET_SETTINGS_KEY } from './db/fleet-settings.js';
|
||||
|
||||
export function fleetSettings(): FleetSettings {
|
||||
const row = db().prepare('SELECT value FROM settings WHERE key = ?').get(FLEET_SETTINGS_KEY) as { value: string };
|
||||
return JSON.parse(row.value);
|
||||
}
|
||||
|
||||
export function saveFleetSettings(input: unknown): FleetSettings {
|
||||
const settings = validateFleetSettings(input);
|
||||
db().prepare('UPDATE settings SET value = ? WHERE key = ?').run(JSON.stringify(settings), FLEET_SETTINGS_KEY);
|
||||
return settings;
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
import { json } from '@sveltejs/kit';
|
||||
import { validateFleetSettings } from '$lib/fleet-settings.js';
|
||||
import { fleetSettings, saveFleetSettings } from '$lib/server/settings.js';
|
||||
|
||||
export function GET() { return json(fleetSettings()); }
|
||||
|
||||
export async function PUT({ request }: { request: Request }) {
|
||||
let settings;
|
||||
try { settings = validateFleetSettings(await request.json()); }
|
||||
catch (error) { return json({ error: (error as Error).message }, { status: 400 }); }
|
||||
return json(saveFleetSettings(settings));
|
||||
}
|
||||
@@ -3,14 +3,71 @@
|
||||
import { availableTimeZones, DEFAULT_TIME_ZONE } from '$lib/time-zone.js';
|
||||
import { onMount } from 'svelte';
|
||||
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
|
||||
import Button from '$ui/components/primitives/Button.svelte';
|
||||
import type { FleetSettings } from '$lib/fleet-settings.js';
|
||||
let exampleDate = $state<Date | null>(null);
|
||||
let settings = $state<FleetSettings | null>(null);
|
||||
let loading = $state(true);
|
||||
let saving = $state(false);
|
||||
let error = $state('');
|
||||
let notice = $state('');
|
||||
const zones = $derived([...new Set([...availableTimeZones(), $timeZone])].sort());
|
||||
onMount(() => { exampleDate = new Date(); });
|
||||
onMount(() => { exampleDate = new Date(); void loadSettings(); });
|
||||
|
||||
async function loadSettings() {
|
||||
loading = true; error = '';
|
||||
try {
|
||||
const response = await fetch('/api/settings');
|
||||
if (!response.ok) throw new Error('Could not load fleet settings.');
|
||||
settings = await response.json();
|
||||
} catch (cause) { error = (cause as Error).message; }
|
||||
finally { loading = false; }
|
||||
}
|
||||
|
||||
async function saveSettings(event: SubmitEvent) {
|
||||
event.preventDefault();
|
||||
saving = true; error = ''; notice = '';
|
||||
try {
|
||||
const response = await fetch('/api/settings', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(settings) });
|
||||
const result = await response.json();
|
||||
if (!response.ok) throw new Error(result.error ?? 'Could not save fleet settings.');
|
||||
settings = result;
|
||||
notice = 'Fleet settings saved. Changes apply to the next queue or cleanup check.';
|
||||
} catch (cause) { error = (cause as Error).message; }
|
||||
finally { saving = false; }
|
||||
}
|
||||
</script>
|
||||
|
||||
<svelte:head><title>Settings · Electric Cherry</title></svelte:head>
|
||||
<section class="page">
|
||||
<SectionHeader title="Settings" subtitle="Choose how dates and times appear across Electric Cherry." />
|
||||
<SectionHeader title="Settings" subtitle="Manage fleet limits, history retention, and display preferences." />
|
||||
<section class="guide" aria-labelledby="fleet-heading">
|
||||
<h2 id="fleet-heading">Fleet settings</h2>
|
||||
<p>Saved for everyone using this Electric Cherry instance. Changes apply without restarting the app.</p>
|
||||
{#if error}<p class="ui-error" role="alert">{error}</p>{/if}
|
||||
{#if notice}<p role="status">{notice}</p>{/if}
|
||||
{#if loading}
|
||||
<p role="status">Loading fleet settings…</p>
|
||||
{:else if settings}
|
||||
<form onsubmit={saveSettings}>
|
||||
<fieldset disabled={saving}>
|
||||
<label for="retention">Campaign history retention (days)</label>
|
||||
<input id="retention" type="number" min="1" max="36500" step="1" required bind:value={settings.deployKeepDays} aria-describedby="retention-help" />
|
||||
<p id="retention-help">Default: 7 days. Removes eligible old campaign records and file diffs on the next cleanup check. Pending, failed, held, and health-recovery-linked campaigns are kept. Provisioning history is retained.</p>
|
||||
<label for="restarts">Concurrent restarts</label>
|
||||
<input id="restarts" type="number" min="1" step="1" required bind:value={settings.restartConcurrency} aria-describedby="restarts-help" />
|
||||
<p id="restarts-help">Default: 1 across the fleet. Campaigns, betas, maintenance, and lifecycle actions share recovery slots until container and game health are verified.</p>
|
||||
<label for="startups">Concurrent server startups</label>
|
||||
<input id="startups" type="number" min="1" step="1" required bind:value={settings.createConcurrency} aria-describedby="startups-help" />
|
||||
<p id="startups-help">Default: 1 across collections and individual launches. Each startup holds its slot through container, game health, and selected-map verification.</p>
|
||||
<p>Lowering a limit lets active operations finish; queued work waits for capacity. Player safety checks still apply.</p>
|
||||
<Button type="submit" disabled={saving}>{saving ? 'Saving…' : 'Save fleet settings'}</Button>
|
||||
</fieldset>
|
||||
</form>
|
||||
{:else}
|
||||
<Button onclick={loadSettings}>Retry loading</Button>
|
||||
{/if}
|
||||
</section>
|
||||
<section class="guide" aria-labelledby="display-heading">
|
||||
<h2 id="display-heading">Display preferences</h2>
|
||||
<label class="timezone">Timezone
|
||||
@@ -32,4 +89,7 @@
|
||||
p { font-size: var(--text-sm); color: var(--text-secondary); line-height: 1.65; }
|
||||
.timezone { display: grid; gap: 8px; margin-top: 16px; font-size: var(--text-sm); max-width: 420px; }
|
||||
.timezone select { width: 100%; padding: 10px 12px; font: inherit; color: var(--text-primary); background: var(--surface-sunken); border: 1px solid var(--border-control); border-radius: var(--radius-sm); }
|
||||
fieldset { border: 0; padding: 0; margin: 20px 0 0; min-width: 0; }
|
||||
fieldset label { display: block; margin: 20px 0 8px; font-size: var(--text-sm); }
|
||||
fieldset input { width: 160px; padding: 10px 12px; font: inherit; color: var(--text-primary); background: var(--surface-sunken); border: 1px solid var(--border-control); border-radius: var(--radius-sm); }
|
||||
</style>
|
||||
|
||||
@@ -66,6 +66,44 @@ async function states() {
|
||||
}
|
||||
|
||||
describe('shared creation queue', () => {
|
||||
it('rechecks a lowered limit after awaiting a startup in the same pass', async () => {
|
||||
await queueServers(3);
|
||||
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||
saveFleetSettings({ ...fleetSettings(), createConcurrency: 3 });
|
||||
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
|
||||
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
|
||||
vi.mocked(komodoCreateDeployment).mockImplementationOnce(async () => {
|
||||
saveFleetSettings({ ...fleetSettings(), createConcurrency: 1 });
|
||||
return { ok: true };
|
||||
});
|
||||
await processCreations();
|
||||
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
|
||||
expect(await states()).toEqual(['awaiting-health', 'queued', 'queued']);
|
||||
});
|
||||
|
||||
it('applies live increases and decreases while preserving active startups', async () => {
|
||||
const jobs = await queueServers(4);
|
||||
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
|
||||
const { komodoInspectHealthy, komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
|
||||
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, reason: 'still starting' });
|
||||
await processCreations();
|
||||
saveFleetSettings({ ...fleetSettings(), createConcurrency: 3 });
|
||||
await processCreations();
|
||||
expect(await states()).toEqual(['awaiting-health', 'awaiting-health', 'awaiting-health', 'queued']);
|
||||
saveFleetSettings({ ...fleetSettings(), createConcurrency: 1 });
|
||||
await processCreations();
|
||||
expect(komodoCreateDeployment).toHaveBeenCalledTimes(3);
|
||||
expect(await states()).toEqual(['awaiting-health', 'awaiting-health', 'awaiting-health', 'queued']);
|
||||
vi.mocked(komodoInspectHealthy).mockImplementation(async server => server === jobs[2].name
|
||||
? { ok: false, reason: 'still starting' } : { ok: true, state: 'running' });
|
||||
await processCreations();
|
||||
expect(await states()).toEqual(['done', 'done', 'awaiting-health', 'queued']);
|
||||
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
|
||||
await processCreations();
|
||||
expect(await states()).toEqual(['done', 'done', 'done', 'awaiting-health']);
|
||||
});
|
||||
|
||||
it('starts only one server per pass and holds its slot until a later health check', async () => {
|
||||
const jobs = await queueServers(3);
|
||||
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
|
||||
@@ -345,13 +383,14 @@ describe('shared creation queue', () => {
|
||||
describe('creation concurrency configuration', () => {
|
||||
it('defaults to one startup', async () => {
|
||||
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', undefined);
|
||||
const { config } = await import('../../src/lib/server/config.js');
|
||||
expect(config.createConcurrency).toBe(1);
|
||||
const { fleetSettings } = await import('../../src/lib/server/settings.js');
|
||||
expect(fleetSettings().createConcurrency).toBe(1);
|
||||
});
|
||||
|
||||
it.each(['0', '-1', '1.5', 'invalid', 'Infinity', '9007199254740992'])('rejects invalid limit %s', async value => {
|
||||
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', value);
|
||||
await expect(import('../../src/lib/server/config.js')).rejects.toThrow(/DEPLOYMENT_CREATE_CONCURRENCY/);
|
||||
const { fleetSettings } = await import('../../src/lib/server/settings.js');
|
||||
expect(() => fleetSettings()).toThrow(/Concurrent server startups/);
|
||||
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '1');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -51,6 +51,16 @@ function keysFor(preview: CollectionPreview) {
|
||||
}
|
||||
|
||||
describe('multi-server blueprints', () => {
|
||||
it('shows the saved startup limit in new previews and existing collection progress', async () => {
|
||||
const { blueprint, preview } = await previewLineup();
|
||||
const { queueCollection, getCollection, previewCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
const result = queueCollection({ previewId: preview.id, keys: keysFor(preview) });
|
||||
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||
saveFleetSettings({ ...fleetSettings(), createConcurrency: 3 });
|
||||
expect(previewCollection(blueprint.id).launchConcurrency).toBe(3);
|
||||
expect(getCollection(result.collectionId!)?.queue.concurrency).toBe(3);
|
||||
});
|
||||
|
||||
it('saves a mixed-game lineup and previews unique slots and global ports without reserving them', async () => {
|
||||
const { blueprint, preview } = await previewLineup();
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
|
||||
let fixture: Fixture;
|
||||
const defaults = { deployKeepDays: 7, restartConcurrency: 1, createConcurrency: 1 };
|
||||
beforeEach(async () => {
|
||||
vi.resetModules();
|
||||
fixture = await newFixture([]);
|
||||
for (const key of ['DEPLOY_KEEP_DAYS', 'DEPLOYMENT_RESTART_CONCURRENCY', 'DEPLOYMENT_CREATE_CONCURRENCY']) vi.stubEnv(key, undefined);
|
||||
});
|
||||
afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); });
|
||||
|
||||
describe('fleet settings API and persistence', () => {
|
||||
it('returns defaults and saves all settings across a module reload and database reopen', async () => {
|
||||
const { GET, PUT } = await import('../../src/routes/api/settings/+server.js');
|
||||
expect(await GET().json()).toEqual(defaults);
|
||||
const saved = { deployKeepDays: 30, restartConcurrency: 3, createConcurrency: 2 };
|
||||
const response = await PUT({ request: new Request('http://localhost/api/settings', { method: 'PUT', body: JSON.stringify(saved) }) });
|
||||
expect(response.status).toBe(200);
|
||||
expect(await response.json()).toEqual(saved);
|
||||
const { closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
closeDb(); vi.resetModules();
|
||||
// Once saved, even invalid legacy environment values are irrelevant.
|
||||
vi.stubEnv('DEPLOY_KEEP_DAYS', '-1');
|
||||
vi.stubEnv('DEPLOYMENT_RESTART_CONCURRENCY', 'bad');
|
||||
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '9');
|
||||
const reopened = await import('../../src/routes/api/settings/+server.js');
|
||||
expect(await reopened.GET().json()).toEqual(saved);
|
||||
});
|
||||
|
||||
it('imports legacy values into an existing database once and preserves unrelated settings', async () => {
|
||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
db().prepare("DELETE FROM settings WHERE key = 'fleet_settings'").run();
|
||||
db().prepare("INSERT INTO settings (key, value) VALUES ('unrelated', 'true')").run();
|
||||
closeDb();
|
||||
vi.stubEnv('DEPLOY_KEEP_DAYS', '14');
|
||||
vi.stubEnv('DEPLOYMENT_RESTART_CONCURRENCY', '4');
|
||||
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '2');
|
||||
const { fleetSettings } = await import('../../src/lib/server/settings.js');
|
||||
expect(fleetSettings()).toEqual({ deployKeepDays: 14, restartConcurrency: 4, createConcurrency: 2 });
|
||||
closeDb();
|
||||
vi.stubEnv('DEPLOY_KEEP_DAYS', '7');
|
||||
expect(fleetSettings().deployKeepDays).toBe(14);
|
||||
expect(db().prepare("SELECT value FROM settings WHERE key = 'unrelated'").get()).toEqual({ value: 'true' });
|
||||
});
|
||||
|
||||
it.each([
|
||||
'null', '[]', '{}', '{',
|
||||
...['deployKeepDays', 'restartConcurrency', 'createConcurrency'].flatMap(key =>
|
||||
[0, -1, 1.5, '2', true, null, 9007199254740992].map(value => JSON.stringify({ ...defaults, [key]: value }))),
|
||||
JSON.stringify({ ...defaults, deployKeepDays: 36501 }),
|
||||
JSON.stringify({ ...defaults, unknown: 1 })
|
||||
])('rejects invalid input without partially saving: %s', async body => {
|
||||
const { GET, PUT } = await import('../../src/routes/api/settings/+server.js');
|
||||
const response = await PUT({ request: new Request('http://localhost/api/settings', { method: 'PUT', body }) });
|
||||
expect(response.status).toBe(400);
|
||||
expect(await response.json()).toHaveProperty('error');
|
||||
expect(await GET().json()).toEqual(defaults);
|
||||
});
|
||||
});
|
||||
|
||||
it('applies saved retention to cleanup while protecting unfinished campaigns and provisioning history', async () => {
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||
const { sweep } = await import('../../src/lib/server/pipeline/sweep.js');
|
||||
const old = new Date(Date.now() - 10 * 86400000).toISOString();
|
||||
for (const id of ['completed', 'pending', 'failed', 'held']) {
|
||||
db().prepare("INSERT INTO deployments (id, action, started_at) VALUES (?, 'test', ?)").run(id, old);
|
||||
db().prepare('INSERT INTO deployment_servers (deployment_id, server, restart_state, files_changed) VALUES (?, ?, ?, 0)')
|
||||
.run(id, 'server-' + id, id === 'completed' ? 'done' : id);
|
||||
}
|
||||
db().prepare("INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by) VALUES ('history', 'removed', 'destroy', 'done', 'Complete', ?, 'ui')").run(old);
|
||||
saveFleetSettings({ ...fleetSettings(), deployKeepDays: 30 });
|
||||
await sweep();
|
||||
expect(db().prepare('SELECT id FROM deployments').all()).toHaveLength(4);
|
||||
saveFleetSettings({ ...fleetSettings(), deployKeepDays: 7 });
|
||||
await sweep();
|
||||
expect(db().prepare('SELECT id FROM deployments ORDER BY id').all()).toEqual([{ id: 'failed' }, { id: 'held' }, { id: 'pending' }]);
|
||||
expect(db().prepare("SELECT server FROM deployment_servers WHERE deployment_id = 'completed'").all()).toEqual([]);
|
||||
expect(db().prepare('SELECT id FROM server_deployments').all()).toEqual([{ id: 'history' }]);
|
||||
});
|
||||
@@ -438,10 +438,8 @@ describe('Provisioning ownership', () => {
|
||||
|
||||
describe('rollout configuration', () => {
|
||||
it('honors a configured limit larger than one', async () => {
|
||||
const { closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
closeDb();
|
||||
vi.resetModules();
|
||||
vi.stubEnv('DEPLOYMENT_RESTART_CONCURRENCY', '2');
|
||||
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||
saveFleetSettings({ ...fleetSettings(), restartConcurrency: 2 });
|
||||
await deployment();
|
||||
const { sweep } = await import('../helpers/provisioning.js');
|
||||
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
|
||||
@@ -451,13 +449,20 @@ describe('rollout configuration', () => {
|
||||
expect(komodoRestart).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it.each(['0', '-1', '1.5', 'abc'])('rejects invalid concurrency %s', async (value) => {
|
||||
const { closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
closeDb();
|
||||
vi.resetModules();
|
||||
vi.stubEnv('DEPLOYMENT_RESTART_CONCURRENCY', value);
|
||||
await expect(import('../../src/lib/server/config.js')).rejects.toThrow(/DEPLOYMENT_RESTART_CONCURRENCY/);
|
||||
vi.stubEnv('DEPLOYMENT_RESTART_CONCURRENCY', '1');
|
||||
vi.resetModules();
|
||||
it('holds new restarts after lowering a live limit without releasing active slots', async () => {
|
||||
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||
const { sweep } = await import('../helpers/provisioning.js');
|
||||
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
|
||||
saveFleetSettings({ ...fleetSettings(), restartConcurrency: 2 });
|
||||
await deployment();
|
||||
await sweep();
|
||||
saveFleetSettings({ ...fleetSettings(), restartConcurrency: 1 });
|
||||
await sweep();
|
||||
expect(komodoRestart).toHaveBeenCalledTimes(2);
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
expect(db().prepare('SELECT server FROM restart_rollout WHERE healthy_at IS NULL').all()).toHaveLength(2);
|
||||
nextTick();
|
||||
await sweep();
|
||||
expect(komodoRestart).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user