josh/electric-cherry
1
0
Fork 0
Code Issues 18 Pull Requests Actions Packages Projects Releases Wiki Activity

Clarify server actions and group maintenance failures

Browse Source
This commit is contained in:
josh
2026-10-07 19:21:53 -04:00
parent bcecc42dd1
commit c39563daef
15 changed files with 559 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guides/maintenance.md
+5 -1
View File
@@ -10,7 +10,11 @@ Campaigns retain managed-file deployment evidence; Collections define lineups.
- **Overview** shows active work, attention, upcoming restarts, and Plutonium version
coverage. Filter by capability or server. **Check now** refreshes health, drift,
the production release, and running revisions.
the production release, and running revisions. Failed checks are grouped by
cause with links to affected servers; raw paths and responses stay in
**Technical details**. Identical revision states share one summary row. Expand
it for each server’s check and pending restart, or use **Retry version check**
when the running revision is unverified.
- **Policies & schedules** controls health recovery, configuration drift, automatic
Plutonium updates, and the optional daily restart schedule independently.
- **Activity** combines recovery findings, update results, and scheduled restarts.
docs/guides/provisioning.md
+11 -7
View File
@@ -5,7 +5,11 @@ activity. **Fleet → Collections** owns reusable lineups and collection launch
progress; their individual operations remain visible in Provisioning.
Provisioning owns the queues, safety checks, execution, and
recovery for all six actions. Create a server using **Create server**. For an existing
server, choose **Manage server** or open its **Server actions** menu.
server, use its visible **Restart server** or **Start server** action, or open
**More actions**. Only actions supported by current host and container evidence
are shown. Confirmations name the server and its current players and round;
disruptive actions still wait for a fresh empty-lobby check. **Manage server**
on the operations page offers the same actions for a selected server.
**Create server** opens one dialog with game and map choices followed by automatic
readiness checks. Blockers explain what needs attention before launch. Runtime
@@ -29,18 +33,18 @@ retries when a response is lost.
| Restart | Wait until empty, restart the existing container, and verify container health and RCON. |
| Rebuild | Wait until empty, replace the instance through the host agent, preserving its server identity and port while advancing its generation. Verify container health, game status, map, and Plutonium revision. |
| Stop | Wait until empty, stop the container, and verify that the host agent reports it stopped. Keep the instance for a later Start. |
| Destroy | Wait until empty, remove the container and its files through the host agent, then retire the instance and registration. |
| Remove | Wait until empty, remove the container and its files through the host agent, then retire the instance and registration. |
## Safety and verification
### Beta ownership and rebuilds
Servers linked to a running or ready-to-promote beta cannot be destroyed or
Servers linked to a running or ready-to-promote beta cannot be removed or
rebuilt. Promote or roll back the beta first, including paused or failed test
plans. Fleet removal and exact collection launches queue nothing if any server
selected for removal belongs to an active beta. Provisioning checks again before
teardown; already queued removals wait while their server belongs to an active
beta. Opening a beta also requires cancelling or finishing any pending destruction
beta. Opening a beta also requires cancelling or finishing any pending removal
or rebuild on its selected servers.
Rebuild requires confirmation because **no instance data is preserved**. Its fresh
@@ -87,7 +91,7 @@ Administrators can add and remove available keys in **Admin → Keys**;
assigned keys cannot be removed. Existing keys are migrated into their game
pool, retaining assignments for existing servers and pending work. If a legacy
rebuild has an uncertain result and used a different key, both possible keys
remain reserved until that server is destroyed. Labels are for identification
remain reserved until that server is removed. Labels are for identification
only and do not change the hostname registered with Plutonium.
### Player checks and administrator overrides
@@ -210,7 +214,7 @@ heartbeat or completion-time promises.
`DELETE /api/servers/:name`.
- `/api/provisioning/:id`, `/:id/resume`, `/:id/cancel`, `/preview`, and
`/house-party` preserve the existing operation, recovery, removal cancellation,
launch preview, and fleet destruction APIs.
launch preview, and fleet removal APIs.
- `POST /api/provisioning/:id/cancel` cancels any active create, start, restart,
rebuild, stop, or destroy job, including health verification. It releases that
job's recovery capacity and prevents further steps; completed effects are not
@@ -250,7 +254,7 @@ See [native host management](native-hosts.md) for command delivery and recovery.
Admin → Settings groups Concurrency separately from history and storage settings,
with one Save changes action for all pending edits. Existing fleet restart, startup
and removal limits continue to apply. Additional per-host restart, startup and destruction limits, a combined
and removal limits continue to apply. Additional per-host restart, startup and removal limits, a combined
lifecycle limit, a rebuild limit, and a limit on hosts preparing game files default
to 0 (no additional cap), preserving existing throughput on upgrade.
src/lib/provisioning.ts
+3 -3
View File
@@ -2,16 +2,16 @@ export const provisioningActions = ['create', 'start', 'restart', 'rebuild', 'st
export type ProvisioningAction = typeof provisioningActions[number];
export type RuntimeAction = Exclude<ProvisioningAction, 'create' | 'destroy'>;
export const actionLabels: Record<ProvisioningAction, string> = {
create: 'Create', start: 'Start', restart: 'Restart', rebuild: 'Rebuild', stop: 'Stop', destroy: 'Destroy'
create: 'Create', start: 'Start', restart: 'Restart', rebuild: 'Rebuild', stop: 'Stop', destroy: 'Remove'
};
export const actionCompleted: Record<ProvisioningAction, string> = {
create: 'Server creation completed', start: 'Server started', restart: 'Server restarted',
rebuild: 'Server rebuilt', stop: 'Server stopped', destroy: 'Server destroyed'
rebuild: 'Server rebuilt', stop: 'Server stopped', destroy: 'Server removed'
};
export const runtimeActions: RuntimeAction[] = ['start', 'restart', 'rebuild', 'stop'];
export const runtimeDescriptions: Record<RuntimeAction, string> = {
start: 'Start the existing container and verify its health.',
restart: 'Restart the container after the server is empty, then verify its health.',
rebuild: 'Destroy the instance and delete all of its server files, then create a fresh instance in the same server slot and port with the latest promoted setup. No instance data is retained. Uses the key saved separately in Keys. Waits until empty, then verifies health and map.',
rebuild: 'Remove the instance and delete all of its server files, then create a fresh instance in the same server slot and port with the latest promoted setup. No instance data is retained. Uses the key saved separately in Keys. Waits until empty, then verifies health and map.',
stop: 'Stop the container after the server is empty. Its files, key, and registration are retained so you can start it again.'
};
src/lib/ui/components/ChatPanel.svelte
+13 -8
View File
@@ -6,8 +6,10 @@
import { pollVisible } from '$ui/poll.js';
import { timeFormat } from '$ui/stores/time-zone.js';
import Button from './primitives/Button.svelte';
import Freshness from './Freshness.svelte';
let { server, history }: { server?: string; history?: ContextualChatEntry[] } = $props();
let { server, history, hostOffline = false, hostName }: { server?: string; history?: ContextualChatEntry[]; hostOffline?: boolean; hostName?: string } = $props();
let updatedAt = $state<number | null>(null);
let liveEntries = $state<ContextualChatEntry[]>([]);
const entries = $derived(history ?? liveEntries);
let timeline = $derived(groupChatTimeline(entries));
@@ -24,7 +26,7 @@
let controller: AbortController | null = null;
async function refresh(manual = false) {
if (!server) return;
if (!server || hostOffline) return;
if (loading) {
if (manual) manualRefreshing = true;
return;
@@ -37,11 +39,11 @@
const response = await fetch(`/api/servers/${encodeURIComponent(server)}/chat`, { signal: request.signal });
const data = await response.json().catch(() => null);
if (!response.ok || !data?.ok) throw new Error('Could not refresh chat. Try again shortly.');
if (request.signal.aborted) return;
if (request.signal.aborted || hostOffline) return;
if (paused && !manual) return;
liveEntries = data.entries ?? [];
missing = !!data.missing;
liveLoaded = true;
liveLoaded = true; updatedAt = Date.now();
error = '';
await tick();
if (follow && body) body.scrollTop = body.scrollHeight;
@@ -50,6 +52,8 @@
} finally { loading = false; manualRefreshing = false; }
}
$effect(() => { if (hostOffline) controller?.abort(); });
function onScroll() {
if (body) follow = body.scrollHeight - body.scrollTop - body.clientHeight < 40;
}
@@ -93,17 +97,17 @@
<section aria-label="Player chat">
<div class="chat-head">
<div class="chat-title"><h2>Chat</h2><span class="update-cadence">{server ? paused ? 'Paused' : 'Live · 5s' : 'Match history'}</span></div>
<div class="chat-title"><h2>Chat</h2>{#if !server}<span class="update-cadence">Match history</span>{:else if hostOffline}<span class="update-cadence">Host offline</span>{:else if paused}<span class="update-cadence">Paused</span>{:else if !error}<Freshness {updatedAt} intervalLabel="every 5 s" />{/if}</div>
{#if server}
<div class="actions">
{#if hasMessages}<Button size="sm" variant="ghost" onclick={() => paused = !paused}>{paused ? 'Resume' : 'Pause'}</Button>{/if}
{#if !follow}<Button size="sm" variant="ghost" onclick={latest}>Latest messages</Button>{/if}
<Button size="sm" onclick={() => refresh(true)} loading={manualRefreshing} ariaLabel={manualRefreshing ? 'Refreshing chat' : 'Refresh chat'}>Refresh</Button>
</div>
{/if}
</div>
{#if error}<p class="ui-error" role="alert">{error} {loaded ? 'Showing the last loaded messages.' : ''}</p>{/if}
{#if !loaded && !error}
{#if hostOffline}<p class="outage">Chat resumes when {hostName ?? 'the host agent'} reconnects.{loaded ? ' Showing the last loaded messages.' : ''}</p>
{:else if error}<p class="ui-error" role="alert">Chat updates paused. {loaded ? 'Showing the last loaded messages.' : 'Could not load messages.'} <Button size="sm" onclick={() => refresh(true)} loading={manualRefreshing}>Retry</Button></p>{/if}
{#if !loaded && !error && !hostOffline}
<p class="empty" role="status">Loading chat…</p>
{:else if loaded && !hasMessages}
<p class="empty">{history !== undefined ? 'No chat messages were saved for this match.' : missing ? 'Waiting for the game log. Player messages will appear here.' : 'No player messages in this match yet.'}</p>
@@ -152,6 +156,7 @@
.chat-title { gap: var(--space-2) var(--space-3); }
h2 { margin: 0; font-size: var(--text-lg); }
.update-cadence { color: var(--text-secondary); font-size: var(--text-xs); }
.outage { color: var(--state-warning); font-size: var(--text-sm); line-height: var(--leading-body); }
.empty { color: var(--text-secondary); font-size: var(--text-sm); padding: 8px 0; margin-bottom: 0; }
.messages { max-height: 360px; overflow: auto; margin-top: var(--space-4); }
.messages:focus-visible { outline: none; box-shadow: inset 0 0 0 2px var(--accent-primary); border-radius: var(--radius-sm); }
src/lib/ui/components/DestroyServerModal.svelte
+20 -29
View File
@@ -1,15 +1,15 @@
<script lang="ts">
import { goto } from '$app/navigation';
import { lifecycleImpact } from '$ui/lifecycle-actions.js';
import { serverTitle } from '$ui/server-labels.js';
import Button from './primitives/Button.svelte';
import Dialog from './primitives/Dialog.svelte';
let { open = $bindable(false), name, players }: { open?: boolean; name: string; players: number | null } = $props();
let dialog = $state<HTMLDialogElement>();
let { open = $bindable(false), name, players, round = null }: { open?: boolean; name: string; players: number | null; round?: number | null } = $props();
let submitting = $state(false);
let error = $state('');
$effect(() => {
if (!dialog) return;
if (open && !dialog.open) { error = ''; dialog.showModal(); }
else if (!open && dialog.open) dialog.close();
if (open) error = '';
});
async function destroy() {
@@ -19,7 +19,7 @@
try {
const res = await fetch(`/api/servers/${encodeURIComponent(name)}`, { method: 'DELETE' });
const body = await res.json();
if (!res.ok || !body.ok || !body.deploymentId) throw new Error(body.error ?? 'Could not queue destruction. Please retry.');
if (!res.ok || !body.ok || !body.deploymentId) throw new Error(body.error ?? 'Could not queue server removal. Please retry.');
await goto(`/provisioning/${encodeURIComponent(body.deploymentId)}`);
open = false;
} catch (err) { error = (err as Error).message; }
@@ -27,33 +27,24 @@
}
</script>
<dialog bind:this={dialog} aria-labelledby="destroy-title" onclose={() => open = false}
oncancel={(event) => { if (submitting) event.preventDefault(); }}>
<form onsubmit={(event) => { event.preventDefault(); void destroy(); }}>
<h2 id="destroy-title">Queue server removal?</h2>
<p class="name">{name}</p>
<p>This stops and removes the container, its writable server files, and campaign details. Shared game libraries, provisioning history, and the saved server key are retained.</p>
<div class="notice">
{#if players !== null && players > 0}<strong>{players} player{players === 1 ? '' : 's'} currently online.</strong>{/if}
Removal waits until a live check confirms the server is empty. Follow progress or cancel the waiting request on the provisioning page.
</div>
<Dialog bind:open title={`Remove ${serverTitle(name)}?`} busy={submitting}>
<div class="fields">
<p class="notice">{lifecycleImpact('destroy', players, round)}</p>
<p>This stops and removes the container, its writable server files, and registration. Shared game libraries, recorded history, and the saved server key are retained.</p>
<p>Follow progress or cancel the waiting request on the provisioning page.</p>
<p class="irreversible">You can cancel while waiting. Once removal begins, it cannot be undone.</p>
{#if error}<p role="alert" class="error">{error}</p>{/if}
<footer>
<Button variant="ghost" disabled={submitting} onclick={() => open = false}>Cancel</Button>
<Button variant="danger" type="submit" loading={submitting}>Queue server removal</Button>
</footer>
</form>
</dialog>
</div>
{#snippet actions()}
<Button disabled={submitting} onclick={() => open = false}>Cancel</Button>
<Button variant="danger" disabled={submitting} loading={submitting} onclick={destroy}>Remove server when empty</Button>
{/snippet}
</Dialog>
<style>
dialog { width: min(520px, calc(100vw - 32px)); padding: 0; border: 1px solid var(--border-strong); border-radius: var(--radius-lg); background: var(--surface); color: var(--text-primary); }
dialog::backdrop { background: rgb(0 0 0 / .65); backdrop-filter: blur(4px); }
form { padding: 28px; } h2 { margin: 0 0 12px; font-size: 22px; }
p { line-height: 1.6; color: var(--text-secondary); }
.name { font-family: var(--font-mono); color: var(--text-primary); overflow-wrap: anywhere; }
.fields { display: grid; gap: 16px; }
p { margin: 0; line-height: 1.6; color: var(--text-secondary); }
.notice { padding: 16px; background: var(--surface-raised); border-radius: var(--radius-md); line-height: 1.6; }
strong { display: block; color: var(--state-warning); margin-bottom: 4px; }
.notice { color: var(--text-primary); }
.irreversible { font-size: 13px; } .error { color: var(--state-danger); }
footer { display: flex; justify-content: flex-end; flex-wrap: wrap; gap: 12px; margin-top: 24px; }
</style>
src/lib/ui/components/HousePartyDialog.svelte
+10 -9
View File
@@ -1,4 +1,5 @@
<script lang="ts">
import { serverTitle } from '$ui/server-labels.js';
import { isDialogBackdrop } from '$ui/dialog.js';
import Button from './primitives/Button.svelte';
@@ -36,7 +37,7 @@
const body = await res.json();
if (!res.ok) {
if (res.status === 409) servers = null;
throw new Error(body.error ?? 'Could not queue destruction.');
throw new Error(body.error ?? 'Could not queue server removal.');
}
open = false;
onqueued(body.deployments.length);
@@ -45,21 +46,21 @@
}
</script>
<dialog bind:this={dialog} aria-label="Destroy fleet" onclick={event => { if (isDialogBackdrop(event, dialog) && !busy) open = false; }} oncancel={event => { if (busy) event.preventDefault(); }} onclose={() => open = false}>
<dialog bind:this={dialog} aria-label="Remove fleet servers" onclick={event => { if (isDialogBackdrop(event, dialog) && !busy) open = false; }} oncancel={event => { if (busy) event.preventDefault(); }} onclose={() => open = false}>
<div class="content">
<header><h2>Queue fleet removal</h2></header>
<p>Queue destruction of every server, including servers being created. Each waits until a fresh player count confirms it is empty. Unreachable servers stay queued.</p>
<p>Destruction removes containers, server files, and fleet registrations. Saved keys and collections are kept.</p>
<header><h2>Remove fleet servers?</h2></header>
<p>Queue removal of every server, including servers being created. Each waits until a fresh player count confirms it is empty. Unreachable servers stay queued.</p>
<p>Removal deletes containers, server files, and fleet registrations. Saved keys and collections are kept.</p>
<p>In Provisioning’s Active tab, cancel individual waiting requests or all waiting removals. Once a server begins removal, that removal cannot be cancelled or undone.</p>
{#if error}<p class="error" role="alert">{error}</p>{/if}
{#if servers}
<strong>{servers.length ? `${servers.length} server${servers.length === 1 ? '' : 's'} to destroy` : 'The house is already empty.'}</strong>
{#if servers.length}<ul>{#each servers as server}<li>{server}</li>{/each}</ul><p class="caption">Existing destruction requests are reused. Servers added after this request are not included.</p>{/if}
<strong>{servers.length ? `${servers.length} server${servers.length === 1 ? '' : 's'} to remove` : 'No servers to remove.'}</strong>
{#if servers.length}<ul>{#each servers as server}<li>{serverTitle(server)}</li>{/each}</ul><p class="caption">Existing removal requests are reused. Servers added after this request are not included.</p>{/if}
{:else}<p>{busy ? 'Reviewing the fleet…' : 'Refresh the preview to review the current fleet.'}</p>{/if}
<div class="actions">
<Button variant="ghost" disabled={busy} onclick={() => open = false}>Cancel</Button>
{#if !servers}<Button disabled={busy} onclick={review}>Refresh preview</Button>
{:else}<Button variant="danger" loading={busy} disabled={busy || !servers.length} onclick={launch}>Queue destruction of {servers.length} server{servers.length === 1 ? '' : 's'}</Button>{/if}
{:else}<Button variant="danger" loading={busy} disabled={busy || !servers.length} onclick={launch}>Remove {servers.length} server{servers.length === 1 ? '' : 's'}</Button>{/if}
</div>
</div>
</dialog>
@@ -70,6 +71,6 @@
.content { display: grid; gap: 20px; padding: 28px; } h2, p { margin: 0; } h2 { font-size: 25px; }
p { color: var(--text-secondary); font-size: 13px; line-height: 1.6; } .error { color: var(--state-danger); } .caption { font-size: 12px; }
ul { margin: 0; padding: 12px 12px 12px 30px; max-height: 220px; overflow: auto; background: var(--surface-sunken); border-radius: var(--radius-md); }
li { padding: 5px 0; font-family: var(--font-mono); font-size: 12px; overflow-wrap: anywhere; }
li { padding: 5px 0; font-family: var(--font-body); font-size: 12px; overflow-wrap: anywhere; }
.actions { display: flex; gap: 12px; justify-content: flex-end; flex-wrap: wrap; }
</style>
src/lib/ui/components/LifecycleDialog.svelte
+73 -22
View File
@@ -1,41 +1,79 @@
<script lang="ts">
import { untrack } from 'svelte';
import { goto } from '$app/navigation';
import { runtimeActions, actionLabels, runtimeDescriptions, type RuntimeAction } from '$lib/provisioning.js';
import { currentRound, freshServer, type FleetServer } from '$lib/fleet-overview.js';
import { actionLabels, runtimeDescriptions, type RuntimeAction } from '$lib/provisioning.js';
import { applicableRuntimeActions, lifecycleImpact } from '$ui/lifecycle-actions.js';
import { serverHealth } from '$ui/server-health.js';
import { serverTitle } from '$ui/server-labels.js';
import { serverTimeNow, syncServerTime, uiNow } from '$ui/stores/servers.js';
import Dialog from './primitives/Dialog.svelte';
import Button from './primitives/Button.svelte';
let { open = $bindable(false), initialServer = '', initialAction = 'start' }:
{ open?: boolean; initialServer?: string; initialAction?: RuntimeAction } = $props();
let servers = $state<{ name: string }[]>([]);
let server = $state('');
let action = $state<RuntimeAction | 'destroy'>('start');
let { open = $bindable(false), initialServer = '', initialAction = 'start', serverContext = null, players = null, round = null }:
{ open?: boolean; initialServer?: string; initialAction?: RuntimeAction; serverContext?: FleetServer | null; players?: number | null; round?: number | null } = $props();
let servers = $state<FleetServer[]>([]);
let server = $state(untrack(() => initialServer));
let action = $state<RuntimeAction | 'destroy'>(untrack(() => initialAction));
let busy = $state(false);
let loading = $state(false);
let error = $state('');
let confirmed = $state(false);
const selected = $derived(serverContext?.name === server ? serverContext : servers.find(item => item.name === server));
const available = $derived(selected ? applicableRuntimeActions(selected, $uiNow) : []);
const applicable = $derived(action === 'destroy' || available.includes(action));
const destructive = $derived(action === 'rebuild' || action === 'destroy');
const health = $derived(selected ? serverHealth(selected, $uiNow) : null);
const impactPlayers = $derived(initialServer ? players : selected && health?.state === 'online' && freshServer(selected, $uiNow) ? selected.last_players : null);
const impactRound = $derived(initialServer ? round : selected ? currentRound(selected, selected.telemetry, $uiNow) : null);
const title = $derived(initialServer ? `${actionLabels[action]} ${serverTitle(initialServer)}?` : 'Manage server');
const submitLabel = $derived(action === 'start' ? 'Start server' : action === 'destroy' ? 'Remove server when empty' : `${actionLabels[action]} when empty`);
$effect(() => {
if (open) { server = initialServer; action = initialAction; confirmed = false; error = ''; void load(); }
if (!open) return;
const fixedServer = initialServer;
const requestedAction = initialAction;
const controller = new AbortController();
// Live context and match reports update the copy and eligibility without
// reopening the dialog or clearing a deliberate confirmation.
untrack(() => {
server = fixedServer; action = requestedAction; confirmed = false; error = ''; servers = [];
loading = false;
if (!fixedServer || serverContext?.name !== fixedServer) void load(fixedServer, controller.signal);
});
async function load() {
return () => controller.abort();
});
async function load(fixedServer: string, signal: AbortSignal) {
loading = true;
try {
const response = await fetch('/api/servers');
const response = await fetch(fixedServer ? `/api/servers/${encodeURIComponent(fixedServer)}` : '/api/servers', { signal });
if (!response.ok) throw new Error('Could not load servers.');
servers = (await response.json()).servers;
} catch (err) { error = (err as Error).message; }
finally { loading = false; }
const body = await response.json();
if (signal.aborted) return;
servers = body.servers;
syncServerTime(body.serverNow);
} catch (err) { if (!signal.aborted) error = (err as Error).message; }
finally { if (!signal.aborted) loading = false; }
}
function chooseServer(event: Event & { currentTarget: EventTarget & HTMLSelectElement }) {
server = event.currentTarget.value;
confirmed = false; error = '';
const item = servers.find(item => item.name === server);
const actions = item ? applicableRuntimeActions(item, serverTimeNow()) : [];
action = actions[0] ?? 'destroy';
}
async function submit() {
if (busy || !server || (destructive && !confirmed)) return;
if (action !== 'destroy' && (!selected || !applicableRuntimeActions(selected, serverTimeNow()).includes(action))) {
error = 'The server status has changed. Close this dialog and choose an action from its current status.';
return;
}
busy = true; error = '';
try {
const response = action === 'destroy'
? await fetch(`/api/servers/${encodeURIComponent(server)}`, { method: 'DELETE' })
: await fetch('/api/provisioning', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ server, action }) });
const body = await response.json();
if (!response.ok) throw new Error(body.error ?? 'Could not queue this operation.');
if (!response.ok || !body.deploymentId) throw new Error(body.error ?? 'Could not queue this operation.');
await goto(`/provisioning/${encodeURIComponent(body.deploymentId)}`);
open = false;
} catch (err) { error = (err as Error).message; }
@@ -43,23 +81,34 @@
}
</script>
<Dialog bind:open title="Manage server" {busy}>
<Dialog bind:open {title} {busy}>
<div class="fields">
<label>Server <select bind:value={server} disabled={busy || loading} onchange={() => confirmed = false}>
{#if !initialServer}
<label>Server <select bind:value={server} disabled={busy || loading} onchange={chooseServer}>
<option value="">{loading ? 'Loading servers…' : 'Choose a server'}</option>
{#each servers as item}<option value={item.name}>{serverTitle(item.name)} · {item.name}</option>{/each}
{#each servers as item (item.name)}<option value={item.name}>{serverTitle(item.name)}</option>{/each}
</select></label>
<label>Action <select bind:value={action} disabled={busy} onchange={() => confirmed = false}>
{#each runtimeActions as value}<option value={value}>{actionLabels[value]}</option>{/each}
<option value="destroy">Destroy</option>
<label>Action <select bind:value={action} disabled={busy || !server || loading} onchange={() => { confirmed = false; error = ''; }}>
{#if !server}<option value="start">Choose a server first</option>{/if}
{#if server && !applicable}<option value={action} disabled>{actionLabels[action]} unavailable</option>{/if}
{#each available as value}<option value={value}>{actionLabels[value]}</option>{/each}
{#if server}<option value="destroy">Remove server</option>{/if}
</select></label>
<p>{action === 'destroy' ? 'Destroy the container and remove its writable server files and registration after the server is empty. Shared game libraries, provisioning history, and the key saved separately in Keys are retained.' : runtimeDescriptions[action]}</p>
{#if destructive}<label class="confirmation"><input type="checkbox" bind:checked={confirmed} disabled={busy} /> I understand that all files and data in this instance will be permanently deleted.</label>{/if}
{/if}
{#if loading}<p role="status">Checking the latest server status…</p>
{:else if server}
{#if applicable}
<p class="impact">{lifecycleImpact(action, impactPlayers, impactRound)}</p>
<p>{action === 'destroy' ? 'Remove the container, its writable server files, and registration after the server is empty. Shared game libraries, recorded history, and the key saved separately in Keys are retained.' : runtimeDescriptions[action]}</p>
{#if action !== 'start'}<p class="help">Follow progress or cancel the waiting request on the provisioning page.</p>{/if}
{:else}<p role="status">{actionLabels[action]} is unavailable. {health?.reason ?? 'The current server status could not be verified.'} Close this dialog and choose an action from its current status.</p>{/if}
{#if applicable && destructive}<label class="confirmation"><input type="checkbox" bind:checked={confirmed} disabled={busy} /> I understand that this instance’s writable files and data will be permanently deleted.</label>{/if}
{/if}
{#if error}<p class="ui-error" role="alert">{error}</p>{/if}
</div>
{#snippet actions()}
<Button disabled={busy} onclick={() => open = false}>Cancel</Button>
<Button variant={destructive ? 'danger' : 'primary'} disabled={busy || loading || !server || (destructive && !confirmed)} loading={busy} onclick={submit}>Queue {actionLabels[action].toLowerCase()}</Button>
<Button variant={destructive ? 'danger' : 'primary'} disabled={busy || loading || !server || !applicable || (destructive && !confirmed)} loading={busy} onclick={submit}>{submitLabel}</Button>
{/snippet}
</Dialog>
@@ -67,4 +116,6 @@
.fields { display: grid; gap: 16px; } label { display: grid; gap: 8px; }
select { padding: 10px; border: 1px solid var(--border-strong); border-radius: var(--radius-sm); background: var(--surface); color: var(--text-primary); }
p { margin: 0; line-height: 1.6; color: var(--text-secondary); } .confirmation { display: flex; align-items: flex-start; gap: 12px; line-height: 1.5; }
.impact { padding: var(--space-4); border-radius: var(--radius-md); background: var(--surface-raised); color: var(--text-primary); }
.help { font-size: var(--text-sm); }
</style>
src/lib/ui/lifecycle-actions.ts
+28
View File
@@ -0,0 +1,28 @@
import type { FleetServer } from '../fleet-overview.js';
import { actionLabels, type RuntimeAction } from '../provisioning.js';
import { freshStatusTimestamp } from '../server-freshness.js';
import { serverHealth } from './server-health.js';
/** UI affordances only: the queue still checks ownership, conflicts and occupancy. */
export function applicableRuntimeActions(server: FleetServer, now: number): RuntimeAction[] {
const health = serverHealth(server, now).state;
if (health === 'stopped') return ['start', 'rebuild'];
const currentRunningContainer = server.container_state === 'running'
&& freshStatusTimestamp(server.container_checked_at ?? null, now)
&& (!server.telemetry_after || Date.parse(server.container_checked_at!) > Date.parse(server.telemetry_after));
if (health === 'online' || health === 'not-responding' && currentRunningContainer) return ['restart', 'stop', 'rebuild'];
return [];
}
/** Counts must come from fresh observations; null must never read as an empty lobby. */
export function lifecycleImpact(action: RuntimeAction | 'destroy', players: number | null, round: number | null): string {
if (action === 'start') return 'Start the existing container, then verify that the game is responding.';
const label = action === 'destroy' ? 'Removal' : actionLabels[action];
if (players === null || !Number.isSafeInteger(players) || players < 0) {
return `Player count is unknown. ${label} waits for a live check confirming the lobby is empty.`;
}
if (players === 0) return `The lobby is currently empty. A live check will confirm this before ${label.toLowerCase()} begins.`;
const subject = `${players.toLocaleString('en-US')} ${players === 1 ? 'player is' : 'players are'}`;
const activity = round !== null && Number.isSafeInteger(round) && round >= 0 ? `in round ${round.toLocaleString('en-US')}` : 'currently online';
return `${subject} ${activity}. ${label} waits until the lobby is empty.`;
}
src/lib/ui/maintenance-presentation.ts
+86
View File
@@ -0,0 +1,86 @@
import type { MaintenanceJob, PlutoniumVersion } from '../plutonium.js';
import type { StatusTone } from './status.js';
export interface ScanErrorGroup {
key: string;
title: string;
description: string;
servers: string[];
details: { server: string | null; message: string }[];
}
const scanCauses = [
{ key: 'host', pattern: /(?:host|agent).*(?:offline|unavailable|disconnect|not connected)/i,
title: 'Host agent unavailable', description: 'The host agent could not complete the checks. Check its connection, then retry.' },
{ key: 'timeout', pattern: /timed?\s*out|timeout|no reply/i,
title: 'Checks timed out', description: 'The check did not receive a response in time. Check the server connection, then retry.' },
{ key: 'permissions', pattern: /permission denied|access denied|EACCES|EPERM/i,
title: 'File access denied', description: 'The host agent could not read files needed for the check. Review file permissions on the affected host.' },
{ key: 'files', pattern: /no such file|file.*not found|ENOENT|missing (?:file|directory)|cannot find.*file/i,
title: 'Files unavailable', description: 'Files needed for the check could not be found. Review the affected server’s files and retry.' },
{ key: 'response', pattern: /bad.reply|invalid (?:response|reply)|malformed|could not parse/i,
title: 'Unexpected response', description: 'The response could not be read. Inspect the connection and host logs before retrying.' },
{ key: 'setup', pattern: /register a native host instance/i,
title: 'No managed servers', description: 'Register a host and create a server before running automatic maintenance checks.' }
];
const otherCause = { key: 'other', title: 'Other check failures', description: 'Some checks could not finish. Review the technical details before retrying.' };
/** Worker errors use "server: detail" lines. Only known names become links. */
export function groupMaintenanceScanErrors(raw: string | null | undefined, knownServers: readonly string[]): ScanErrorGroup[] {
if (!raw?.trim()) return [];
const names = [...new Set(knownServers)].sort((a, b) => b.length - a.length);
const groups = new Map<string, ScanErrorGroup>();
for (const line of raw.split(/\r?\n/).map(value => value.trim()).filter(Boolean)) {
const server = names.find(name => line.startsWith(`${name}:`)) ?? null;
const message = server ? line.slice(server.length + 1).trim() : line;
const cause = scanCauses.find(cause => cause.pattern.test(message)) ?? otherCause;
let group = groups.get(cause.key);
if (!group) {
group = { key: cause.key, title: cause.title, description: cause.description, servers: [], details: [] };
groups.set(cause.key, group);
}
if (server && !group.servers.includes(server)) group.servers.push(server);
if (!group.details.some(detail => detail.server === server && detail.message === message)) group.details.push({ server, message });
}
return [...groups.values()];
}
type VersionStatus = NonNullable<PlutoniumVersion['update_status']>;
const versionStatuses: Record<VersionStatus, { label: string; tone: StatusTone }> = {
current: { label: 'Up to date', tone: 'online' },
outdated: { label: 'Update available', tone: 'warning' },
ahead: { label: 'Ahead of published release', tone: 'neutral' },
stopped: { label: 'Intentionally stopped', tone: 'neutral' },
unknown: { label: 'Unverified', tone: 'warning' }
};
export interface MaintenanceVersionGroup {
key: string;
revision: number | null;
status: VersionStatus;
label: string;
tone: StatusTone;
versions: PlutoniumVersion[];
pending: MaintenanceJob[];
updatable: string[];
}
/** Keep verification and pending work distinct even when revision numbers match. */
export function groupMaintenanceVersions(versions: readonly PlutoniumVersion[], jobs: readonly MaintenanceJob[]): MaintenanceVersionGroup[] {
const groups = new Map<string, MaintenanceVersionGroup>();
const pending = jobs.filter(job => !job.server_deleted_at && ['queued', 'restarting'].includes(job.state));
for (const version of versions) {
const status = version.update_status ?? (version.stopped ? 'stopped' : 'unknown');
const key = `${version.revision ?? 'unknown'}:${status}`;
let group = groups.get(key);
if (!group) {
group = { key, revision: version.revision, status, ...versionStatuses[status], versions: [], pending: [], updatable: [] };
groups.set(key, group);
}
group.versions.push(version);
const work = pending.filter(job => job.server === version.server);
group.pending.push(...work);
if (status === 'outdated' && !work.length) group.updatable.push(version.server);
}
const order: Record<VersionStatus, number> = { unknown: 0, outdated: 1, current: 2, ahead: 3, stopped: 4 };
return [...groups.values()].sort((a, b) => order[a.status] - order[b.status] || (b.revision ?? 0) - (a.revision ?? 0));
}
src/routes/maintenance/+page.svelte
+69 -15
View File
@@ -15,9 +15,13 @@
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
import Tabs from '$ui/components/primitives/Tabs.svelte';
import Badge from '$ui/components/primitives/Badge.svelte';
import StatusBadge from '$ui/components/primitives/StatusBadge.svelte';
import StatusPip from '$ui/components/primitives/StatusPip.svelte';
import Button from '$ui/components/primitives/Button.svelte';
import Freshness from '$ui/components/Freshness.svelte';
import EmptyState from '$ui/components/primitives/EmptyState.svelte';
import ListFooter from '$ui/components/primitives/ListFooter.svelte';
import { groupMaintenanceScanErrors, groupMaintenanceVersions } from '$ui/maintenance-presentation.js';
let data = $state<MaintenanceOverview | null>(null);
let error = $state('');
@@ -34,6 +38,7 @@
let scheduleLoaded = false;
let displayedZone = $state('');
let request = 0;
let checkedAt = $state<number | null>(null);
const tabs = [{ id: 'overview', label: 'Overview' }, { id: 'policies', label: 'Policies & schedules' }, { id: 'activity', label: 'Activity' }];
const policies: { id: MaintenancePolicy; title: string; description: string }[] = [
{ id: 'health', title: 'Health recovery', description: 'Restart unhealthy servers, then rebuild once if recovery fails. Rebuilding deletes instance files and installs the latest promoted setup.' },
@@ -44,6 +49,9 @@
const outdated = $derived(versions.filter(v => v.update_status === 'outdated'));
const unknown = $derived(versions.filter(v => v.update_status === 'unknown'));
const activeJobs = $derived(data?.runtime.jobs.filter(j => !j.server_deleted_at && ['queued', 'restarting'].includes(j.state)) ?? []);
const scanErrors = $derived(groupMaintenanceScanErrors(data?.recovery.scanError, data?.runtime.servers.map(version => version.server) ?? []));
const versionGroups = $derived(groupMaintenanceVersions(versions, activeJobs));
const updatableServers = $derived(versionGroups.flatMap(group => group.updatable));
const activities = $derived([
...(data?.recovery.incidents ?? []).map(i => ({
key: i.id, kind: i.kind, server: i.server, server_deleted_at: i.server_deleted_at, at: i.first_seen_at,
@@ -92,6 +100,7 @@
}
function receive(body: MaintenanceOverview) {
data = body;
checkedAt = Date.now();
if (!scheduleLoaded) { scheduleEnabled = body.runtime.schedule.enabled; scheduleLoaded = true; }
}
async function load() {
@@ -129,8 +138,9 @@
{/snippet}
</SectionHeader>
{#if error}<p class="ui-error" role="alert">{error} {data ? 'Showing the last successful refresh.' : ''} <Button size="sm" onclick={load} disabled={busy}>Retry refresh</Button></p>{/if}
{#if !error}<Freshness updatedAt={checkedAt} intervalLabel="Updates every 5 seconds" />{/if}
{#if notice}<p role="status">{notice}</p>{/if}
{#if data?.policies.paused}<div class="paused"><strong>All maintenance is paused.</strong> Checks and verification continue. New actions, scheduled runs, and queued maintenance are held. Submitted operations finish recovery. Campaigns, betas, and manual Provisioning actions continue.</div>{/if}
{#if data?.policies.paused}<div class="paused"><strong>All maintenance is paused.</strong> Checks and verification continue. New actions, scheduled runs, and queued maintenance are held. Submitted operations finish recovery. Campaigns, betas, and manual server operations continue.</div>{/if}
{#if $page.url.searchParams.has('returnTo')}<BackLink fallback="/provisioning" />{/if}
<Tabs {tabs} bind:value={view} onchange={() => saveContext()} ariaLabel="Maintenance views" idPrefix="maintenance-tabs" panelId="maintenance-panel" />
<div id="maintenance-panel" role="tabpanel" aria-labelledby={`maintenance-tabs-${view}`} class="panels">
@@ -159,10 +169,21 @@
</section>
{:else}
{#if view === 'overview'}
<div class="summary"><span><strong>{attention}</strong> need attention</span><span><strong>{waiting}</strong> active</span><span><strong>{outdated.length}</strong> outdated</span><span><strong>{unknown.length}</strong> versions unverified</span></div>
<div class="summary"><span><strong>{attention}</strong> {attention === 1 ? 'needs' : 'need'} attention</span><span><strong>{waiting}</strong> active</span><span><strong>{outdated.length}</strong> outdated</span><span><strong>{unknown.length}</strong> {unknown.length === 1 ? 'version' : 'versions'} unverified</span></div>
{#if data.runtime.schedule.nextAt}<p class="muted">Next scheduled restart: {$timeFormat.dateTime(data.runtime.schedule.nextAt)} · <a href="/maintenance?view=policies#schedule">Manage schedule</a></p>{/if}
{#if data.recovery.lastScanAt}<p class="muted">Health and drift checked {$timeFormat.dateTime(data.recovery.lastScanAt)}</p>{/if}
{#if data.recovery.scanError}<p class="ui-error" role="alert">{data.recovery.scanError}</p>{/if}
{#if scanErrors.length}
<section class="scan-errors" aria-labelledby="scan-errors-heading">
<div class="row-heading"><h2 id="scan-errors-heading">Checks need attention</h2><Button size="sm" disabled={busy} onclick={() => act({ action: 'scan' })}>Retry checks</Button></div>
<ul class="error-groups">{#each scanErrors as group (group.key)}
<li><h3><StatusPip tone="warning" />{group.title}{group.servers.length ? ` · ${group.servers.length} ${group.servers.length === 1 ? 'server' : 'servers'}` : ''}</h3>
<p>{group.description}</p>
{#if group.servers.length}<ul class="affected-servers">{#each group.servers as name}<li><a href={contextHref(`/servers/${encodeURIComponent(name)}?tab=diagnostics`, $page.url)}>{serverTitle(name)}</a></li>{/each}</ul>{/if}
<details><summary>Technical details</summary><ul class="technical-details">{#each group.details as detail}<li>{#if detail.server}<strong>{serverTitle(detail.server)}</strong>{/if}<pre>{detail.message}</pre></li>{/each}</ul></details>
</li>
{/each}</ul>
</section>
{/if}
{/if}
<div class="filters">
<label>Capability<select bind:value={kind} onchange={() => saveContext()}><option value="all">All capabilities</option><option value="health">Health recovery</option><option value="drift">Configuration drift</option><option value="updates">Plutonium updates</option><option value="schedule">Scheduled restarts</option></select></label>
@@ -171,19 +192,34 @@
</div>
{#if view === 'overview' && ['all', 'updates'].includes(kind)}
<section class="card" id="updates">
<div class="row-heading"><h2>Plutonium updates</h2><Button disabled={busy || data.policies.paused || !outdated.some(v => !activeJobs.some(j => j.server === v.server))} onclick={() => act({ action: 'update', servers: outdated.map(v => v.server) })}>Update outdated servers on restart</Button></div>
<div class="row-heading"><h2>Plutonium updates</h2>{#if updatableServers.length}<Button disabled={busy || data.policies.paused} onclick={() => act({ action: 'update', servers: updatableServers })}>Update outdated servers on restart</Button>{/if}</div>
<p>Published revision: <strong>{data.runtime.release.revision ? `r${data.runtime.release.revision}` : 'Unknown'}</strong> · <a href={PLUTONIUM_MANIFEST_URL} target="_blank" rel="noreferrer">Production manifest</a> · <a href="https://plutonium.pw/docs/changelog/" target="_blank" rel="noreferrer">Changelog</a></p>
{#if data.runtime.release.checkedAt}<p class="muted">Checked {$timeFormat.dateTime(data.runtime.release.checkedAt)}</p>{/if}
{#if data.runtime.release.error}<p class="ui-error" role="alert">{data.runtime.release.error} Displayed release information is cached.</p>{/if}
<p class="muted">Plutainer applies the available release at startup when auto-update is enabled. Maintenance verifies the running revision after recovery. Unknown versions require a successful check before automatic updates.</p>
<div class="table-wrap"><table><thead><tr><th>Server</th><th>Running revision</th><th>Status</th><th>Action</th></tr></thead><tbody>
{#each versions as version (version.server)}
{@const pending = activeJobs.find(j => j.server === version.server)}
<tr><td><a href={`/servers/${encodeURIComponent(version.server)}`}><ServerIdentity name={version.server} /></a></td><td>{version.revision ? `r${version.revision}` : 'Unknown'}{#if version.checked_at}<small>{$timeFormat.dateTime(version.checked_at)}</small>{/if}{#if version.error}<small>{version.error}</small>{/if}</td>
<td>{version.update_status === 'outdated' ? 'Update available' : version.update_status === 'current' ? 'Up to date' : version.update_status === 'ahead' ? 'Ahead of published release' : version.update_status === 'stopped' ? 'Intentionally stopped' : 'Unverified'}</td>
<td>{#if pending}<a href={pending.provisioning_id ? `/provisioning/${encodeURIComponent(pending.provisioning_id)}` : '#activity'}>Restart pending</a>{:else if version.update_status === 'outdated'}<Button size="sm" disabled={busy || data.policies.paused} onclick={() => act({ action: 'update', servers: [version.server] })}>Update on restart</Button>{/if}</td></tr>
{:else}<tr><td colspan="4">No matching fleet servers.</td></tr>{/each}
</tbody></table></div>
<div class="version-groups">{#each versionGroups as group (group.key)}
<article class="version-group">
<div class="row-heading">
<div class="version-summary"><strong>{group.versions.length} {group.versions.length === 1 ? 'server' : 'servers'} · {group.revision ? `reported r${group.revision}` : 'revision unknown'}</strong><Badge variant={group.tone}><StatusPip tone={group.tone} size="sm" />{group.label}</Badge></div>
<div class="version-actions">
{#if group.pending.length}<span>{group.pending.length} {group.pending.length === 1 ? 'restart' : 'restarts'} pending</span>{/if}
{#if group.status === 'unknown'}<Button size="sm" disabled={busy} onclick={() => act({ action: 'scan' })}>Retry version check</Button>
{:else if group.updatable.length}<Button size="sm" disabled={busy || data.policies.paused} onclick={() => act({ action: 'update', servers: group.updatable })}>Update {group.updatable.length === 1 ? 'server' : `${group.updatable.length} servers`} on restart</Button>{/if}
</div>
</div>
<details><summary>{group.versions.length === 1 ? 'Server and check details' : `Show ${group.versions.length} servers and check details`}</summary>
<ul class="version-members">{#each group.versions as version (version.server)}
{@const pending = group.pending.find(job => job.server === version.server)}
<li><div class="row-heading"><a href={contextHref(`/servers/${encodeURIComponent(version.server)}`, $page.url)}><ServerIdentity name={version.server} /></a>
{#if pending}<span class="pending-job"><StatusBadge state={pending.state} />{#if pending.provisioning_id}<a href={contextHref(`/provisioning/${encodeURIComponent(pending.provisioning_id)}`, $page.url)}>View restart</a>{:else}<a href="#activity">View pending restart</a>{/if}</span>{/if}
</div>
{#if version.checked_at}<small>Last checked <time datetime={version.checked_at}>{$timeFormat.dateTime(version.checked_at)}</time></small>{/if}
{#if version.error}<pre class="version-error">{version.error}</pre>{/if}
</li>
{/each}</ul>
</details>
</article>
{:else}<EmptyState title="No matching servers" description="Choose another server name to see its reported Plutonium revision." />{/each}</div>
</section>
{/if}
<section id="activity">
@@ -215,11 +251,29 @@
.summary, .paused { padding: 20px; background: var(--surface-raised); border: 1px solid var(--border-strong); border-radius: var(--radius-md); }
.paused { line-height: 1.7; }
h2 { font-size: var(--text-md); } h3 { font-size: var(--text-sm); } h2, h3, p { margin: 0; }
p, label, td, th { font-size: var(--text-sm); line-height: 1.6; }
p, label { font-size: var(--text-sm); line-height: 1.6; }
.muted, small { color: var(--text-secondary); } small { display: block; }
a { color: var(--accent-primary-hover); } label { display: grid; gap: 6px; } .toggle { display: flex; align-items: center; }
input:not([type='checkbox']), select { color: inherit; background: var(--surface); padding: 9px; border: 1px solid var(--border-strong); border-radius: var(--radius-sm); max-width: 100%; }
.table-wrap { overflow-x: auto; } table { width: 100%; border-collapse: collapse; text-align: left; } td, th { padding: 12px 8px; border-bottom: 1px solid var(--border-subtle); }
input:not([type='checkbox']), select { color: inherit; background: var(--surface-sunken); padding: 9px; border: 1px solid var(--border-control); border-radius: var(--radius-sm); max-width: 100%; }
.scan-errors { padding: var(--space-4); border: 1px solid color-mix(in srgb, var(--state-warning) 35%, var(--border-subtle)); border-radius: var(--radius-md); background: var(--surface); }
.error-groups, .technical-details, .version-members { list-style: none; padding: 0; margin: 0; }
.error-groups { display: grid; gap: var(--space-4); }
.error-groups > li + li { border-top: 1px solid var(--border-subtle); padding-top: var(--space-4); }
.error-groups h3 { display: flex; align-items: center; gap: var(--space-2); margin-bottom: var(--space-2); }
.error-groups p { color: var(--text-secondary); }
.affected-servers { display: flex; flex-wrap: wrap; gap: var(--space-2) var(--space-4); padding: 0; margin: var(--space-3) 0; list-style: none; font-size: var(--text-sm); }
details { margin-top: var(--space-2); min-width: 0; }
summary { cursor: pointer; color: var(--text-secondary); font-size: var(--text-xs); }
.technical-details > li { margin-top: var(--space-3); font-size: var(--text-xs); }
pre { max-width: 100%; white-space: pre-wrap; overflow-wrap: anywhere; font-family: var(--font-mono); font-size: var(--text-xs); line-height: var(--leading-mono); color: var(--text-secondary); margin: var(--space-2) 0 0; }
.version-groups { display: grid; gap: var(--space-2); }
.version-group { min-width: 0; padding: var(--space-3) var(--space-4); border: 1px solid var(--border-subtle); border-radius: var(--radius-md); }
.version-summary, .version-actions, .pending-job { display: flex; align-items: center; flex-wrap: wrap; gap: var(--space-2); font-size: var(--text-sm); }
.version-summary strong { font-weight: var(--weight-medium); font-variant-numeric: tabular-nums; }
.version-actions { font-size: var(--text-xs); color: var(--text-secondary); }
.version-members > li { padding: var(--space-3) 0; }
.version-members > li + li { border-top: 1px solid var(--border-subtle); }
.version-members small { margin-top: var(--space-1); font-size: var(--text-xs); }
.activity article, section { scroll-margin-top: 100px; }
@media (max-width: 640px) { .card { padding: 18px; } }
</style>
src/routes/provisioning/+page.svelte
+1 -1
View File
@@ -158,7 +158,7 @@
<Button onclick={() => lifecycleOpen = true}>Server actions</Button><Button variant="primary" onclick={() => open = true}>Create server</Button>
<details class="fleet-actions" bind:open={fleetActionsOpen}><summary>Fleet actions</summary><div class="fleet-menu">
{#if active.some(row => row.canCancel)}<Button disabled={cancelling} onclick={() => cancelRequests(active.filter(row => row.canCancel && !row.server_deleted_at), true)}>Cancel all active operations</Button>{/if}
<Button variant="danger" onclick={() => { fleetActionsOpen = false; housePartyOpen = true; }}>Destroy fleet…</Button>
<Button variant="danger" onclick={() => { fleetActionsOpen = false; housePartyOpen = true; }}>Remove fleet servers…</Button>
</div></details>
{/snippet}
</SectionHeader>
src/routes/servers/[name]/+page.svelte
+37 -16
View File
@@ -23,9 +23,11 @@
import { page } from '$app/stores';
import { goto } from '$app/navigation';
import LifecycleDialog from '$ui/components/LifecycleDialog.svelte';
import { runtimeActions, actionLabels, type RuntimeAction } from '$lib/provisioning.js';
import { actionLabels, type RuntimeAction } from '$lib/provisioning.js';
let lifecycleOpen = $state(false);
let lifecycleAction = $state<RuntimeAction>('start');
import { applicableRuntimeActions } from '$ui/lifecycle-actions.js';
import { serverHealth } from '$ui/server-health.js';
import DestroyServerModal from '$ui/components/DestroyServerModal.svelte';
import { telemetryByServer, uiNow } from '$ui/stores/servers.js';
import type { RosterPlayer, Health, RconEndpoint, RconFailure } from '$ui/stores/servers.js';
@@ -39,7 +41,6 @@
import Badge from '$ui/components/primitives/Badge.svelte';
import Button from '$ui/components/primitives/Button.svelte';
import Tabs from '$ui/components/primitives/Tabs.svelte';
import StatusPip from '$ui/components/primitives/StatusPip.svelte';
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
import EmptyState from '$ui/components/primitives/EmptyState.svelte';
@@ -91,7 +92,24 @@
});
const snapshot = $derived($fleet.receivedAt ? $fleet : data.serverSnapshot);
const visibleFleet = $derived($fleet.receivedAt ? $fleetServers : data.serverSnapshot.servers);
let server = $derived((visibleFleet.find(s => s.name === name) as Server | undefined) ?? null);
const serverContext = $derived(visibleFleet.find(s => s.name === name) ?? null);
let server = $derived((serverContext as Server | null));
const health = $derived(serverContext ? serverHealth(serverContext, $uiNow) : null);
const canOperate = $derived($page.data.user?.role === 'admin' || $page.data.user?.role === 'operator');
const availableActions = $derived(serverContext ? applicableRuntimeActions(serverContext, $uiNow) : []);
const primaryAction = $derived(availableActions.includes('restart') ? 'restart' : availableActions.includes('start') ? 'start' : null);
function openAction(action: RuntimeAction) { if (canOperate && availableActions.includes(action)) { lifecycleAction = action; lifecycleOpen = true; } }
let consumedAction = '';
$effect(() => {
const requested = $page.url.searchParams.get('action');
const key = `${name}:${requested}`;
if (requested !== 'start') { consumedAction = ''; return; }
if (requested === 'start' && canOperate && availableActions.includes('start') && consumedAction !== key) {
consumedAction = key; openAction('start');
const url = new URL($page.url); url.searchParams.delete('action');
void goto(url, { replaceState: true, noScroll: true, keepFocus: true });
}
});
const native = $derived(!!server?.native_instance_id);
let telemetryOpen = $state(false);
let history = $state<DeploymentRow[]>([]);
@@ -159,7 +177,7 @@
$effect(() => { name; rconProbe = null; });
const fresh = $derived(server ? freshServer(server, $uiNow) : false);
let effectivePlayers = $derived(fresh ? server?.last_players ?? null : null);
let effectivePlayers = $derived(fresh && health?.state === 'online' ? server?.last_players ?? null : null);
const scoreboard = $derived(server?.telemetry ?? $telemetryByServer.get(name));
let effectiveHealth = $derived<Health>(server?.health === 'online' && !fresh ? 'stale' : server?.health ?? 'offline');
let effectiveSeen = $derived(server?.last_seen_at ?? null);
@@ -182,7 +200,8 @@
let gameInProgress = $derived(!!effectiveGameStartedAt && !effectiveGameEndedAt && !!effectiveActiveSince);
let gameElapsed = $derived(fresh && gameInProgress ? elapsedSince(effectiveGameStartedAt, $uiNow) : null);
let hasDiagnosticIssue = $derived(effectiveHealth !== 'online' && effectiveHealth !== 'starting' && effectiveHealth !== 'provisioning');
const currentRound = $derived(effectivePlayers !== null && scoreboard && 'round' in scoreboard && freshTelemetry(scoreboard, $uiNow, server?.telemetry_after) ? scoreboard.round : null);
let hasDiagnosticIssue = $derived(!!health && !['online', 'starting', 'provisioning', 'stopped'].includes(health.state));
function ago(iso: string | null, now: number): string {
if (!iso) return 'never';
@@ -218,30 +237,33 @@
</script>
<svelte:head><title>{serverTitle(name)} · {activeTab.charAt(0).toUpperCase() + activeTab.slice(1)} · Electric Cherry</title></svelte:head>
<LifecycleDialog bind:open={lifecycleOpen} initialServer={name} initialAction={lifecycleAction} />
<DestroyServerModal bind:open={destroyOpen} {name} players={effectivePlayers} />
<LifecycleDialog bind:open={lifecycleOpen} initialServer={name} initialAction={lifecycleAction} {serverContext} players={effectivePlayers} round={currentRound} />
<DestroyServerModal bind:open={destroyOpen} {name} players={effectivePlayers} round={currentRound} />
{#if server}
<div class="server-page" style="view-transition-name: server-{name.replace(/[^a-z0-9_-]/gi, '-')};">
<header class="instance-header">
<Breadcrumbs items={[{ label: 'Fleet', href: '/' }, { label: serverTitle(name) }]} />
<Breadcrumbs items={[{ label: 'Servers', href: '/' }, { label: serverTitle(name) }]} />
<SectionHeader title={serverTitle(server.name)}>
{#snippet actions()}
<details class="server-actions"><summary>Instance actions</summary><div class="action-menu">{#each runtimeActions as action}<Button onclick={() => { lifecycleAction = action; lifecycleOpen = true; }}>{actionLabels[action]}</Button>{/each}<Button variant="danger" onclick={() => destroyOpen = true}>Queue instance removal</Button></div></details>
{#if canOperate}
{#if primaryAction}<Button onclick={() => openAction(primaryAction)}>{actionLabels[primaryAction]} server</Button>{/if}
<details class="server-actions"><summary>More actions</summary><div class="action-menu">{#each availableActions.filter(action => action !== primaryAction) as action}<Button onclick={() => openAction(action)}>{actionLabels[action]} server</Button>{/each}<Button variant="danger" onclick={() => destroyOpen = true}>Remove server</Button></div></details>
{/if}
{/snippet}
</SectionHeader>
<div class="instance-meta">
<span class="health"><StatusPip health={effectiveHealth} size="md" />{effectiveHealth === 'online' ? 'Online' : effectiveHealth === 'starting' ? 'Starting' : effectiveHealth === 'provisioning' ? 'Provisioning' : effectiveHealth === 'stale' ? 'Stale' : 'Offline'}</span>
<StatusBadge state={health?.state ?? 'unknown'} />
{#if server.host_id}<a href={contextHref(`/hosts/${server.host_id}`, $page.url)}>{server.host?.name ?? server.host_id}</a>{/if}
<details><summary>Details</summary><div class="instance-details"><CopyValue value={name} label="Instance ID" /><RuntimeVersion {name} version={server.plutonium} after={server.telemetry_after} ready={effectiveHealth === 'online'} /></div></details>
</div>
</header>
{#if $fleetError}<div class="ui-error" role="alert">{$fleetError}<Button size="sm" onclick={() => refreshFleet(true)}>Retry</Button></div>{/if}
{#if hasDiagnosticIssue}
{#if hasDiagnosticIssue && health}
<section class="connection-issue" aria-label="Server connection issue">
<div><strong>{effectiveHealth === 'stale' ? 'Server report is outdated' : 'Server is not responding'}</strong><p>Player count is unknown. {effectiveSeen ? `Last successful report ${ago(effectiveSeen, $uiNow)}.` : 'No successful report has been received.'} {native && !server.host?.connected ? 'The host agent is offline; the game may still be running.' : ''}</p></div>
<div class="issue-actions"><Button size="sm" onclick={() => changeTab('diagnostics')}>Diagnose connection</Button><Button size="sm" variant="ghost" onclick={() => changeTab('logs')}>Open logs</Button>{#if native && server.host_id}<Button size="sm" href={contextHref(`/hosts/${server.host_id}`, $page.url)}>Check host</Button>{:else}<Button size="sm" href="/settings">Check system setup</Button>{/if}</div>
<div><strong>{health.label}</strong><p>{health.reason} {health.since ? `Last evidence ${ago(health.since, $uiNow)}.` : 'No current report is available.'}</p></div>
<div class="issue-actions">{#if health.state === 'host-offline' && server.host_id}<Button size="sm" href={contextHref(`/hosts/${server.host_id}`, $page.url)}>Check host</Button>{:else}<Button size="sm" onclick={() => changeTab('diagnostics')}>Open diagnostics</Button>{/if}<Button size="sm" variant="ghost" onclick={() => changeTab('logs')}>Open logs</Button></div>
</section>
{/if}
<div class="tabs-row">
@@ -259,7 +281,7 @@
</section>
<section id="chat" aria-label="Chat">
<Card>
{#key `${name}:${scoreboard?.timing?.matchId ?? ''}:${effectiveGameStartedAt ?? ''}`}<ChatPanel server={name} />{/key}
{#key `${name}:${scoreboard?.timing?.matchId ?? ''}:${effectiveGameStartedAt ?? ''}`}<ChatPanel server={name} hostName={server.host?.name} hostOffline={health?.state === 'host-offline'} />{/key}
</Card>
</section>
{:else if activeSection === 'configuration'}
@@ -529,8 +551,7 @@
.live-context { margin: 0; font-size: var(--text-sm); color: var(--text-secondary); }
.instance-header { display: grid; gap: 12px; }
.instance-meta, .health { display: flex; align-items: center; flex-wrap: wrap; gap: 12px; font-size: var(--text-sm); }
.health { gap: 8px; }
.instance-meta { display: flex; align-items: center; flex-wrap: wrap; gap: 12px; font-size: var(--text-sm); }
.instance-details { display: grid; gap: 8px; padding: 12px 0; }
details > summary { cursor: pointer; color: var(--text-secondary); font-size: var(--text-sm); }
.server-actions { position: relative; }
tests/unit/maintenance-presentation.test.ts
+93
View File
@@ -0,0 +1,93 @@
import { describe, expect, it } from 'vitest';
import { groupMaintenanceScanErrors, groupMaintenanceVersions } from '../../src/lib/ui/maintenance-presentation.js';
import type { MaintenanceJob, PlutoniumVersion } from '../../src/lib/plutonium.js';
const theater = 't5_zombie_theater_1';
const moon = 't5_zombie_moon_1';
const version = (server: string, changes: Partial<PlutoniumVersion> = {}): PlutoniumVersion => ({
server, revision: 4878, checked_at: '2026-10-07T12:00:00Z', error: null, update_status: 'unknown', ...changes
});
const job = (server: string, changes: Partial<MaintenanceJob> = {}): MaintenanceJob => ({
id: 1, server, state: 'queued', target_revision: 4878, requested_at: '2026-10-07T12:00:00Z',
restarted_at: null, provisioning_id: 'restart-1', finished_at: null, detail: null,
kind: 'update', origin: 'manual', ...changes
});
describe('maintenance scan summaries', () => {
it('groups shared failures into one cause while retaining each affected server and diagnostic message', () => {
const errors = `${theater}: Host agent is offline; /srv/private-one unavailable\n${moon}: Host agent is offline; /srv/private-two unavailable`;
const groups = groupMaintenanceScanErrors(errors, [theater, moon]);
expect(groups).toHaveLength(1);
expect(groups[0]).toMatchObject({ key: 'host', title: 'Host agent unavailable', servers: [theater, moon] });
expect(groups[0].details).toHaveLength(2);
expect(groups[0].description).not.toContain('/srv/');
expect(groups[0].description).not.toContain(theater);
expect(groups[0].details[0].message).toContain('/srv/private-one');
});
it('separates file, permission, timeout and unknown causes without exposing raw messages as headings', () => {
const groups = groupMaintenanceScanErrors([
`${theater}: ENOENT: no such file /srv/missing`,
`${moon}: EACCES: permission denied /srv/private`,
`${theater}: timeout: no reply`,
`${moon}: Unrecognized diagnostic code 123`
].join('\n'), [theater, moon]);
expect(groups.map(group => group.key)).toEqual(['files', 'permissions', 'timeout', 'other']);
expect(groups.every(group => !/ENOENT|EACCES|123/.test(group.title))).toBe(true);
});
it('supports custom names containing colons, avoids linking unknown identifiers and deduplicates identical failures', () => {
const message = 'Weekend: west: timeout: No reply';
const groups = groupMaintenanceScanErrors(`${message}\n${message}\nretired_server: Agent unavailable`, ['Weekend', 'Weekend: west']);
expect(groups[0].servers).toEqual(['Weekend: west']);
expect(groups[0].details).toEqual([{ server: 'Weekend: west', message: 'timeout: No reply' }]);
expect(groups[1].servers).toEqual([]);
expect(groups[1].details[0].server).toBeNull();
});
it('handles setup messages and empty scans without inventing an affected server', () => {
expect(groupMaintenanceScanErrors(null, [])).toEqual([]);
expect(groupMaintenanceScanErrors(' \n ', [])).toEqual([]);
expect(groupMaintenanceScanErrors('Register a native host instance before automatic recovery.', []))
.toEqual([expect.objectContaining({ key: 'setup', servers: [] })]);
});
});
describe('maintenance revision groups', () => {
it('collapses identical revisions and verification states without claiming that unknown versions are current', () => {
const versions = Array.from({ length: 12 }, (_, index) => version(`t5_zombie_theater_${index + 1}`));
const groups = groupMaintenanceVersions(versions, []);
expect(groups).toHaveLength(1);
expect(groups[0]).toMatchObject({ revision: 4878, label: 'Unverified', tone: 'warning', updatable: [] });
expect(groups[0].versions).toHaveLength(12);
});
it('retains separate verification states and puts failures and available updates first', () => {
const groups = groupMaintenanceVersions([
version('current', { update_status: 'current' }), version('unknown'),
version('outdated', { revision: 4800, update_status: 'outdated' }),
version('ahead', { revision: 5000, update_status: 'ahead' })
], []);
expect(groups.map(group => group.status)).toEqual(['unknown', 'outdated', 'current', 'ahead']);
expect(groups[0].revision).toBe(groups[2].revision);
});
it('excludes pending restarts from group update actions while preserving the pending operation links', () => {
const pending = job(theater, { state: 'restarting' });
const [group] = groupMaintenanceVersions([
version(theater, { update_status: 'outdated' }), version(moon, { update_status: 'outdated' })
], [pending, job(moon, { id: 2, state: 'done' })]);
expect(group.updatable).toEqual([moon]);
expect(group.pending).toEqual([pending]);
});
it('does not let deleted-server jobs suppress a current server update and preserves stopped legacy evidence', () => {
const groups = groupMaintenanceVersions([
version(theater, { update_status: 'outdated' }),
version(moon, { update_status: undefined, stopped: true, revision: null })
], [job(theater, { server_deleted_at: '2026-10-06' })]);
expect(groups[0].updatable).toEqual([theater]);
expect(groups[0].pending).toEqual([]);
expect(groups[1]).toMatchObject({ status: 'stopped', label: 'Intentionally stopped', revision: null, updatable: [] });
});
});
tests/unit/ui-lifecycle-actions.test.ts
+102
View File
@@ -0,0 +1,102 @@
import { describe, expect, it, vi } from 'vitest';
import { render } from 'svelte/server';
import { get } from 'svelte/store';
import type { FleetServer } from '../../src/lib/fleet-overview.js';
import { applicableRuntimeActions, lifecycleImpact } from '../../src/lib/ui/lifecycle-actions.js';
import { actionCompleted, actionLabels } from '../../src/lib/provisioning.js';
import { uiNow } from '../../src/lib/ui/stores/servers.js';
import LifecycleDialog from '../../src/lib/ui/components/LifecycleDialog.svelte';
import DestroyServerModal from '../../src/lib/ui/components/DestroyServerModal.svelte';
vi.mock('$app/navigation', () => ({ goto: vi.fn() }));
const now = get(uiNow);
const iso = (offset = 0) => new Date(now + offset).toISOString();
const server: FleetServer = {
name: 't5_zombie_theater_1', game: 't5', map: 'zombie_theater', last_map: 'zombie_theater',
last_gametype: 'zclassic', last_players: 4, last_seen_at: iso(-10_000), last_error: null,
container_state: 'running', container_checked_at: iso(-5_000), last_checked_at: iso(-5_000),
roster: [], health: 'online', held: [], active_since: iso(-120_000), inactive_since: null,
game_started_at: iso(-120_000), game_ended_at: null, host_id: 'host-1',
host: { id: 'host-1', name: 'Dallas 1', connected: true, connectionState: 'online', lastHeartbeat: iso(-1_000) }
};
describe('lifecycle action applicability', () => {
it('offers Restart, Stop and Rebuild for a responding game, without Start', () => {
expect(applicableRuntimeActions(server, now)).toEqual(['restart', 'stop', 'rebuild']);
});
it('offers Start only for a confirmed stopped container with a reachable host', () => {
const stopped = { ...server, health: 'offline' as const, container_state: 'stopped' };
expect(applicableRuntimeActions(stopped, now)).toEqual(['start', 'rebuild']);
for (const change of [
{ container_checked_at: iso(-91_000) }, { container_checked_at: null },
{ telemetry_after: iso(-2_000) }, { last_seen_at: iso(-2_000) },
{ host: null }, { host: { ...server.host!, connected: false } }
]) expect(applicableRuntimeActions({ ...stopped, ...change }, now)).not.toContain('start');
});
it('permits recovery of an unresponsive game only with a current running-container observation', () => {
const unresponsive = { ...server, health: 'stale' as const, last_seen_at: iso(-300_000), last_error: 'timeout:no reply' };
expect(applicableRuntimeActions(unresponsive, now)).toEqual(['restart', 'stop', 'rebuild']);
expect(applicableRuntimeActions({ ...unresponsive, container_checked_at: iso(-91_000) }, now)).toEqual([]);
expect(applicableRuntimeActions({ ...unresponsive, telemetry_after: iso(-2_000) }, now)).toEqual([]);
});
it('does not offer runtime mutations during startup or without evidence', () => {
for (const change of [
{ health: 'starting' as const }, { health: 'provisioning' as const },
{ host: { ...server.host!, connected: false } },
{ health: 'offline' as const, container_state: 'unknown', last_seen_at: null }
]) expect(applicableRuntimeActions({ ...server, ...change }, now)).toEqual([]);
});
});
describe('lifecycle impact copy', () => {
it('names current players and round with correct singular grammar', () => {
expect(lifecycleImpact('restart', 4, 30)).toBe('4 players are in round 30. Restart waits until the lobby is empty.');
expect(lifecycleImpact('stop', 1, null)).toBe('1 player is currently online. Stop waits until the lobby is empty.');
});
it('does not confuse unknown or invalid player counts with an empty lobby', () => {
for (const players of [null, -1, NaN, 1.5]) expect(lifecycleImpact('destroy', players, 30))
.toBe('Player count is unknown. Removal waits for a live check confirming the lobby is empty.');
expect(lifecycleImpact('rebuild', 0, 30)).toBe('The lobby is currently empty. A live check will confirm this before rebuild begins.');
});
it('does not claim a round or occupancy wait when starting a stopped container', () => {
expect(lifecycleImpact('start', null, null)).toBe('Start the existing container, then verify that the game is responding.');
expect(lifecycleImpact('restart', 4, NaN)).toContain('4 players are currently online.');
});
});
describe('contextual lifecycle confirmation', () => {
it('names the server and impact without re-asking the instance or action', () => {
const body = render(LifecycleDialog, { props: { open: true, initialServer: server.name,
initialAction: 'restart', serverContext: server, players: 4, round: 30 } }).body;
expect(body).toContain('Restart Kino Der Toten · Lobby 1?');
expect(body).toContain('4 players are in round 30.');
expect(body).toContain('Restart when empty');
expect(body).not.toContain('<select');
expect(body).not.toContain('>Manage server<');
});
it('explains an unavailable action when the evidence no longer supports it', () => {
const body = render(LifecycleDialog, { props: { open: true, initialServer: server.name,
initialAction: 'start', serverContext: server } }).body;
expect(body).toContain('Start is unavailable. The game is responding.');
const startButton = [...body.matchAll(/<button\b[^>]*>[\s\S]*?<\/button>/g)].find(match => match[0].includes('Start server'))?.[0];
expect(startButton).toContain('disabled');
expect(body).not.toContain('Start the existing container');
});
it('uses Remove server consistently and preserves recorded history in the explanation', () => {
const body = render(DestroyServerModal, { props: { open: true, name: server.name, players: 4, round: 30 } }).body;
expect(body).toContain('Remove Kino Der Toten · Lobby 1?');
expect(body).toContain('4 players are in round 30.');
expect(body).toContain('Remove server when empty');
expect(body).toContain('recorded history');
expect(body).not.toContain('campaign details');
expect(actionLabels.destroy).toBe('Remove');
expect(actionCompleted.destroy).toBe('Server removed');
});
});
tests/unit/ui-match-chat.test.ts
+8
View File
@@ -30,6 +30,14 @@ describe('historical match chat', () => {
expect(body).not.toContain('Loading chat');
});
it('explains an offline host without claiming live updates or offering a manual refresh', () => {
const body = render(ChatPanel, { props: { server: 't5_zombie_theater_1', hostOffline: true, hostName: 'Dallas 1' } }).body.replaceAll(/<!--.*?-->/g, '');
expect(body).toContain('Chat resumes when Dallas 1 reconnects.');
expect(body).not.toContain('Live ·');
expect(body).not.toContain('Loading chat');
expect(body).not.toContain('Refresh chat');
});
it('explains when a historical match has no saved messages', () => {
const body = render(ChatPanel, { props: { history: [] } }).body;
expect(body).toContain('No chat messages were saved for this match.');