Commit Graph
56 Commits
Author SHA1 Message Date
josh 36cf2b9c05 Add autoscaling dry run and policy controls 2026-10-06 22:52:50 -04:00
josh 1b42244858 Supplement telemetry rosters with fresh RCON pings 2026-10-06 21:13:02 -04:00
josh 57d2bd11db Preserve verified health when provisioning completes 2026-10-06 15:48:38 -04:00
josh d99e5979fa Paginate beta history and refresh pages from events 2026-10-06 14:50:47 -04:00
josh daefb22be6 Share status monitoring snapshots across each fleet pass 2026-10-06 14:47:14 -04:00
josh 88a950692a Fix retention query performance and cleanup scheduling 2026-10-06 10:39:32 -04:00
josh 3952c4c938 Add configurable history and storage retention policies 2026-10-05 22:48:04 -04:00
josh f8d0ba3936 Allow revising active betas with fresh tests and history 2026-10-05 21:02:58 -04:00
josh 4b0f341d64 Remove Komodo backend in favor of native hosts 2026-10-05 18:07:07 -04:00
josh b1b3e8b958 Relax freshness limits and tolerate transient RCON failures 2026-10-05 16:36:23 -04:00
josh ae703617af Preserve server status when batch probes miss the deadline 2026-10-05 16:17:29 -04:00
josh 87c5cf4b4b Use status monitor evidence for routine watchdog health checks 2026-10-05 16:13:47 -04:00
josh 030794635e Coalesce fleet refreshes and suppress unchanged sweep events 2026-10-05 15:29:03 -04:00
josh 239eceb779 Reuse heartbeat container state and batch host RCON status 2026-10-05 15:14:44 -04:00
josh 3f5cc9d3ba Limit status checks to two per host 2026-10-05 15:02:19 -04:00
josh a6c40cde9b Add filters and context to host operation history 2026-10-05 13:23:11 -04:00
josh e6912ceb57 Add configurable retention for successful routine operations 2026-10-05 13:09:44 -04:00
josh 2d069907f2 Share recent health observations between background monitors 2026-10-05 13:00:53 -04:00
josh 725d41c112 Automatically remove unused game-file versions 2026-10-04 23:05:32 -04:00
josh 81f088739c Fix native health polling and isolate file operation traffic 2026-10-04 21:50:29 -04:00
josh ed004ddbcf Improve application responsiveness and background processing 2026-10-04 19:56:06 -04:00
josh 2adadb0d05 Fix native provisioning scheduling and progress reporting 2026-10-04 14:50:59 -04:00
josh 65d3a99846 Fix transient health checks and avoid unnecessary recovery 2026-10-02 13:06:18 -04:00
josh fb48f64de1 Defer server file changes until guarded restart activation
Persist beta, promotion, drift, and rollback payloads until occupancy and restart-capacity checks pass. Preserve rollback snapshots and campaign ordering, and allow held rollback campaigns to resume safely.

Fixes #28

Validation: 805 tests passed, Svelte checks clean, production build passed.
2026-09-27 23:06:16 -04:00
josh 981fb27c1c Preserve deleted server history
Retain historical server identities, results, and evidence after deletion. Mark archived entries as Deleted and prevent old work from targeting servers that reuse the same name.

Fixes #16.
2026-09-27 21:48:55 -04:00
josh 4fe8613678 Move fleet retention and concurrency controls into Settings
Persist settings in SQLite, import legacy environment values once, and apply changes to cleanup and deployment queues without restarting.

Closes #51
2026-09-27 16:11:45 -04:00
josh fc46cb9da4 feat: unify recovery, updates, and schedules under Maintenance 2026-09-27 16:00:18 -04:00
josh 8ea696417f fix: honor Komodo health and prefer fresh telemetry across fleet status 2026-09-27 15:34:30 -04:00
josh b9f5902945 feat: add Watchdog recovery and drift rebuild workflows 2026-09-27 15:13:38 -04:00
josh df265d1900 refactor: make provisioning own instance lifecycle operations 2026-09-27 14:45:00 -04:00
josh cfed584614 feat: add provisioning lifecycle actions
Rename Deployments to Provisioning with compatible redirects and APIs. Add durable Start, Restart, Stop, and Rebuild operations alongside Create and Destroy; rebuild wipes the instance and recreates it in the same slot and port.

Add lifecycle, migration, recovery, and Komodo command tests. Type checks and production build pass; the full suite retains a telemetry assertion failure reproduced on the unchanged branch.

Refs #39
2026-09-27 14:20:04 -04:00
josh 47d6dc07b7 Add Plutonium version checks and scheduled rolling restarts 2026-09-21 18:47:24 -04:00
josh 3a63e6d3ce Add Metrics tab with historical server utilization and map capacity 2026-09-20 21:58:08 -04:00
josh 44b6415821 Show provisioning status during server creation 2026-09-19 22:53:24 -04:00
josh 5985959882 Show resetting state after game end 2026-09-19 22:50:17 -04:00
josh 5bb49a776e Simplify telemetry to three game events 2026-09-18 15:35:21 -04:00
josh 6d903adb31 Fix fleet stale-status flicker during polling 2026-09-18 15:11:49 -04:00
josh a50eae9d21 Add configurable beta test plans and staged rollout 2026-09-17 21:35:28 -04:00
josh 885abbdffe Use database fleet registry and gate RCON polling on container readiness 2026-09-17 17:28:38 -04:00
josh c2ad5cc95a Queue server creation and verify health and selected map before completion 2026-09-17 17:14:36 -04:00
josh a611850649 new destruction flow 2026-09-17 16:57:08 -04:00
josh 858704bb10 adds round telemetry support 2026-09-17 14:35:42 -04:00
josh 8e04a190b3 Roll deployment restarts through a shared fleet-wide recovery limit 2026-09-17 11:25:06 -04:00
josh efd4a4fa66 Fix game timer: cycle-key tracking, wall-clock only, active gate
games_sp.log is not wiped on restart, so old J; lines from prior
sessions were poisoning the anchor. Now the scanner walks the tail
backward to the LATEST InitGame and only considers lines after it,
returning an opaque cycle key (the first J; line) plus a state. The
worker stamps game_started_at/ended_at with Date.now() at observation
time instead of inferring from the log's own timestamps. Same-players
game restarts produce a new key and cleanly re-anchor. When the server
transitions to 0 players, game state is cleared server-side and the
UI also gates the timer display on activeSince.

Adds servers.game_cycle_key column with additive migration.
2026-09-13 22:47:03 -04:00
josh 3d111880d3 Add active/inactive/game-time timers to fleet + server pages
Fleet cards and the per-server detail hero now show three live-ticking
timers: Active (since players > 0), Inactive (since server emptied), and
Game (since InitGame in games_sp.log, back-computed from the log's own
MM:SS prefix so it stays accurate across restarts). Anchors are
persisted on `servers` and fanned out via the SSE bus; a single 1s
ticker feeds every card. Idle map cycles are filtered out so the game
timer doesn't flicker.
2026-09-13 21:45:54 -04:00
josh 85bcb05a7d multi script beta 2026-09-12 18:23:56 -04:00
joshandClaude Opus 4.7 1afdfddc87 Promote as a real deployment; version state and scope dropdowns
Turns promotion into a first-class deployment (action='promote') so it lands
in the deployments UI with before/after content, per-server restart tracking,
and one-click rollback. Non-beta matching-scope servers get restart_state=
pending so the existing sweep drains and restarts them the same way drift does.

Also adds a per-version state (untested/tested/test-failed) so the upload
flow can visibly signal that a beta must land before promotion, and swaps
the scope-value text box for game/map dropdowns populated from the fleet.
Beta rollback at ready-promote is now labelled Reject and marks the version
test-failed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-09-12 14:24:35 -04:00
joshandClaude Opus 4.7 11e47b2a3f Widen Komodo health probe: try container endpoints, snippet on failure
GetDeployment returns config, not runtime state, so the health check was
always coming back with 'no container state found in response'. For
deployment-type servers the probe now tries GetDeploymentContainer,
InspectDeploymentContainer, and GetDeploymentStatus first, falling back
to GetDeployment only if those aren't available. The extractor also
recognizes more shapes (Health.Status, container_state, deployment_state,
etc.) and accepts more healthy strings (running, healthy, ok, started,
up).

When nothing matches, the truncated raw response snippet rides along in
the error field so the deployment/beta UI shows exactly which field to
teach the extractor next time.

Currently-stuck deployment_server rows are still 'pending', so the sweep
will retry them on its next 15s tick — no manual reset needed. Once one
of the widened endpoints resolves, those rows advance to 'done'
automatically.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-09-12 13:51:04 -04:00
joshandClaude Opus 4.7 edfe4dfc09 Add drift detection and removal as a real deployment
A drift deployment scans each server's sp/ tree for files not in the
server_scripts inventory (untracked scripts left over from before EC was
managing the fleet), deletes them, and records the removed bytes in
deployment_files for audit. The existing sweep then runs the standard
wait-for-empty → Komodo restart cycle on each affected server.

To make that cycle safer, the sweep now has a second phase: after Komodo
restart succeeds we wait for komodoInspectHealthy + RCON to both come
back green before marking the deployment_server row 'done'. If either
stays broken the row stays pending and keeps retrying.

New: /api/drift (GET scan, POST run), drift panel on /deployments.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-09-12 13:44:52 -04:00
joshandClaude Opus 4.7 93d5a8e2e5 Rework as scripts-centric, upload-only, gated beta lifecycle
Scripts become first-class: named, versioned, uploaded via the UI, optionally
scoped to a game or map. Any change that would land on scope-matched servers
(marking-required or new version of a scoped script) must first pass a beta.
The beta runs a per-server state machine — deploy -beta.gsc → wait empty →
Komodo restart → wait healthy → optionally wait for a real match end
(games_sp.log 'ExitLevel: executed') — before it's eligible for promotion.
Rollback is always available and restores from .OLD.

The shared/scripts mirror and its betas/ staging folder are gone. Every
server now has a tracked script inventory (server_scripts table), populated
only by explicit upload flows.

New: /scripts, /scripts/[id], /betas/[id] UI; /api/scripts + /api/betas +
/api/servers/[name]/scripts endpoints; scripts.ts, inventory.ts, beta.ts
(rewritten), beta-runner.ts, match-log.ts, komodoInspectHealthy().
Removed: push.ts, /api/push, BetaPanel, legacy importer.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-09-12 13:25:06 -04:00