Commit Graph
7 Commits
Author SHA1 Message Date
josh b9f5902945 feat: add Watchdog recovery and drift rebuild workflows 2026-09-27 15:13:38 -04:00
josh df265d1900 refactor: make provisioning own instance lifecycle operations 2026-09-27 14:45:00 -04:00
josh 8e04a190b3 Roll deployment restarts through a shared fleet-wide recovery limit 2026-09-17 11:25:06 -04:00
joshandClaude Opus 4.7 11e47b2a3f Widen Komodo health probe: try container endpoints, snippet on failure
GetDeployment returns config, not runtime state, so the health check was
always coming back with 'no container state found in response'. For
deployment-type servers the probe now tries GetDeploymentContainer,
InspectDeploymentContainer, and GetDeploymentStatus first, falling back
to GetDeployment only if those aren't available. The extractor also
recognizes more shapes (Health.Status, container_state, deployment_state,
etc.) and accepts more healthy strings (running, healthy, ok, started,
up).

When nothing matches, the truncated raw response snippet rides along in
the error field so the deployment/beta UI shows exactly which field to
teach the extractor next time.

Currently-stuck deployment_server rows are still 'pending', so the sweep
will retry them on its next 15s tick — no manual reset needed. Once one
of the widened endpoints resolves, those rows advance to 'done'
automatically.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-09-12 13:51:04 -04:00
joshandClaude Opus 4.7 edfe4dfc09 Add drift detection and removal as a real deployment
A drift deployment scans each server's sp/ tree for files not in the
server_scripts inventory (untracked scripts left over from before EC was
managing the fleet), deletes them, and records the removed bytes in
deployment_files for audit. The existing sweep then runs the standard
wait-for-empty → Komodo restart cycle on each affected server.

To make that cycle safer, the sweep now has a second phase: after Komodo
restart succeeds we wait for komodoInspectHealthy + RCON to both come
back green before marking the deployment_server row 'done'. If either
stays broken the row stays pending and keeps retrying.

New: /api/drift (GET scan, POST run), drift panel on /deployments.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-09-12 13:44:52 -04:00
josh 854f5532a9 Replace IW4MAdmin with direct RCON for player counts
IW4M kept hitting a network wall (container -> host LAN IP -> IW4M
loopback bind) that we couldn't cleanly punch through, and the two-minute
!rt token added enough operational friction that the pause-all-deployments
gate never got a fair test. RCON gives us the same fact — how many
players are on a server right now — direct from each game server over
UDP, with no session, no token, no proxy.

- Add integrations/rcon.ts: dgram UDP client. rconGetPlayerCount() and
  rconGetStatus() parse both `status` (print + scoreboard) and
  `statusResponse` (info line + rows) dialects. Timeout via
  RCON_TIMEOUT_MS.
- Add komodoResolveGamePort() + extractUdpPort(). Reads Komodo's read
  API for a server, walks both docker-compose-style port objects and
  docker-run-style "host:container/udp" strings, caches into
  servers.game_port on first success.
- Replace pipelineReady() guard in sweep with a simple
  integrations-configured check. Same held/failed/manual outcomes,
  but keyed on RCON+Komodo instead of IW4M+auth-state.
- Worker's status refresh fans out per-server over RCON at concurrency=4
  so a 15-server fleet doesn't spike UDP.
- Schema: drop iw4m_match/iw4m_value, add game_port + rcon_host.
- Delete every auth artifact: TokenModal, auth store, auth-state module,
  /api/auth/token, /api/events/auth, iw4m.test.ts. Layout no longer
  mounts the modal, pipeline store no longer tracks a frozen state.
- README + .env.example + docker-compose.yml + komodo/stack.toml swap
  IW4M_* env for RCON_PASSWORD / RCON_HOST / RCON_TIMEOUT_MS. Same
  host.docker.internal:host-gateway extra_hosts entry, repurposed
  for RCON reach.
- Tests: add rcon.test.ts and komodo.test.ts, update sweep.test.ts to
  drop the auth-freeze case. 23/23 passing.
2026-09-12 01:33:06 -04:00
josh d99711563e Initial commit — SvelteKit fleet manager for T5 Plutonium servers
Ports push_scripts.sh (1497 lines of bash) to a SvelteKit + SQLite web app
that manages a fleet of T5 Plutonium servers running on Komodo. Same push
and beta workflow as the shell script, with a live SSE pipeline view, a
!rt-token-gated auth modal that hard-pauses deployments on stale tokens,
per-file audit diffs, and a Komodo-container deployment target.

- Push/beta engines port the exact filter, held-by-beta, conflict, and
  stale-backup rules from the shell script
- SQLite (better-sqlite3) replaces script_metadata.json; a one-shot
  importer moves legacy state over on first boot
- 16 vitest tests cover push add/update/skip, beta open→rollback,
  open→promote, conflict guards, sweep frozen/manual branches, and IW4M
  login/status parsing
- docker/ + komodo/stack.toml deploy as a managed container on the same
  host as the game servers, /root/zombies bind-mounted at the same path
  the shell script used
2026-09-12 00:23:51 -04:00