Introduce an outbound Python host agent, Debian installer, durable remote
operations, immutable game-file libraries, host-aware deployments and ports,
and occupancy-safe campaigns and draining. Native hosts do not require Komodo.
Add administrator, operator and viewer accounts, protected management routes,
secure sessions, password recovery, audit history and separate telemetry tokens.
Document first-admin setup, reporter migration and live-host acceptance checks.
Validation: 1,343 TypeScript tests and 39 agent tests passed, along with type
checking, the production build, compiled SSR/API smoke tests and administrator
recovery. Real Debian/Docker T5/T6 acceptance remains unrun.
Keep maintenance observations and automatic requests clear of active lifecycle operations and retained recovery capacity while preserving owned recovery and explicit retries.
Fixes#55
Rename Deployments to Provisioning with compatible redirects and APIs. Add durable Start, Restart, Stop, and Rebuild operations alongside Create and Destroy; rebuild wipes the instance and recreates it in the same slot and port.
Add lifecycle, migration, recovery, and Komodo command tests. Type checks and production build pass; the full suite retains a telemetry assertion failure reproduced on the unchanged branch.
Refs #39
read/ListTags with an empty body was returning something we couldn't
parse, so every tag lookup silently missed and the CreateTag duplicate
fallback dead-ended with no diagnostic.
Now the tag lookup tries two request styles in order — the per-endpoint
`read/ListTags {}` used by the rest of this file, then the tagged-union
`read {type: "ListTags", params: {}}` used by mbecker20/komodo's
canonical API. If both fail to yield a match, both raw response bodies
get logged so the actual Komodo shape is diagnosable on the next run
without another commit-and-deploy cycle.
Two real-world Komodo responses broke the first pass:
1. read/ListTags didn't surface tag "t5" in a matchable shape, so we
fell through to CreateTag and got HTTP 500 "duplicate key: t5".
2. write/CreateTag returned 200 with an id field named "id" (not "_id"),
which the parser rejected as an unexpected shape even on success.
Fixes:
- extractTagId accepts either "id" or "_id"
- unwrapTagList peels back common list wrappers (bare array, {tags},
{data}, {results}, {items}) instead of requiring a top-level array
- CreateTag duplicate-key error triggers a re-list so the existing tag
can be resolved to its id instead of failing the whole tag pass
- Both ListTags and CreateTag branches log the raw body when parsing
fails, so the next unfamiliar shape is diagnosable at a glance
New deployments get two tags attached (e.g. "t5" and "zombie_pentagon")
so operators can filter the Komodo deployment list by game and by map.
Tags are ensured via read/ListTags + write/CreateTag fallback, then
attached with write/UpdateTagsOnResource against the deployment name.
Tag application runs fire-and-forget after CreateDeployment: any failure
logs a warning and the server creation proceeds. Tags are cosmetic
organization inside Komodo, not a correctness requirement.
Full server lifecycle management from Electric Cherry: pick game/mode/map/pluto
key in the Fleet page, EC provisions the deployment via Komodo, allocates a
port in 28960-28980, creates the 0o777 host dir, and starts the container.
Destroy button tears everything down (Komodo Destroy+Delete, rm -rf host dir,
transactional DB cleanup) and surfaces any partial-failure warnings inline.
Same phase cuts over to the new naming scheme (t5_zombie_cosmodrome_1 etc.).
Komodo entity names now match EC server names 1:1 (no more _/- transform), and
the map catalog carries friendly/slug/canonical for each of the 11 t5 zombie
maps. Server rows gain game_mode, map_slug, host_dir, config_file, game_id,
image, created_by, and destroyed_at columns for provisioning + audit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
GetDeployment returns config, not runtime state, so the health check was
always coming back with 'no container state found in response'. For
deployment-type servers the probe now tries GetDeploymentContainer,
InspectDeploymentContainer, and GetDeploymentStatus first, falling back
to GetDeployment only if those aren't available. The extractor also
recognizes more shapes (Health.Status, container_state, deployment_state,
etc.) and accepts more healthy strings (running, healthy, ok, started,
up).
When nothing matches, the truncated raw response snippet rides along in
the error field so the deployment/beta UI shows exactly which field to
teach the extractor next time.
Currently-stuck deployment_server rows are still 'pending', so the sweep
will retry them on its next 15s tick — no manual reset needed. Once one
of the widened endpoints resolves, those rows advance to 'done'
automatically.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Scripts become first-class: named, versioned, uploaded via the UI, optionally
scoped to a game or map. Any change that would land on scope-matched servers
(marking-required or new version of a scoped script) must first pass a beta.
The beta runs a per-server state machine — deploy -beta.gsc → wait empty →
Komodo restart → wait healthy → optionally wait for a real match end
(games_sp.log 'ExitLevel: executed') — before it's eligible for promotion.
Rollback is always available and restores from .OLD.
The shared/scripts mirror and its betas/ staging folder are gone. Every
server now has a tracked script inventory (server_scripts table), populated
only by explicit upload flows.
New: /scripts, /scripts/[id], /betas/[id] UI; /api/scripts + /api/betas +
/api/servers/[name]/scripts endpoints; scripts.ts, inventory.ts, beta.ts
(rewritten), beta-runner.ts, match-log.ts, komodoInspectHealthy().
Removed: push.ts, /api/push, BetaPanel, legacy importer.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The raw hex dump showed Plutonium T5's `status` reply is a 9-column
format, not the 5-column shape the parser assumed:
num score ping guid name lastmsg address qport rate
--- ----- ---- ------- ---------------- ------- --------------------- ----- -----
0 3040 36 2741626 JoshWright^7 0 68.235.46.100:28960 -14733 25000
Three bugs in the old regex: it required 16+ hex chars for guid (T5
uses 7-digit decimal IDs), it expected name to run to end-of-line
(4 trailing columns get glued on), and it left Plutonium color codes
(^0..^9) in the rendered name.
Rewrote parsePlayerRoster to detect column boundaries from the header
line word positions instead — column N runs from word N's start to
word N+1's start, matching how CoD's status output actually lays out
(values borrow the padding between adjacent columns). Same code path
handles the old 5-column format and the T5 9-column format cleanly.
stripColorCodes() removes ^\d sequences from names in both dialects.
PlayerRoster component only appends `…` when the guid is actually
long enough to need truncation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Player count says 1 but roster shows 0 for t5_ascension_1 — a parser
mismatch (likely Plutonium's status body uses Steam IDs instead of the
16+ char hex GUIDs my roster regex expects). To diagnose these without
guessing, always keep the raw RCON reply around and expose it on the
detail page.
rconStatusResult now includes rawReply on success too (used to only
persist it on bad-reply). Worker + probe-rcon endpoint persist it to
last_reply_snippet on every tick, so the byte-level truth about the
reply survives page loads.
New "RCON reply" panel on the detail page (visible whenever we have a
snippet or an endpoint) shows:
- Probe RCON button (was buried inside the offline-only diagnostic
panel; now accessible when healthy too)
- Parsed values in a grid: endpoint, players, map, gametype, roster
row count, last seen — so mismatches between "what the parser
extracted" and "what the reply actually said" are one glance apart
- Raw hex+ASCII dump of the last reply so the operator can read
literal player rows and see why the roster parser gave 0
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The hex dump surfaced Plutonium's actual reply:
ff ff ff ff 70 72 69 6e 74 0a ....print.
55 6e 6b 6e 6f 77 6e 20 63 6f Unknown co
6d 6d 61 6e 64 20 22 67 65 74 mmand "get
73 74 61 74 75 73 22 0a status".
Auth was fine — Plutonium just doesn't recognize `getstatus` as an rcon
subcommand. In Q3-derived engines, `getstatus` is an anonymous query
(no rcon wrapper); we were sending `rcon <password> getstatus`, which
made Plutonium treat it as an unknown subcommand.
Switch rconStatusResult to use `status`, same command rconGetPlayerCount
has been using successfully all along. The response is a `print` body
with a `key: value` header (`map: mp_kino\n...`) followed by the
scoreboard, not the `\info\` line the statusResponse dialect uses.
New parseStatusHeader extracts key:value dvars from the header, stopping
at the `--- ---` separator (or the first player row) so guids and names
can't leak into the metadata. rconStatusResult now merges header values
with the statusResponse info line so both dialects produce map/gametype
in the RconStatus shape.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
"unparseable reply (34 bytes)" told us Plutonium answered but hid the
actual bytes, forcing another guess-and-check round. rconStatusResult
now emits a formatRawReply() hex + ASCII dump of the raw UDP payload on
every bad-reply, persisted in a new last_reply_snippet column so it
survives page loads.
Wrong-password detection widens beyond the "bad rcon" phrasing to
"(bad|invalid|wrong)…(rcon|password|pass)"; print-body replies that
don't match get their first non-header line surfaced verbatim so at
least *some* server-supplied text reaches the UI.
Probe RCON button (POST /api/servers/[name]/probe-rcon) fires the same
code path as the periodic sweep on demand, cuts the RCON iteration loop
from 30s per attempt to sub-second, and refreshes the header/roster/
health when it succeeds. Snippet <details> auto-opens on bad-reply so
the bytes are one glance away.
migrateServers() picks up the new column automatically via the
introspection path added last commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Real Komodo GetDeployment responses render config.ports as a single
newline-delimited string ("28963:28963/udp\n28963:28963/tcp\n"), not as
the array shapes we already handled. The walker only descended into
arrays and objects, so the string was skipped entirely and every Komodo
probe on a real deployment fell through to no-udp-in-response.
When the walker hits a value under a ports-keyed field that's a string,
split it on \r\n , ; and treat the resulting parts as a preferred string
array — the existing "host:container/udp" and "container/udp" regexes
handle each part.
Fixture pulled verbatim from the t5-nacht-1 response the fleet UI
surfaced. Also covers single-line, host-network (no host prefix), and
TCP-only-shouldn't-match cases.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Komodo probe answers a specific question (can we resolve a port from
Komodo's API), but leaves ambiguity when it fails: is DNS broken, is the
route blocked, is the target port unreachable, or is the target refusing?
A 10-second timeout looks the same for all of those.
Adds a second button, "diagnose network", that fires parallel raw TCP
connect attempts (no HTTP, no auth, no bytes sent) at a fixed matrix of
candidate hosts against the configured Komodo port:
- 127.0.0.1 (sanity — container loopback)
- host.docker.internal (bridge-gateway alias)
- 172.17.0.1 (raw default bridge gateway)
- the KOMODO_ADDRESS host (if different)
- RCON_HOST (if different)
Each row shows resolved IP, outcome (connected/refused/timeout/dns-fail/
unreachable/error), elapsed ms, and detail. Together they pinpoint which
network layer the traffic dies at without guessing.
New helper: src/lib/server/integrations/netProbe.ts — DNS lookup + net.
createConnection with explicit timeout, classifying errors by errno
(ECONNREFUSED, EHOSTUNREACH, ENOTFOUND). Runs trials in parallel so the
whole diagnostic completes in one probe-timeout budget.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The probe surfaced "unreachable: This operation was aborted" — technically
correct (an AbortController fired the fetch timeout) but useless for
diagnosing whether it's a wrong URL, blocked route, or firewall. Node's
fetch also hides real network causes (ECONNREFUSED, ENOTFOUND,
EHOSTUNREACH) under `.cause`, and the default message just says
"fetch failed".
describeFetchError() now unwraps AbortError to "timed out after 10000ms
dialing <url>" and pulls `.cause.code/.cause.message` for network
failures. KomodoPortResult gains a `url` field, always populated, so the
probe panel can surface the exact URL we dialed in a code block —
operators can then `curl` it from the container to confirm reachability.
komodoRestart benefits from the same unwrapping since it uses the same
komodoCall() transport.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
komodoResolveGamePort silently returned null for six distinct failure
modes (unconfigured, no-name, unreachable, http-error, unparseable,
no-udp-in-response), which made "no game_port — Komodo did not return
one" indistinguishable from an auth failure or an unrecognized response
shape. Adds komodoResolvePortDetailed returning a discriminated union
with the failure reason, HTTP status, and a truncated response snippet.
extractUdpPort broadens to cover four more shapes seen in the wild:
extra host-port field names (HostPort/hostPort/host_port/PublicPort),
`type` as a protocol alias, host-network strings ("28962/udp"), and the
docker-inspect container-keyed form ({ "28960/udp": [{ HostPort }] }).
Arrays reached via a ports-flavored key now win over accidental matches
elsewhere in the tree.
New POST /api/servers/[name]/probe-komodo triggers a fresh lookup,
caches the port on success, and returns the structured result. The RCON
diagnostic panel gains a Probe Komodo button (shown when endpoint is
unresolved and Komodo is configured) that renders the snippet inline —
so unrecognized shapes surface directly instead of vanishing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
rconGetStatus previously collapsed at least five distinct failure modes
(no password, no port, socket error, timeout, bad reply) into a single
null return, so the UI could only say "rcon unreachable" — useless when
diagnosing a real Alcatraz outage.
Refactored to rconStatusResult() returning a discriminated union with a
RconFailure reason and human-facing detail. The worker persists these as
"<reason>:<detail>" in last_error (no schema change). /api/servers now
returns the derived endpoint (host:port) per server plus a top-level
integrations block (rcon/komodo configured, timeout, default host).
Server detail page gains an RCON diagnostic panel when health is not
online: shows the exact endpoint being dialed, integration status,
timeout, a color-coded reason tag, the detail, and a targeted fix hint.
Wrong-password replies are detected via "bad rcon" text in the print
body.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Themes the fleet manager around the Electric Cherry perk-a-cola from Mob of
the Dead — cherry-red art-deco label with electric cyan discharge. New
design tokens (Alfa Slab One + Inter + JetBrains Mono, self-hosted), full
redesign of layout/dashboard/cards/detail page, tokens pass on
deployments/betas/pipeline/beta/diff, cherry+lightning favicon.
Lights up server data that RCON was already fetching but the UI dropped.
parsePlayerRoster() extracts num/score/ping/guid/name from both status
dialects. Worker persists last_map/last_gametype/last_roster/last_seen_at/
last_error (new columns with idempotent boot-time ALTERs) and emits them
over SSE. Fleet cards now show health dot, map · gametype, top-3 roster
preview, and last-seen. Detail page adds a sortable full roster with
distinct empty-lobby vs offline states. New health classifier
(online/stale/offline) drives the visual state.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
IW4M kept hitting a network wall (container -> host LAN IP -> IW4M
loopback bind) that we couldn't cleanly punch through, and the two-minute
!rt token added enough operational friction that the pause-all-deployments
gate never got a fair test. RCON gives us the same fact — how many
players are on a server right now — direct from each game server over
UDP, with no session, no token, no proxy.
- Add integrations/rcon.ts: dgram UDP client. rconGetPlayerCount() and
rconGetStatus() parse both `status` (print + scoreboard) and
`statusResponse` (info line + rows) dialects. Timeout via
RCON_TIMEOUT_MS.
- Add komodoResolveGamePort() + extractUdpPort(). Reads Komodo's read
API for a server, walks both docker-compose-style port objects and
docker-run-style "host:container/udp" strings, caches into
servers.game_port on first success.
- Replace pipelineReady() guard in sweep with a simple
integrations-configured check. Same held/failed/manual outcomes,
but keyed on RCON+Komodo instead of IW4M+auth-state.
- Worker's status refresh fans out per-server over RCON at concurrency=4
so a 15-server fleet doesn't spike UDP.
- Schema: drop iw4m_match/iw4m_value, add game_port + rcon_host.
- Delete every auth artifact: TokenModal, auth store, auth-state module,
/api/auth/token, /api/events/auth, iw4m.test.ts. Layout no longer
mounts the modal, pipeline store no longer tracks a frozen state.
- README + .env.example + docker-compose.yml + komodo/stack.toml swap
IW4M_* env for RCON_PASSWORD / RCON_HOST / RCON_TIMEOUT_MS. Same
host.docker.internal:host-gateway extra_hosts entry, repurposed
for RCON reach.
- Tests: add rcon.test.ts and komodo.test.ts, update sweep.test.ts to
drop the auth-freeze case. 23/23 passing.
Ports push_scripts.sh (1497 lines of bash) to a SvelteKit + SQLite web app
that manages a fleet of T5 Plutonium servers running on Komodo. Same push
and beta workflow as the shell script, with a live SSE pipeline view, a
!rt-token-gated auth modal that hard-pauses deployments on stale tokens,
per-file audit diffs, and a Komodo-container deployment target.
- Push/beta engines port the exact filter, held-by-beta, conflict, and
stale-backup rules from the shell script
- SQLite (better-sqlite3) replaces script_metadata.json; a one-shot
importer moves legacy state over on first boot
- 16 vitest tests cover push add/update/skip, beta open→rollback,
open→promote, conflict guards, sweep frozen/manual branches, and IW4M
login/status parsing
- docker/ + komodo/stack.toml deploy as a managed container on the same
host as the game servers, /root/zombies bind-mounted at the same path
the shell script used
josh
