electric-cherry/docs/development/README.md

# Development

The controller uses SvelteKit, TypeScript and SQLite (`better-sqlite3`), with the
Node adapter and server-sent events. The host agent uses Python 3's standard
library and runs as a systemd service; release verification calls `gpgv`.
Hosts do not need Node.js.

Agents connect outbound to the controller over HTTPS and manage Docker/Plutainer
and local RCON on each host. The controller owns fleet metadata and queues; it
does not require a Docker socket or direct access to host filesystems.

## Run locally

Use Node.js 22.13+ (the container uses Node.js 22), npm and Python 3. Install
GnuPG's `gpg`, `gpg-agent` and `gpgv` to run real release-signing integration tests; local test
runs skip those tests when either executable is absent.

```sh
npm ci
export EC_SETUP_TOKEN="$(openssl rand -hex 32)"
export BETA_ARCHIVE_DIR=./data/beta-archive
EC_AUTH_ALLOW_HTTP=true npm run dev
```

Use the exported token at `/setup` to create the first administrator, then remove
it from the environment. Local metadata defaults to `./data/cherry.db`; use a
separate database when testing enrollment. HTTP login is for local development;
production agent transport requires HTTPS. See [authentication](../guides/authentication.md)
for access and recovery details.

## Synthetic UI preview

Create a disposable fixture without connecting game hosts:

```sh
npx svelte-kit sync
fixture_parent=$(mktemp -d "${TMPDIR:-/tmp}/ec-ui.XXXXXX")
npm run seed:fixture -- --db "$fixture_parent/preview/cherry.db" --watch
```

The sync step generates the SvelteKit TypeScript configuration that the seeder
needs in a fresh checkout.

The seeder prints the generated paths. Leave that terminal open to refresh
synthetic observations every 15 seconds. In a second terminal, load the printed
environment file and start the controller on localhost:

```sh
source /absolute/path/printed/above/preview.env
npm run dev -- --host 127.0.0.1 --port 3000
```

Open `http://127.0.0.1:3000/login`. The private `fixture.json` next to the database
contains the `demo` account's generated password. It is not printed in command
output. To run a fixed snapshot instead of a live feeder, use a fresh directory
and replace `--watch` with `--now 2026-10-07T12:00:00Z`. Domain data is deterministic
for that timestamp; account credentials are always generated separately. Old
snapshots naturally show stale observations when viewed against today's clock.

The fixture contains 3 hosts, 12 canonical Black Ops I/II servers, and 500 match
records: 494 historical and 6 live. It includes solo through four-player squads,
unknown starting counts, enough players and matches to exercise pagination,
rosters, seven days of occupancy samples with unknown player limits, and retained Activity
from campaigns, betas, host operations, maintenance, recovery, and autoscaling.
The fleet has 6 live matches, 2 empty lobbies, a preparing server, and 3 problems:
a stopped server, a server not responding, and a server on the offline host.
Script and config downloads contain harmless synthetic text.

`--db` must be an absolute path in a **new directory** whose parent exists. The
seeder refuses existing directories, databases, sidecars and symlinks, and never
uses an inherited `DB_PATH`. It runs the application's database migrations with
isolated paths, without starting the web server or workers. A failed seed removes
only its newly reserved directory. Completed fixtures remain after Ctrl+C.

The preview controller runs its usual local workers. Automatic maintenance and
autoscaling are paused; all seeded operations are terminal. Hostnames, runtime
images, library IDs and credentials are synthetic, so there are no runnable game
files or authenticated agents. Use this environment only for local UI review.
The feeder checks the fixture marker and restores only its known observations,
including the deliberately offline host. Stop both terminals when finished and
remove the generated fixture directory.

## Checks and builds

```sh
npm run lint
npm run check
npm test
npm run test:agent
npm run build
```

The [Gitea Actions workflow](../../.gitea/workflows/ci.yml) runs these checks on
every push and pull request after `npm ci`. It uses a Node.js 22 Debian container
with Python 3, `gpg`, `gpg-agent`, `gpgv` and native build tools for `better-sqlite3`. Signing
tests create disposable keys in temporary directories; CI never receives release
private keys. The runner must support Docker jobs with the `ubuntu-latest` label.
`npm run check` runs before the tests
to generate the SvelteKit configuration needed in a fresh checkout. Application
tests use two workers in CI to bound resource use for database-heavy fixtures.
The build step runs the same `npm run build` command used by the Dockerfile,
including agent packaging and the report and launch bundles.

Run `npm run format` to apply Prettier formatting before committing. It uses
four-space indentation and a 100-column target for supported source, config and
documentation files, including Svelte components. Python, shell and game scripts
are outside Prettier's supported languages. Generated agent downloads and the npm
lockfile keep their generator's formatting.

`npm run lint` checks formatting without changing files, then runs the recommended
JavaScript, TypeScript and Svelte ESLint rules. Generated build output and local
runtime data are excluded. Type checking remains part of `npm run check`.
The ESLint config documents exceptions for existing Svelte navigation, list and
reactivity patterns. Individual suppressions explain intentional validation and
test fixtures. Node.js 22.13 is the minimum supported 22.x version for the lint
toolchain.

The `package.json` overrides keep SvelteKit 2 on patched `devalue` 5.x (5.9.4 or
newer) and `cookie` 0.7.x releases until its dependency ranges include those fixes.
These dependencies ship in the server bundle even though SvelteKit is a
development dependency.
Dependency security checks must include development dependencies: use `npm audit`
without `--omit=dev`.

`npm run dev` and `npm run build` regenerate the downloadable agent, installer,
standalone updater and checksum manifest. Packaging preserves an existing
`static/agent/manifest.json.sig` only when its manifest bytes remain identical;
changes to the agent, updater or installer remove the stale signature. Builds
never sign, and hosts reject unsigned releases while retaining installed code.
Finalize and review release code, run `npm run agent:package`, and explicitly
sign it on the offline publisher machine:

```sh
python3 agent/sign_release.py --gnupghome /secure/electric-cherry-signing --fingerprint FULL_SIGNING_KEY_FINGERPRINT
```

Commit or distribute the signed public artifacts before building the controller
image from the same source revision. Keep the signing home outside the repository, Docker context, controller
and CI. Follow [Publishing signed releases](../guides/native-hosts.md#publishing-signed-releases)
for key creation, host provisioning and rotation.

Builds also bundle report workers
under `build/reports` and launch preparation under `build/launch`. Development
and test commands compile those workers automatically. Restart `npm run dev`
after changing agent or worker code, or run `npm run agent:package` to refresh
only the downloadable agent assets while the development server is running.

To run the production build locally, run these commands from the repository root
with the controller environment configured as above:

```sh
npm run build
NODE_ENV=production node build
```

`NODE_ENV=production` makes the report and launch workers load from `build/reports`
and `build/launch`. Without it, they look under `.svelte-kit/`, where the production
build does not create worker bundles. The Docker image sets this variable already.

Automated tests simulate Docker and game hosts. Real Debian installation and
T5/T6 server validation use the [host acceptance checklist](../guides/native-hosts.md#development-and-validation).

## Code layout

Paths below are relative to the repository root.

| Path                                    | Responsibility                                              |
| --------------------------------------- | ----------------------------------------------------------- |
| `src/routes`                            | Pages and API endpoints                                     |
| `src/lib/ui`                            | Shared components, stores and [UI patterns](ui-patterns.md) |
| `src/lib/server/hosts`                  | Enrollment, readiness and command transport                 |
| `src/lib/server/backends`               | Instance identity and remote file access                    |
| `src/lib/server/provisioning/native.ts` | Lifecycle and campaign queues connected to agents           |
| `src/lib/server/game-files`             | Immutable uploads and streaming distribution                |
| `src/lib/server/db`                     | SQLite schema, migrations and shared artifact storage       |
| `src/lib/server/reports`                | Metrics, leaderboard and capacity workers                   |
| `agent`                                 | Python service, installer, packaging and tests              |

## Performance diagnostics

Set `EC_PERF_LOG=true` to log slow requests, report computation/cache timings and
event-loop delay. Responses expose `Server-Timing`; browser navigation records an
`ec:navigation` performance measure. Report workers use separate read-only SQLite
connections and bounded caches. Metrics snapshots can be reused for up to 30
seconds when their inputs are unchanged.

Benchmark the leaderboard against an isolated synthetic database:

```sh
npm run benchmark:leaderboard
npm run benchmark:leaderboard -- --scenario all --baseline-ref <commit>
```

The comparison checks response parity and warm query timings.

Set `EC_SLOW_SQL_MS=20` to log runtime statements taking at least 20 ms. Logging
omits bound values and string literals and is disabled by default. Capture timings
under the same fleet load before and after database changes. Inspect database
size, large tables/indexes, WAL size and free pages inside the controller with:

```sh
npm run diagnostics:database -- /data/cherry.db
```

This opens an existing database read-only without running migrations. Substitute
`./data/cherry.db` for a default local installation.

## Storage and upgrades

Game-file uploads live in dedicated filesystem storage. Script, plugin and
configuration artifact bodies are stored once by SHA-256 in SQLite; versions,
staging plans, rollback snapshots and launch commands reference those shared
bytes. Startup transactionally backfills older rows. Current agents fetch
assigned artifacts through an authenticated endpoint and verify their hashes;
older agents receive inline content until they update.

Back up the persistent data directory before upgrading. Startup logs show
migration phases, elapsed time and resident memory. Completed migrations are
retained across restarts; interrupted transactions roll back and retry on startup.
The one-time vacuum needs disk space for a temporary database copy. **Do not
remove `cherry.db-wal` to recover from an interrupted upgrade.**

Database upgrades are ordered in `src/lib/server/db/migrations.ts` and tracked by
SQLite's `PRAGMA user_version`. Startup applies only pending versions, including
host, authentication and game-file schemas, before serving requests or starting
workers. Databases from older releases begin at version 0; the compatibility
steps honor their existing migration markers so completed backfills are preserved.
An application cannot open a database with a newer version than it supports.

For a schema or data upgrade, append a consecutive version to that list. Do not
edit an applied migration or rely on changing `schema.sql` to upgrade databases.
Each step and its version commit in one transaction. Table rebuilds can select
`foreign-keys-off`, which disables enforcement before the transaction and checks
references before committing. The resumable incremental-vacuum step uses
`autocommit` because SQLite forbids `VACUUM` inside a transaction; its version is
recorded only after the rebuild and host preparation boundary recovery succeed.
Connection tuning and query optimization still run on every open.

The controller uses a 128 MiB page cache; each reader worker uses 32 MiB.
Main-database migrations use 16 MiB with memory mapping disabled. Temporary
storage stays disk-backed. Containers use `/data/.sqlite-tmp` for scratch files;
override `SQLITE_TMPDIR` only with another writable, disk-backed directory.

For a local data disk, `EC_SQLITE_MMAP_MB=256` enables memory-mapped reads; leave it
at zero for network mounts. Query statistics are optimized at startup and hourly.
WAL reuse is capped at 64 MiB after checkpoints, though active transactions can
temporarily grow it larger.