Clean up README and organize documentation

docs/development/README.md

@@ -0,0 +1,114 @@
+# Development
+
+The controller uses SvelteKit, TypeScript and SQLite (`better-sqlite3`), with the
+Node adapter and server-sent events. The host agent uses Python 3's standard
+library and runs as a systemd service; hosts do not need Node.js.
+
+Agents connect outbound to the controller over HTTPS and manage Docker/Plutainer
+and local RCON on each host. The controller owns fleet metadata and queues; it
+does not require a Docker socket or direct access to host filesystems.
+
+## Run locally
+
+Use Node.js 22 (the container's version), npm and Python 3.
+
+```sh
+npm ci
+npm run agent:package
+export EC_SETUP_TOKEN="$(openssl rand -hex 32)"
+export BETA_ARCHIVE_DIR=./data/beta-archive
+EC_AUTH_ALLOW_HTTP=true npm run dev
+```
+
+Use the exported token at `/setup` to create the first administrator, then remove
+it from the environment. Local metadata defaults to `./data/cherry.db`; use a
+separate database when testing enrollment. HTTP login is for local development;
+production agent transport requires HTTPS. See [authentication](../guides/authentication.md)
+for access and recovery details.
+
+## Checks and builds
+
+```sh
+npm run check
+npm test
+npm run test:agent
+npm run build
+```
+
+Builds regenerate the downloadable agent, standalone updater and checksum
+manifest. They also bundle report workers under `build/reports` and launch
+preparation under `build/launch`. Development and test commands compile those
+workers automatically. Restart `npm run dev` after changing worker code.
+
+Automated tests simulate Docker and game hosts. Real Debian installation and
+T5/T6 server validation use the [host acceptance checklist](../guides/native-hosts.md#development-and-validation).
+
+## Code layout
+
+Paths below are relative to the repository root.
+
+| Path | Responsibility |
+| --- | --- |
+| `src/routes` | Pages and API endpoints |
+| `src/lib/ui` | Shared components, stores and [UI patterns](ui-patterns.md) |
+| `src/lib/server/hosts` | Enrollment, readiness and command transport |
+| `src/lib/server/backends` | Instance identity and remote file access |
+| `src/lib/server/provisioning/native.ts` | Lifecycle and campaign queues connected to agents |
+| `src/lib/server/game-files` | Immutable uploads and streaming distribution |
+| `src/lib/server/db` | SQLite schema, migrations and shared artifact storage |
+| `src/lib/server/reports` | Metrics, leaderboard and capacity workers |
+| `agent` | Python service, installer, packaging and tests |
+
+## Performance diagnostics
+
+Set `EC_PERF_LOG=true` to log slow requests, report computation/cache timings and
+event-loop delay. Responses expose `Server-Timing`; browser navigation records an
+`ec:navigation` performance measure. Report workers use separate read-only SQLite
+connections and bounded caches. Metrics snapshots can be reused for up to 30
+seconds when their inputs are unchanged.
+
+Benchmark the leaderboard against an isolated synthetic database:
+
+```sh
+npm run benchmark:leaderboard
+npm run benchmark:leaderboard -- --scenario all --baseline-ref <commit>
+```
+
+The comparison checks response parity and warm query timings.
+
+Set `EC_SLOW_SQL_MS=20` to log runtime statements taking at least 20 ms. Logging
+omits bound values and string literals and is disabled by default. Capture timings
+under the same fleet load before and after database changes. Inspect database
+size, large tables/indexes, WAL size and free pages inside the controller with:
+
+```sh
+npm run diagnostics:database -- /data/cherry.db
+```
+
+This opens an existing database read-only without running migrations. Substitute
+`./data/cherry.db` for a default local installation.
+
+## Storage and upgrades
+
+Game-file uploads live in dedicated filesystem storage. Script, plugin and
+configuration artifact bodies are stored once by SHA-256 in SQLite; versions,
+staging plans, rollback snapshots and launch commands reference those shared
+bytes. Startup transactionally backfills older rows. Current agents fetch
+assigned artifacts through an authenticated endpoint and verify their hashes;
+older agents receive inline content until they update.
+
+Back up the persistent data directory before upgrading. Startup logs show
+migration phases, elapsed time and resident memory. Completed migrations are
+retained across restarts; interrupted transactions roll back and retry on startup.
+The one-time vacuum needs disk space for a temporary database copy. **Do not
+remove `cherry.db-wal` to recover from an interrupted upgrade.**
+
+The controller uses a 128 MiB page cache; each reader worker uses 32 MiB.
+Main-database migrations use 16 MiB with memory mapping disabled. Temporary
+storage stays disk-backed. Containers use `/data/.sqlite-tmp` for scratch files;
+override `SQLITE_TMPDIR` only with another writable, disk-backed directory.
+
+For a local data disk, `EC_SQLITE_MMAP_MB=256` enables memory-mapped reads; leave it
+at zero for network mounts. Query statistics are optimized at startup and hourly.
+WAL reuse is capped at 64 MiB after checkpoints, though active transactions can
+temporarily grow it larger.