Vetting/docs/architecture.md

Architecture

A single Go binary runs the orchestrator. A second Go binary runs inside a custom Debian live image (built with mkosi) and becomes the per-run test agent. The two talk over HTTP + SSE.

Operator browser (HTMX + SSE, admin login)
   │ HTTPS
   ▼
┌───────────────────────────────────────────────────────────────┐
│  Orchestrator LXC — single Go binary `vetting`                │
│                                                               │
│   UI (Templ) ─┬─ Agent API ─┬─ SSE hub                        │
│               │             │                                 │
│         Orchestrator core (state machine, dispatcher sem=3,   │
│         stage executors, WoL sender, token issuer)            │
│               │                                               │
│         ┌─────┴─────┬──────────┐                              │
│         ▼           ▼          ▼                              │
│     SQLite   flat-file logs   dnsmasq subprocess              │
│                                (DHCP+TFTP+HTTP, MAC allowlist)│
│                                                               │
│         Janitor goroutine (retention-based cleanup)           │
│         Notifier registry (ntfy/discord/smtp)                 │
└─────────────────────────────────────────┬─────────────────────┘
                                          │ LAN
                                          ▼
                               Host under test (×2–3)
                               PXE → iPXE → Linux live image
                                 └─ vetting-agent (HTTP+SSE back)

Packages

Package	Purpose
cmd/vetting	Orchestrator entrypoint. Wires config, stores, runner, dispatcher, iperf supervisor, PXE supervisor, janitor, HTTP router.
cmd/vetting-agent	In-image agent entrypoint. Reads kernel cmdline params, starts the agent loop.
internal/config	YAML loader + types.
internal/db	SQLite open + embedded migrations. Pure Go via modernc.org/sqlite.
internal/model	Plain structs: Host, Run, Stage, Measurement, SpecDiff, Artifact.
internal/store	Repository layer; SQL is hand-written.
internal/orchestrator	State machine, dispatcher, per-run runner, WoL sender, HMAC run tokens, iperf supervisor.
internal/api	HTTP handlers: agent_handlers.go (the agent-facing API) and ui_handlers.go (HTMX fragments + SSE).
internal/httpserver	chi router assembly — lives here to avoid api ↔ orchestrator cyclic imports.
internal/web	Embedded static assets + compiled Templ templates.
internal/pxe	dnsmasq subprocess supervisor + per-MAC iPXE script generator.
internal/events	In-process SSE hub (fan-out to live browser clients).
internal/logs	Per-run flat-file writer + SSE fan-out of live log tail.
internal/spec	Expected-vs-actual diff engine with severity classification.
internal/notify	Pluggable notifier registry (ntfy, Discord webhook, SMTP).
internal/report	HTML + JSON report generation (html/template, self-contained).
internal/hold	Per-run SSH key issuance for FailedHolding.
internal/janitor	Retention-based cleanup of old artifact files + log files.
agent/	In-image agent: claim loop, stage dispatch, heartbeat, log forwarder, thermal sidecar.
agent/probes	lshw, dmidecode, smartctl, lspci, hwmon, nvidia-smi wrappers.
agent/tests	Per-stage test implementations (SMART, CPUStress, Storage, Network, GPU, PSU).
live-image/	mkosi config + postinst for the Debian live image.
deploy/	systemd unit + example config + install.sh.
test/e2e/	Build-tagged (-tags=e2e) QEMU + PXE full-stack test.

State machine

Per-run state is the single source of truth; the UI is a pure projection of DB + event stream.

Registered → Queued → WaitingWoL → Booting → InventoryCheck
  → SpecValidate → SMART → CPUStress → Storage → Network
  → GPU → PSU → Reporting → Completed

any stage → Failed → FailedHolding → Released

Key points:

• Transitions are table-driven (internal/orchestrator/statemachine.go). Each (state, event) → (next, action) is encoded once.
• Orchestrator-owned stages resolve inside /result: SpecValidate and Reporting flip state forward as part of the preceding stage's result handler, so the agent never sees them as "its turn".
• Stage rows persist before SSE fan-out — the UI can re-derive state by reading SQLite, and an SSE reconnect mid-run just fetches fresh tile fragments.

Agent ↔ orchestrator protocol

GET  /ipxe/{MAC}                     → per-MAC iPXE script
POST /api/v1/runs/{id}/hello         → "I booted; here's my address"
POST /api/v1/runs/{id}/claim         → validate token, receive stage list
POST /api/v1/runs/{id}/heartbeat     → liveness ping; response carries cmd
POST /api/v1/runs/{id}/log           → batch of log lines
POST /api/v1/runs/{id}/sensor        → batch of measurements (thermals, throughput)
POST /api/v1/runs/{id}/result        → stage result; response says next_state
POST /api/v1/runs/{id}/hold          → on FailedHolding, receive authorized_key

Auth on every /api/v1/* call: the bearer token is stored as a bcrypt hash in runs.agent_token_hash and compared in constant time. The plaintext is in the kernel cmdline — unforgeable by anyone not on the trusted bridge, because the iPXE script is issued per-MAC and the MAC must already be in the dnsmasq allowlist.

Heartbeat control channel

The heartbeat response carries a cmd field the agent acts on:

cmd	When fired	Agent action
continue	Normal case	No-op; keep running current stage
shutdown	Run reached Completed	systemctl poweroff
abort	Run in FailedHolding or Released	Stop heartbeat loop; let the operator drive
retry_stage	Operator pressed "Override wipe"	Re-enter the named stage with override_flags armed

Safety: destructive disk tests

Four layered gates:

1. MAC allowlist — dnsmasq only answers DHCP for registered MACs.
2. Signed run token — orchestrator issues a per-run HMAC token in the iPXE kernel cmdline; the agent submits it on /claim and the orchestrator verifies before handing back the stage list.
3. Wipe probe — before badblocks, the agent scans for filesystem signatures / LVM metadata / partition tables. Anything found → FailedHolding on Storage. The operator explicitly clicks Override wipe-probe to proceed.
4. Device allowlist — the agent only targets block devices matching the inventory's expected_disks. USB sticks and surprise disks are skipped.

Notifications

Fire-and-forget. The orchestrator fires four event kinds:

Kind	Severity	When
StageFailed	critical	Any stage returns passed=false
SpecMismatch	critical	SpecValidate finds critical diffs
HoldingOpened	critical	Agent POSTs /hold (operator can SSH in)
RunCompleted	info	Pipeline reaches Completed

The config maps event kinds and severities to one or more notifiers (ntfy, Discord webhook, SMTP). Each notifier gets one attempt per event with a 10s timeout; delivery failures are logged, nothing is persisted.

Why a separate notify package?

Keeps the /result and /hold handlers non-blocking. Each dispatch starts a goroutine per target; a slow ntfy server doesn't back up an SMTP notifier or delay the HTTP response to the agent.

Data retention

The janitor goroutine (internal/janitor) runs a sweep every janitor.interval_minutes (default 60) and deletes:

• artifact files older than artifacts.retention_days, plus their artifacts table rows
• log files older than logs.retention_days

runs, hosts, stages, measurements, spec_diffs rows are never deleted by the janitor — host histories and aggregate metrics survive cleanups.

Reproducible builds

The orchestrator and agent are pure Go; make orchestrator-linux cross-compiles to linux-amd64 from Windows or macOS.

The live image requires Linux-side tooling (mkosi, debootstrap, squashfs-tools) so make live-image fails loudly on Windows and redirects to wsl make live-image. Pinning to snapshot.debian.org in live-image/mkosi.conf keeps image bits stable across time for a given git SHA.