josh/Vetting

Fork 0

Files

T

josh 8367ec2a9f

CI / Lint + build + test (push) Successful in 1m36s

Details

Release / detect (push) Successful in 5s

Details

Release / build-live-image (push) Has been skipped

Details

Release / bundle (push) Successful in 49s

Details

docs: comprehensive documentation expansion

Add 4 new doc files (configuration reference, development guide, API
reference with full request/response schemas, database schema), expand
the README with a feature list and how-it-works walkthrough, fix
missing Firmware and Burn stages in architecture.md and test-suite.md,
add threshold engine and host-mode agent sections, and add godoc
comments to 11 packages and 6 model types.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-04-23 18:37:26 -04:00

13 KiB

Raw Permalink Blame History

Configuration reference

The orchestrator reads a single YAML file at startup. Production installs use /etc/vetting/vetting.yaml; the dev default is deploy/vetting.example.yaml. Pass the path with --config:

vetting --config /etc/vetting/vetting.yaml

Every key has a compile-time default (see internal/config/config.go), so an empty file produces a working orchestrator bound to 127.0.0.1:8080 with PXE disabled.

`server`

Key	Type	Default	Description
`bind`	string	`127.0.0.1:8080`	Address and port the HTTP server listens on.
`public_url`	string	(empty)	External URL the orchestrator is reachable at from a browser. Used in notification click-throughs (e.g. `https://vetting.lan:8443`).
`tls.enabled`	bool	`false`	Terminate TLS at the orchestrator.
`tls.cert_file`	string	(empty)	Path to the PEM-encoded certificate.
`tls.key_file`	string	(empty)	Path to the PEM-encoded private key.

`database`

Key	Type	Default	Description
`path`	string	`./var/vetting.db`	SQLite database file. Created on first run.

`artifacts`

Key	Type	Default	Description
`dir`	string	`./var/artifacts`	Directory for per-run files (reports, fio logs, iperf logs, hold keys).
`retention_days`	int	`30`	Days to keep artifact files before the janitor prunes them. `0` = keep forever. DB rows are never pruned.

`logs`

Key	Type	Default	Description
`dir`	string	`./var/logs`	Directory for per-run append-only log files.
`retention_days`	int	`30`	Days to keep log files. `0` = keep forever.

`janitor`

Key	Type	Default	Description
`interval_minutes`	int	`60`	Minutes between cleanup sweeps. `0` defaults to `60`.

`dispatcher`

Key	Type	Default	Description
`max_concurrent_runs`	int	`3`	Semaphore limiting how many vetting runs execute in parallel.

`network`

Key	Type	Default	Description
`iperf_port`	int	`5201`	Port the orchestrator-supervised `iperf3 -s` binds to. The agent connects here during the Network stage.

`pxe`

PXE is disabled by default. Enable it after running vetting-pxe-setup.

Key	Type	Default	Description
`enabled`	bool	`false`	Enable dnsmasq + iPXE serving.
`interface`	string	(empty)	LAN NIC the dnsmasq proxy-DHCP binds to (e.g. `eth0`).
`subnet`	string	(empty)	LAN CIDR (e.g. `192.168.1.0/24`). Scopes the proxy-DHCP responses.
`orchestrator_url`	string	(empty)	URL the live-image agent uses to reach the orchestrator (e.g. `http://192.168.1.135:8080`). Baked into the iPXE kernel cmdline.
`tftp_root`	string	(empty)	Directory containing `ipxe.efi` + `undionly.kpxe`.
`live_dir`	string	(empty)	Directory containing `vmlinuz` + `initrd.img`. Served at `/live/*`.

dnsmasq runs in proxy-DHCP mode: it coexists with your existing router's DHCP server and only supplements PXE options. See operations.md for the full setup walkthrough.

`agent`

Key	Type	Default	Description
`asset_dir`	string	`<database.dir>/../assets`	Directory containing `vetting-agent-linux-amd64`. Served at `/assets/*` so the quick-register one-liner can download the agent binary. Empty string disables the route.

`notifiers`

An array of notification targets. Each entry declares a named notifier with a type-specific set of fields. Delivery is fire-and-forget (one attempt per event, 10 s timeout, failures logged).

ntfy

notifiers:
  - name: ops-ntfy
    type: ntfy
    server: https://ntfy.sh
    topic: vetting-YOUR-TOPIC

Field	Type	Description
`name`	string	Identifier referenced by `routes[].notifier`.
`type`	string	`ntfy`
`server`	string	ntfy server URL.
`topic`	string	Topic to publish to.

Discord

notifiers:
  - name: ops-discord
    type: discord
    webhook_url: https://discord.com/api/webhooks/XXX/YYY

Field	Type	Description
`name`	string	Identifier referenced by `routes[].notifier`.
`type`	string	`discord`
`webhook_url`	string	Discord webhook URL.

SMTP

notifiers:
  - name: ops-email
    type: smtp
    smtp:
      host: mail.lan
      port: 25
      from: vetting@lan.local
      to: [ops@lan.local]

Field	Type	Description
`name`	string	Identifier referenced by `routes[].notifier`.
`type`	string	`smtp`
`smtp.host`	string	SMTP server hostname.
`smtp.port`	int	SMTP server port.
`smtp.from`	string	Sender address.
`smtp.to`	string[]	Recipient addresses.

`routes`

Routes map notification events to notifiers by kind and severity. Each route is evaluated independently; an event can match multiple routes and fire on multiple notifiers.

routes:
  - match_severity: [critical]
    notifier: ops-ntfy
  - match_severity: [critical]
    notifier: ops-discord
  - match_kind: [RunCompleted]
    notifier: ops-ntfy

Field	Type	Description
`match_kind`	string[]	Event kinds to match: `StageFailed`, `SpecMismatch`, `HoldingOpened`, `RunCompleted`. Omit to match all kinds.
`match_severity`	string[]	Severities to match: `critical`, `warning`, `info`. Omit to match all severities.
`notifier`	string	Name of a declared notifier to deliver to.

`vetting`

Shared pipeline defaults that apply to all profiles.

`vetting.stages`

Ordered list of stage names the pipeline walks. Default:

vetting:
  stages:
    - Inventory
    - Firmware
    - SpecValidate
    - SMART
    - CPUStress
    - Storage
    - Network
    - Burn
    - GPU
    - PSU
    - Reporting

`vetting.thresholds`

Array of threshold rules evaluated against every /sensor batch. Rules apply across all profiles — a 92 C CPU limit fails both a 2-minute quick run and a 12-hour soak.

Field	Type	Description
`stage`	string	Stage selector. `*` matches any stage; exact name (e.g. `PSU`) limits to that stage.
`kind`	string	Measurement kind to match: `temp`, `psu_volt`, `iperf`, `fio_p99_us`, `nic_retrans`, `edac_ue`, `edac_ce`, `mce`, `smart_attr`, `fan`.
`key`	string	Key selector. Glob-ish matching: `` matches all, `cpu/` matches keys starting with `cpu/`, exact string for specific keys.
`op`	string	Comparison operator (see table below).
`value`	float	Threshold limit.
`nominal`	float	Reference value, only used by `within_pct` (e.g. `12.0` for a +12 V rail).
`unit`	string	Display unit (e.g. `C`, `V`, `Mbps`). Informational only.
`severity`	string	`critical` = fail the run immediately. `warning` = record for the report only.

Threshold operators:

Operator	Pass condition	Typical use
`lt`	`observed < value`	CPU temp < 92 C
`lte`	`observed <= value`	EDAC UE count <= 0
`gt`	`observed > value`	—
`gte`	`observed >= value`	iperf throughput >= 900 Mbps
`within_pct`	`abs(observed - nominal) / nominal * 100 <= value`	+12 V rail within 5 % of 12.0 V

Default thresholds (from deploy/vetting.example.yaml):

thresholds:
  - { stage: "*",       kind: temp,        key: "cpu/*",         op: lt,         value: 92,    unit: C,    severity: critical }
  - { stage: PSU,       kind: psu_volt,    key: "+12V",          op: within_pct, value: 5,     nominal: 12.0, severity: critical }
  - { stage: PSU,       kind: psu_volt,    key: "+5V",           op: within_pct, value: 5,     nominal: 5.0,  severity: critical }
  - { stage: PSU,       kind: psu_volt,    key: "+3.3V",         op: within_pct, value: 5,     nominal: 3.3,  severity: critical }
  - { stage: Storage,   kind: fio_p99_us,  key: "*",             op: lt,         value: 50000, severity: warning  }
  - { stage: Network,   kind: iperf,       key: throughput_mbps, op: gte,        value: 900,   severity: critical }
  - { stage: Network,   kind: nic_retrans, key: "*/rate",        op: lt,         value: 0.001, severity: warning  }
  - { stage: CPUStress, kind: edac_ue,     key: "*",             op: lte,        value: 0,     severity: critical }
  - { stage: CPUStress, kind: mce,         key: "*",             op: lte,        value: 0,     severity: critical }

`profiles`

Three built-in profiles control per-stage durations and probe knobs. Every profile exercises every probe and gate — only the durations scale. Quick is a ~10-minute same-day sanity check; deep is the 8-12 hour overnight soak; soak is the opt-in 36-40 hour extreme run.

Profile inheritance

A profile can declare inherit: <parent> to merge the parent's timeouts and defaults before applying its own overrides. Child keys win. The default soak profile inherits from deep.

`stage_timeouts`

Per-stage time limits. The orchestrator kills the agent's stage subprocess when a timeout fires.

Stage	quick	deep	soak
CPUStress	5 m	2 h	14 h
Storage	5 m	4 h	8 h
Network	2 m	35 m	2 h 30 m
Burn	3 m	3 h	20 h
PSU	1 m	10 m	15 m

`defaults`

Per-stage probe knobs shipped to the agent on /claim. Empty values mean "fall back to the agent's compile-time default".

`cpustress`

Knob	Type	Description	quick	deep	soak
`cpu_pass`	duration	`stress-ng --cpu` duration	2 m	60 m	12 h
`mem_pass`	duration	`stress-ng --vm` duration	2 m	60 m	(inherit)
`edac_poll`	duration	EDAC error counter polling interval	10 s	10 s	(inherit)

`storage`

Knob	Type	Description	quick	deep	soak
`mode`	string	`fio_sample` (skip badblocks) or `full_disk` (badblocks + fio)	fio_sample	full_disk	full_disk
`fio_size`	string	fio test file size (only in `fio_sample` mode)	1 GiB	(inherit)	(inherit)
`fio_time`	duration	fio runtime	3 m	2 h	6 h
`fio_bs`	string	fio block size	4 k	4 k	(inherit)
`fio_rw`	string	fio I/O pattern	randrw	randrw	(inherit)
`verify`	string	fio integrity mode (`md5` or empty)	md5	md5	(inherit)

`network`

Knob	Type	Description	quick	deep	soak
`duration`	duration	`iperf3` test duration	60 s	30 m	2 h

`burn`

Knob	Type	Description	quick	deep	soak
`duration`	duration	Total burn-in window (CPU + mem + disk + net simultaneously)	2 m	2 h	18 h
`cpu_workers`	string	`all` (= `runtime.NumCPU()`) or a numeric string	all	all	(inherit)
`mem_pct`	int	Percentage of MemAvailable to stress	50	70	(inherit)
`fio_on_spare`	bool	Run fio inside Burn (requires a spare partition)	true	true	(inherit)
`iperf_parallel`	int	Parallel stream count fed to `iperf3 -P`	2	4	8

Example profile block

profiles:
  quick:
    stage_timeouts:
      CPUStress: 5m
      Storage:   5m
      Network:   2m
    defaults:
      cpustress: { cpu_pass: 2m, mem_pass: 2m, edac_poll: 10s }
      storage:   { mode: fio_sample, fio_size: 1GiB, fio_time: 3m, fio_bs: 4k, fio_rw: randrw, verify: md5 }
      network:   { duration: 60s }
      burn:      { duration: 2m, cpu_workers: all, mem_pct: 50, fio_on_spare: true, iperf_parallel: 2 }
  deep:
    stage_timeouts:
      CPUStress: 2h
      Storage:   4h
      Network:   35m
    defaults:
      cpustress: { cpu_pass: 60m, mem_pass: 60m, edac_poll: 10s }
      storage:   { mode: full_disk, fio_time: 2h, fio_bs: 4k, fio_rw: randrw, verify: md5 }
      network:   { duration: 30m }
      burn:      { duration: 2h, cpu_workers: all, mem_pct: 70, fio_on_spare: true, iperf_parallel: 4 }
  soak:
    inherit: deep
    stage_timeouts:
      CPUStress: 14h
      Storage:   8h
      Network:   2h30m
    defaults:
      cpustress: { cpu_pass: 12h }
      storage:   { mode: full_disk, fio_time: 6h }
      network:   { duration: 2h }
      burn:      { duration: 18h, iperf_parallel: 8 }

Host-mode agent config

The persistent host-mode agent reads a separate file at /etc/vetting/host-agent.yaml. This is installed by the quick-register one-liner and is distinct from the orchestrator config.

Key	Type	Default	Description
`orchestrator_url`	string	(required)	URL of the orchestrator (e.g. `http://192.168.1.135:8080`).
`mac`	string	(auto-detected)	MAC address to heartbeat as. Auto-detected from the default route NIC if omitted.
`interval`	duration	`30s`	Heartbeat interval.

13 KiB Raw Permalink Blame History

Configuration reference

server

database

artifacts

logs

janitor

dispatcher

network

pxe

agent

notifiers

ntfy

Discord

SMTP

routes

vetting

vetting.stages

vetting.thresholds

profiles

Profile inheritance

stage_timeouts

defaults

cpustress

storage

network

burn

Example profile block

Host-mode agent config

13 KiB

Raw Permalink Blame History

`server`

`database`

`artifacts`

`logs`

`janitor`

`dispatcher`

`network`

`pxe`

`agent`

`notifiers`

`routes`

`vetting`

`vetting.stages`

`vetting.thresholds`

`profiles`

`stage_timeouts`

`defaults`

`cpustress`

`storage`

`network`

`burn`