Wrap the three install scripts in a shared inline style block (TTY/UTF-8/
NO_COLOR-aware) so the one-liner install looks and feels intentional:
banner on start, timed step lines, braille spinner over silent apt/
systemctl calls with failure log dumps, single-line curl progress bars
with size-prefixed headers, and a summary box at the end with live-image
version + service state + next steps. install.sh defers banner/summary
to proxmox-install.sh when VETTING_INSTALL_WRAPPED is set so the two
scripts compose without duplication.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Splits the release workflow into three jobs (detect, build-live-image,
bundle) so the ~9 min mkosi build only runs when live-image/VERSION
bumps. The slim bundle (~30 MB: orchestrator + agent + deploy scripts
+ a live-image/VERSION pointer) rebuilds every push; the ~300 MB
vmlinuz+initrd.img are published separately under the immutable
live-image/<version>/ path. install.sh compares the pointer to
/var/lib/vetting/live/VERSION and fetches the files only on mismatch,
cutting repeat-install wall-clock from ~30 s + 300 MB to ~10 s + 0 MB
on the common no-live-image-change release.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A stale /etc/vetting/vetting.yaml (e.g. pxe.interface=eth1 after an
LXC rebuild renamed the NIC to eth0) blocks vetting.service startup
with "pxe.interface 'eth1' not found on host", requiring the operator
to ssh in and hand-edit the yaml after every rebuild.
install.sh now validates the pxe block against the host's actual
network state on every install/upgrade run. If pxe.enabled is true and
pxe.interface doesn't exist (or pxe.subnet is missing/malformed), the
script auto-detects the primary NIC via the default route, reads its
subnet from the kernel-scope route, and patches both values in place.
Valid configs are left exactly as the operator had them; fresh
installs with pxe.enabled=false skip the check entirely.
The one-liner install/update is now self-healing for the most common
stale-config failure mode.
Ships all five phases of the deep-profile overhaul together. Runs now
carry a profile (quick/deep/soak); every profile walks the same
11-stage order — Inventory → Firmware → SpecValidate → SMART →
CPUStress → Storage → Network → Burn → GPU → PSU → Reporting —
with only per-stage durations and concurrency scaled.
Phase 1: profiles.ProfileRegistry loaded from vetting.yaml; runs.profile
column + CreateWithProfile; threshold table + evaluator seeded per-run
from the shared vetting.thresholds block; breach flips result at
/sensor + /result.
Phase 2: upgraded CPUStress (stress-ng --cpu-method=all --verify +
EDAC/MCE poll), Storage (fio --verify=md5 + SMART start/end delta),
Network (sustained iperf + /proc/net/dev deltas) with per-profile
knobs from Deps.
Phase 3: Burn super-stage with goroutine fan-out for CPU + memory +
fio + iperf, PSU rails sampled across the Burn window, SensorMux
(2 s flush, 500-sample cap) to absorb backpressure.
Phase 4: Firmware stage + firmware_snapshots table; probes dmidecode
(BIOS), ipmitool (BMC), ethtool -i (NIC), nvme (sysfs + id-ctrl),
lspci (HBA), /proc/cpuinfo (microcode). spec.DiffFirmware folds into
SpecValidate with pin-by-identifier and fan-out-across-component
matching; mismatches park the run in FailedHolding.
Phase 5: profile radio on the host start form, profile chip on the
run header, Firmware section in the HTML report, coverage artifact
uploaded from CI, agent/tests/fakes/ scaffold with Deps.LookPath
seam + stress_ng and dmidecode example fakes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop --progress-bar (curl's minimal hash meter) in favor of the default
progress output, which includes transfer rate and time remaining.
Bundles grew from ~30 MB to ~300 MB with the full-rootfs initrd, and
a percentage-only bar with no speed hint makes a slow registry look
indistinguishable from a hang.
proxmox-install.sh tarball-extracts into a tempdir that gets wiped on
EXIT, so after the one-liner there's no pxe-setup.sh on disk for the
operator to run. Have install.sh drop the script + ipxe-shas.txt into
/usr/local/share/vetting/ and symlink it as
/usr/local/sbin/vetting-pxe-setup (in PATH).
pxe-setup.sh now readlink -f's BASH_SOURCE so the symlink resolves to
the share dir where ipxe-shas.txt lives, and gracefully handles the
case where install.sh already staged vmlinuz + initrd.img into
LIVE_DIR (no bundle live-image/ needed at that point).
Update the trailing hint in proxmox-install.sh and the operations
runbook to surface the new `sudo vetting-pxe-setup ...` command.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously the orchestrator ran a full DHCP server on a dedicated
br-vetting bridge (10.77.0.0/24), which required a hypervisor-level
bridge + physical cabling onto that bridge for every repaired host.
Real-world bite: the LXC's br-vetting had no L2 path to the target
host's PXE NIC, so DHCPDISCOVERs never reached eth1 and PXE silently
timed out.
dnsmasq's proxy-DHCP mode is the idiomatic answer: it coexists with
the LAN's existing DHCP server (UniFi, etc.), never assigns an IP
itself, and only supplements the PXE options. No dedicated bridge,
no VLAN, no cabling changes \u2014 dnsmasq binds to the LAN interface
and layers option 66/67 + the PXE BINL on top of the real DHCP
exchange. The MAC allowlist still gates replies, so random LAN
clients booting from network get nothing.
Template switches dhcp-range=<start,end,lease> to
dhcp-range=<cidr>,proxy and replaces dhcp-boot= for first-boot ROM
clients with pxe-service= directives (the correct proxy-mode
chainload form). Validation drops the dhcp_range regex for a
net.ParseCIDR check on pxe.subnet. Config, production/example yaml,
and pxe-setup.sh swap --dhcp-range for --subnet.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
-fsSL suppresses all output during the ~30 MB download, which
leaves the operator staring at 'fetching bundle...' for up to a
minute on a cold registry. Drop -s and add --progress-bar so there
is a live indicator; keep -fL so we still fail on HTTP errors and
follow redirects. Print the downloaded size alongside 'extracting'
for quick sanity-checking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single-command upgrades were leaving /var/lib/vetting/live/ stale on
PXE-enabled LXCs because install.sh explicitly punted live-image
staging to pxe-setup.sh. That was right when make-release ran on a
dev box, but the new registry-pull flow ships vmlinuz+initrd.img
inside the bundle — they should land in place during every install.
install.sh now:
- auto-detects live-image/{vmlinuz,initrd.img} (release bundle
layout) or ../live-image/build/ (repo dev checkout) and stages
them into --live-dir (default /var/lib/vetting/live).
- restarts vetting.service when already enabled, so the
curl | sudo bash one-liner is the full upgrade loop. First-
install path still leaves the service stopped for config edits.
pxe-setup.sh's own live-image copy is now redundant on upgrade but
still runs for first-time PXE setup (it also writes the pxe: block
of vetting.yaml, which install.sh has no business touching).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drops the per-install Go toolchain dance + source build. The installer
now just curls the bundle from
${REGISTRY_URL}/api/packages/${PACKAGE_OWNER}/generic/vetting/${VETTING_VERSION}/vetting-bundle.tar.gz,
extracts it, and hands off to the bundled install.sh with explicit
--binary / --agent-binary paths so the in-bundle layout is picked up.
Default version is `latest` (rolling alias, overwritten by release.yml
on each push to main). Pin via `VETTING_VERSION=sha-abc1234 curl ... |
bash` when rolling back or testing a specific commit.
Removes the `apt install build-essential git` + Go toolchain download
+ templ install + `make orchestrator-linux agent-linux` path — the CI
workflow already produced all of that. Install time on a cold LXC
drops from minutes to under a minute, and live-image kernel/initrd
now arrive with every install instead of requiring a separate WSL
build.
Also rewrites docs/operations.md's install section around the
one-liner, keeps the `make release` + scp path as the offline
fallback, and swaps the upgrade section to just "rerun the one-liner."
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The production yaml ships `interface: "" # e.g. "eth0"`.
The old extractor did `gsub(/^"|"$/, "")` which only strips outer quotes, so
with an inline comment containing quotes it produced garbage like
`" # e.g. "eth0`, tripping the idempotency check.
Replaces the two inline extractors with one `extract_yaml_value` helper
that first tries to match `"[^"]*"` (grabbing only the first quoted
value), falling back to strip-trailing-comment + trim for unquoted
values.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Git on Windows dropped the exec bit when the files were first committed,
so `sudo ./pxe-setup.sh` on the LXC errored with "command not found".
Fix via `git update-index --chmod=+x`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Collapses the LXC side of PXE enablement from a six-step manual dance
(build, fetch iPXE, scp, bridge, hand-edit yaml) into:
make release # dev box (Linux/WSL)
scp bundle.tar.gz lxc:/tmp/
sudo ./install.sh # base install, unchanged
sudo ./pxe-setup.sh --interface ... --dhcp-range ... --orchestrator-url ...
pxe-setup.sh fetches iPXE from boot.ipxe.org, verifies against pinned
SHA256s in deploy/ipxe-shas.txt (fail-closed), places vmlinuz/initrd.img
from the bundle, and rewrites only the pxe: block of vetting.yaml.
Idempotent; --force gates overwriting a hand-edited block.
Adds Supervisor.Validate() — called before dnsmasq spawn — so typo'd
configs fail at orchestrator startup with clear errors naming the
missing file or yaml key, instead of silently serving broken TFTP
until a real host tries to PXE-boot. Nine tests cover missing files,
bogus interface, malformed dhcp_range, bad orchestrator_url, and
aggregate reporting.
Hypervisor bridge creation stays documented (LXC can't do it) but
everything downstream of the bridge is now scripted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The agent binary is never run on the LXC, but it has to be present
so /assets/vetting-agent-linux-amd64 can serve it to target hosts
via the quick-register one-liner. Install was failing because only
orchestrator-linux was being built.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Regenerated _templ.go files embed the templ source path at runtime,
which differs between the dev machine and /opt/vetting-src on the
target. That left tracked files modified after every build, and the
next upgrade-run hit "local changes would be overwritten by checkout"
and aborted. /opt/vetting-src is script-managed, so `git reset --hard
origin/<branch>` is the right semantics.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
vetting-agent gains a `host` subcommand that runs as a systemd service
installed by the quick-register one-liner, POSTing every 30s to
/api/v1/hosts/{mac}/heartbeat so the dashboard tile shows "online" or
"Nm ago" without waiting on WoL. Ships dormant client code for the
Phase 2 reboot_for_vetting command so the server can flip it on later
without a binary redeploy.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Can't log in from a fresh LXC deploy, and the service is LAN-only by
design. Rip out the whole bcrypt-password / signed-cookie session
layer: internal/auth, login templates, gen-admin-password binary +
Makefile targets, auth config block, login/logout routes and the
RequireSession middleware wrap. Agent bearer-token auth on
/api/v1/runs/{id}/* is untouched.
Operators who want a password can front the service with a reverse
proxy — noted in README and docs/operations.md.
Service was crashing on every boot because vetting.example.yaml uses
./var/... relative paths that resolve to / under ProtectSystem=strict.
Ship a separate vetting.production.yaml with absolute /var/lib/vetting
+ /var/log/vetting paths that match the unit's ReadWritePaths, and
have install.sh copy that one. Also move StartLimit* keys into [Unit]
to silence the 'Unknown key' warning on modern systemd.
proxmox-install.sh + install.sh left operators with no way to
generate the bcrypt hash on the LXC — 'vetting gen-admin-password'
was suggested in the post-install message but the binary has no
subcommands. Cross-build gen-admin-password-linux-amd64 during the
one-liner flow and drop it into /usr/local/bin.
deploy/proxmox-install.sh bootstraps a fresh LXC end-to-end: apt
prereqs, Go toolchain (if missing), git clone, build, then hands off
to deploy/install.sh. README documents the curl|bash invocation.
josh