pxe.Supervisor.Reload() was defined but never wired up. After a host
was registered in the UI or via the quick-register JSON endpoint, the
dnsmasq conf still held only the hosts that existed at orchestrator
startup. The new MAC wasn't tagged `known`, so when the host PXE'd,
dnsmasq logged "PXE(eth0) <mac> proxy-ignored" and the boot timed out
back to the BIOS.
Add an optional PXEReloader interface to api.UI, wire it from main
when pxe is enabled, and call u.reloadPXE() after successful Create
and Delete. Logs-and-continues on failure — host registration itself
has already committed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
dnsmasq's proxy-DHCP syntax is `dhcp-range=<network-ip>,proxy[,<mask>]`,
not a CIDR. Passing "192.168.1.0/24,proxy" made dnsmasq refuse to start
with "bad dhcp-range at line 12". Parse the CIDR once in writeConf()
and render Network + Netmask as separate template fields.
The config surface (pxe.subnet) stays CIDR because that's the right
shape for humans; we just unpack it before handing to dnsmasq.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
proxmox-install.sh tarball-extracts into a tempdir that gets wiped on
EXIT, so after the one-liner there's no pxe-setup.sh on disk for the
operator to run. Have install.sh drop the script + ipxe-shas.txt into
/usr/local/share/vetting/ and symlink it as
/usr/local/sbin/vetting-pxe-setup (in PATH).
pxe-setup.sh now readlink -f's BASH_SOURCE so the symlink resolves to
the share dir where ipxe-shas.txt lives, and gracefully handles the
case where install.sh already staged vmlinuz + initrd.img into
LIVE_DIR (no bundle live-image/ needed at that point).
Update the trailing hint in proxmox-install.sh and the operations
runbook to surface the new `sudo vetting-pxe-setup ...` command.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rewrites the PXE section of the ops runbook around the new proxy-DHCP
model (no dedicated bridge, coexists with UniFi/pfSense/etc.) and
swaps the e2e test's default bridge + orchestrator URL to match. The
e2e file now calls out the LAN-DHCP precondition in its header so
future-me (or CI) doesn't hang at PXE wondering why nothing answers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously the orchestrator ran a full DHCP server on a dedicated
br-vetting bridge (10.77.0.0/24), which required a hypervisor-level
bridge + physical cabling onto that bridge for every repaired host.
Real-world bite: the LXC's br-vetting had no L2 path to the target
host's PXE NIC, so DHCPDISCOVERs never reached eth1 and PXE silently
timed out.
dnsmasq's proxy-DHCP mode is the idiomatic answer: it coexists with
the LAN's existing DHCP server (UniFi, etc.), never assigns an IP
itself, and only supplements the PXE options. No dedicated bridge,
no VLAN, no cabling changes \u2014 dnsmasq binds to the LAN interface
and layers option 66/67 + the PXE BINL on top of the real DHCP
exchange. The MAC allowlist still gates replies, so random LAN
clients booting from network get nothing.
Template switches dhcp-range=<start,end,lease> to
dhcp-range=<cidr>,proxy and replaces dhcp-boot= for first-boot ROM
clients with pxe-service= directives (the correct proxy-mode
chainload form). Validation drops the dhcp_range regex for a
net.ParseCIDR check on pxe.subnet. Config, production/example yaml,
and pxe-setup.sh swap --dhcp-range for --subnet.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
-fsSL suppresses all output during the ~30 MB download, which
leaves the operator staring at 'fetching bundle...' for up to a
minute on a cold registry. Drop -s and add --progress-bar so there
is a live indicator; keep -fL so we still fail on HTTP errors and
follow redirects. Print the downloaded size alongside 'extracting'
for quick sanity-checking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Without explicit dhcp-leasefile and pid-file, dnsmasq reaches for
its distro defaults (/var/lib/misc/dnsmasq.leases,
/run/dnsmasq.pid) — both outside the systemd unit's
ReadWritePaths=/var/lib/vetting /var/log/vetting sandbox, causing
'Read-only file system' on every start.
RuntimeDir is already writable by construction (Supervisor.Start
mkdir's it), so writing both files there keeps dnsmasq entirely
inside the sandbox.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously tftp_root defaulted to logs.dir/../tftp and the pxe
runtime dir to logs.dir/../pxe. On a production install that
resolves to /var/log/tftp and /var/log/pxe, both outside the
systemd unit's ReadWritePaths=/var/lib/vetting /var/log/vetting
sandbox. The service crash-looped with "mkdir /var/log/pxe:
read-only file system" as soon as PXE was enabled.
Switch the anchor to filepath.Dir(cfg.Artifacts.Dir) — typically
/var/lib/vetting — which the sandbox already allows.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The linux-image-amd64 postinst creates /vmlinuz but the paired
/initrd.img symlink only shows up via an initramfs-tools hook that
doesn't fire when we call update-initramfs ourselves. Without it,
the top-level Makefile's `cp live-image/build/initrd.img` fails and
`make release` aborts with a broken bundle.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two bugs chained together to ship a broken bundle:
1. With Bootable=no, mkosi skips update-initramfs, so no
/boot/initrd.img-<kver> ever gets generated inside the rootfs.
The postinst now runs update-initramfs via chroot to produce it.
2. The `make release` recipe chained its `cp` calls with `;`, so
a missing live-image/build/initrd.img silently failed and the
bundle still got tarred + uploaded. Adding `set -e` at the top
of the recipe makes any missing component fail the build loudly
instead of shipping a half-bundle.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single-command upgrades were leaving /var/lib/vetting/live/ stale on
PXE-enabled LXCs because install.sh explicitly punted live-image
staging to pxe-setup.sh. That was right when make-release ran on a
dev box, but the new registry-pull flow ships vmlinuz+initrd.img
inside the bundle — they should land in place during every install.
install.sh now:
- auto-detects live-image/{vmlinuz,initrd.img} (release bundle
layout) or ../live-image/build/ (repo dev checkout) and stages
them into --live-dir (default /var/lib/vetting/live).
- restarts vetting.service when already enabled, so the
curl | sudo bash one-liner is the full upgrade loop. First-
install path still leaves the service stopped for config edits.
pxe-setup.sh's own live-image copy is now redundant on upgrade but
still runs for first-time PXE setup (it also writes the pxe: block
of vetting.yaml, which install.sh has no business touching).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mkosi was failing with "systemd-boot was not found at
usr/lib/systemd/boot/efi" because Bootable=yes expects systemd-boot
installed *inside* the image for EFI boot. This image is only ever
PXE-booted — iPXE loads vmlinuz+initrd from TFTP directly, so the
rootfs itself needs no bootloader.
Switching to Bootable=no drops the EFI-image assembly step; the
linux-image-amd64 postinst still creates /vmlinuz and /initrd.img
symlinks that the top-level Makefile copies into the bundle.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mkosi only mounts live-image/ as /work/src, so the postinst couldn't
reach the repo-root bin/vetting-agent.linux-amd64 — the build failed
in CI with `install: cannot stat '/work/src/bin/vetting-agent.linux-amd64'`.
The Makefile now copies the prebuilt agent into mkosi.extra/, which
mkosi merges into the image root automatically. The postinst is
reduced to creating the multi-user.target.wants symlink.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After installing the kernel package into the live image, mkosi runs
depmod on the host against the image's module tree. depmod ships in
the kmod package, which isn't in the runner container by default.
mkosi Bootable=yes shells out to bootctl kernel-identify on the host,
which ships in the systemd-boot package on Ubuntu (not in systemd
itself). Without it, the live-image build fails at the end with
"bootctl: not found" after successfully installing all packages.
mkosi's apt-get (inside the mkosi workspace) couldn't verify Debian's
InRelease signatures because the act_runner's Ubuntu base image ships
Ubuntu's keyring, not Debian's. Adding `debian-archive-keyring` to the
apt install list exposes /usr/share/keyrings/debian-archive-keyring.gpg
which debootstrap and apt need for the bookworm repos.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mkosi refused with "Could not find 'ukify'". The live image's mkosi.conf
sets Bootable=yes, and mkosi invokes ukify to package the Unified
Kernel Image alongside vmlinuz+initrd.img. On Debian/Ubuntu, ukify
ships in the `systemd-ukify` apt package (not in `systemd`).
Added to both release.yml and e2e.yml's live-image dep lists.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mkosi refuses to run a non-executable postinst. git was tracking it
as 100644 because it was added from Windows (no POSIX exec bit on the
FS), so CI saw a non-executable file even though WSL/Linux had been
treating it fine locally. Same fix applied earlier to install.sh +
pxe-setup.sh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI's `go mod tidy` check caught the drift. The module was used
transitively but never recorded.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The action tries to restore from 172.18.0.2:36061 (Gitea's cache
server), times out, falls through to a fresh download anyway. Pure
waste since the runner already has the toolchain in
/opt/hostedtoolcache. Turn cache off.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Unquoted `(default: main)` trips Gitea Actions' strict YAML parser
with "mapping values are not allowed in this context" because the
inline colon reads as a nested mapping. GitHub Actions' parser was
lenient about this; Gitea's isn't.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The self-hosted Gitea runner advertises itself as `ubuntu-latest`,
not `self-hosted`, so the jobs were never getting picked up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drops the per-install Go toolchain dance + source build. The installer
now just curls the bundle from
${REGISTRY_URL}/api/packages/${PACKAGE_OWNER}/generic/vetting/${VETTING_VERSION}/vetting-bundle.tar.gz,
extracts it, and hands off to the bundled install.sh with explicit
--binary / --agent-binary paths so the in-bundle layout is picked up.
Default version is `latest` (rolling alias, overwritten by release.yml
on each push to main). Pin via `VETTING_VERSION=sha-abc1234 curl ... |
bash` when rolling back or testing a specific commit.
Removes the `apt install build-essential git` + Go toolchain download
+ templ install + `make orchestrator-linux agent-linux` path — the CI
workflow already produced all of that. Install time on a cold LXC
drops from minutes to under a minute, and live-image kernel/initrd
now arrive with every install instead of requiring a separate WSL
build.
Also rewrites docs/operations.md's install section around the
one-liner, keeps the `make release` + scp path as the offline
fallback, and swaps the upgrade section to just "rerun the one-liner."
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds `.gitea/workflows/{ci,e2e,release}.yml` and removes the old
`.github/workflows/` counterparts. Gitea reads both paths, so keeping
them would double-run every job on every push.
- ci.yml / e2e.yml are 1:1 ports of the GitHub versions, just with
`runs-on: self-hosted` (Gitea has no hosted runners).
- release.yml is new: fires on push to main, runs `make release`, then
publishes `vetting-bundle.tar.gz` to the Gitea generic package
registry under two versions — `sha-<short-sha>` (immutable, pinnable)
and `latest` (rolling alias, DELETE+PUT on each run). Auth via a
REGISTRY_TOKEN secret + REGISTRY_URL variable configured on the Gitea
side.
The runner is being reconfigured to privileged so `mkosi` + `debootstrap`
can build the live image inside CI.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The production yaml ships `interface: "" # e.g. "eth0"`.
The old extractor did `gsub(/^"|"$/, "")` which only strips outer quotes, so
with an inline comment containing quotes it produced garbage like
`" # e.g. "eth0`, tripping the idempotency check.
Replaces the two inline extractors with one `extract_yaml_value` helper
that first tries to match `"[^"]*"` (grabbing only the first quoted
value), falling back to strip-trailing-comment + trim for unquoted
values.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Git on Windows dropped the exec bit when the files were first committed,
so `sudo ./pxe-setup.sh` on the LXC errored with "command not found".
Fix via `git update-index --chmod=+x`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Collapses the LXC side of PXE enablement from a six-step manual dance
(build, fetch iPXE, scp, bridge, hand-edit yaml) into:
make release # dev box (Linux/WSL)
scp bundle.tar.gz lxc:/tmp/
sudo ./install.sh # base install, unchanged
sudo ./pxe-setup.sh --interface ... --dhcp-range ... --orchestrator-url ...
pxe-setup.sh fetches iPXE from boot.ipxe.org, verifies against pinned
SHA256s in deploy/ipxe-shas.txt (fail-closed), places vmlinuz/initrd.img
from the bundle, and rewrites only the pxe: block of vetting.yaml.
Idempotent; --force gates overwriting a hand-edited block.
Adds Supervisor.Validate() — called before dnsmasq spawn — so typo'd
configs fail at orchestrator startup with clear errors naming the
missing file or yaml key, instead of silently serving broken TFTP
until a real host tries to PXE-boot. Nine tests cover missing files,
bogus interface, malformed dhcp_range, bad orchestrator_url, and
aggregate reporting.
Hypervisor bridge creation stays documented (LXC can't do it) but
everything downstream of the bridge is now scripted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Re-running quick.sh on a host where vetting-reporter was already
running failed with curl error 23 because curl can't overwrite a
busy executable. Download to a staging path, then use `install(1)`
which unlinks the target before writing. Swap `enable --now` for
`enable` + `restart` so the service picks up the new binary.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Every supported host runs vetting-reporter in-OS and heartbeats every
30s. WoL was never the thing that started vetting — the heartbeat
response's reboot_for_vetting command was. Firing WoL first only
crowded the run log with misleading diagnostics when the real failure
mode is "reporter isn't installed."
- StartRun 409s if the host hasn't heartbeated within 60s, pointing
the operator at /register/quick.sh.
- Dispatcher re-checks LastSeenAt at dispatch time (run may sit in
Queued long enough for the host to go offline); stale hosts mark
the run Failed with failed_stage=dispatch instead of looping.
- New StateWaitingReboot + TriggerRebootCommanded capture the actual
semantics. StateWaitingWoL kept as the hook point for a future
manual-override button.
- Tile disables the Start button with a quick.sh tooltip when the
host is offline, matching the server-side 409.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Operators who installed vetting before agent.asset_dir existed keep
their config preserved by install.sh on upgrade, which left them
with AssetDir="" — the router silently dropped the /assets/*
mount and the quick-register one-liner hit 404 fetching the agent
binary. Default AssetDir alongside the database file so the same
directory install.sh already creates + drops the agent binary into
is picked up automatically.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pipeline now always renders all 13 nodes (3 pre-stage + 9 stage +
Completed), synthesising ghosts from run state when stage rows
aren't seeded yet. Makes a WaitingWoL host show the full timeline
ahead of it instead of just 4 dots.
Agent tags each log line with its stage; logs.Hub fans out to both
log-{runID} and log-{runID}-{stage} SSE events so the detail page
can show per-stage tabs with a pure-CSS radio-sibling switch. Flat
run log prepends [stage] so grep still works.
Dispatcher writes picked/sent-WoL/heartbeat lines into the per-run
log — the operator opens the detail page, sees WaitingWoL stuck,
and reads exactly what the dispatcher did and why nothing's
progressing, instead of having to tail journalctl on the LXC.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The agent binary is never run on the LXC, but it has to be present
so /assets/vetting-agent-linux-amd64 can serve it to target hosts
via the quick-register one-liner. Install was failing because only
orchestrator-linux was being built.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Regenerated _templ.go files embed the templ source path at runtime,
which differs between the dev machine and /opt/vetting-src on the
target. That left tracked files modified after every build, and the
next upgrade-run hit "local changes would be overwritten by checkout"
and aborted. /opt/vetting-src is script-managed, so `git reset --hard
origin/<branch>` is the right semantics.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Click a tile to open /hosts/{id} — the canonical control surface per
host. Timeline renders every pre-stage, stage, and terminal node in
order, with the current one pulsing, failed ones flagged, and
downstream ones dimmed as skipped. Detail page shows summary, hold
card (when holding), all action buttons, spec diffs, a full-height
log pane, and a collapsed expected-spec YAML.
Tile slims to name, last-seen, status, and one primary action; a
CSS-overlay <a> makes the whole card clickable while buttons stay
receptive via z-index.
Runner.publishTileUpdate now also emits pipeline-{runID} fragments,
and CompleteStage wraps Stages.CompleteByName so stage completions
advance the timeline live — without this the dots only moved on
state transitions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When the operator clicks Start vetting and the host is heartbeating,
the heartbeat response now carries cmd=reboot_for_vetting + run_id.
The handler drives the Queued → WaitingWoL transition via the existing
state machine, so a benign race with the 2s dispatcher poll is refused
by the state machine (not double-dispatched). WaitingWoL retries for
10 minutes to cover a crashed-mid-reboot case, then falls back to
operator action.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
vetting-agent gains a `host` subcommand that runs as a systemd service
installed by the quick-register one-liner, POSTing every 30s to
/api/v1/hosts/{mac}/heartbeat so the dashboard tile shows "online" or
"Nm ago" without waiting on WoL. Ships dormant client code for the
Phase 2 reboot_for_vetting command so the server can flip it on later
without a binary redeploy.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two bugs compounded on Proxmox hosts: primary_iface walked
`ip link show` and picked the physical NIC (e.g. enp1s0), which has
no IPv4 on Proxmox because the address lives on vmbr0. Even if vmbr0
had been picked, the kernel reports its broadcast as 0.0.0.0, so the
script fell all the way back to 255.255.255.255.
Now we prefer the default-route interface (vmbr0 on Proxmox, eno1 on
bare metal) and, when `ip` doesn't surface a usable `brd`, compute
the broadcast from the inet CIDR instead of giving up.
Operator pastes `curl -fsSL $ORCH/register/quick.sh | sudo bash` on the
target host (pre-wipe). The script probes MAC + CPU/RAM/disks/NICs/GPUs,
emits an expected-spec YAML, and POSTs to a new LAN-trusted JSON
endpoint /api/v1/hosts. The register page shows the command prefilled
with the orchestrator URL; the manual form moves into a collapsible
"Register manually" disclosure.
Can't log in from a fresh LXC deploy, and the service is LAN-only by
design. Rip out the whole bcrypt-password / signed-cookie session
layer: internal/auth, login templates, gen-admin-password binary +
Makefile targets, auth config block, login/logout routes and the
RequireSession middleware wrap. Agent bearer-token auth on
/api/v1/runs/{id}/* is untouched.
Operators who want a password can front the service with a reverse
proxy — noted in README and docs/operations.md.
Service was crashing on every boot because vetting.example.yaml uses
./var/... relative paths that resolve to / under ProtectSystem=strict.
Ship a separate vetting.production.yaml with absolute /var/lib/vetting
+ /var/log/vetting paths that match the unit's ReadWritePaths, and
have install.sh copy that one. Also move StartLimit* keys into [Unit]
to silence the 'Unknown key' warning on modern systemd.
proxmox-install.sh + install.sh left operators with no way to
generate the bcrypt hash on the LXC — 'vetting gen-admin-password'
was suggested in the post-install message but the binary has no
subcommands. Cross-build gen-admin-password-linux-amd64 during the
one-liner flow and drop it into /usr/local/bin.
deploy/proxmox-install.sh bootstraps a fresh LXC end-to-end: apt
prereqs, Go toolchain (if missing), git clone, build, then hands off
to deploy/install.sh. README documents the curl|bash invocation.
josh