feat(cancel): allow cancel from FailedHolding, reboot to local disk
CI / Lint + build + test (push) Successful in 1m38s
Release / release (push) Successful in 6m10s

A held run sits indefinitely at an SSH prompt waiting for operator
investigation. Previously the only exits were Override (re-enter the
failed stage) or leaving the host on forever — Cancel rejected any
terminal state, including FailedHolding, and there was no button in
the UI anyway.

Add a dedicated exit path:
  - statemachine: TriggerOperatorCancelled now accepts FailedHolding
    as a valid source, transitioning to Cancelled like any other
    live state.
  - CancelRun handler: treats FailedHolding as cancellable even
    though IsTerminal reports true.
  - heartbeat: Cancelled runs fork on FailedStage. Set means the
    agent is parked in waitForOverride with no subprocess in
    flight, so cmd=reboot tells it to systemctl reboot; the host
    falls through iPXE's no-active-run script to the local disk.
    Empty FailedStage keeps the pre-existing cmd=cancel_stage path
    for mid-stage cancels (kill stage ctx, then power off).
  - UI: canCancel now returns true for FailedHolding, and the
    run-detail page renders a distinct "Cancel & reboot" button
    with a hold-specific confirm message so the action doesn't
    look identical to a mid-run cancel.

Tests cover the new statemachine transition, the heartbeat fork
(reboot vs cancel_stage), and keep the pre-existing mid-run cancel
behaviour locked in.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-04-19 22:59:34 -04:00
parent 21014c1268
commit 62bddac110
9 changed files with 287 additions and 149 deletions
+14 -3
View File
@@ -272,9 +272,20 @@ func (a *Agent) Heartbeat(w http.ResponseWriter, r *http.Request) {
// intervention.
cmd = "reboot"
case run.State == model.StateCancelled:
// Operator clicked Cancel — agent cancels the active stage ctx,
// posts a cancelled outcome, and powers off.
cmd = "cancel_stage"
// Operator clicked Cancel. Two sub-cases:
// - FailedStage set → run was sitting in FailedHolding with no
// in-flight stage subprocess; the agent is parked in
// waitForOverride. Send cmd=reboot so the heartbeat loop
// reboots the host, falls through iPXE's no-active-run
// script and boots local disk.
// - FailedStage empty → cancel mid-stage; kill the stage ctx
// first so the running subprocess exits cleanly, then the
// agent powers off via its existing cancel path.
if run.FailedStage != "" {
cmd = "reboot"
} else {
cmd = "cancel_stage"
}
case run.State == model.StateFailedHolding || run.State == model.StateReleased:
cmd = "abort"
case run.FailedStage == "Storage" && overrideWipeSet(run.OverrideFlagsJSON):