feat(cancel): allow cancel from FailedHolding, reboot to local disk
A held run sits indefinitely at an SSH prompt waiting for operator
investigation. Previously the only exits were Override (re-enter the
failed stage) or leaving the host on forever — Cancel rejected any
terminal state, including FailedHolding, and there was no button in
the UI anyway.
Add a dedicated exit path:
- statemachine: TriggerOperatorCancelled now accepts FailedHolding
as a valid source, transitioning to Cancelled like any other
live state.
- CancelRun handler: treats FailedHolding as cancellable even
though IsTerminal reports true.
- heartbeat: Cancelled runs fork on FailedStage. Set means the
agent is parked in waitForOverride with no subprocess in
flight, so cmd=reboot tells it to systemctl reboot; the host
falls through iPXE's no-active-run script to the local disk.
Empty FailedStage keeps the pre-existing cmd=cancel_stage path
for mid-stage cancels (kill stage ctx, then power off).
- UI: canCancel now returns true for FailedHolding, and the
run-detail page renders a distinct "Cancel & reboot" button
with a hold-specific confirm message so the action doesn't
look identical to a mid-run cancel.
Tests cover the new statemachine transition, the heartbeat fork
(reboot vs cancel_stage), and keep the pre-existing mid-run cancel
behaviour locked in.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -73,7 +73,7 @@ var table = map[Trigger]transition{
|
||||
TriggerStageMismatch: {from: stageExecutionStates(), to: model.StateFailedHolding},
|
||||
TriggerAllStagesPassed: {from: []model.RunState{model.StateReporting}, to: model.StateCompleted},
|
||||
TriggerOperatorReleased: {from: []model.RunState{model.StateFailedHolding}, to: model.StateReleased},
|
||||
TriggerOperatorCancelled: {from: allActiveStates(), to: model.StateCancelled},
|
||||
TriggerOperatorCancelled: {from: append(allActiveStates(), model.StateFailedHolding), to: model.StateCancelled},
|
||||
}
|
||||
|
||||
// Next computes the target state for a trigger against the current state.
|
||||
|
||||
@@ -142,6 +142,30 @@ func TestStageNameForState(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestTriggerOperatorCancelledFromHold: cancelling a held run must be
|
||||
// allowed so the operator can walk away from a FailedHolding SSH prompt
|
||||
// and have the host reboot to local disk. Before this, FailedHolding
|
||||
// was considered terminal and Cancel errored out with "trigger not
|
||||
// allowed from FailedHolding".
|
||||
func TestTriggerOperatorCancelledFromHold(t *testing.T) {
|
||||
got, err := orchestrator.Next(model.StateFailedHolding, orchestrator.TriggerOperatorCancelled)
|
||||
if err != nil {
|
||||
t.Fatalf("FailedHolding + OperatorCancelled: %v", err)
|
||||
}
|
||||
if got != model.StateCancelled {
|
||||
t.Fatalf("got %q, want %q", got, model.StateCancelled)
|
||||
}
|
||||
// Sanity: other terminal states still reject the trigger so we don't
|
||||
// accidentally allow Cancel after Completed/Cancelled/Released.
|
||||
for _, bad := range []model.RunState{
|
||||
model.StateCompleted, model.StateCancelled, model.StateReleased, model.StateFailed,
|
||||
} {
|
||||
if _, err := orchestrator.Next(bad, orchestrator.TriggerOperatorCancelled); err == nil {
|
||||
t.Fatalf("OperatorCancelled from %q: expected error, got none", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNextStageWalk(t *testing.T) {
|
||||
// Walking StageCompleted from each stage should land on the next
|
||||
// one in the canonical order, and from Reporting onto Completed.
|
||||
|
||||
Reference in New Issue
Block a user