cpustress+orchestrator: serial CPU/RAM passes + silent-skip guard
Orion's run (log 20:49 → 20:54) shipped GREEN while silently skipping CPUStress. Two compounding bugs: 1. CPUStress ran --cpu N AND --vm N --vm-bytes 90% concurrently. On a 4-core 8 GiB N95, that's 360% RAM overcommit; the OOM-killer fired, usually on the agent itself. Replaced with two sequential passes — CPU (all methods, --verify) for 3 min, then RAM (--vm 1, --vm-bytes capped to MemAvailable − 1.5 GiB, floor 256 MiB, --verify) for 3 min. Each pass now also asserts elapsed ≥ target − 2s so a premature clean exit counts as failure instead of a silent pass. 2. On systemd-restart after the OOM, the agent hardcoded nextStage := "Inventory" and re-ran it. The orchestrator's /result handler advances run state via TriggerStageCompleted against the *current* RunState, not against body.Stage — so an Inventory result posted while the run was in StateCPUStress silently advanced CPUStress → Storage and marked CPUStress passed without it ever running. Two-layer defense for #2: - agent-side: /claim response now carries current_state; agent resumes at the matching stage on a re-claim (happy path). - server-side: new TriggerStageMismatch + StageNameForState helper backstop. If body.Stage doesn't match the run's current stage, /result parks the run in FailedHolding with failed_stage labeled "<got> (expected <expected>)" and returns 409. Other stages audited for similar unbounded concurrency — none found; only CPUStress was unsafe. Tests: - cpustress_test.go — parseMemAvailable parses real meminfo, errors on missing/malformed; cap calc hits floor on tiny boxes, uses 1.5 GiB headroom on normal/huge boxes. - statemachine_test.go — TriggerStageMismatch lands at FailedHolding from every stage state and is rejected from pre-stage/terminal states; StageNameForState round-trips the stageStates map. - agent_handlers_test.go — TestResult_RejectsMismatchedStage proves the Orion scenario now 409s + FailedHolding; TestResult_AcceptsMatchingStage proves the guard doesn't break the happy path. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
+15
-2
@@ -69,7 +69,7 @@ func Run(ctx context.Context, p *bootstate.Params) error {
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
fwd.info(fmt.Sprintf("claimed run; stages=%v", claim.Stages))
|
||||
fwd.info(fmt.Sprintf("claimed run; stages=%v current_state=%s", claim.Stages, claim.CurrentState))
|
||||
|
||||
go thermalSidecar(ctx, c, fwd)
|
||||
|
||||
@@ -80,7 +80,20 @@ func Run(ctx context.Context, p *bootstate.Params) error {
|
||||
// orchestrator (SpecValidate, Reporting) resolve inside /result and
|
||||
// flip next_state forward past themselves, so they simply never match
|
||||
// our dispatch table.
|
||||
nextStage := "Inventory"
|
||||
//
|
||||
// Start stage comes from claim.CurrentState so a re-claim after an
|
||||
// agent crash resumes at the stage the run was parked at, instead of
|
||||
// blindly replaying Inventory and letting the orchestrator silently
|
||||
// advance past the crashed stage (the Orion OOM bug). A fresh claim
|
||||
// naturally lands on InventoryCheck, which maps back to "Inventory".
|
||||
nextStage := stageForState(claim.CurrentState)
|
||||
if nextStage == "" {
|
||||
nextStage = "Inventory"
|
||||
}
|
||||
if nextStage != "Inventory" {
|
||||
fwd.warn(fmt.Sprintf("resuming mid-pipeline at %s (claim current_state=%s) — likely agent restart after crash",
|
||||
nextStage, claim.CurrentState))
|
||||
}
|
||||
for nextStage != "" {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
|
||||
Reference in New Issue
Block a user