Beta healthy-minute progress resets while servers remain healthy #149

Closed
opened 2026-10-12 18:41:30 -04:00 by goddard · 1 comment
Collaborator

Beta 84 repeatedly loses healthy-minute progress despite its evidence showing healthy servers. This moves servers from ready back to observing and prevents stable promotion readiness.

  • src/lib/server/fleet/beta-evaluation.ts:87 requires consecutive samples to be no more than exactly 60 seconds apart; longer gaps reset healthy minutes to zero.
  • src/lib/server-freshness.ts:3 allows health evidence to remain fresh for 90 seconds.
  • src/lib/server/pipeline/beta-plan-runner.ts:257 revokes readiness when the reset makes the requirement incomplete.

Observed in Beta 84:

Server Gap between healthy samples Progress reset
Buried 60.048 seconds 2.50 minutes → 0
Verrückt 60.005 seconds 2.00 minutes → 0

Both samples remain healthy, so these resets produce no unhealthy event or outage increment. This is separate from #148's telemetry-to-RCON monitoring gap.

Fix

  • Align beta healthy-minute continuity with the shared health-freshness policy, tolerating normal polling variation.
  • Preserve progress and promotion readiness across healthy samples within the supported freshness window.
  • Continue resetting progress on genuine unhealthy observations or expired evidence.
  • Add regression tests for gaps just above 60 seconds, the freshness boundary, and readiness after a previously satisfied requirement.

Validation: Replaying production samples through the evaluator reproduced the displayed counters exactly. All nine existing evaluator tests passed; they do not cover this timing boundary.

Beta 84 repeatedly loses healthy-minute progress despite its evidence showing healthy servers. This moves servers from ready back to observing and prevents stable promotion readiness. - `src/lib/server/fleet/beta-evaluation.ts:87` requires consecutive samples to be no more than exactly 60 seconds apart; longer gaps reset healthy minutes to zero. - `src/lib/server-freshness.ts:3` allows health evidence to remain fresh for 90 seconds. - `src/lib/server/pipeline/beta-plan-runner.ts:257` revokes readiness when the reset makes the requirement incomplete. Observed in [Beta 84](https://ec.thewrightserver.net/betas/84): | Server | Gap between healthy samples | Progress reset | |---|---:|---:| | Buried | 60.048 seconds | 2.50 minutes → 0 | | Verrückt | 60.005 seconds | 2.00 minutes → 0 | Both samples remain healthy, so these resets produce no unhealthy event or outage increment. This is separate from #148's telemetry-to-RCON monitoring gap. **Fix** - [x] Align beta healthy-minute continuity with the shared health-freshness policy, tolerating normal polling variation. - [x] Preserve progress and promotion readiness across healthy samples within the supported freshness window. - [x] Continue resetting progress on genuine unhealthy observations or expired evidence. - [x] Add regression tests for gaps just above 60 seconds, the freshness boundary, and readiness after a previously satisfied requirement. **Validation:** Replaying production samples through the evaluator reproduced the displayed counters exactly. All nine existing evaluator tests passed; they do not cover this timing boundary.
goddard added the
priority
medium
type/bug
labels 2026-10-12 18:41:30 -04:00
josh was assigned by goddard 2026-10-12 18:41:30 -04:00
Author
Collaborator

Reviewed 272055a1ed, f91ec4d152. All fix items are complete; closing this issue.

Checklist

  • Shared policy. beta-evaluation.ts:88 uses ONLINE_MAX_MS rather than an exact 60-second interval.
  • Readiness preserved. Evaluator and durable-plan tests replay 60,005 ms, 60,048 ms and 90,000 ms gaps after the healthy-minute requirement has passed.
  • Failure boundaries retained. Explicit unhealthy samples and expired observations reset progress and revoke promotion readiness. Re-reading the same sample does not extend its lifetime.
  • Regressions. beta-evaluation.test.ts and beta-plans.test.ts cover the original production gaps, freshness boundary, real failures and readiness transitions.

Review notes

  • Reviewed current EC main at fa0baa8db5. Full application suite passed: 230 files, 3,120 tests, including migration-memory coverage; no failures or skips.
  • Python agent suite: 332 tests passed, no skips. Type/style checks: zero errors and warnings.
Reviewed 272055a1ed6ef6ce1fbd537fcc8bcdc276179361, f91ec4d15201a91923e34501e067e506b685b4f2. All fix items are complete; closing this issue. **Checklist** - [x] **Shared policy.** `beta-evaluation.ts:88` uses `ONLINE_MAX_MS` rather than an exact 60-second interval. - [x] **Readiness preserved.** Evaluator and durable-plan tests replay 60,005 ms, 60,048 ms and 90,000 ms gaps after the healthy-minute requirement has passed. - [x] **Failure boundaries retained.** Explicit unhealthy samples and expired observations reset progress and revoke promotion readiness. Re-reading the same sample does not extend its lifetime. - [x] **Regressions.** `beta-evaluation.test.ts` and `beta-plans.test.ts` cover the original production gaps, freshness boundary, real failures and readiness transitions. **Review notes** - Reviewed current EC main at fa0baa8db5f7629de3f58f2e6bf8da0568018be0. Full application suite passed: 230 files, 3,120 tests, including migration-memory coverage; no failures or skips. - Python agent suite: 332 tests passed, no skips. Type/style checks: zero errors and warnings.
Sign in to join this conversation.