Servers briefly become unhealthy after the last player leaves #148

Closed
opened 2026-10-12 18:37:01 -04:00 by goddard · 1 comment
Collaborator

When the last player leaves, EC can briefly mark a responsive server unhealthy during the transition from script telemetry to RCON monitoring. This resets beta continuous healthy minutes and can revoke readiness for promotion.

Observed in Beta 84 on October 12, 2026. All times Eastern:

Server Empty report Unhealthy Healthy again
Buried 6:29:17 PM 6:30:47 PM 6:31:28 PM
Verrückt 6:30:35 PM 6:32:13 PM 6:32:28 PM

Both unhealthy events had no recorded error. Beta 84 changed from “Test plan passed” back to active during this sequence.

  • src/lib/server/pipeline/worker.ts:218,335 skips RCON monitoring while fresh telemetry reports zero players.
  • src/lib/server-freshness.ts:3 defines the 90-second freshness window. Once the final empty report expires, the previous RCON result can also be stale, leaving a gap until another check completes.
  • src/lib/server/fleet/beta-tests.ts:129 consumes that observation, and src/lib/server/fleet/beta-evaluation.ts:84 resets healthy minutes and increments the outage count.

The monitoring handoff can therefore count missing fresh evidence as an outage even when the game server remains responsive.

Fix

  • Resume RCON monitoring before empty-server telemetry expires, allowing time for polling and agent response.
  • Preserve continuous health evidence across the transition for a responsive server, without resetting beta healthy minutes, incrementing outages, or revoking promotion readiness.
  • Continue detecting genuine failures within the intended detection window; do not extend stale evidence indefinitely or assume an empty server is healthy.
  • Add regression coverage for the complete transition from the final empty report through resumed RCON monitoring, including polling timing around the 90-second expiry and delayed or failed RCON responses.

Validation

Reproduced the stale-health transition and beta counter reset locally. All 113 existing tests across server-observation, server-status, host-status-batch, and beta-plans passed at revision c296769; the relevant health code was unchanged on main at ffda04f. Existing tests cover skipped RCON checks and eventual fallback, but miss the transition gap.

When the last player leaves, EC can briefly mark a responsive server unhealthy during the transition from script telemetry to RCON monitoring. This resets beta continuous healthy minutes and can revoke readiness for promotion. Observed in [Beta 84](https://ec.thewrightserver.net/betas/84) on October 12, 2026. All times Eastern: | Server | Empty report | Unhealthy | Healthy again | |---|---|---|---| | Buried | 6:29:17 PM | 6:30:47 PM | 6:31:28 PM | | Verrückt | 6:30:35 PM | 6:32:13 PM | 6:32:28 PM | Both unhealthy events had no recorded error. Beta 84 changed from “Test plan passed” back to active during this sequence. - `src/lib/server/pipeline/worker.ts:218,335` skips RCON monitoring while fresh telemetry reports zero players. - `src/lib/server-freshness.ts:3` defines the 90-second freshness window. Once the final empty report expires, the previous RCON result can also be stale, leaving a gap until another check completes. - `src/lib/server/fleet/beta-tests.ts:129` consumes that observation, and `src/lib/server/fleet/beta-evaluation.ts:84` resets healthy minutes and increments the outage count. The monitoring handoff can therefore count missing fresh evidence as an outage even when the game server remains responsive. **Fix** - [x] Resume RCON monitoring before empty-server telemetry expires, allowing time for polling and agent response. - [x] Preserve continuous health evidence across the transition for a responsive server, without resetting beta healthy minutes, incrementing outages, or revoking promotion readiness. - [x] Continue detecting genuine failures within the intended detection window; do not extend stale evidence indefinitely or assume an empty server is healthy. - [x] Add regression coverage for the complete transition from the final empty report through resumed RCON monitoring, including polling timing around the 90-second expiry and delayed or failed RCON responses. **Validation** Reproduced the stale-health transition and beta counter reset locally. All 113 existing tests across `server-observation`, `server-status`, `host-status-batch`, and `beta-plans` passed at revision `c296769`; the relevant health code was unchanged on main at `ffda04f`. Existing tests cover skipped RCON checks and eventual fallback, but miss the transition gap.
goddard added the
priority
medium
type/bug
labels 2026-10-12 18:37:01 -04:00
josh was assigned by goddard 2026-10-12 18:37:01 -04:00
Author
Collaborator

Reviewed f91ec4d152. All fix items are complete; closing this issue.

Checklist

  • Early RCON handoff. MonitoringSnapshot.canSkipRcon stops skipping at 30 seconds after the final empty report, reserving two polling intervals before the 90-second expiry. Both worker paths use it.
  • Continuous beta evidence. beta-tests.ts:140 accepts a newer successful RCON observation while telemetry still owns the displayed roster. Healthy-minute continuity uses the shared freshness window.
  • Real failures still expire. Pending, failed and skipped RCON replies do not prolong stale telemetry; tests require lost health/readiness at 90,001 ms.
  • Complete transition coverage. host-status-batch.test.ts covers probes at 30,000 and 59,999 ms, replies 25 seconds later, continuous readiness and the transition to RCON.

Review notes

  • Reviewed current EC main at fa0baa8db5. Full application suite passed: 230 files, 3,120 tests, including migration-memory coverage; no failures or skips.
  • Python agent suite: 332 tests passed, no skips. Type/style checks: zero errors and warnings.
Reviewed f91ec4d15201a91923e34501e067e506b685b4f2. All fix items are complete; closing this issue. **Checklist** - [x] **Early RCON handoff.** `MonitoringSnapshot.canSkipRcon` stops skipping at 30 seconds after the final empty report, reserving two polling intervals before the 90-second expiry. Both worker paths use it. - [x] **Continuous beta evidence.** `beta-tests.ts:140` accepts a newer successful RCON observation while telemetry still owns the displayed roster. Healthy-minute continuity uses the shared freshness window. - [x] **Real failures still expire.** Pending, failed and skipped RCON replies do not prolong stale telemetry; tests require lost health/readiness at 90,001 ms. - [x] **Complete transition coverage.** `host-status-batch.test.ts` covers probes at 30,000 and 59,999 ms, replies 25 seconds later, continuous readiness and the transition to RCON. **Review notes** - Reviewed current EC main at fa0baa8db5f7629de3f58f2e6bf8da0568018be0. Full application suite passed: 230 files, 3,120 tests, including migration-memory coverage; no failures or skips. - Python agent suite: 332 tests passed, no skips. Type/style checks: zero errors and warnings.
Sign in to join this conversation.