Autoscaling lifecycle test fails about half the time on a same-millisecond timestamp race #79

Closed
opened 2026-10-08 18:08:54 -04:00 by goddard · 1 comment
Collaborator

autoscaling-lifecycle.test.ts fails in about half of all runs, in whichever of the four "allows … without scaling up during work or recovery" cases hits a same-millisecond race. Only the test is affected.

  • autoscaling-lifecycle.test.ts:66-67 records the restart's requested_at with new Date().
  • autoscaling-lifecycle.test.ts:72 takes a second new Date() for the server's status times, after a single runAutoscaling() call, often in the same millisecond.
  • lifecycle-evidence.ts:16 counts the restart request as the server's latest submission, and server-observation.ts:70 only trusts a status reading strictly newer than it. When both timestamps are equal, the server reads as unknown and the final runAutoscaling() waits instead of creating t6_zombie_town_2 (autoscaling-lifecycle.test.ts:82-83).

It failed 7 of 15 runs before the dependency upgrade in #70 and 8 of 15 after it. With a 2 ms pause before line 72, 0 of 15 runs failed. In production a status read comes seconds after a restart, so the strict comparison is correct; the test needs fixing.

Fix

  • Set the test's status times strictly after the restart's requested_at (for example, one second later) instead of relying on two new Date() calls
  • Confirm autoscaling-lifecycle.test.ts passes 20 runs in a row
`autoscaling-lifecycle.test.ts` fails in about half of all runs, in whichever of the four "allows … without scaling up during work or recovery" cases hits a same-millisecond race. Only the test is affected. - `autoscaling-lifecycle.test.ts:66-67` records the restart's `requested_at` with `new Date()`. - `autoscaling-lifecycle.test.ts:72` takes a second `new Date()` for the server's status times, after a single `runAutoscaling()` call, often in the same millisecond. - `lifecycle-evidence.ts:16` counts the restart request as the server's latest submission, and `server-observation.ts:70` only trusts a status reading strictly newer than it. When both timestamps are equal, the server reads as unknown and the final `runAutoscaling()` waits instead of creating `t6_zombie_town_2` (`autoscaling-lifecycle.test.ts:82-83`). It failed 7 of 15 runs before the dependency upgrade in #70 and 8 of 15 after it. With a 2 ms pause before line 72, 0 of 15 runs failed. In production a status read comes seconds after a restart, so the strict comparison is correct; the test needs fixing. **Fix** - [x] Set the test's status times strictly after the restart's `requested_at` (for example, one second later) instead of relying on two `new Date()` calls - [x] Confirm `autoscaling-lifecycle.test.ts` passes 20 runs in a row
goddard added the
priority
low
type/bug
labels 2026-10-08 18:08:54 -04:00
josh was assigned by goddard 2026-10-08 18:08:54 -04:00
Author
Collaborator

Reviewed 40ad5f01a9. Both fix items are done, so I'm closing this.

Checklist

  • Status times strictly after the restart. The test now freezes Date with fake timers (autoscaling-lifecycle.test.ts:48), records the restart's requested_at once (:67) and sets the status times one second later (:75-76). afterEach restores real timers (:8), so the other cases aren't affected.
  • 20 runs in a row. autoscaling-lifecycle.test.ts passed 20 of 20 runs locally. Before the fix it failed 8 of 15.

Review notes

  • The full test suite now passes locally (205 files, 2,591 tests). Before the fix, this test was its only failure.
Reviewed 40ad5f01a971e5a5aa086be5f453629ac023508b. Both fix items are done, so I'm closing this. **Checklist** - [x] **Status times strictly after the restart.** The test now freezes `Date` with fake timers (`autoscaling-lifecycle.test.ts:48`), records the restart's `requested_at` once (`:67`) and sets the status times one second later (`:75-76`). `afterEach` restores real timers (`:8`), so the other cases aren't affected. - [x] **20 runs in a row.** `autoscaling-lifecycle.test.ts` passed 20 of 20 runs locally. Before the fix it failed 8 of 15. **Review notes** - The full test suite now passes locally (205 files, 2,591 tests). Before the fix, this test was its only failure.
Sign in to join this conversation.