migration-memory.test.ts rebuilds a 256 MiB database while other test files run beside it. In CI, a test that happens to overlap it can slow down past its 5-second timeout. Run 842 failed this way, on a commit that doesn't touch the code the failing test uses.
migration-memory.test.ts:20-30 starts tests/helpers/vacuum-memory.ts in a child process. That helper writes 256 MiB of rows (vacuum-memory.ts:10-12), checkpoints them into the database file (:14), then rebuilds the whole file (:20). That's well over 750 MiB of disk writes with fsyncs, far more than any other test.
vite.config.ts:6-7 has one test configuration, and CI runs it with two workers (ci.yml:34), so this test runs alongside whichever file the other worker has picked up.
In run 842, library-metadata.test.ts ran entirely inside migration-memory's 10.7 s window and timed out at 6,493 ms. migration-memory itself was slower than usual too (8.7 s in run 841). The test normally takes about 0.4 s: 397 ms in run 841, where it started just after migration-memory finished, and 428–464 ms locally on the same commit.
Overlap doesn't always hurt. In run 840 the same test overlapped and took 283 ms, so it likely depends on which part of the rebuild it lands on. I couldn't reproduce 6.5 s locally either: saturating this machine's disk only took the test from about 0.45 s to about 1 s, and this disk is faster than the runner's.
Raising timeouts would hide this and real slowdowns along with it. Running the rebuild test on its own keeps the 256 MiB check intact and takes it out of the way of everything else.
Fix
Move migration-memory.test.ts into its own Vitest project (test.projects) with a later sequence.groupOrder, so it runs after the other test files instead of beside them
Confirm in a CI log that migration-memory.test.ts now starts after every other test file has finished
`migration-memory.test.ts` rebuilds a 256 MiB database while other test files run beside it. In CI, a test that happens to overlap it can slow down past its 5-second timeout. [Run 842](https://gitea.thewrightserver.net/josh/electric-cherry/actions/runs/842) failed this way, on a commit that doesn't touch the code the failing test uses.
- `migration-memory.test.ts:20-30` starts `tests/helpers/vacuum-memory.ts` in a child process. That helper writes 256 MiB of rows (`vacuum-memory.ts:10-12`), checkpoints them into the database file (`:14`), then rebuilds the whole file (`:20`). That's well over 750 MiB of disk writes with fsyncs, far more than any other test.
- `vite.config.ts:6-7` has one test configuration, and CI runs it with two workers (`ci.yml:34`), so this test runs alongside whichever file the other worker has picked up.
- In run 842, `library-metadata.test.ts` ran entirely inside `migration-memory`'s 10.7 s window and timed out at 6,493 ms. `migration-memory` itself was slower than usual too (8.7 s in run 841). The test normally takes about 0.4 s: 397 ms in run 841, where it started just after `migration-memory` finished, and 428–464 ms locally on the same commit.
- Overlap doesn't always hurt. In run 840 the same test overlapped and took 283 ms, so it likely depends on which part of the rebuild it lands on. I couldn't reproduce 6.5 s locally either: saturating this machine's disk only took the test from about 0.45 s to about 1 s, and this disk is faster than the runner's.
Raising timeouts would hide this and real slowdowns along with it. Running the rebuild test on its own keeps the 256 MiB check intact and takes it out of the way of everything else.
**Fix**
- [x] Move `migration-memory.test.ts` into its own Vitest project (`test.projects`) with a later `sequence.groupOrder`, so it runs after the other test files instead of beside them
- [x] Confirm in a CI log that `migration-memory.test.ts` now starts after every other test file has finished
Another failure from the same cause: run 855 on 0bfda267ec, a metrics-only commit. One test out of 2,843 failed: launch-recipe-summary-migration.test.ts › "lazily upgrades legacy snapshots once without changing fingerprints or retained bytes" timed out at 5,124 ms.
In the run's log, migration-memory.test.ts ran from about 21:01:47.0 to 21:01:57.0 (10,033 ms). The failing test ran from about 21:01:52.2 to 21:01:57.3, so 4.8 s of its 5.1 s overlapped the rebuild.
Its own startup migrations weren't the slow part: about 150 ms in both run 854 and run 855. The extra time goes to its disk work, which writes, closes and reopens a database holding a 2.7 MB legacy recipe and waits on the same disk as the rebuild.
Across the last ten runs, its duration follows the overlap:
Run
launch-recipe-summary-migration
Overlap with migration-memory
846, 851, 853, 854
408–548 ms
0–0.4 s
847, 850
550–738 ms
0.6–0.7 s
848
1,400 ms
1.2 s
849
3,582 ms (a near miss)
3.3 s
855
5,126 ms (failed)
4.8 s
Run 852 took 1,222 ms with no overlap, so other load adds some variation, but every run above 1.5 s overlapped the rebuild.
As in the original report, I couldn't reproduce the timeout locally. Running the two files together with VITEST_MAX_WORKERS=2 only took this test from 1.25–1.37 s alone to 1.49–1.56 s, because this container's disk is faster than the runner's.
The fix above still stands. Running migration-memory.test.ts after every other test file would also have prevented this failure, and raising this test's timeout would only move the problem to the next file that overlaps the rebuild.
Another failure from the same cause: [run 855](https://gitea.thewrightserver.net/josh/electric-cherry/actions/runs/855) on 0bfda267ecd3b18d35aafb670b8a4cdb871ba41d, a metrics-only commit. One test out of 2,843 failed: `launch-recipe-summary-migration.test.ts` › "lazily upgrades legacy snapshots once without changing fingerprints or retained bytes" timed out at 5,124 ms.
- In the run's log, `migration-memory.test.ts` ran from about 21:01:47.0 to 21:01:57.0 (10,033 ms). The failing test ran from about 21:01:52.2 to 21:01:57.3, so 4.8 s of its 5.1 s overlapped the rebuild.
- Its own startup migrations weren't the slow part: about 150 ms in both run 854 and run 855. The extra time goes to its disk work, which writes, closes and reopens a database holding a 2.7 MB legacy recipe and waits on the same disk as the rebuild.
Across the last ten runs, its duration follows the overlap:
| Run | `launch-recipe-summary-migration` | Overlap with `migration-memory` |
|---|---|---|
| 846, 851, 853, 854 | 408–548 ms | 0–0.4 s |
| 847, 850 | 550–738 ms | 0.6–0.7 s |
| 848 | 1,400 ms | 1.2 s |
| 849 | 3,582 ms (a near miss) | 3.3 s |
| 855 | 5,126 ms (failed) | 4.8 s |
Run 852 took 1,222 ms with no overlap, so other load adds some variation, but every run above 1.5 s overlapped the rebuild.
As in the original report, I couldn't reproduce the timeout locally. Running the two files together with `VITEST_MAX_WORKERS=2` only took this test from 1.25–1.37 s alone to 1.49–1.56 s, because this container's disk is faster than the runner's.
The fix above still stands. Running `migration-memory.test.ts` after every other test file would also have prevented this failure, and raising this test's timeout would only move the problem to the next file that overlaps the rebuild.
Reviewed 7fcf1cded2. Both fix items are done, so I'm closing this.
Checklist
Own Vitest project with a later groupOrder.vite.config.ts now defines an application project (groupOrder: 0) that excludes migration-memory.test.ts, and a migration-memory project (groupOrder: 1) that includes only that file.
Confirm the order in a CI log. In run 856 on this commit, the other 216 test files had all finished by 22:21:09.45 (the last was files.test.ts). migration-memory.test.ts started 0.18 s later, at about 22:21:09.62, and ran alone for 8,655 ms. No other file overlaps it. All 217 files and 2,866 tests passed.
Review notes
launch-recipe-summary-migration.test.ts, which failed in run 855, took 363 ms in run 856.
Running the rebuild last didn't slow the suite down: 197.95 s in run 856, against 186–199 s for runs 850–855.
Locally, npm test -- tests/unit/migration-memory.test.ts tests/unit/launch-recipe-summary-migration.test.ts still runs both files under the new projects (2 files, 5 tests pass).
Reviewed 7fcf1cded20647459b50cac820bf83e4377ddf45. Both fix items are done, so I'm closing this.
**Checklist**
- [x] **Own Vitest project with a later `groupOrder`.** `vite.config.ts` now defines an `application` project (`groupOrder: 0`) that excludes `migration-memory.test.ts`, and a `migration-memory` project (`groupOrder: 1`) that includes only that file.
- [x] **Confirm the order in a CI log.** In [run 856](https://gitea.thewrightserver.net/josh/electric-cherry/actions/runs/856) on this commit, the other 216 test files had all finished by 22:21:09.45 (the last was `files.test.ts`). `migration-memory.test.ts` started 0.18 s later, at about 22:21:09.62, and ran alone for 8,655 ms. No other file overlaps it. All 217 files and 2,866 tests passed.
**Review notes**
- `launch-recipe-summary-migration.test.ts`, which failed in run 855, took 363 ms in run 856.
- Running the rebuild last didn't slow the suite down: 197.95 s in run 856, against 186–199 s for runs 850–855.
- Locally, `npm test -- tests/unit/migration-memory.test.ts tests/unit/launch-recipe-summary-migration.test.ts` still runs both files under the new projects (2 files, 5 tests pass).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
migration-memory.test.tsrebuilds a 256 MiB database while other test files run beside it. In CI, a test that happens to overlap it can slow down past its 5-second timeout. Run 842 failed this way, on a commit that doesn't touch the code the failing test uses.migration-memory.test.ts:20-30startstests/helpers/vacuum-memory.tsin a child process. That helper writes 256 MiB of rows (vacuum-memory.ts:10-12), checkpoints them into the database file (:14), then rebuilds the whole file (:20). That's well over 750 MiB of disk writes with fsyncs, far more than any other test.vite.config.ts:6-7has one test configuration, and CI runs it with two workers (ci.yml:34), so this test runs alongside whichever file the other worker has picked up.library-metadata.test.tsran entirely insidemigration-memory's 10.7 s window and timed out at 6,493 ms.migration-memoryitself was slower than usual too (8.7 s in run 841). The test normally takes about 0.4 s: 397 ms in run 841, where it started just aftermigration-memoryfinished, and 428–464 ms locally on the same commit.Raising timeouts would hide this and real slowdowns along with it. Running the rebuild test on its own keeps the 256 MiB check intact and takes it out of the way of everything else.
Fix
migration-memory.test.tsinto its own Vitest project (test.projects) with a latersequence.groupOrder, so it runs after the other test files instead of beside themmigration-memory.test.tsnow starts after every other test file has finishedAnother failure from the same cause: run 855 on
0bfda267ec, a metrics-only commit. One test out of 2,843 failed:launch-recipe-summary-migration.test.ts› "lazily upgrades legacy snapshots once without changing fingerprints or retained bytes" timed out at 5,124 ms.migration-memory.test.tsran from about 21:01:47.0 to 21:01:57.0 (10,033 ms). The failing test ran from about 21:01:52.2 to 21:01:57.3, so 4.8 s of its 5.1 s overlapped the rebuild.Across the last ten runs, its duration follows the overlap:
launch-recipe-summary-migrationmigration-memoryRun 852 took 1,222 ms with no overlap, so other load adds some variation, but every run above 1.5 s overlapped the rebuild.
As in the original report, I couldn't reproduce the timeout locally. Running the two files together with
VITEST_MAX_WORKERS=2only took this test from 1.25–1.37 s alone to 1.49–1.56 s, because this container's disk is faster than the runner's.The fix above still stands. Running
migration-memory.test.tsafter every other test file would also have prevented this failure, and raising this test's timeout would only move the problem to the next file that overlaps the rebuild.Reviewed
7fcf1cded2. Both fix items are done, so I'm closing this.Checklist
groupOrder.vite.config.tsnow defines anapplicationproject (groupOrder: 0) that excludesmigration-memory.test.ts, and amigration-memoryproject (groupOrder: 1) that includes only that file.files.test.ts).migration-memory.test.tsstarted 0.18 s later, at about 22:21:09.62, and ran alone for 8,655 ms. No other file overlaps it. All 217 files and 2,866 tests passed.Review notes
launch-recipe-summary-migration.test.ts, which failed in run 855, took 363 ms in run 856.npm test -- tests/unit/migration-memory.test.ts tests/unit/launch-recipe-summary-migration.test.tsstill runs both files under the new projects (2 files, 5 tests pass).