A completed host rebuild can leave EC’s instance record on the previous generation. Health checks, log reads and destroy requests then target an obsolete instance, leaving the replacement unmanaged through EC and the rebuild stuck in verification.
Observed on October 14, 2026, for t6_zombie_tomb_1 (Origins) on ec-2, instance 80edb630-ba7f-422e-8f6a-411b999796a0. All times below are EDT:
1:33:23 a.m.: The agent reported rebuild 38d86909… completed. Verification immediately began failing with “Instance generation has been superseded or removed.”
The rebuild recorded 1,786 occurrences of that error before cancellation at 8:01:24 a.m.
8:03:40 a.m.:Destroy ce1408c0… failed with the same error. Its agent operation, fb6e0635-4182-47dd-8eba-54f61a4a58c0, explicitly targeted generation 2.
8:05 a.m.: EC still recorded generation 2 and reported the server offline, while ec-2’s inventory reported the same instance at generation 3, with its container running and healthy. Public joinability was not verified.
Relevant code at 09396ab584961519b664c8f165cdce24c3cf25c5:
src/lib/server/provisioning/native.ts:1496 advances the registered generation after a completed rebuild; verification uses the resulting instance at line 1545.
src/lib/server/provisioning/native.ts:816 also advances generations when settling completed operations after cancellation.
agent/ec_agent.py:2780 rejects commands targeting superseded generations, including removal commands.
The live evidence confirms a controller/agent generation mismatch. The exact reason the controller failed to retain the replacement generation remains to be established. The mismatch predates cancellation; bypassing player-safety checks does not resolve it.
Fix
Reproduce the completed-rebuild mismatch and fix receipt reconciliation so the replacement generation is durably recorded before verification or subsequent commands use it. Applying the same receipt repeatedly must not increment the generation again.
Preserve reconciliation across controller restarts, delayed completion reports and cancellation after the host has completed the rebuild.
Provide a supported recovery path for existing mismatches. Validate the host, immutable instance ID and completed rebuild evidence before adopting the replacement generation; retain the agent’s protection against stale commands.
Replace indefinite verification retries for a confirmed generation mismatch with an actionable recovery state that identifies the expected and observed generations.
After reconciliation, ensure health checks, log reads and newly requested lifecycle operations target the replacement generation. Require fresh evidence before reporting the server online.
Add regression tests covering this incident, duplicate receipt processing, controller restart, cancellation after host completion, and rejection of evidence belonging to another instance or host.
A completed host rebuild can leave EC’s instance record on the previous generation. Health checks, log reads and destroy requests then target an obsolete instance, leaving the replacement unmanaged through EC and the rebuild stuck in verification.
Observed on October 14, 2026, for `t6_zombie_tomb_1` (Origins) on `ec-2`, instance `80edb630-ba7f-422e-8f6a-411b999796a0`. All times below are EDT:
- **1:33:23 a.m.:** The agent reported [rebuild `38d86909…`](https://ec.thewrightserver.net/provisioning/38d86909-661f-4350-a789-638e04b9afe6) completed. Verification immediately began failing with “Instance generation has been superseded or removed.”
- The rebuild recorded **1,786 occurrences** of that error before cancellation at **8:01:24 a.m.**
- **8:03:40 a.m.:** [Destroy `ce1408c0…`](https://ec.thewrightserver.net/provisioning/ce1408c0-bd62-4338-aec2-877626e21090) failed with the same error. Its agent operation, `fb6e0635-4182-47dd-8eba-54f61a4a58c0`, explicitly targeted **generation 2**.
- **8:05 a.m.:** EC still recorded generation 2 and reported the server offline, while ec-2’s inventory reported the same instance at **generation 3**, with its container running and healthy. Public joinability was not verified.
Relevant code at `09396ab584961519b664c8f165cdce24c3cf25c5`:
- `src/lib/server/provisioning/native.ts:1496` advances the registered generation after a completed rebuild; verification uses the resulting instance at line 1545.
- `src/lib/server/provisioning/native.ts:816` also advances generations when settling completed operations after cancellation.
- `agent/ec_agent.py:2780` rejects commands targeting superseded generations, including removal commands.
The live evidence confirms a controller/agent generation mismatch. The exact reason the controller failed to retain the replacement generation remains to be established. The mismatch predates cancellation; bypassing player-safety checks does not resolve it.
**Fix**
- [x] Reproduce the completed-rebuild mismatch and fix receipt reconciliation so the replacement generation is durably recorded before verification or subsequent commands use it. Applying the same receipt repeatedly must not increment the generation again.
- [x] Preserve reconciliation across controller restarts, delayed completion reports and cancellation after the host has completed the rebuild.
- [x] Provide a supported recovery path for existing mismatches. Validate the host, immutable instance ID and completed rebuild evidence before adopting the replacement generation; retain the agent’s protection against stale commands.
- [x] Replace indefinite verification retries for a confirmed generation mismatch with an actionable recovery state that identifies the expected and observed generations.
- [x] After reconciliation, ensure health checks, log reads and newly requested lifecycle operations target the replacement generation. Require fresh evidence before reporting the server online.
- [x] Add regression tests covering this incident, duplicate receipt processing, controller restart, cancellation after host completion, and rejection of evidence belonging to another instance or host.
Reviewed dedf15d9af and be7f6c77c1. All six fix items are complete, so I am closing this issue.
Checklist
Durable, idempotent reconciliation.rebuild-receipt.ts derives the replacement from the completed command rather than a stale provisioning snapshot and adopts it transactionally before verification. Command submission now saves the generation actually targeted. Replaying a receipt preserves both generation and its evidence timestamp.
Restart and cancellation. Durable command evidence survives database reopen; completion before or after cancellation is reconciled while cancellation stays terminal. Only untouched successor jobs can be retargeted; submitted commands retain their immutable identity. Older applied receipts cannot undo a later replacement.
Supported recovery. Provisioning detail exposes the reconciliation action. Its endpoint requires completed rebuild evidence, matching host and immutable instance ID, the requested replacement generation, and matching inventory no more than 30 seconds old. Foreign, stale and contradictory evidence is rejected; agent stale-command protection remains intact.
Actionable mismatch state. A confirmed mismatch or stale-generation rejection records an interrupted generation-recovery blocker with expected and observed generations. Subsequent worker passes stop repeating verification until recovery clears the blocker.
Replacement targets and fresh health. Reconciliation clears old server observations and records a durable generation freshness boundary used by health readers. Regression tests confirm subsequent inspection, log reads and removal target the replacement; previous health cannot report it online.
Incident and regression tests. Coverage includes a stale job binding against a correctly targeted completed rebuild, duplicate receipts, database reopen, delayed completion on either side of cancellation, existing-mismatch recovery, wrong-host/instance/result/payload evidence, and bounded retries. Migration and retention tests protect the recovery evidence.
Review notes
At be7f6c77c1d7c33138e2adcdc49b28799b6d4d07, all 490 tests across 14 affected and related files pass, with no skips: native provisioning, migrations, retention, permissions, provisioning history, recovery cancellation, cancellation, health observations, server status, native observations/safety, and provisioning detail/refresh/UI.
npm run lint and npm run check pass; Svelte reports zero errors and warnings. The full agent suite also passes all 357 tests, and the signed 1.10.1 release was verified separately for #154.
Recovery was verified with isolated fixtures. I did not invoke the production reconciliation action or claim the affected production instance has been recovered.
Reviewed dedf15d9af755d32bbda2c42f12fe4b00c1a20bb and be7f6c77c1d7c33138e2adcdc49b28799b6d4d07. All six fix items are complete, so I am closing this issue.
**Checklist**
- [x] **Durable, idempotent reconciliation.** `rebuild-receipt.ts` derives the replacement from the completed command rather than a stale provisioning snapshot and adopts it transactionally before verification. Command submission now saves the generation actually targeted. Replaying a receipt preserves both generation and its evidence timestamp.
- [x] **Restart and cancellation.** Durable command evidence survives database reopen; completion before or after cancellation is reconciled while cancellation stays terminal. Only untouched successor jobs can be retargeted; submitted commands retain their immutable identity. Older applied receipts cannot undo a later replacement.
- [x] **Supported recovery.** Provisioning detail exposes the reconciliation action. Its endpoint requires completed rebuild evidence, matching host and immutable instance ID, the requested replacement generation, and matching inventory no more than 30 seconds old. Foreign, stale and contradictory evidence is rejected; agent stale-command protection remains intact.
- [x] **Actionable mismatch state.** A confirmed mismatch or stale-generation rejection records an interrupted `generation-recovery` blocker with expected and observed generations. Subsequent worker passes stop repeating verification until recovery clears the blocker.
- [x] **Replacement targets and fresh health.** Reconciliation clears old server observations and records a durable generation freshness boundary used by health readers. Regression tests confirm subsequent inspection, log reads and removal target the replacement; previous health cannot report it online.
- [x] **Incident and regression tests.** Coverage includes a stale job binding against a correctly targeted completed rebuild, duplicate receipts, database reopen, delayed completion on either side of cancellation, existing-mismatch recovery, wrong-host/instance/result/payload evidence, and bounded retries. Migration and retention tests protect the recovery evidence.
**Review notes**
- At `be7f6c77c1d7c33138e2adcdc49b28799b6d4d07`, all 490 tests across 14 affected and related files pass, with no skips: native provisioning, migrations, retention, permissions, provisioning history, recovery cancellation, cancellation, health observations, server status, native observations/safety, and provisioning detail/refresh/UI.
- `npm run lint` and `npm run check` pass; Svelte reports zero errors and warnings. The full agent suite also passes all 357 tests, and the signed 1.10.1 release was verified separately for #154.
- Recovery was verified with isolated fixtures. I did not invoke the production reconciliation action or claim the affected production instance has been recovered.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
A completed host rebuild can leave EC’s instance record on the previous generation. Health checks, log reads and destroy requests then target an obsolete instance, leaving the replacement unmanaged through EC and the rebuild stuck in verification.
Observed on October 14, 2026, for
t6_zombie_tomb_1(Origins) onec-2, instance80edb630-ba7f-422e-8f6a-411b999796a0. All times below are EDT:38d86909…completed. Verification immediately began failing with “Instance generation has been superseded or removed.”ce1408c0…failed with the same error. Its agent operation,fb6e0635-4182-47dd-8eba-54f61a4a58c0, explicitly targeted generation 2.Relevant code at
09396ab584961519b664c8f165cdce24c3cf25c5:src/lib/server/provisioning/native.ts:1496advances the registered generation after a completed rebuild; verification uses the resulting instance at line 1545.src/lib/server/provisioning/native.ts:816also advances generations when settling completed operations after cancellation.agent/ec_agent.py:2780rejects commands targeting superseded generations, including removal commands.The live evidence confirms a controller/agent generation mismatch. The exact reason the controller failed to retain the replacement generation remains to be established. The mismatch predates cancellation; bypassing player-safety checks does not resolve it.
Fix
Reviewed
dedf15d9afandbe7f6c77c1. All six fix items are complete, so I am closing this issue.Checklist
rebuild-receipt.tsderives the replacement from the completed command rather than a stale provisioning snapshot and adopts it transactionally before verification. Command submission now saves the generation actually targeted. Replaying a receipt preserves both generation and its evidence timestamp.generation-recoveryblocker with expected and observed generations. Subsequent worker passes stop repeating verification until recovery clears the blocker.Review notes
be7f6c77c1d7c33138e2adcdc49b28799b6d4d07, all 490 tests across 14 affected and related files pass, with no skips: native provisioning, migrations, retention, permissions, provisioning history, recovery cancellation, cancellation, health observations, server status, native observations/safety, and provisioning detail/refresh/UI.npm run lintandnpm run checkpass; Svelte reports zero errors and warnings. The full agent suite also passes all 357 tests, and the signed 1.10.1 release was verified separately for #154.