Agent 1.9.3 can remove a working server container, then fail to create its replacement because it downloads already-verified artifacts again. A transient controller connection failure therefore leaves the lobby offline despite successful preflight validation.
Source references at revision 82e07ea:
agent/ec_agent.py:1682,2138 downloads and verifies replacement files during rebuild preflight but discards their bytes.
agent/ec_agent.py:2152,2153,2156 stops and removes the existing container before calling replacement creation.
agent/ec_agent.py:1919,1927,1984 downloads the files again during replacement validation and writing.
agent/ec_agent.py:1993,2455,2462 opens each artifact request with a 60-second timeout.
Production evidence from October 14, 2026 confirms both failure paths: Farm Lobby 1 failed while writing replacement files, and Mob of the Dead Lobby 1 failed during replacement validation. Both recorded URLError caused by TimeoutError after the old container was removed.
Successful preflight should eliminate subsequent artifact-download dependencies during destructive replacement. The underlying intermittent network timeout remains unconfirmed; this issue addresses the verified rebuild defect.
Fix
Retain the complete verified replacement file set before disrupting the existing container, within the existing 64 MiB limit.
Reuse those bytes for staged-file comparison, replacement validation, and writing, with no artifact downloads after container removal.
Preserve checksum, path, size, player-safety, bank-settlement, and immutable-operation checks.
Add regressions proving that preflight download failure preserves the original container, and staged/unstaged rebuilds succeed when artifact connectivity disappears after successful preflight.
Verify prepared bytes cannot leak into or be reused by another operation.
Run the agent suite and publish a new signed agent release newer than 1.9.3.
Agent 1.9.3 can remove a working server container, then fail to create its replacement because it downloads already-verified artifacts again. A transient controller connection failure therefore leaves the lobby offline despite successful preflight validation.
Source references at revision `82e07ea`:
- `agent/ec_agent.py:1682,2138` downloads and verifies replacement files during rebuild preflight but discards their bytes.
- `agent/ec_agent.py:2152,2153,2156` stops and removes the existing container before calling replacement creation.
- `agent/ec_agent.py:1919,1927,1984` downloads the files again during replacement validation and writing.
- `agent/ec_agent.py:1993,2455,2462` opens each artifact request with a 60-second timeout.
Production evidence from October 14, 2026 confirms both failure paths: [Farm Lobby 1](https://ec.thewrightserver.net/provisioning/36c9e6a1-8db4-43cc-b2b8-74b35984704b) failed while writing replacement files, and [Mob of the Dead Lobby 1](https://ec.thewrightserver.net/provisioning/b11ef916-9a90-4509-b251-e3a4b19c4f82) failed during replacement validation. Both recorded `URLError` caused by `TimeoutError` after the old container was removed.
Successful preflight should eliminate subsequent artifact-download dependencies during destructive replacement. The underlying intermittent network timeout remains unconfirmed; this issue addresses the verified rebuild defect.
**Fix**
- [x] Retain the complete verified replacement file set before disrupting the existing container, within the existing 64 MiB limit.
- [x] Reuse those bytes for staged-file comparison, replacement validation, and writing, with no artifact downloads after container removal.
- [x] Preserve checksum, path, size, player-safety, bank-settlement, and immutable-operation checks.
- [x] Add regressions proving that preflight download failure preserves the original container, and staged/unstaged rebuilds succeed when artifact connectivity disappears after successful preflight.
- [x] Verify prepared bytes cannot leak into or be reused by another operation.
- [x] Run the agent suite and publish a new signed agent release newer than 1.9.3.
Reviewed 61dbfb2c5f and be7f6c77c1. All six fix items are complete, so I am closing this issue.
Checklist
Complete preflight set.validate_spec() retains every verified replacement file before disruption, preserving the 32 MiB per-file and 64 MiB aggregate limits.
Reuse through replacement.rebuild_before() compares prepared bytes, replacement validation rechecks those bytes without fetching, and write_initial_files() writes the same verified set. No artifact request is needed after container removal.
Safety and integrity. Checksum, canonical path, duplicate-file and size checks remain. Player checks still run after preparation and staged snapshots. Bank settlement remains before replacement creation, with unresolved decisions preserving runtime storage. Immutable request fingerprints and generation checks are unchanged.
Failure regressions. Tests cover staged and unstaged rebuilds after artifact connectivity disappears, plus partial preparation failure that leaves the old container running. Corruption, size limits, invalid paths and mismatched staged bytes are also covered.
Operation isolation. Prepared bytes stay local to the current call. Tests verify later operations and occupied retries prepare their own files and that saved requests and receipts do not acquire those bytes.
Agent suite and signed release. All 357 Python tests pass with no skips. Agent 1.10.1 source and packaged bytes match the manifest, whose signature validates against the repository public key. Public controller release assets checked on October 14 at approximately 16:44 UTC match that reviewed release.
Review notes
Reviewed current upstream be7f6c77c1d7c33138e2adcdc49b28799b6d4d07; shared lint and type/style checks also pass, with zero Svelte errors or warnings.
Signed publication is verified. Installation on every host and a real game-host outage exercise were not tested. The original intermittent network timeout remains unexplained; this fix removes the repeated artifact downloads that caused the verified rebuild failures.
Reviewed 61dbfb2c5f26da264bb812f88c40de20561cc966 and be7f6c77c1d7c33138e2adcdc49b28799b6d4d07. All six fix items are complete, so I am closing this issue.
**Checklist**
- [x] **Complete preflight set.** `validate_spec()` retains every verified replacement file before disruption, preserving the 32 MiB per-file and 64 MiB aggregate limits.
- [x] **Reuse through replacement.** `rebuild_before()` compares prepared bytes, replacement validation rechecks those bytes without fetching, and `write_initial_files()` writes the same verified set. No artifact request is needed after container removal.
- [x] **Safety and integrity.** Checksum, canonical path, duplicate-file and size checks remain. Player checks still run after preparation and staged snapshots. Bank settlement remains before replacement creation, with unresolved decisions preserving runtime storage. Immutable request fingerprints and generation checks are unchanged.
- [x] **Failure regressions.** Tests cover staged and unstaged rebuilds after artifact connectivity disappears, plus partial preparation failure that leaves the old container running. Corruption, size limits, invalid paths and mismatched staged bytes are also covered.
- [x] **Operation isolation.** Prepared bytes stay local to the current call. Tests verify later operations and occupied retries prepare their own files and that saved requests and receipts do not acquire those bytes.
- [x] **Agent suite and signed release.** All 357 Python tests pass with no skips. Agent 1.10.1 source and packaged bytes match the manifest, whose signature validates against the repository public key. Public controller release assets checked on October 14 at approximately 16:44 UTC match that reviewed release.
**Review notes**
- Reviewed current upstream `be7f6c77c1d7c33138e2adcdc49b28799b6d4d07`; shared lint and type/style checks also pass, with zero Svelte errors or warnings.
- Signed publication is verified. Installation on every host and a real game-host outage exercise were not tested. The original intermittent network timeout remains unexplained; this fix removes the repeated artifact downloads that caused the verified rebuild failures.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Agent 1.9.3 can remove a working server container, then fail to create its replacement because it downloads already-verified artifacts again. A transient controller connection failure therefore leaves the lobby offline despite successful preflight validation.
Source references at revision
82e07ea:agent/ec_agent.py:1682,2138downloads and verifies replacement files during rebuild preflight but discards their bytes.agent/ec_agent.py:2152,2153,2156stops and removes the existing container before calling replacement creation.agent/ec_agent.py:1919,1927,1984downloads the files again during replacement validation and writing.agent/ec_agent.py:1993,2455,2462opens each artifact request with a 60-second timeout.Production evidence from October 14, 2026 confirms both failure paths: Farm Lobby 1 failed while writing replacement files, and Mob of the Dead Lobby 1 failed during replacement validation. Both recorded
URLErrorcaused byTimeoutErrorafter the old container was removed.Successful preflight should eliminate subsequent artifact-download dependencies during destructive replacement. The underlying intermittent network timeout remains unconfirmed; this issue addresses the verified rebuild defect.
Fix
Reviewed
61dbfb2c5fandbe7f6c77c1. All six fix items are complete, so I am closing this issue.Checklist
validate_spec()retains every verified replacement file before disruption, preserving the 32 MiB per-file and 64 MiB aggregate limits.rebuild_before()compares prepared bytes, replacement validation rechecks those bytes without fetching, andwrite_initial_files()writes the same verified set. No artifact request is needed after container removal.Review notes
be7f6c77c1d7c33138e2adcdc49b28799b6d4d07; shared lint and type/style checks also pass, with zero Svelte errors or warnings.