A failed campaign target remains in Fleet’s “Needs attention” section after a separate successful rebuild installs the intended release. The campaign continues referencing its failed operation, and its resume action does not support failed targets.
Observed on Shi No Numa · Lobby 1 (t5_zombie_sumpf_1) on ec-3, October 19, 2026.
Standalone rebuild completed, replacing generation 5 with generation 6 and passing container, RCON, map, and Plutonium-version checks.
6:46pm
Server remained online with no health error, but the campaign target remained failed, referencing the original operation.
The successful rebuild’s recipe and recorded inventory match the campaign’s Common 1.7.1 and Leaderboard 1.4.1, including both SHA-256 hashes. The campaign remains unfinished with 19 targets done and one failed.
The original timeout occurred while downloading replacement files, before container teardown. Its underlying network cause is undetermined; the persistent warning after verified recovery is independently reproducible.
Cause
Inspected at 0f05be3a59f74f013c7f465b39c2747640c1fb3d:
src/lib/server/provisioning/requests.ts:64–84 returns the campaign’s previously linked operation, including a terminal failure.
src/lib/server/pipeline/sweep.ts mirrors that operation’s failure back into the campaign target; it does not reconcile a later successful standalone rebuild.
src/lib/server/fleet/overview.ts:28–34 includes unfinished campaign targets independently of newer successful lifecycle operations.
src/routes/api/campaigns/[id]/resume/+server.ts:12,23 accepts only held and manual targets. A failed target receives HTTP 409.
src/routes/+page.svelte:411–418 displays the operation warning alongside the server-health timestamp.
An isolated reproduction using the current application code confirmed that a later completed rebuild leaves campaign attention present and campaign resume returns 409.
Screenshot evidence
The supplied screenshot shows 20/20 healthy, Shi No Numa marked “Update needs attention”, and “Evidence 8s ago.” That timestamp describes a recent health observation, not the afternoon’s update failure.
Fix
Reconcile failed campaign targets against later successful operations when the current instance generation and affected file hashes demonstrably satisfy the campaign. Record the recovery operation and preserve the original failed attempt.
Clear actionable campaign attention and complete the campaign once all remaining targets are satisfied. A healthy server or unrelated successful restart alone must not qualify.
Provide an explicit retry path for failed targets that still need activation. Preserve campaign association, use the current generation, and retain normal occupancy and startup-capacity checks.
Show the relevant operation failure or recovery timestamp on attention cards instead of the latest server-health timestamp.
Add regression coverage for matching recovery, mismatched files, stale-generation evidence, unrelated restarts, failed-target retry, campaign completion, and retained failure history.
A failed campaign target remains in Fleet’s “Needs attention” section after a separate successful rebuild installs the intended release. The campaign continues referencing its failed operation, and its resume action does not support failed targets.
Observed on **Shi No Numa · Lobby 1** (`t5_zombie_sumpf_1`) on **ec-3**, October 19, 2026.
**Evidence**
All times are EDT:
| Time | Event |
|---|---|
| 2:38pm | [Beta 104 promotion](https://ec.thewrightserver.net/campaigns/dep-20261019T183806Z-promote-104) started; both changed files staged successfully. |
| 2:47:26pm | Campaign rebuild began after waiting for startup capacity. |
| 2:48:42pm | [Campaign rebuild failed](https://ec.thewrightserver.net/provisioning/5cfe3bfb-4a36-4f4f-90f7-4ab3dd8aa0e1) with `TimeoutError; at artifact_open:2738`. |
| 5:29pm | Josh requested a standalone rebuild. |
| 5:31pm | [Standalone rebuild completed](https://ec.thewrightserver.net/provisioning/9118664c-20af-461f-86fa-6d409699b0e0), replacing generation 5 with generation 6 and passing container, RCON, map, and Plutonium-version checks. |
| 6:46pm | Server remained online with no health error, but the campaign target remained `failed`, referencing the original operation. |
The successful rebuild’s recipe and recorded inventory match the campaign’s **Common 1.7.1** and **Leaderboard 1.4.1**, including both SHA-256 hashes. The campaign remains unfinished with **19 targets done and one failed**.
The original timeout occurred while downloading replacement files, before container teardown. Its underlying network cause is undetermined; the persistent warning after verified recovery is independently reproducible.
**Cause**
Inspected at `0f05be3a59f74f013c7f465b39c2747640c1fb3d`:
- `src/lib/server/provisioning/requests.ts:64–84` returns the campaign’s previously linked operation, including a terminal failure.
- `src/lib/server/pipeline/sweep.ts` mirrors that operation’s failure back into the campaign target; it does not reconcile a later successful standalone rebuild.
- `src/lib/server/fleet/overview.ts:28–34` includes unfinished campaign targets independently of newer successful lifecycle operations.
- `src/routes/api/campaigns/[id]/resume/+server.ts:12,23` accepts only `held` and `manual` targets. A `failed` target receives HTTP 409.
- `src/routes/+page.svelte:411–418` displays the operation warning alongside the server-health timestamp.
An isolated reproduction using the current application code confirmed that a later completed rebuild leaves campaign attention present and campaign resume returns 409.
**Screenshot evidence**
The supplied screenshot shows **20/20 healthy**, Shi No Numa marked **“Update needs attention”**, and **“Evidence 8s ago.”** That timestamp describes a recent health observation, not the afternoon’s update failure.
**Fix**
- [ ] Reconcile failed campaign targets against later successful operations when the current instance generation and affected file hashes demonstrably satisfy the campaign. Record the recovery operation and preserve the original failed attempt.
- [ ] Clear actionable campaign attention and complete the campaign once all remaining targets are satisfied. A healthy server or unrelated successful restart alone must not qualify.
- [x] Provide an explicit retry path for failed targets that still need activation. Preserve campaign association, use the current generation, and retain normal occupancy and startup-capacity checks.
- [x] Show the relevant operation failure or recovery timestamp on attention cards instead of the latest server-health timestamp.
- [x] Add regression coverage for matching recovery, mismatched files, stale-generation evidence, unrelated restarts, failed-target retry, campaign completion, and retained failure history.
Reviewed b5ed0831ae. The original Shi No Numa campaign is fixed in production, but a successful rebuild that needed an automatic recovery restart still leaves its campaign failed, so this remains open.
Checklist
Reconcile verified recovery. Direct rebuild recovery checks the instance, host, generation, recipe, staged overrides and recorded inventory, and retains both operations. One valid recovery path is excluded: a rebuild that completes after its automatic unhealthy-container restart.
Clear attention and finish the campaign. Works for direct rebuild recovery, including waiting for other targets. The recovery-restart case below still leaves restart_state='failed', finished_at=NULL, and actionable campaign attention.
Explicit failed-target retry. Retry releases the terminal operation binding, retains its history, requeues desired files, and creates a rebuild against the current generation. Native provisioning tests verify startup-capacity and occupied-server checks for both pending and previously applied files.
Relevant evidence timestamp. Fleet operation records supply their own evidence time, and healthy-server attention cards use it instead of the health poll time. Rendering tests cover both operation and health warnings.
Requested regression coverage. Tests cover matching recovery, mismatched files and overrides, stale generations, unrelated restarts, failed-target retry, campaign completion, and retained failure history.
Review notes
Live verification: campaign dep-20261019T183806Z-promote-104 completed on October 19 at 8:45:29pm EDT, with 20/20 targets done. Shi No Numa now references successful rebuild 9118664c-20af-461f-86fa-6d409699b0e0; original failure 5cfe3bfb-4a36-4f4f-90f7-4ab3dd8aa0e1 remains linked with its original failed outcome and diagnostic. Its recovery event records generation 6 and the original successful verification time.
Remaining defect:campaign-recovery.ts:41–47 requires the operation currently linked by native_provisioning.operation_id to be instance.rebuild, with current generation equal to that operation's generation plus one. Automatic unhealthy recovery replaces that binding with instance.restart at the already advanced generation (native.ts:982). The overall rebuild can then pass health verification and finish successfully, but campaign reconciliation rejects it.
Reproduced with an isolated fixture using the native provisioning test helpers: promote the desired script; fail its campaign activation; run a standalone rebuild; report its container unhealthy; complete its automatic restart and report healthy. The rebuild is done, generation 2 and both recipe/inventory hashes match, but the campaign stays failed. The otherwise identical control without the recovery restart passes. No application source changes were made.
Validation: 8 affected/adjacent test files passed, 239 tests. npm run check passed with zero errors or warnings; npm run lint passed. The additional two-case reproduction had one passing control and one failing recovery-restart assertion.
Next steps
Correlate the completed replacement receipt and its generation through the rebuild's automatic recovery chain, while retaining the current file, instance, host and health-verification checks. An unrelated restart must still be rejected.
Add a regression for a matching rebuild that finishes after automatic recovery restart; assert campaign completion, cleared attention, and preservation of both original failure and recovery history.
Reviewed b5ed0831ae2f7b2b44753a8af0384a1d04b66580. The original Shi No Numa campaign is fixed in production, but a successful rebuild that needed an automatic recovery restart still leaves its campaign failed, so this remains open.
**Checklist**
- [ ] **Reconcile verified recovery.** Direct rebuild recovery checks the instance, host, generation, recipe, staged overrides and recorded inventory, and retains both operations. One valid recovery path is excluded: a rebuild that completes after its automatic unhealthy-container restart.
- [ ] **Clear attention and finish the campaign.** Works for direct rebuild recovery, including waiting for other targets. The recovery-restart case below still leaves `restart_state='failed'`, `finished_at=NULL`, and actionable campaign attention.
- [x] **Explicit failed-target retry.** Retry releases the terminal operation binding, retains its history, requeues desired files, and creates a rebuild against the current generation. Native provisioning tests verify startup-capacity and occupied-server checks for both pending and previously applied files.
- [x] **Relevant evidence timestamp.** Fleet operation records supply their own evidence time, and healthy-server attention cards use it instead of the health poll time. Rendering tests cover both operation and health warnings.
- [x] **Requested regression coverage.** Tests cover matching recovery, mismatched files and overrides, stale generations, unrelated restarts, failed-target retry, campaign completion, and retained failure history.
**Review notes**
- Live verification: campaign `dep-20261019T183806Z-promote-104` completed on October 19 at **8:45:29pm EDT**, with **20/20 targets done**. Shi No Numa now references successful rebuild `9118664c-20af-461f-86fa-6d409699b0e0`; original failure `5cfe3bfb-4a36-4f4f-90f7-4ab3dd8aa0e1` remains linked with its original failed outcome and diagnostic. Its recovery event records generation 6 and the original successful verification time.
- **Remaining defect:** [`campaign-recovery.ts:41–47`](https://gitea.thewrightserver.net/josh/electric-cherry/src/commit/b5ed0831ae2f7b2b44753a8af0384a1d04b66580/src/lib/server/pipeline/campaign-recovery.ts#L41) requires the operation currently linked by `native_provisioning.operation_id` to be `instance.rebuild`, with current generation equal to that operation's generation plus one. Automatic unhealthy recovery replaces that binding with `instance.restart` at the already advanced generation ([`native.ts:982`](https://gitea.thewrightserver.net/josh/electric-cherry/src/commit/b5ed0831ae2f7b2b44753a8af0384a1d04b66580/src/lib/server/provisioning/native.ts#L982)). The overall rebuild can then pass health verification and finish successfully, but campaign reconciliation rejects it.
- Reproduced with an isolated fixture using the native provisioning test helpers: promote the desired script; fail its campaign activation; run a standalone rebuild; report its container unhealthy; complete its automatic restart and report healthy. The rebuild is `done`, generation 2 and both recipe/inventory hashes match, but the campaign stays failed. The otherwise identical control without the recovery restart passes. No application source changes were made.
- **Validation:** 8 affected/adjacent test files passed, **239 tests**. `npm run check` passed with zero errors or warnings; `npm run lint` passed. The additional two-case reproduction had one passing control and one failing recovery-restart assertion.
**Next steps**
- Correlate the completed replacement receipt and its generation through the rebuild's automatic recovery chain, while retaining the current file, instance, host and health-verification checks. An unrelated restart must still be rejected.
- Add a regression for a matching rebuild that finishes after automatic recovery restart; assert campaign completion, cleared attention, and preservation of both original failure and recovery history.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
A failed campaign target remains in Fleet’s “Needs attention” section after a separate successful rebuild installs the intended release. The campaign continues referencing its failed operation, and its resume action does not support failed targets.
Observed on Shi No Numa · Lobby 1 (
t5_zombie_sumpf_1) on ec-3, October 19, 2026.Evidence
All times are EDT:
TimeoutError; at artifact_open:2738.failed, referencing the original operation.The successful rebuild’s recipe and recorded inventory match the campaign’s Common 1.7.1 and Leaderboard 1.4.1, including both SHA-256 hashes. The campaign remains unfinished with 19 targets done and one failed.
The original timeout occurred while downloading replacement files, before container teardown. Its underlying network cause is undetermined; the persistent warning after verified recovery is independently reproducible.
Cause
Inspected at
0f05be3a59f74f013c7f465b39c2747640c1fb3d:src/lib/server/provisioning/requests.ts:64–84returns the campaign’s previously linked operation, including a terminal failure.src/lib/server/pipeline/sweep.tsmirrors that operation’s failure back into the campaign target; it does not reconcile a later successful standalone rebuild.src/lib/server/fleet/overview.ts:28–34includes unfinished campaign targets independently of newer successful lifecycle operations.src/routes/api/campaigns/[id]/resume/+server.ts:12,23accepts onlyheldandmanualtargets. Afailedtarget receives HTTP 409.src/routes/+page.svelte:411–418displays the operation warning alongside the server-health timestamp.An isolated reproduction using the current application code confirmed that a later completed rebuild leaves campaign attention present and campaign resume returns 409.
Screenshot evidence
The supplied screenshot shows 20/20 healthy, Shi No Numa marked “Update needs attention”, and “Evidence 8s ago.” That timestamp describes a recent health observation, not the afternoon’s update failure.
Fix
Reviewed
b5ed0831ae. The original Shi No Numa campaign is fixed in production, but a successful rebuild that needed an automatic recovery restart still leaves its campaign failed, so this remains open.Checklist
restart_state='failed',finished_at=NULL, and actionable campaign attention.Review notes
dep-20261019T183806Z-promote-104completed on October 19 at 8:45:29pm EDT, with 20/20 targets done. Shi No Numa now references successful rebuild9118664c-20af-461f-86fa-6d409699b0e0; original failure5cfe3bfb-4a36-4f4f-90f7-4ab3dd8aa0e1remains linked with its original failed outcome and diagnostic. Its recovery event records generation 6 and the original successful verification time.campaign-recovery.ts:41–47requires the operation currently linked bynative_provisioning.operation_idto beinstance.rebuild, with current generation equal to that operation's generation plus one. Automatic unhealthy recovery replaces that binding withinstance.restartat the already advanced generation (native.ts:982). The overall rebuild can then pass health verification and finish successfully, but campaign reconciliation rejects it.done, generation 2 and both recipe/inventory hashes match, but the campaign stays failed. The otherwise identical control without the recovery restart passes. No application source changes were made.npm run checkpassed with zero errors or warnings;npm run lintpassed. The additional two-case reproduction had one passing control and one failing recovery-restart assertion.Next steps