Add persistent AFK reserves and recovery (Refs #151)
CI / checks (push) Successful in 7m7s

This commit is contained in:
2026-10-17 16:57:26 -04:00
parent 7f2205e9c4
commit d723f23d67
22 changed files with 2646 additions and 16 deletions
+1
View File
@@ -26,6 +26,7 @@ and launch your first server.
- [Plugin releases](guides/plugin-releases.md) — GitHub release discovery and imports.
- [Metrics](guides/metrics.md) — utilization, capacity pressure, and hosting experiments.
- [Player bank](guides/bank.md) — shared balances, per-server credentials, transfers and recovery.
- [AFK reserve](guides/afk.md) — persistent break time, gameplay rewards, purchases and session recovery.
- [Player moderation](guides/moderation.md) — restrictions, live actions, cases, evidence and game integration.
- [Leaderboard and matches](guides/leaderboard.md) — player statistics, match history, and the read API.
+142
View File
@@ -0,0 +1,142 @@
# Persistent AFK time
AFK time is a GUID account shared by T5 and T6, separate from the points bank. The scripts own `.afk`, `.back`, explicit purchase confirmation, actual gameplay contribution detection, protection and controls, the center-screen countdown, and all-player-AFK / match-ending behavior. Electric Cherry owns policy, durable receipts, atomic points purchases, reservations, and reporting. The immutable native instance's existing bank credential authenticates both APIs; the shared telemetry token and browser sessions cannot authorize game requests.
## Policy and accounting
All balances are integer seconds. Defaults:
| Setting | Environment variable | Default |
| ---------------------------------------------- | ---------------------------------------------------------------- | ----------- |
| Free rolling allowance | `EC_AFK_FREE_SECONDS` | 1800 |
| Combined earned/purchased capacity | `EC_AFK_CAPACITY_SECONDS` | 21600 |
| Active play interval / reward | `EC_AFK_ACTIVE_INTERVAL_SECONDS`, `EC_AFK_ACTIVE_REWARD_SECONDS` | 300 / 60 |
| Participating round reward | `EC_AFK_ROUND_REWARD_SECONDS` | 15 |
| Kill interval / reward | `EC_AFK_KILL_INTERVAL`, `EC_AFK_KILL_REWARD_SECONDS` | 100 / 30 |
| Headshot interval / reward | `EC_AFK_HEADSHOT_INTERVAL`, `EC_AFK_HEADSHOT_REWARD_SECONDS` | 25 / 15 |
| Successful teammate revive | `EC_AFK_REVIVE_REWARD_SECONDS` | 30 |
| Performance reward cap per rolling active hour | `EC_AFK_BONUS_HOURLY_SECONDS` | 480 |
| Points / seconds per purchase pack | `EC_AFK_PURCHASE_POINTS`, `EC_AFK_PURCHASE_SECONDS` | 10000 / 600 |
Free time is spent first. Its available amount is the allowance minus outstanding free reservations and confirmed free consumption charged within the preceding 24 hours. Consumption is charged at the controller's receipt of a durable stop, so free time replenishes 24 hours after that receipt. This deliberately conservative rolling window cannot grant more than the allowance in a rolling day, even with delayed delivery or inaccurate server clocks. Each charge expires independently; unused days never accumulate. Reads, reconnects, map changes, and changing game/server do not reset anything. An unclosed reservation never expires, even after 24 hours. Profile `freeRenewsAt` is the next known charge expiration, not a promise to release unresolved authorization.
Active seconds and partial kill/headshot thresholds persist by GUID across matches and servers. Cumulative snapshots contribute only their increase over the last accepted snapshot for that originating immutable instance, match key, and GUID. Active play rewards use every completed interval. Participating rounds, kill thresholds, and headshot thresholds share one rolling 3600-_active-second_ budget: each performance grant occupies budget until another 3600 verified active seconds elapses. Kill and headshot rewards can stack. Spectating, AFK, idle connections, wall-clock waiting, and raw score increases do not advance this clock. Each legitimate successful teammate revive earns the full configured revive reward, including repeated revives of the same teammate; there is no revive count limit or performance cap. Only overall saved capacity applies.
Capacity includes saved seconds currently reserved by sessions. Overflow rewards and performance bonuses above the rolling budget are discarded, while partial threshold progress persists; capacity does not bank deferred whole rewards. The scripts report only actual gameplay events, eligible active seconds, and rounds with participation. The backend requires `activity:"playing"`, rejects current reward submissions while a session is reserved or started (an unresolved, stopped crash may resume gameplay), enforces monotonic counters, and rejects cumulative active time above the match's wall duration plus a 60-second telemetry tolerance. This scoped trusted-server API cannot independently observe a game engine's successful revive or verify its eligibility. The native/GSC collection paths are part of the trust boundary.
## Machine protocol
Use `POST /api/afk/{balance,event,purchase,reserve,start,settle}` with `Content-Type: application/json` and `Authorization: Bearer <instance bank credential>`. The body limit is 4096 bytes; responses have `Cache-Control: no-store`. Purchases, reservations, start acknowledgments, and balance reads require the exact current active telemetry match and a present GUID on the authenticated server. Gameplay backlog can additionally reference a retained historical match containing that GUID when the immutable instance lifetime proves origin. Historical active time is bounded to the recorded match end or last received telemetry, and its grants still respect held saved capacity. Gameplay whose historical membership evidence has been removed is rejected without granting time. Settlement remains valid for the original instance after disconnect, match completion, or match/server history deletion.
All requests contain `server`, `guid`, and `match` (the existing telemetry `reporter_match_key`, not a separate AFK match identifier). Mutations additionally require `request_id`. GUID, match, request and session IDs are 1–64 characters from `[A-Za-z0-9_.:-]`. Optional `event_time` is a parseable timestamp no more than five minutes in the future; it records the reported event time, while `created_at` records controller receipt time. Keep the complete body immutable across retries, including `event_time`.
| Action | Additional request fields |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `balance` | None |
| `event` | `seq` positive integer; `active_seconds`, `kills`, `headshots`, `rounds`, `revives` nonnegative cumulative integer counters (maximum 2147483647); `activity:"playing"` |
| `purchase` | `packs` integer 1–1000; `confirmed:true`; `expected_purchase_points` and `expected_purchase_seconds`, copied from the displayed quote, per pack |
| `reserve` | `session_id`; `seconds` integer 1–604800, the requested maximum authorization |
| `start` | `session_id` |
| `settle` | `session_id`; `consumed_seconds` integer 0–604800 or `null`; `end_reason`; optional `confirmed_seconds` checkpoint |
`end_reason` is one of `manual`, `disconnect`, `death`, `match_end`, `expired`, `never_started`, or `crash`. `expired` requires exact consumption of the authorization. `never_started` requires zero. An unknown crash duration uses `consumed_seconds:null,end_reason:"crash",confirmed_seconds:<last durable elapsed lower bound>`. Confirmed consumption must fit the original authorization and cannot fall below an already-recorded checkpoint.
Every application result has:
```json
{
"ok": true,
"state": "reserved",
"code": "ok",
"request_id": "request-123",
"free_seconds": 0,
"saved_seconds": 600,
"available_seconds": 600,
"reserved_seconds": 1800,
"capacity_seconds": 21600,
"authorized_seconds": 1800,
"bank_balance": 20000,
"purchase_points": 10000,
"purchase_seconds": 600,
"max_purchase_packs": 2,
"session_id": "session-123",
"consumed_seconds": null
}
```
`available_seconds` excludes held authorizations; `reserved_seconds` is held time for this GUID across all servers. `authorized_seconds` is meaningful for session results and is the original authorization, never an extension. `session_id` and `consumed_seconds` appear on session results; `awarded_seconds` appears on reward and purchase results. `max_purchase_packs` is the smaller of affordable packs and capacity, or zero when a bank transfer is prepared. Recheck the quote through `balance` before confirmation; changed pricing rejects with `price_changed` without spending points. There is no automatic purchase.
States are `balance`, `rewarded`, `purchased`, `reserved`, `started`, `settled`, `unresolved`, `corrected`, and `rejected`. HTTP 200 can contain an application rejection (`ok:false`), such as `empty`, `session_pending`, `bank_pending`, `insufficient_balance`, `reserve_capacity`, `price_changed`, `invalid_active_time`, `invalid_consumption`, `progress_regression`, or `progress_conflict`. Authentication failures are 401/403; invalid syntax/amounts are 400. `not_in_match` is a nonreceipted HTTP 409 for unknown/current membership publication lag on new work, allowing the same immutable request to retry after telemetry catches up. Known completed/obsolete matches and recorded players who have departed (`present=0`) are terminal HTTP 200 rejections for new work. Only never-observed membership remains eligible for the publication-lag retry. Gameplay missing historical membership/origin proof is also a terminal HTTP 200 rejection so an unrecoverable old snapshot cannot block the durable queue; subsequent cumulative snapshots can still include its counters if membership is established. Changed request identity or ownership is HTTP 409 `request_conflict`/`session_conflict`. Transaction failures are 503 and can retry unchanged. Early authentication/body failures may contain only `ok` and `code`.
Purchase debits the existing bank account and grants saved time in the same immediate SQLite transaction, with linked point/time ledger entries. A prepared bank transfer blocks purchase until its own existing commit/cancel workflow completes. No second points bank is created.
## Ordering, authorization, and outages
1. Native code durably writes the immutable request before sending it. Use one stable request ID per operation, with separate IDs for reserve, start, and settle. Stable session IDs bind all session operations.
2. `reserve` atomically takes free time first and saved time second, authorizing `min(requested,available)` seconds. Only one outstanding session can hold a GUID's funds. An identical retry returns the saved result. A new operation referring to an already-known session cannot reset or extend it.
3. Durable native state records the reservation and the start intent. `start` must return the `started` acknowledgment before protection is enabled. Scripts additionally recheck current player/map eligibility and activate only once. A delayed acknowledgment cannot reactivate an ended local session. The countdown uses the locally recorded authorization and elapsed clock; it does not require heartbeat renewal.
4. Manual return, disconnect, death, match end, or zero countdown removes protection and restores controls before producing a durable stop. Native code records the confirmed elapsed seconds and immutable `settle` request, then retries delivery until acknowledged. Round elapsed time up to whole seconds and cap it to authorization. At zero the scripts immediately restore normal vulnerability and controls; there is no grace, automatic buy, or spectator transition. Manual `.back` retains its separate gameplay behavior.
5. `settle` records confirmed consumption and releases unused reserved seconds exactly once. It accepts legitimate delayed reports after membership disappears. `never_started` settlement can arrive before its reserve and installs a tombstone; delayed reserve/start cannot reopen that session. Completed/corrected sessions remain terminal. A retry of an earlier operation returns that operation's original receipt, which may predate settlement; native terminal journal state must win over such an old response.
6. If the controller is unavailable, an already-started session keeps its locally authorized countdown to zero. If reserve/start was never acknowledged, do not enable protection. A crashed process must first be proven stopped before recovery. A checkpoint is only a lower bound on elapsed time; an absent heartbeat or old checkpoint is never evidence for a refund. Recover unknown elapsed time as `unresolved`, retaining authorization until reliable stop evidence or an explicit administrator correction resolves it.
Receipts bind the complete body and action to server, immutable instance, and request ID. Reusing an ID with any changed parameter is a conflict. Accepted same/lower-sequence gameplay counters award no additional time; same sequence with changed counters conflicts, a later sequence with a decreased total is rejected, and a lower sequence cannot introduce a higher counter. Native/GSC code must preserve cumulative per-GUID totals across reconnects within the same match. A distinct revive increments `revives` even for the same teammate; retrying a snapshot never increments it.
For host teardown, `POST /api/agent/afk/settle` uses the assigned host credential and `{operationId,action,transfer}`. Only an unfinished `instance.remove`, `instance.rebuild`, or `instance.cancel-create` for that host/immutable instance is accepted. `action:"settle"` reconciles session evidence; `event` and `purchase` are receipt-only recovery and return `not_applied` if the original request never committed. Teardown cannot create a new purchase or reward. Instance retirement revokes normal game API credentials, including reads; the assigned host recovery path permits settlement after retirement without granting a replacement instance ownership of old sessions.
## Crash correction and reporting
An administrator can reconcile an explicitly stopped `unresolved` session with `POST /api/afk/corrections` using their browser session and same-origin protections:
```json
{
"session_id": "session-123",
"request_id": "admin-case-456",
"consumed_seconds": 120,
"reason": "Confirmed process stopped; reviewed native checkpoint and host logs.",
"stopped": true
}
```
The selected charge must be at least the confirmed checkpoint and no more than authorization. Correction cannot release a live reserved/started session, and does not convert an uncertain duration into an exact one: `state` becomes `corrected`, `consumedSeconds` remains null, `chargedSeconds` records the accounting decision, and `confirmedSeconds` retains the evidence lower bound. The audit ledger records the actor, reason, charge, and release. The correction ID is idempotent and rejects changed parameters. A true confirmed stop report can instead settle an unresolved session with its verified exact duration.
The existing player profile access controls protect balances and recent time-ledger entries alongside bank information. Reports distinguish cumulative verified active seconds, confirmed AFK duration, unresolved sessions, expiration count, manual returns, purchases, and corrected charges. Session starts, end reasons, operation/session IDs, server, match when retained, reported event time, receipt time, resulting balances, free grants, rewards, point-linked purchases, reservations, consumption, releases, and corrections survive match retention and server deletion.
## Coordinated rollout and validation
Back up the controller database and persistent game journals. Deploy EC first;
migration 30 creates the AFK accounts, operations, progress, sessions, bonus budget
and ledger tables without match/server cascading deletion. Install host agent
**1.12.0+**, including its `afk.settlement` capability, before activating the game
scripts. Package and offline-sign the agent release using the normal host update
workflow; a local build does not sign or deploy it.
Install TWS AFK scripts **2.0.0** with matching native plugins **1.4.0+** for both
engines, common **1.4.0+**, telemetry **1.0.0+**, and Plutonium **r5334+**. Fully
restart each game-server process for the plugin update. Keep the existing private
bank credential configuration and persistent `tws-afk/t5/` / `tws-afk/t6/` storage.
See the [scripts AFK guide](https://gitea.thewrightserver.net/josh/tws-zombies-scripts/src/branch/main/docs/afk.md)
for commands, active-play criteria, protection ownership, game dvars and the
in-game validation matrix.
Run `npx vitest run tests/unit/afk.test.ts tests/unit/bank.test.ts tests/unit/ui-afk-profile.test.ts`,
the normal EC checks/build, and `python3 -m unittest discover -s agent/tests -v`.
The scripts repository supplies `node tools/check_afk.mjs` and
`python3 tools/test_plugin_afk.py` with the pinned native JSON dependency headers.
Portable native tests use sanitizers and engine adapters; script tests use engine
shims. Neither proves Windows DLL linking or live game behavior.
Before production rollout, exercise both T5 and T6 with a short test allowance:
manual return, immediate zero-time expiry, downed/dead/spectator states, `.hud`
hidden, repeated revives, partial reward thresholds across reconnects, explicit
purchase confirmation and capacity rejection. Compare the center countdown to
the authorized seconds, ensure controls/weapons and normal vulnerability return
at zero, and confirm active-play metrics exclude AFK time (manual `.back` restores reward eligibility immediately, including during its separate protection grace). Test last-active
death/disconnect and stock afterlife/Grief recovery on deployed maps.
Drop reserve/start/purchase/settle responses, reorder deliveries, disconnect EC
after authorization and restart game/controller/host processes. Confirm the local
deadline survives the outage, durable stops release unused time once, and ambiguous
crashes remain unconfirmed until an audited correction. Rebuild/remove an instance
with both games' journals and a retired game credential; host recovery must retain
files until acknowledged and must never create a new purchase or gameplay reward.
Live T5/T6 gameplay and deployment validation require the actual game servers.
+19
View File
@@ -1101,3 +1101,22 @@ host credential; journal URLs never determine request destinations. Upgrade
agents first, then [rebuild servers changing bank endpoints](bank.md#upgrading-from-a-controller-derived-game-endpoint)
to drain pending decisions before the new scripts start. See the
[agent changelog](../../agent/CHANGELOG.md) for release and validation details.
### AFK journals during server teardown
Install agent **1.12.0+** before enabling [persistent AFK reserves](afk.md).
The `afk.settlement` capability adds stopped-process recovery for
`tws-afk/t5/` and `tws-afk/t6/` journals. The agent uses its enrolled controller
and assigned unfinished teardown operation at `/api/agent/afk/settle`.
Journal endpoints must match the fixed public AFK endpoint; they never control
the destination of an agent request.
Removal, rebuild and creation cancellation preserve runtime files until every
decision is acknowledged. Prepared sessions that never activated return their
reservation; confirmed stops retain their original elapsed seconds and IDs.
Active sessions interrupted by a crash become unresolved, retaining their
checkpoint and balance hold for administrator reconciliation. The agent does
not invent a duration from the time of its next heartbeat. Pending purchase or
reward requests only retrieve existing receipts during teardown; they cannot
create new spending after the game stopped. Corrupt, oversized, symlinked or
unacknowledged journals block runtime deletion.