Compare commits
6
Commits
c9a604303d
...
daf78bc11c
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
daf78bc11c | ||
|
|
7ede98ff87 | ||
|
|
76aa377ba5 | ||
|
|
13f818963b | ||
|
|
a7a600b687 | ||
|
|
eadf97a428 |
+1
-1
@@ -38,7 +38,7 @@ proposes calendar leaderboards, personal-best highlights, shareable result cards
|
|||||||
and consistent map and status presentation.
|
and consistent map and status presentation.
|
||||||
|
|
||||||
[Lobby autoscaling](plans/lobby-autoscaling-plan.md) records the implemented design
|
[Lobby autoscaling](plans/lobby-autoscaling-plan.md) records the implemented design
|
||||||
and the production dry run, pilot measurements, and rollout still to complete.
|
and the production pilot measurements and rollout still to complete.
|
||||||
Current operating instructions live in the guides above.
|
Current operating instructions live in the guides above.
|
||||||
|
|
||||||
## Research
|
## Research
|
||||||
|
|||||||
+17
-18
@@ -2,30 +2,30 @@
|
|||||||
|
|
||||||
Open **Fleet → Autoscaling** to keep empty lobbies available for new groups. Policies apply to one map variant or every map in a game. A game-wide policy applies the same minimum, maximum, and spare targets independently to each map, including all variants. A partly occupied Zombies lobby is not an empty spare, even when it still has player slots.
|
Open **Fleet → Autoscaling** to keep empty lobbies available for new groups. Policies apply to one map variant or every map in a game. A game-wide policy applies the same minimum, maximum, and spare targets independently to each map, including all variants. A partly occupied Zombies lobby is not an empty spare, even when it still has player slots.
|
||||||
|
|
||||||
Autoscaling is opt-in. Installing the feature creates no policies and changes no lobby counts. Use a two-week dry run before enabling live actions.
|
Autoscaling is opt-in. Installing the feature creates no policies and changes no lobby counts. Choose On and save a policy to enable autoscaling.
|
||||||
|
|
||||||
T5 game-wide policies cover the ten standard Zombies maps. Dead Ops Arcade is not supported.
|
T5 game-wide policies cover the ten standard Zombies maps. Dead Ops Arcade is not supported.
|
||||||
|
|
||||||
## Configure a policy
|
## Configure a policy
|
||||||
|
|
||||||
Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.
|
Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Off, with a minimum of one running lobby, one empty spare, and a 15-minute empty interval. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.
|
||||||
|
|
||||||
Choose **Preview changes** to inspect the existing lobbies, game key availability, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving.
|
Choose **Save policy** to apply changes directly. Saving checks the current fleet, policy revision, ownership conflicts, and host readiness. If saving is blocked, the form keeps your settings and shows the reason. Saved policies show their lobbies, key availability, current decision, and blockers.
|
||||||
|
|
||||||
| Mode | Behavior |
|
| Autoscaling | Behavior |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| Off | No new actions. Unsubmitted work is withdrawn and delivered operations remain tracked until settled. |
|
| Off | No new actions. Unsubmitted work is withdrawn and delivered operations remain tracked until settled. |
|
||||||
| Dry run | Records hypothetical decisions without reserving identities, hosts, or ports. Collections retain control of the lineup. |
|
| On | Adds lobbies and removes surplus lobbies after continuous empty evidence and all removal checks pass. |
|
||||||
| Create only | Adds a lobby when eligible. Surplus removal remains advisory. |
|
|
||||||
| Create and destroy | Adds lobbies and removes surplus lobbies after continuous empty evidence and all removal checks pass. |
|
|
||||||
|
|
||||||
Plutonium keys belong to a game pool shared by every map, manual launch, collection, beta, and autoscaling policy for that game. An eligible creation uses the lowest available lobby number from 1–20 and reserves any free game key atomically. Key labels do not select a map or lobby. The preview shows saved and available game keys; policy maxima can exceed pool size, and a policy can be saved before any keys are added. An empty or exhausted pool holds new growth with a clear reason; it does not prevent surplus removal. Queued launches reserve their key, stopped instances retain it, and confirmed cleanup returns it to the pool. Failed or uncertain launches keep their reservation until safely reconciled. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
|
Plutonium keys belong to a game pool shared by every map, manual launch, collection, beta, and autoscaling policy for that game. An eligible creation uses the lowest available lobby number from 1–20 and reserves any free game key atomically. Key labels do not select a map or lobby. The policy shows saved and available game keys; policy maxima can exceed pool size, and a policy can be saved before any keys are added. An empty or exhausted pool holds new growth with a clear reason; it does not prevent surplus removal. Queued launches reserve their key, stopped instances retain it, and confirmed cleanup returns it to the pool. Failed or uncertain launches keep their reservation until safely reconciled. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
|
||||||
|
|
||||||
The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies share one cooldown and one outstanding operation across their maps, prioritizing additions before removals. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.
|
The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies allow one outstanding operation across their maps, prioritizing additions before removals. Once an operation settles, the next evaluation can act immediately on another eligible lobby. Surplus lobbies must still satisfy the configured continuous empty duration before removal. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.
|
||||||
|
|
||||||
|
Existing policies saved in the removed Dry run or Create only modes switch to Off on upgrade. Review and turn autoscaling On to resume; pending work remains tracked until settled.
|
||||||
|
|
||||||
## Ownership and blockers
|
## Ownership and blockers
|
||||||
|
|
||||||
A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and apply each map’s targets independently while sharing the game’s key pool. Pool capacity is a game-wide limit; targets across maps are not guaranteed when all keys are in use. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish.
|
A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected. Game-wide rules include every catalog map and apply each map’s targets independently while sharing the game’s key pool. Pool capacity is a game-wide limit; targets across maps are not guaranteed when all keys are in use. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish.
|
||||||
|
|
||||||
Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and game key waits.
|
Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and game key waits.
|
||||||
|
|
||||||
@@ -33,7 +33,7 @@ Current positive occupancy from either telemetry or RCON vetoes an empty claim.
|
|||||||
|
|
||||||
## Pause and cancellation
|
## Pause and cancellation
|
||||||
|
|
||||||
Use **Pause all** to stop new decisions globally, or **Edit or pause → Off** for one policy. A global pause retains each live policy's ownership; setting it Off releases ownership once its outstanding operation settles. Resuming the global switch preserves each policy's selected mode.
|
Use **Pause all** to stop new decisions globally, or **Edit or pause → Off** for one policy. A global pause retains each live policy's ownership; setting it Off releases ownership once its outstanding operation settles. Resuming the global switch preserves each policy's On or Off setting.
|
||||||
|
|
||||||
Unsubmitted creates and queued removals are withdrawn where possible. Cancelling an autoscaling operation from Provisioning also turns its policy Off so it cannot immediately request a replacement.
|
Unsubmitted creates and queued removals are withdrawn where possible. Cancelling an autoscaling operation from Provisioning also turns its policy Off so it cannot immediately request a replacement.
|
||||||
|
|
||||||
@@ -49,15 +49,14 @@ Choose **Delete policy** and confirm to remove its settings, decisions, and meas
|
|||||||
|
|
||||||
Deletion requires the current policy revision and no unsettled autoscaling operation. If work is outstanding, turn the policy Off and wait for its operation to settle before deleting. Delivered or uncertain host commands retain their policy and admission evidence until reconciliation finishes.
|
Deletion requires the current policy revision and no unsettled autoscaling operation. If work is outstanding, turn the policy Off and wait for its operation to settle before deleting. Delivered or uncertain host commands retain their policy and admission evidence until reconciliation finishes.
|
||||||
|
|
||||||
## Dry run and pilot
|
## Pilot
|
||||||
|
|
||||||
The production observation period and live pilot have not been run as part of implementation. Begin them on the operator-selected maps after deploying the controller and compatible agents.
|
The production observation period and live pilot have not been run as part of implementation. Begin them on the operator-selected maps after deploying the controller and compatible agents.
|
||||||
|
|
||||||
1. Select one or two representative maps and save policies in Dry run for two weeks. Record the baseline lineup, keys, versions, host CPU and memory, player-hours, and comparable days of the week.
|
1. Select one or two representative maps and record the baseline lineup, keys, versions, host CPU and memory, and player-hours.
|
||||||
2. Review recent hypothetical actions, empty lobby-hours, minutes below the spare target, unknown-status minutes, capacity/key blockers, and projected action frequency. Dry run observes the actual lineup; it cannot predict how players would redistribute after a removal.
|
2. Select On after resolving blockers and checking agent capability. Set the minimum, maximum, and spare target, then choose Save policy.
|
||||||
3. Enable Create only on the pilot maps after resolving blockers. Measure request-to-ready time from Provisioning, browser visibility, and successful external joins separately. Increase the spare target if creation latency leaves new groups without a suitable lobby.
|
3. Monitor request-to-ready time from Provisioning, browser visibility, successful external joins, empty lobby-hours, spare deficits, and removals. Increase the spare target if creation latency leaves new groups without a suitable lobby.
|
||||||
4. Agree the acceptable spare-deficit duration for the pilot, verify agent capability, and enable Create and destroy. Keep normal gameplay and joins under observation while comparing actual host CPU and empty lobby-hours with the baseline.
|
4. Expand one map at a time when spare availability and gameplay meet your requirements. Pause if they regress.
|
||||||
5. Expand one map at a time only when savings are measurable and spare availability and gameplay meet the agreed tolerance. Pause immediately if those conditions regress.
|
|
||||||
|
|
||||||
The page retains up to 31 days of decision evidence and observations, with the 30 most recent decision changes shown. Repeated unchanged evaluations update the decision's last-seen time instead of adding a row. Unresolved intent survives retention. Provisioning history retains the operation and policy backlink separately.
|
The page retains up to 31 days of decision evidence and observations, with the 30 most recent decision changes shown. Repeated unchanged evaluations update the decision's last-seen time instead of adding a row. Unresolved intent survives retention. Provisioning history retains the operation and policy backlink separately.
|
||||||
|
|
||||||
@@ -67,6 +66,6 @@ Measurement totals exclude collection gaps. Request-to-verified-readiness summar
|
|||||||
|
|
||||||
`GET /api/autoscaling` returns policies, current evidence, recent decisions, measurements, and global pause state. Viewers may read it; mutations require an operator or administrator, using the existing authentication and origin checks.
|
`GET /api/autoscaling` returns policies, current evidence, recent decisions, measurements, and global pause state. Viewers may read it; mutations require an operator or administrator, using the existing authentication and origin checks.
|
||||||
|
|
||||||
`POST /api/autoscaling` accepts `preview`, `save`, `delete`, or `global-pause`. Delete requires `id` and the current `revision`. Preview/save requests contain `settings`, plus `id` and `revision` when editing. Use `mapSlug: '*'` for every map in the selected game; existing map-specific requests are unchanged. Scope cannot change when editing. Save also requires the preview `token`. Global pause uses a boolean `paused` field. Agent removal claims use a separate host-authenticated endpoint and are bound to the exact command, host, instance generation, policy revision, and expiry.
|
`POST /api/autoscaling` accepts `save`, `delete`, or `global-pause`. Delete requires `id` and the current `revision`. Save requests contain `settings`, plus `id` and `revision` when editing. Use `mapSlug: '*'` for every map in the selected game; existing map-specific requests are unchanged. Scope cannot change when editing. Save validates the current fleet atomically with the policy update; no preview or token is required. Global pause uses a boolean `paused` field. Agent removal claims use a separate host-authenticated endpoint and are bound to the exact command, host, instance generation, policy revision, and expiry.
|
||||||
|
|
||||||
Shared contracts live in `src/lib/autoscaling.ts`; the evaluator, snapshots, persistence, ownership, and authorization live in `src/lib/server/autoscaling/`. Provisioning remains the only lifecycle executor. See the [approved implementation plan](../plans/lobby-autoscaling-plan.md) for the design and rollout criteria.
|
Shared contracts live in `src/lib/autoscaling.ts`; the evaluator, snapshots, persistence, ownership, and authorization live in `src/lib/server/autoscaling/`. Provisioning remains the only lifecycle executor. See the [approved implementation plan](../plans/lobby-autoscaling-plan.md) for the design and rollout criteria.
|
||||||
|
|||||||
@@ -105,4 +105,4 @@ published game files. Preserve that directory across upgrades. Review
|
|||||||
[Fleet settings](settings.md) for concurrency and retention controls.
|
[Fleet settings](settings.md) for concurrency and retention controls.
|
||||||
|
|
||||||
For lobby autoscaling, start with the documented
|
For lobby autoscaling, start with the documented
|
||||||
[dry run and pilot](autoscaling.md#dry-run-and-pilot) before enabling live actions.
|
[pilot procedure](autoscaling.md#pilot) before enabling live actions.
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Lobby autoscaling implementation plan
|
# Lobby autoscaling implementation plan
|
||||||
|
|
||||||
Status: Approved and implemented October 6, 2026. Controller support and agent 1.7.0 are complete; production dry run, pilot measurements, and live rollout remain pending. See [usage and the pilot procedure](../guides/autoscaling.md).
|
Status: Approved and implemented October 6, 2026. Controller support and agent 1.7.0 are complete; production pilot measurements and live rollout remain pending. See [usage and the pilot procedure](../guides/autoscaling.md).
|
||||||
|
|
||||||
The design below preserves the approved scope and subsequent implementation
|
The design below preserves the approved scope and subsequent implementation
|
||||||
record. Use the linked guide for current policy behavior, including game-wide
|
record. Use the linked guide for current policy behavior, including game-wide
|
||||||
@@ -10,6 +10,8 @@ Introduce an opt-in policy for each game and map variant that keeps a small numb
|
|||||||
|
|
||||||
Start with Create/Destroy on existing native hosts. Run the evaluator in dry run for two weeks, then enable creation for a small pilot before enabling removal. Host acquisition, VM shutdown, Stop/Start pooling, scheduled demand prediction, and waking servers from incoming packets are outside the first release.
|
Start with Create/Destroy on existing native hosts. Run the evaluator in dry run for two weeks, then enable creation for a small pilot before enabling removal. Host acquisition, VM shutdown, Stop/Start pooling, scheduled demand prediction, and waking servers from incoming packets are outside the first release.
|
||||||
|
|
||||||
|
Update October 7, 2026: Dry run and Create only have been removed. Current policies support Off and On, defaulting to Off. Existing policies in removed modes switch to Off on upgrade. Action cooldowns have also been removed; policies retain the continuous empty duration before removal and one outstanding operation at a time. Policies now save directly with server-side validation; no preview or confirmation token is required. The original implementation stages below record the initial design; follow the current [usage and pilot procedure](../guides/autoscaling.md) for operation.
|
||||||
|
|
||||||
## Existing foundations and required extensions
|
## Existing foundations and required extensions
|
||||||
|
|
||||||
| Foundation | Reuse and required work |
|
| Foundation | Reuse and required work |
|
||||||
@@ -35,7 +37,7 @@ Use one policy per `(game, mapSlug)`, preserving configured variants such as T6
|
|||||||
| Minimum running lobbies | Default 1; a floor for desired ready capacity, subject to available keys and host capacity. |
|
| Minimum running lobbies | Default 1; a floor for desired ready capacity, subject to available keys and host capacity. |
|
||||||
| Maximum lobbies | Required explicit limit, at most 20. Count occupied lobbies, registered identities, reservations, and unresolved creations toward admission. |
|
| Maximum lobbies | Required explicit limit, at most 20. Count occupied lobbies, registered identities, reservations, and unresolved creations toward admission. |
|
||||||
| Empty spare target | Default 1. Count empty ready lobbies, not unfilled player slots. |
|
| Empty spare target | Default 1. Count empty ready lobbies, not unfilled player slots. |
|
||||||
| Empty duration before removal | Default 20 minutes; adjustable after the pilot. |
|
| Empty duration before removal | Default 15 minutes; adjustable after the pilot. |
|
||||||
| Action cooldown | Default 5 minutes between new policy actions, measured from durable admission. Cancellation and safety checks do not wait for cooldown. |
|
| Action cooldown | Default 5 minutes between new policy actions, measured from durable admission. Cancellation and safety checks do not wait for cooldown. |
|
||||||
| Launch settings | Reuse normal map recipes and automatic placement on eligible active hosts. Use current promoted files when provisioning begins. |
|
| Launch settings | Reuse normal map recipes and automatic placement on eligible active hosts. Use current promoted files when provisioning begins. |
|
||||||
|
|
||||||
|
|||||||
@@ -216,7 +216,7 @@ styling, and document their use.
|
|||||||
| --- | --- |
|
| --- | --- |
|
||||||
| Fleet cards | Map/lobby/game first; health, player count, and round next; roster and match duration below; lower-priority timing and technical identity in supporting details. Preserve compact empty cards, search, pins, table mode, and the match spotlight. |
|
| Fleet cards | Map/lobby/game first; health, player count, and round next; roster and match duration below; lower-priority timing and technical identity in supporting details. Preserve compact empty cards, search, pins, table mode, and the match spotlight. |
|
||||||
| Server detail | Add map/game identity to the shared header with health. Keep player counts, round, and timing in the Live content so Configuration and Diagnostics remain focused on their task. |
|
| Server detail | Add map/game identity to the shared header with health. Keep player counts, round, and timing in the Live content so Configuration and Diagnostics remain focused on their task. |
|
||||||
| Autoscaling | Use shared identity and mode badges, a compact count strip, and a prominent current decision/blocker. Keep per-map targets, Dry run/live mode, ownership, global pause, and unsettled operations visible. Reuse existing evidence/history disclosures. |
|
| Autoscaling | Use shared identity and mode badges, a compact count strip, and a prominent current decision/blocker. Keep per-map targets, Off/On setting, ownership, global pause, and unsettled operations visible. Reuse existing evidence/history disclosures. |
|
||||||
| Provisioning and host operations | Use consistent labels and status components while preserving action, target, result, elapsed time, and specific wait reasons. Put long IDs, source paths, and low-level metadata behind explicit details/copy controls. |
|
| Provisioning and host operations | Use consistent labels and status components while preserving action, target, result, elapsed time, and specific wait reasons. Put long IDs, source paths, and low-level metadata behind explicit details/copy controls. |
|
||||||
| Player and match results | Reuse map identity and achievement treatment across profile highlights, match results, and downloaded cards. Distinguish a current record from a record earned at that time. |
|
| Player and match results | Reuse map identity and achievement treatment across profile highlights, match results, and downloaded cards. Distinguish a current record from a record earned at that time. |
|
||||||
|
|
||||||
|
|||||||
@@ -1,16 +1,16 @@
|
|||||||
import { zombieMapBySlug, type ZombieGame } from './map-aliases.js';
|
import { zombieMapBySlug, type ZombieGame } from './map-aliases.js';
|
||||||
|
|
||||||
export const autoscalingModes = ['off', 'dry-run', 'create-only', 'create-destroy'] as const;
|
export const autoscalingModes = ['off', 'create-destroy'] as const;
|
||||||
export type AutoscalingMode = typeof autoscalingModes[number];
|
export type AutoscalingMode = typeof autoscalingModes[number];
|
||||||
export interface ScalingSettings {
|
export interface ScalingSettings {
|
||||||
game: ZombieGame;
|
game: ZombieGame;
|
||||||
/** '*' applies the targets independently to every map. */
|
/** '*' applies the targets independently to every map. */
|
||||||
mapSlug: string;
|
mapSlug: string;
|
||||||
mode: AutoscalingMode;
|
mode: AutoscalingMode;
|
||||||
minimum: number; maximum: number; spares: number; emptyMinutes: number; cooldownMinutes: number;
|
minimum: number; maximum: number; spares: number; emptyMinutes: number;
|
||||||
}
|
}
|
||||||
export interface ScalingPolicy extends ScalingSettings {
|
export interface ScalingPolicy extends ScalingSettings {
|
||||||
id: string; revision: number; nextActionAt: number; pausedReason: string | null; updatedAt: string;
|
id: string; revision: number; pausedReason: string | null; updatedAt: string;
|
||||||
}
|
}
|
||||||
export interface ScalingMember {
|
export interface ScalingMember {
|
||||||
name: string; mapSlug?: string; identity: string; state: 'occupied' | 'empty' | 'creating' | 'removing' | 'stopped' | 'unknown' | 'protected';
|
name: string; mapSlug?: string; identity: string; state: 'occupied' | 'empty' | 'creating' | 'removing' | 'stopped' | 'unknown' | 'protected';
|
||||||
@@ -25,9 +25,9 @@ export interface ScalingDecision {
|
|||||||
action: 'create' | 'destroy' | 'wait'; reason: string; server: string | null;
|
action: 'create' | 'destroy' | 'wait'; reason: string; server: string | null;
|
||||||
desired: number; deficit: number; occupied: number; empty: number; pending: number; unknown: number;
|
desired: number; deficit: number; occupied: number; empty: number; pending: number; unknown: number;
|
||||||
}
|
}
|
||||||
export interface ScalingPreview { policy: ScalingPolicy; snapshot: ScalingSnapshot; decision: ScalingDecision; token: string; blockers: string[] }
|
export interface ScalingEvaluation { policy: ScalingPolicy; snapshot: ScalingSnapshot; decision: ScalingDecision; blockers: string[] }
|
||||||
export interface ScalingHistory { id: string; at: number; lastSeenAt: number; decision: ScalingDecision; operationId: string | null; operationState: string | null }
|
export interface ScalingHistory { id: string; at: number; lastSeenAt: number; decision: ScalingDecision; operationId: string | null; operationState: string | null }
|
||||||
export interface ScalingView extends ScalingPreview {
|
export interface ScalingView extends ScalingEvaluation {
|
||||||
history: ScalingHistory[]; lastEvaluatedAt: number | null;
|
history: ScalingHistory[]; lastEvaluatedAt: number | null;
|
||||||
totals: { observedMinutes: number; emptyLobbyHours: number; deficitMinutes: number; unknownMinutes: number };
|
totals: { observedMinutes: number; emptyLobbyHours: number; deficitMinutes: number; unknownMinutes: number };
|
||||||
operations: { creates: number; removals: number; cancelled: number; failed: number; readinessSamples: number; readinessMeanSeconds: number | null; readinessMaxSeconds: number | null };
|
operations: { creates: number; removals: number; cancelled: number; failed: number; readinessSamples: number; readinessMeanSeconds: number | null; readinessMaxSeconds: number | null };
|
||||||
@@ -35,7 +35,7 @@ export interface ScalingView extends ScalingPreview {
|
|||||||
unsettled: { id: string; action: string; state: string; stage: string }[];
|
unsettled: { id: string; action: string; state: string; stage: string }[];
|
||||||
}
|
}
|
||||||
export interface ScalingReport { policies: ScalingView[]; globalPaused: boolean; liveAvailable: boolean; removalAvailable: boolean }
|
export interface ScalingReport { policies: ScalingView[]; globalPaused: boolean; liveAvailable: boolean; removalAvailable: boolean }
|
||||||
export const liveScaling = (mode: AutoscalingMode): boolean => mode === 'create-only' || mode === 'create-destroy';
|
export const liveScaling = (mode: AutoscalingMode): boolean => mode === 'create-destroy';
|
||||||
|
|
||||||
export function scalingSettings(input: unknown): ScalingSettings {
|
export function scalingSettings(input: unknown): ScalingSettings {
|
||||||
if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('Provide an autoscaling policy.');
|
if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('Provide an autoscaling policy.');
|
||||||
@@ -52,5 +52,5 @@ export function scalingSettings(input: unknown): ScalingSettings {
|
|||||||
const minimum = integer('minimum', 1, 20), maximum = integer('maximum', undefined, 20), spares = integer('spares', 1, 20);
|
const minimum = integer('minimum', 1, 20), maximum = integer('maximum', undefined, 20), spares = integer('spares', 1, 20);
|
||||||
if (minimum > maximum || spares > maximum) throw new Error('Minimum and spare target cannot exceed the maximum.');
|
if (minimum > maximum || spares > maximum) throw new Error('Minimum and spare target cannot exceed the maximum.');
|
||||||
return { game, mapSlug, mode: mode as AutoscalingMode, minimum, maximum, spares,
|
return { game, mapSlug, mode: mode as AutoscalingMode, minimum, maximum, spares,
|
||||||
emptyMinutes: integer('emptyMinutes', 20, 1440), cooldownMinutes: integer('cooldownMinutes', 5, 1440) };
|
emptyMinutes: integer('emptyMinutes', 15, 1440) };
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,9 +17,8 @@ export function scalingAdmission(operationId: string): string | null {
|
|||||||
const paused = (db().prepare("SELECT value FROM settings WHERE key='autoscaling_paused'").get() as { value: string } | undefined)?.value === '1';
|
const paused = (db().prepare("SELECT value FROM settings WHERE key='autoscaling_paused'").get() as { value: string } | undefined)?.value === '1';
|
||||||
if (!row || !liveScaling(row.mode) || !row.owns_map || row.revision !== intent.revision || row.paused_reason || paused) reason = 'Autoscaling policy is paused or changed.';
|
if (!row || !liveScaling(row.mode) || !row.owns_map || row.revision !== intent.revision || row.paused_reason || paused) reason = 'Autoscaling policy is paused or changed.';
|
||||||
else {
|
else {
|
||||||
const policy: ScalingPolicy = { ...JSON.parse(row.settings), id: row.id, mode: row.mode, revision: row.revision, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
const policy: ScalingPolicy = { ...JSON.parse(row.settings), id: row.id, mode: row.mode, revision: row.revision, pausedReason: null, updatedAt: '' };
|
||||||
const requested = JSON.parse(intent.decision) as ScalingDecision;
|
const requested = JSON.parse(intent.decision) as ScalingDecision;
|
||||||
if (requested.action === 'destroy' && row.mode !== 'create-destroy') return 'Automatic removal is disabled.';
|
|
||||||
const snapshot = scalingSnapshot(policy, Date.now(), operationId);
|
const snapshot = scalingSnapshot(policy, Date.now(), operationId);
|
||||||
if (requested.action === 'create') {
|
if (requested.action === 'create') {
|
||||||
snapshot.reservedNames = snapshot.reservedNames.filter(n => n !== requested.server);
|
snapshot.reservedNames = snapshot.reservedNames.filter(n => n !== requested.server);
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
import { createHash, randomUUID } from 'node:crypto';
|
import { createHash, randomUUID } from 'node:crypto';
|
||||||
import { db } from '../db/index.js';
|
import { db } from '../db/index.js';
|
||||||
import { liveScaling, scalingSettings, type ScalingPolicy, type ScalingPreview, type ScalingReport, type ScalingView } from '../../autoscaling.js';
|
import { liveScaling, scalingSettings, type ScalingPolicy, type ScalingEvaluation, type ScalingReport, type ScalingView } from '../../autoscaling.js';
|
||||||
import { emptyObservation, planScaling } from './planner.js';
|
import { emptyObservation, planScaling } from './planner.js';
|
||||||
import { parseServerName } from '../../map-aliases.js';
|
import { parseServerName } from '../../map-aliases.js';
|
||||||
import { scalingSnapshot } from './snapshot.js';
|
import { scalingSnapshot } from './snapshot.js';
|
||||||
@@ -15,19 +15,19 @@ import { scalingOwner } from './ownership.js';
|
|||||||
import { getServer } from '../fleet/servers.js';
|
import { getServer } from '../fleet/servers.js';
|
||||||
|
|
||||||
export class ScalingError extends Error { constructor(message: string, public status = 400) { super(message); } }
|
export class ScalingError extends Error { constructor(message: string, public status = 400) { super(message); } }
|
||||||
interface PolicyRow { id: string; revision: number; settings: string; mode: ScalingPolicy['mode']; next_action_at: number; paused_reason: string | null; updated_at: string }
|
interface PolicyRow { id: string; revision: number; settings: string; mode: ScalingPolicy['mode']; paused_reason: string | null; updated_at: string }
|
||||||
const present = (row: PolicyRow): ScalingPolicy => ({ ...JSON.parse(row.settings), id: row.id, revision: row.revision, mode: row.mode, nextActionAt: row.next_action_at, pausedReason: row.paused_reason, updatedAt: row.updated_at });
|
const present = (row: PolicyRow): ScalingPolicy => ({ ...JSON.parse(row.settings), id: row.id, revision: row.revision, mode: row.mode, pausedReason: row.paused_reason, updatedAt: row.updated_at });
|
||||||
export const listScalingPolicies = (): ScalingPolicy[] => (db().prepare('SELECT * FROM autoscaling_policies ORDER BY game,map_slug').all() as PolicyRow[]).map(present);
|
export const listScalingPolicies = (): ScalingPolicy[] => (db().prepare('SELECT * FROM autoscaling_policies ORDER BY game,map_slug').all() as PolicyRow[]).map(present);
|
||||||
export function getScalingPolicy(id: string): ScalingPolicy | undefined { const row = db().prepare('SELECT * FROM autoscaling_policies WHERE id=?').get(id) as PolicyRow | undefined; return row && present(row); }
|
export function getScalingPolicy(id: string): ScalingPolicy | undefined { const row = db().prepare('SELECT * FROM autoscaling_policies WHERE id=?').get(id) as PolicyRow | undefined; return row && present(row); }
|
||||||
export const scalingGloballyPaused = (): boolean => (db().prepare("SELECT value FROM settings WHERE key='autoscaling_paused'").get() as { value: string } | undefined)?.value === '1';
|
export const scalingGloballyPaused = (): boolean => (db().prepare("SELECT value FROM settings WHERE key='autoscaling_paused'").get() as { value: string } | undefined)?.value === '1';
|
||||||
|
|
||||||
export function previewScaling(input: unknown, id?: string, revision?: number): ScalingPreview {
|
export function evaluateScaling(input: unknown, id?: string, revision?: number): ScalingEvaluation {
|
||||||
const settings = scalingSettings(input), existing = id ? getScalingPolicy(id) : undefined;
|
const settings = scalingSettings(input), existing = id ? getScalingPolicy(id) : undefined;
|
||||||
if (id && !existing) throw new ScalingError('Policy not found.', 404);
|
if (id && !existing) throw new ScalingError('Policy not found.', 404);
|
||||||
if (existing && existing.revision !== revision) throw new ScalingError('Policy changed. Refresh before editing.', 409);
|
if (existing && existing.revision !== revision) throw new ScalingError('Policy changed. Refresh before editing.', 409);
|
||||||
if (existing && (existing.game !== settings.game || existing.mapSlug !== settings.mapSlug)) throw new ScalingError('A policy cannot change its game or map.');
|
if (existing && (existing.game !== settings.game || existing.mapSlug !== settings.mapSlug)) throw new ScalingError('A policy cannot change its game or map.');
|
||||||
const policy: ScalingPolicy = { ...settings, id: existing?.id ?? 'preview', revision: existing?.revision ?? 0,
|
const policy: ScalingPolicy = { ...settings, id: existing?.id ?? 'new-policy', revision: existing?.revision ?? 0,
|
||||||
nextActionAt: existing?.nextActionAt ?? 0, pausedReason: null, updatedAt: existing?.updatedAt ?? '' };
|
pausedReason: null, updatedAt: existing?.updatedAt ?? '' };
|
||||||
const snapshot = scalingSnapshot(policy);
|
const snapshot = scalingSnapshot(policy);
|
||||||
const blockers: string[] = [];
|
const blockers: string[] = [];
|
||||||
if (liveScaling(policy.mode)) {
|
if (liveScaling(policy.mode)) {
|
||||||
@@ -43,17 +43,14 @@ export function previewScaling(input: unknown, id?: string, revision?: number):
|
|||||||
try { nativeLaunchSelection({ automatic: true }, policy.game); } catch (error) { blockers.push((error as Error).message); }
|
try { nativeLaunchSelection({ automatic: true }, policy.game); } catch (error) { blockers.push((error as Error).message); }
|
||||||
}
|
}
|
||||||
const decision = planScaling(policy, snapshot, Date.now());
|
const decision = planScaling(policy, snapshot, Date.now());
|
||||||
const token = createHash('sha256').update(JSON.stringify({ settings, revision: policy.revision, globalPaused: scalingGloballyPaused(),
|
return { policy, snapshot, decision, blockers };
|
||||||
members: snapshot.members.map(m => [m.name, m.identity, m.state]), keyPool: snapshot.keyPool, reservations: snapshot.reservedNames.sort(), blockers: snapshot.blockers, decision })).digest('hex');
|
|
||||||
return { policy, snapshot, decision, token, blockers };
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export function saveScaling(input: unknown, id?: string, revision?: number, token?: string): ScalingPolicy {
|
export function saveScaling(input: unknown, id?: string, revision?: number): ScalingPolicy {
|
||||||
return db().transaction(() => {
|
return db().transaction(() => {
|
||||||
const preview = previewScaling(input, id, revision);
|
const evaluation = evaluateScaling(input, id, revision);
|
||||||
if (!token || preview.token !== token) throw new ScalingError('The policy preview changed. Review it again before saving.', 409);
|
if (evaluation.blockers.length) throw new ScalingError(evaluation.blockers[0]);
|
||||||
if (preview.blockers.length) throw new ScalingError(preview.blockers[0]);
|
const policy = evaluation.policy, settings = scalingSettings(policy), nextId = id ?? randomUUID(), at = new Date().toISOString();
|
||||||
const policy = preview.policy, settings = scalingSettings(policy), nextId = id ?? randomUUID(), at = new Date().toISOString();
|
|
||||||
if (id) revokePolicyRemovals(id);
|
if (id) revokePolicyRemovals(id);
|
||||||
if (!id && db().prepare('SELECT 1 FROM autoscaling_policies WHERE game=? AND map_slug=?').get(policy.game, policy.mapSlug)) throw new ScalingError('A policy already exists for this scope.', 409);
|
if (!id && db().prepare('SELECT 1 FROM autoscaling_policies WHERE game=? AND map_slug=?').get(policy.game, policy.mapSlug)) throw new ScalingError('A policy already exists for this scope.', 409);
|
||||||
db().prepare(`INSERT INTO autoscaling_policies(id,game,map_slug,settings,mode,updated_at) VALUES(?,?,?,?,?,?)
|
db().prepare(`INSERT INTO autoscaling_policies(id,game,map_slug,settings,mode,updated_at) VALUES(?,?,?,?,?,?)
|
||||||
@@ -61,7 +58,7 @@ export function saveScaling(input: unknown, id?: string, revision?: number, toke
|
|||||||
// An edit starts a new observation interval; earlier inactivity is not authorization.
|
// An edit starts a new observation interval; earlier inactivity is not authorization.
|
||||||
db().prepare('DELETE FROM autoscaling_members WHERE policy_id=?').run(nextId);
|
db().prepare('DELETE FROM autoscaling_members WHERE policy_id=?').run(nextId);
|
||||||
db().prepare('UPDATE autoscaling_policies SET owns_map=CASE WHEN ? THEN 1 ELSE owns_map END WHERE id=?').run(liveScaling(policy.mode) ? 1 : 0, nextId);
|
db().prepare('UPDATE autoscaling_policies SET owns_map=CASE WHEN ? THEN 1 ELSE owns_map END WHERE id=?').run(liveScaling(policy.mode) ? 1 : 0, nextId);
|
||||||
if (liveScaling(policy.mode)) for (const member of preview.snapshot.members) db().prepare('INSERT INTO autoscaling_members(policy_id,server,identity,observation,adopted) VALUES(?,?,?,?,1)')
|
if (liveScaling(policy.mode)) for (const member of evaluation.snapshot.members) db().prepare('INSERT INTO autoscaling_members(policy_id,server,identity,observation,adopted) VALUES(?,?,?,?,1)')
|
||||||
.run(nextId, member.name, member.identity, JSON.stringify({ identity: member.identity, observedAt: member.observedAt, emptySince: null, evaluatedAt: Date.now() }));
|
.run(nextId, member.name, member.identity, JSON.stringify({ identity: member.identity, observedAt: member.observedAt, emptySince: null, evaluatedAt: Date.now() }));
|
||||||
reconcileScaling(getScalingPolicy(nextId)!);
|
reconcileScaling(getScalingPolicy(nextId)!);
|
||||||
return getScalingPolicy(nextId)!;
|
return getScalingPolicy(nextId)!;
|
||||||
@@ -150,7 +147,7 @@ export async function runAutoscaling(): Promise<boolean> {
|
|||||||
const evidence = JSON.stringify({ members: snapshot.members, reservations: snapshot.reservedNames, keyPool: snapshot.keyPool, blockers: snapshot.blockers });
|
const evidence = JSON.stringify({ members: snapshot.members, reservations: snapshot.reservedNames, keyPool: snapshot.keyPool, blockers: snapshot.blockers });
|
||||||
const fingerprint = createHash('sha256').update(JSON.stringify([policy.revision, decision, snapshot.members.map(m => [m.name, m.identity, m.state])])).digest('hex');
|
const fingerprint = createHash('sha256').update(JSON.stringify([policy.revision, decision, snapshot.members.map(m => [m.name, m.identity, m.state])])).digest('hex');
|
||||||
const last = db().prepare('SELECT id,fingerprint FROM autoscaling_decisions WHERE policy_id=? ORDER BY evaluated_at DESC,rowid DESC LIMIT 1').get(policy.id) as { id: string; fingerprint: string } | undefined;
|
const last = db().prepare('SELECT id,fingerprint FROM autoscaling_decisions WHERE policy_id=? ORDER BY evaluated_at DESC,rowid DESC LIMIT 1').get(policy.id) as { id: string; fingerprint: string } | undefined;
|
||||||
const execute = liveScaling(policy.mode) && (decision.action === 'create' || policy.mode === 'create-destroy' && decision.action === 'destroy');
|
const execute = liveScaling(policy.mode) && (decision.action === 'create' || decision.action === 'destroy');
|
||||||
const decisionId = randomUUID();
|
const decisionId = randomUUID();
|
||||||
if (!execute && last?.fingerprint === fingerprint) db().prepare('UPDATE autoscaling_decisions SET last_seen_at=?,evidence=? WHERE id=?').run(now, evidence, last.id);
|
if (!execute && last?.fingerprint === fingerprint) db().prepare('UPDATE autoscaling_decisions SET last_seen_at=?,evidence=? WHERE id=?').run(now, evidence, last.id);
|
||||||
else db().prepare('INSERT INTO autoscaling_decisions(id,policy_id,revision,evaluated_at,last_seen_at,fingerprint,decision,evidence) VALUES(?,?,?,?,?,?,?,?)')
|
else db().prepare('INSERT INTO autoscaling_decisions(id,policy_id,revision,evaluated_at,last_seen_at,fingerprint,decision,evidence) VALUES(?,?,?,?,?,?,?,?)')
|
||||||
@@ -167,7 +164,6 @@ export async function runAutoscaling(): Promise<boolean> {
|
|||||||
if (!result.deploymentId) throw new Error('Provisioning did not return an operation identity.');
|
if (!result.deploymentId) throw new Error('Provisioning did not return an operation identity.');
|
||||||
db().prepare('UPDATE autoscaling_decisions SET operation_id=? WHERE id=?').run(result.deploymentId, decisionId);
|
db().prepare('UPDATE autoscaling_decisions SET operation_id=? WHERE id=?').run(result.deploymentId, decisionId);
|
||||||
db().prepare("INSERT INTO provisioning_sources(operation_id,kind,source_id,server) VALUES(?,'autoscaling',?,?)").run(result.deploymentId, policy.id, decision.server);
|
db().prepare("INSERT INTO provisioning_sources(operation_id,kind,source_id,server) VALUES(?,'autoscaling',?,?)").run(result.deploymentId, policy.id, decision.server);
|
||||||
db().prepare('UPDATE autoscaling_policies SET next_action_at=? WHERE id=?').run(now + policy.cooldownMinutes * 60_000, policy.id);
|
|
||||||
submitted = true;
|
submitted = true;
|
||||||
})();
|
})();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
@@ -214,7 +210,7 @@ export function scalingReport(): ScalingReport {
|
|||||||
MAX(CASE WHEN d.action='create' AND d.state IN ('done','warning') THEN (julianday(d.finished_at)-julianday(d.started_at))*86400 END) AS readinessMaxSeconds
|
MAX(CASE WHEN d.action='create' AND d.state IN ('done','warning') THEN (julianday(d.finished_at)-julianday(d.started_at))*86400 END) AS readinessMaxSeconds
|
||||||
FROM server_deployments d JOIN provisioning_sources s ON s.operation_id=d.id
|
FROM server_deployments d JOIN provisioning_sources s ON s.operation_id=d.id
|
||||||
WHERE s.kind='autoscaling' AND s.source_id=? AND d.started_at>=?`).get(policy.id, new Date(Date.now() - 31 * 86400_000).toISOString()) as ScalingView['operations'];
|
WHERE s.kind='autoscaling' AND s.source_id=? AND d.started_at>=?`).get(policy.id, new Date(Date.now() - 31 * 86400_000).toISOString()) as ScalingView['operations'];
|
||||||
return { policy, snapshot, decision, token: '', blockers: [], history, lastEvaluatedAt: history[0]?.lastSeenAt ?? null, totals, operations, unsettled, ownsMap };
|
return { policy, snapshot, decision, blockers: [], history, lastEvaluatedAt: history[0]?.lastSeenAt ?? null, totals, operations, unsettled, ownsMap };
|
||||||
});
|
});
|
||||||
return { policies, globalPaused: scalingGloballyPaused(), liveAvailable: true, removalAvailable: true };
|
return { policies, globalPaused: scalingGloballyPaused(), liveAvailable: true, removalAvailable: true };
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -47,7 +47,6 @@ export function planScaling(policy: ScalingPolicy, snapshot: ScalingSnapshot, no
|
|||||||
const blocked = members.find(m => !['occupied', 'empty'].includes(m.state));
|
const blocked = members.find(m => !['occupied', 'empty'].includes(m.state));
|
||||||
if (blocked) return result('wait', blocked.blocker ?? `Waiting for ${blocked.name}: ${blocked.state}.`);
|
if (blocked) return result('wait', blocked.blocker ?? `Waiting for ${blocked.name}: ${blocked.state}.`);
|
||||||
if (members.some(m => m.observedAt === null || m.observedAt > now || now - m.observedAt > ONLINE_MAX_MS)) return result('wait', 'Waiting for fresh occupancy evidence.');
|
if (members.some(m => m.observedAt === null || m.observedAt > now || now - m.observedAt > ONLINE_MAX_MS)) return result('wait', 'Waiting for fresh occupancy evidence.');
|
||||||
if (policy.nextActionAt > now) return result('wait', 'Action cooldown is active.');
|
|
||||||
if (occupied + empty < desired) {
|
if (occupied + empty < desired) {
|
||||||
if (snapshot.reservedNames.length >= policy.maximum) return result('wait', 'Maximum lobby reservations reached.');
|
if (snapshot.reservedNames.length >= policy.maximum) return result('wait', 'Maximum lobby reservations reached.');
|
||||||
if (!snapshot.keyPool.available) return result('wait', snapshot.keyPool.total
|
if (!snapshot.keyPool.available) return result('wait', snapshot.keyPool.total
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ export function ensureAutoscalingSchema(handle: Database.Database): void {
|
|||||||
CREATE TABLE IF NOT EXISTS autoscaling_policies (
|
CREATE TABLE IF NOT EXISTS autoscaling_policies (
|
||||||
id TEXT PRIMARY KEY, game TEXT NOT NULL, map_slug TEXT NOT NULL,
|
id TEXT PRIMARY KEY, game TEXT NOT NULL, map_slug TEXT NOT NULL,
|
||||||
revision INTEGER NOT NULL DEFAULT 1, settings TEXT NOT NULL,
|
revision INTEGER NOT NULL DEFAULT 1, settings TEXT NOT NULL,
|
||||||
mode TEXT NOT NULL DEFAULT 'off', next_action_at INTEGER NOT NULL DEFAULT 0,
|
mode TEXT NOT NULL DEFAULT 'off',
|
||||||
paused_reason TEXT, owns_map INTEGER NOT NULL DEFAULT 0, updated_at TEXT NOT NULL,
|
paused_reason TEXT, owns_map INTEGER NOT NULL DEFAULT 0, updated_at TEXT NOT NULL,
|
||||||
UNIQUE(game,map_slug)
|
UNIQUE(game,map_slug)
|
||||||
);
|
);
|
||||||
@@ -34,4 +34,22 @@ export function ensureAutoscalingSchema(handle: Database.Database): void {
|
|||||||
);
|
);
|
||||||
CREATE INDEX IF NOT EXISTS autoscaling_sample_retention ON autoscaling_samples(sampled_at);
|
CREATE INDEX IF NOT EXISTS autoscaling_sample_retention ON autoscaling_samples(sampled_at);
|
||||||
`);
|
`);
|
||||||
|
handle.transaction(() => {
|
||||||
|
const columns = handle.pragma('table_info(autoscaling_policies)') as { name: string }[];
|
||||||
|
if (columns.some(column => column.name === 'next_action_at')) {
|
||||||
|
handle.exec('ALTER TABLE autoscaling_policies DROP COLUMN next_action_at');
|
||||||
|
}
|
||||||
|
handle.exec(`UPDATE autoscaling_policies SET settings=json_remove(settings,'$.cooldownMinutes')
|
||||||
|
WHERE json_type(settings,'$.cooldownMinutes') IS NOT NULL`);
|
||||||
|
// Removed modes must not silently gain permission to destroy lobbies.
|
||||||
|
// Keep ownership while delivered work settles through normal reconciliation.
|
||||||
|
handle.prepare(`UPDATE autoscaling_authorizations SET revoked_at=?
|
||||||
|
WHERE claimed_at IS NULL AND revoked_at IS NULL AND policy_id IN
|
||||||
|
(SELECT id FROM autoscaling_policies WHERE mode IN ('dry-run','create-only'))`).run(Date.now());
|
||||||
|
handle.prepare(`UPDATE autoscaling_policies SET mode='off', settings=json_set(settings,'$.mode','off'),
|
||||||
|
revision=revision+1, paused_reason='This autoscaling mode was removed. Review and turn autoscaling On to resume.',
|
||||||
|
owns_map=CASE WHEN EXISTS (SELECT 1 FROM autoscaling_decisions WHERE policy_id=autoscaling_policies.id AND resolved=0)
|
||||||
|
THEN owns_map ELSE 0 END, updated_at=?
|
||||||
|
WHERE mode IN ('dry-run','create-only')`).run(new Date().toISOString());
|
||||||
|
})();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { json } from '@sveltejs/kit';
|
import { json } from '@sveltejs/kit';
|
||||||
import { previewScaling, saveScaling, deleteScaling, scalingReport, setScalingGlobalPause, ScalingError } from '$lib/server/autoscaling/index.js';
|
import { saveScaling, deleteScaling, scalingReport, setScalingGlobalPause, ScalingError } from '$lib/server/autoscaling/index.js';
|
||||||
|
|
||||||
export function GET() { return json(scalingReport()); }
|
export function GET() { return json(scalingReport()); }
|
||||||
export async function POST({ request }: { request: Request }) {
|
export async function POST({ request }: { request: Request }) {
|
||||||
@@ -7,8 +7,7 @@ export async function POST({ request }: { request: Request }) {
|
|||||||
const body = await request.json();
|
const body = await request.json();
|
||||||
if (!body || typeof body !== 'object' || Array.isArray(body)) throw new ScalingError('Provide a policy action.');
|
if (!body || typeof body !== 'object' || Array.isArray(body)) throw new ScalingError('Provide a policy action.');
|
||||||
if (body.id !== undefined && typeof body.id !== 'string') throw new ScalingError('Invalid policy identity.');
|
if (body.id !== undefined && typeof body.id !== 'string') throw new ScalingError('Invalid policy identity.');
|
||||||
if (body.action === 'preview') return json(previewScaling(body.settings, body.id, body.revision));
|
if (body.action === 'save') return json({ policy: saveScaling(body.settings, body.id, body.revision) });
|
||||||
if (body.action === 'save') return json({ policy: saveScaling(body.settings, body.id, body.revision, body.token) });
|
|
||||||
if (body.action === 'delete') {
|
if (body.action === 'delete') {
|
||||||
if (typeof body.id !== 'string' || !body.id || !Number.isInteger(body.revision) || body.revision < 1) throw new ScalingError('Provide the policy identity and current revision.');
|
if (typeof body.id !== 'string' || !body.id || !Number.isInteger(body.revision) || body.revision < 1) throw new ScalingError('Provide the policy identity and current revision.');
|
||||||
deleteScaling(body.id, body.revision);
|
deleteScaling(body.id, body.revision);
|
||||||
|
|||||||
@@ -7,19 +7,18 @@
|
|||||||
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
|
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
|
||||||
import Button from '$ui/components/primitives/Button.svelte';
|
import Button from '$ui/components/primitives/Button.svelte';
|
||||||
import { zombieMaps, zombieMapBySlug } from '$lib/map-aliases.js';
|
import { zombieMaps, zombieMapBySlug } from '$lib/map-aliases.js';
|
||||||
import type { ScalingSettings, ScalingPolicy, ScalingPreview, ScalingReport } from '$lib/autoscaling.js';
|
import type { ScalingSettings, ScalingPolicy, ScalingReport } from '$lib/autoscaling.js';
|
||||||
let { data }: { data: import('./$types').PageData } = $props();
|
let { data }: { data: import('./$types').PageData } = $props();
|
||||||
let snapshot = $state<ScalingReport | null>(null);
|
let snapshot = $state<ScalingReport | null>(null);
|
||||||
const report = $derived(snapshot ?? data.autoscaling);
|
const report = $derived(snapshot ?? data.autoscaling);
|
||||||
const canOperate = $derived(page.data.user?.role === 'admin' || page.data.user?.role === 'operator');
|
const canOperate = $derived(page.data.user?.role === 'admin' || page.data.user?.role === 'operator');
|
||||||
const defaults = (): ScalingSettings => ({ game: 't5', mapSlug: 'theater', mode: 'dry-run', minimum: 1, maximum: 4, spares: 1, emptyMinutes: 20, cooldownMinutes: 5 });
|
const defaults = (): ScalingSettings => ({ game: 't5', mapSlug: 'theater', mode: 'off', minimum: 1, maximum: 3, spares: 1, emptyMinutes: 15 });
|
||||||
let settings = $state<ScalingSettings>(defaults());
|
let settings = $state<ScalingSettings>(defaults());
|
||||||
let editing = $state<ScalingPolicy | null>(null);
|
let editing = $state<ScalingPolicy | null>(null);
|
||||||
let editorOpen = $state(false);
|
let editorOpen = $state(false);
|
||||||
let preview = $state<ScalingPreview | null>(null);
|
|
||||||
let busy = $state(false), actionError = $state(''), refreshError = $state(''), notice = $state('');
|
let busy = $state(false), actionError = $state(''), refreshError = $state(''), notice = $state('');
|
||||||
let generation = 0;
|
let generation = 0;
|
||||||
const modeLabel = (mode: string) => ({ off: 'Off', 'dry-run': 'Dry run', 'create-only': 'Create only', 'create-destroy': 'Create and destroy' })[mode] ?? mode;
|
const modeLabel = (mode: string) => ({ off: 'Off', 'create-destroy': 'On' })[mode] ?? mode;
|
||||||
async function refresh(signal?: AbortSignal) {
|
async function refresh(signal?: AbortSignal) {
|
||||||
const version = ++generation;
|
const version = ++generation;
|
||||||
try {
|
try {
|
||||||
@@ -32,7 +31,7 @@
|
|||||||
onMount(() => pollVisible(refresh, 15_000, false));
|
onMount(() => pollVisible(refresh, 15_000, false));
|
||||||
function edit(policy: ScalingPolicy | null) {
|
function edit(policy: ScalingPolicy | null) {
|
||||||
editing = policy; settings = policy ? { ...policy } : defaults();
|
editing = policy; settings = policy ? { ...policy } : defaults();
|
||||||
preview = null; actionError = ''; notice = ''; editorOpen = true;
|
actionError = ''; notice = ''; editorOpen = true;
|
||||||
}
|
}
|
||||||
async function post(body: object) {
|
async function post(body: object) {
|
||||||
const response = await fetch('/api/autoscaling', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
|
const response = await fetch('/api/autoscaling', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
|
||||||
@@ -40,13 +39,12 @@
|
|||||||
if (!response.ok) throw new Error(result.error ?? 'Policy action failed.');
|
if (!response.ok) throw new Error(result.error ?? 'Policy action failed.');
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
async function submit(action: 'preview' | 'save') {
|
async function submit() {
|
||||||
busy = true; actionError = ''; notice = ''; ++generation;
|
busy = true; actionError = ''; notice = ''; ++generation;
|
||||||
try {
|
try {
|
||||||
const result = await post({ action, settings, id: editing?.id, revision: editing?.revision, token: preview?.token });
|
await post({ action: 'save', settings, id: editing?.id, revision: editing?.revision });
|
||||||
if (action === 'preview') preview = result;
|
notice = 'Policy saved. Evaluation runs every 15 seconds.'; editorOpen = false; await refresh();
|
||||||
else { notice = 'Policy saved. Evaluation runs every 15 seconds.'; editorOpen = false; preview = null; await refresh(); }
|
} catch (error) { actionError = (error as Error).message; }
|
||||||
} catch (error) { actionError = (error as Error).message; preview = null; }
|
|
||||||
finally { busy = false; }
|
finally { busy = false; }
|
||||||
}
|
}
|
||||||
async function remove(policy: ScalingPolicy) {
|
async function remove(policy: ScalingPolicy) {
|
||||||
@@ -55,7 +53,7 @@
|
|||||||
busy = true; actionError = ''; notice = ''; ++generation;
|
busy = true; actionError = ''; notice = ''; ++generation;
|
||||||
try {
|
try {
|
||||||
await post({ action: 'delete', id: policy.id, revision: policy.revision });
|
await post({ action: 'delete', id: policy.id, revision: policy.revision });
|
||||||
if (editing?.id === policy.id) { editing = null; editorOpen = false; preview = null; }
|
if (editing?.id === policy.id) { editing = null; editorOpen = false; }
|
||||||
snapshot = { ...report, policies: report.policies.filter(view => view.policy.id !== policy.id) };
|
snapshot = { ...report, policies: report.policies.filter(view => view.policy.id !== policy.id) };
|
||||||
notice = 'Policy deleted.';
|
notice = 'Policy deleted.';
|
||||||
await refresh();
|
await refresh();
|
||||||
@@ -82,52 +80,39 @@
|
|||||||
{#if editorOpen && canOperate}
|
{#if editorOpen && canOperate}
|
||||||
<section class="panel" aria-labelledby="policy-editor">
|
<section class="panel" aria-labelledby="policy-editor">
|
||||||
<h2 id="policy-editor">{editing ? 'Edit policy' : 'New policy'}</h2>
|
<h2 id="policy-editor">{editing ? 'Edit policy' : 'New policy'}</h2>
|
||||||
<form onsubmit={event => { event.preventDefault(); void submit('preview'); }} oninput={() => preview = null} onchange={() => preview = null}>
|
<form onsubmit={event => { event.preventDefault(); void submit(); }}>
|
||||||
<fieldset disabled={busy}>
|
<fieldset disabled={busy}>
|
||||||
<div class="fields">
|
<div class="fields">
|
||||||
<label>Game<select bind:value={settings.game} disabled={!!editing} onchange={() => { if (settings.mapSlug !== '*') settings.mapSlug = zombieMaps(settings.game)[0].slug; }}><option value="t5">Black Ops I</option><option value="t6">Black Ops II</option></select></label>
|
<label>Game<select bind:value={settings.game} disabled={!!editing} onchange={() => { if (settings.mapSlug !== '*') settings.mapSlug = zombieMaps(settings.game)[0].slug; }}><option value="t5">Black Ops I</option><option value="t6">Black Ops II</option></select></label>
|
||||||
<label>Scope<select bind:value={settings.mapSlug} disabled={!!editing}><option value="*">Every map in this game</option>{#each zombieMaps(settings.game) as map}<option value={map.slug}>{map.friendly}</option>{/each}</select></label>
|
<label>Scope<select bind:value={settings.mapSlug} disabled={!!editing}><option value="*">Every map in this game</option>{#each zombieMaps(settings.game) as map}<option value={map.slug}>{map.friendly}</option>{/each}</select></label>
|
||||||
<label>Mode<select bind:value={settings.mode}><option value="off">Off</option><option value="dry-run">Dry run</option>{#if report.liveAvailable}<option value="create-only">Create only</option>{/if}{#if report.removalAvailable}<option value="create-destroy">Create and destroy</option>{/if}</select></label>
|
<label>Autoscaling<select bind:value={settings.mode}><option value="off">Off</option>{#if report.removalAvailable}<option value="create-destroy">On</option>{/if}</select></label>
|
||||||
<label>Minimum running lobbies<input type="number" min="1" max="20" required bind:value={settings.minimum} /></label>
|
<label>Minimum running lobbies<input type="number" min="1" max="20" required bind:value={settings.minimum} /></label>
|
||||||
<label>Maximum lobbies<input type="number" min="1" max="20" required bind:value={settings.maximum} /></label>
|
<label>Maximum lobbies<input type="number" min="1" max="20" required bind:value={settings.maximum} /></label>
|
||||||
<label>Empty spare target<input type="number" min="1" max="20" required bind:value={settings.spares} /></label>
|
<label>Empty spare target<input type="number" min="1" max="20" required bind:value={settings.spares} /></label>
|
||||||
<label>Empty duration in minutes<input type="number" min="1" max="1440" required bind:value={settings.emptyMinutes} /></label>
|
<label>Empty duration in minutes<input type="number" min="1" max="1440" required bind:value={settings.emptyMinutes} /></label>
|
||||||
<label>Action cooldown in minutes<input type="number" min="1" max="1440" required bind:value={settings.cooldownMinutes} /></label>
|
|
||||||
</div>
|
</div>
|
||||||
{#if settings.mapSlug === '*'}<p>Targets apply separately to every map, including all variants. All maps share this game’s key pool. Actions share one cooldown and run one at a time across the game.</p>{/if}
|
{#if settings.mapSlug === '*'}<p>Targets apply separately to every map, including all variants. All maps share this game’s key pool. Actions run one at a time across the game.</p>{/if}
|
||||||
<p>Each new lobby needs an available key from the game pool. Creation waits when the pool is exhausted; keys return after confirmed removal.</p>
|
<p>Each new lobby needs an available key from the game pool. Creation waits when the pool is exhausted; keys return after confirmed removal.</p>
|
||||||
<div class="actions"><Button type="submit" disabled={busy}>Preview changes</Button><Button onclick={() => editorOpen = false}>Close editor</Button></div>
|
<div class="actions"><Button type="submit" variant="primary" disabled={busy}>Save policy</Button><Button onclick={() => editorOpen = false}>Close editor</Button></div>
|
||||||
</fieldset>
|
</fieldset>
|
||||||
</form>
|
</form>
|
||||||
{#if preview}
|
|
||||||
<div class="preview" aria-live="polite">
|
|
||||||
<h3>Preview · {modeLabel(preview.policy.mode)}</h3>
|
|
||||||
<p>Target: {preview.decision.desired} running lobbies{preview.policy.mapSlug === '*' ? ' across all maps' : ''}. {preview.snapshot.keyPool.available} of {preview.snapshot.keyPool.total} game keys available. {preview.snapshot.members.length} existing or pending lobbies.</p>
|
|
||||||
<p>{preview.decision.action === 'wait' ? 'Wait' : `${preview.decision.action === 'create' ? 'Create' : 'Remove'} ${preview.decision.server}`}: {preview.decision.reason}</p>
|
|
||||||
{#if preview.policy.mode === 'create-only' || preview.policy.mode === 'create-destroy'}<p>Saving gives this policy exclusive control of the selected maps' lobby counts. Pause it before launching a competing collection or manual lifecycle action.</p>{/if}
|
|
||||||
{#if preview.snapshot.members.length}<ul>{#each preview.snapshot.members as member}<li>{member.name} · {member.state}{member.blocker ? ` · ${member.blocker}` : ''}</li>{/each}</ul>{/if}
|
|
||||||
{#each [...preview.blockers, ...preview.snapshot.blockers] as blocker}<p>{blocker}</p>{/each}
|
|
||||||
<Button variant="primary" disabled={busy || preview.blockers.length > 0} onclick={() => submit('save')}>Save reviewed policy</Button>
|
|
||||||
</div>
|
|
||||||
{/if}
|
|
||||||
</section>
|
</section>
|
||||||
{/if}
|
{/if}
|
||||||
{#if !report.policies.length}<p class="panel">No autoscaling policies. Add a policy to observe a map or a whole game in dry run.</p>{/if}
|
{#if !report.policies.length}<p class="panel">No autoscaling policies. Add a policy to manage empty lobbies for a map or a whole game.</p>{/if}
|
||||||
{#each report.policies as view (view.policy.id)}
|
{#each report.policies as view (view.policy.id)}
|
||||||
<section class="panel" id={view.policy.id} aria-labelledby={`title-${view.policy.id}`}>
|
<section class="panel" id={view.policy.id} aria-labelledby={`title-${view.policy.id}`}>
|
||||||
<div class="heading"><h2 id={`title-${view.policy.id}`}>{view.policy.mapSlug === '*' ? 'Every map' : zombieMapBySlug(view.policy.game, view.policy.mapSlug)?.friendly ?? view.policy.mapSlug} · {view.policy.game.toUpperCase()}</h2><span>{modeLabel(view.policy.mode)}</span>{#if canOperate}<Button disabled={busy} onclick={() => edit(view.policy)}>Edit or pause</Button><Button variant="danger" disabled={busy || view.unsettled.length > 0} title={view.unsettled.length ? 'Pause the policy and settle outstanding work before deleting it.' : undefined} onclick={() => remove(view.policy)}>Delete policy</Button>{/if}</div>
|
<div class="heading"><h2 id={`title-${view.policy.id}`}>{view.policy.mapSlug === '*' ? 'Every map' : zombieMapBySlug(view.policy.game, view.policy.mapSlug)?.friendly ?? view.policy.mapSlug} · {view.policy.game.toUpperCase()}</h2><span>{modeLabel(view.policy.mode)}</span>{#if canOperate}<Button disabled={busy} onclick={() => edit(view.policy)}>Edit or pause</Button><Button variant="danger" disabled={busy || view.unsettled.length > 0} title={view.unsettled.length ? 'Pause the policy and settle outstanding work before deleting it.' : undefined} onclick={() => remove(view.policy)}>Delete policy</Button>{/if}</div>
|
||||||
<p class="counts">{view.decision.occupied} occupied · {view.decision.empty} empty and ready · {view.decision.pending} pending · {view.decision.unknown} unknown or stopped</p>
|
<p class="counts">{view.decision.occupied} occupied · {view.decision.empty} empty and ready · {view.decision.pending} pending · {view.decision.unknown} unknown or stopped</p>
|
||||||
{#if view.unsettled.length}<div class="banner"><strong>Work is still settling.</strong><p>Pause this policy and settle its outstanding work before deleting it.</p>{#each view.unsettled as operation}<p><a href={`/provisioning/${operation.id}`}>{operation.action} · {operation.state}</a> — {operation.stage}</p>{/each}{#if view.ownsMap}<p>This policy retains ownership of its scope until its outstanding work settles.</p>{/if}</div>{/if}
|
{#if view.unsettled.length}<div class="banner"><strong>Work is still settling.</strong><p>Pause this policy and settle its outstanding work before deleting it.</p>{#each view.unsettled as operation}<p><a href={`/provisioning/${operation.id}`}>{operation.action} · {operation.state}</a> — {operation.stage}</p>{/each}{#if view.ownsMap}<p>This policy retains ownership of its scope until its outstanding work settles.</p>{/if}</div>{/if}
|
||||||
<p>{view.policy.mapSlug === '*' ? 'Per map: ' : ''}Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyPool.available} / {view.snapshot.keyPool.total} game keys available</p>
|
<p>{view.policy.mapSlug === '*' ? 'Per map: ' : ''}Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyPool.available} / {view.snapshot.keyPool.total} game keys available</p>
|
||||||
<p><strong>{view.policy.mode === 'dry-run' ? 'Hypothetical decision: ' : 'Current decision: '}</strong>{view.decision.action}{view.decision.server ? ` ${view.decision.server}` : ''}. {view.decision.reason}</p>
|
<p><strong>Current decision: </strong>{view.decision.action}{view.decision.server ? ` ${view.decision.server}` : ''}. {view.decision.reason}</p>
|
||||||
{#if view.policy.mode === 'create-only' && view.decision.action === 'destroy'}<p>Removal is advisory in Create only mode.</p>{/if}
|
<p class="muted">Last evaluated: {view.lastEvaluatedAt ? $timeFormat.dateTime(new Date(view.lastEvaluatedAt).toISOString()) : 'Not evaluated yet'}</p>
|
||||||
<p class="muted">Last evaluated: {view.lastEvaluatedAt ? $timeFormat.dateTime(new Date(view.lastEvaluatedAt).toISOString()) : 'Not evaluated yet'}{view.policy.nextActionAt > Date.now() ? ` · Next action after ${$timeFormat.dateTime(new Date(view.policy.nextActionAt).toISOString())}` : ''}</p>
|
|
||||||
<details><summary>Lobbies and evidence</summary><ul>{#each view.snapshot.members as member}<li><a href={`/servers/${encodeURIComponent(member.name)}`}>{member.name}</a> · {member.state}{member.blocker ? ` · ${member.blocker}` : ''}<br />{member.source ?? 'No current source'} · {member.observedAt ? $timeFormat.dateTime(new Date(member.observedAt).toISOString()) : 'Unknown observation time'}{#if member.operationId} · <a href={`/provisioning/${member.operationId}`}>Provisioning</a>{/if}</li>{/each}</ul>{#if !view.snapshot.members.length}<p>No registered or pending lobbies.</p>{/if}{#each view.snapshot.blockers as blocker}<p>{blocker}</p>{/each}</details>
|
<details><summary>Lobbies and evidence</summary><ul>{#each view.snapshot.members as member}<li><a href={`/servers/${encodeURIComponent(member.name)}`}>{member.name}</a> · {member.state}{member.blocker ? ` · ${member.blocker}` : ''}<br />{member.source ?? 'No current source'} · {member.observedAt ? $timeFormat.dateTime(new Date(member.observedAt).toISOString()) : 'Unknown observation time'}{#if member.operationId} · <a href={`/provisioning/${member.operationId}`}>Provisioning</a>{/if}</li>{/each}</ul>{#if !view.snapshot.members.length}<p>No registered or pending lobbies.</p>{/if}{#each view.snapshot.blockers as blocker}<p>{blocker}</p>{/each}</details>
|
||||||
<details><summary>Recent decisions and measurements</summary>
|
<details><summary>Recent decisions and measurements</summary>
|
||||||
<p>{view.totals.observedMinutes.toFixed(1)} observed minutes · {view.totals.emptyLobbyHours.toFixed(2)} empty lobby-hours · {view.totals.deficitMinutes.toFixed(1)} minutes below spare target · {view.totals.unknownMinutes.toFixed(1)} unknown minutes</p>
|
<p>{view.totals.observedMinutes.toFixed(1)} observed minutes · {view.totals.emptyLobbyHours.toFixed(2)} empty lobby-hours · {view.totals.deficitMinutes.toFixed(1)} minutes below spare target · {view.totals.unknownMinutes.toFixed(1)} unknown minutes</p>
|
||||||
<p>Last 31 days: {view.operations.creates} create requests · {view.operations.removals} removal requests · {view.operations.cancelled} cancelled · {view.operations.failed} failed</p>
|
<p>Last 31 days: {view.operations.creates} create requests · {view.operations.removals} removal requests · {view.operations.cancelled} cancelled · {view.operations.failed} failed</p>
|
||||||
{#if view.operations.readinessSamples}<p>Request to verified readiness: mean {view.operations.readinessMeanSeconds?.toFixed(0)} seconds, maximum {view.operations.readinessMaxSeconds?.toFixed(0)} seconds across {view.operations.readinessSamples} completed creates. Includes queue time.</p>{/if}
|
{#if view.operations.readinessSamples}<p>Request to verified readiness: mean {view.operations.readinessMeanSeconds?.toFixed(0)} seconds, maximum {view.operations.readinessMaxSeconds?.toFixed(0)} seconds across {view.operations.readinessSamples} completed creates. Includes queue time.</p>{/if}
|
||||||
<p class="muted">Up to 31 days of bounded observations; gaps are excluded. Dry run measures the actual fleet, not a simulated replacement lineup. Readiness does not certify browser visibility or successful joins.</p>
|
<p class="muted">Up to 31 days of bounded observations; gaps are excluded. Readiness does not certify browser visibility or successful joins.</p>
|
||||||
<ol>{#each view.history as item}<li>{$timeFormat.dateTime(new Date(item.at).toISOString())} · {item.decision.action} {item.decision.server ?? ''}<br />{item.decision.reason}{#if item.operationId} · <a href={`/provisioning/${item.operationId}`}>{item.operationState ?? 'View operation'}</a>{/if}</li>{/each}</ol>
|
<ol>{#each view.history as item}<li>{$timeFormat.dateTime(new Date(item.at).toISOString())} · {item.decision.action} {item.decision.server ?? ''}<br />{item.decision.reason}{#if item.operationId} · <a href={`/provisioning/${item.operationId}`}>{item.operationState ?? 'View operation'}</a>{/if}</li>{/each}</ol>
|
||||||
</details>
|
</details>
|
||||||
</section>
|
</section>
|
||||||
@@ -138,13 +123,13 @@
|
|||||||
.page { max-width: 1200px; margin: 0 auto; padding: 24px; }
|
.page { max-width: 1200px; margin: 0 auto; padding: 24px; }
|
||||||
.panel { margin: 20px 0; padding: 20px; border: 1px solid var(--border-subtle); border-radius: var(--radius-lg); background: var(--surface); overflow-wrap: anywhere; }
|
.panel { margin: 20px 0; padding: 20px; border: 1px solid var(--border-subtle); border-radius: var(--radius-lg); background: var(--surface); overflow-wrap: anywhere; }
|
||||||
.heading, .actions { display: flex; gap: 12px; align-items: center; flex-wrap: wrap; }
|
.heading, .actions { display: flex; gap: 12px; align-items: center; flex-wrap: wrap; }
|
||||||
h2 { font-size: var(--text-lg); margin: 0; } h3 { font-size: var(--text-md); }
|
h2 { font-size: var(--text-lg); margin: 0; }
|
||||||
.heading h2 { flex: 1 1 200px; } p, li { line-height: 1.6; } .muted { color: var(--text-secondary); font-size: var(--text-sm); }
|
.heading h2 { flex: 1 1 200px; } p, li { line-height: 1.6; } .muted { color: var(--text-secondary); font-size: var(--text-sm); }
|
||||||
.fields { display: grid; grid-template-columns: repeat(auto-fit, minmax(min(100%, 220px), 1fr)); gap: 16px; margin-bottom: 20px; }
|
.fields { display: grid; grid-template-columns: repeat(auto-fit, minmax(min(100%, 220px), 1fr)); gap: 16px; margin-bottom: 20px; }
|
||||||
label { display: flex; flex-direction: column; gap: 8px; font-size: var(--text-sm); }
|
label { display: flex; flex-direction: column; gap: 8px; font-size: var(--text-sm); }
|
||||||
input, select { padding: 10px; min-width: 0; color: var(--text-primary); background: var(--surface); border: 1px solid var(--border-subtle); border-radius: var(--radius-md); font: inherit; }
|
input, select { padding: 10px; min-width: 0; color: var(--text-primary); background: var(--surface); border: 1px solid var(--border-subtle); border-radius: var(--radius-md); font: inherit; }
|
||||||
fieldset { border: 0; padding: 0; margin: 16px 0; min-width: 0; }
|
fieldset { border: 0; padding: 0; margin: 16px 0; min-width: 0; }
|
||||||
.preview, .banner { padding: 16px; border: 1px solid var(--border-subtle); border-radius: var(--radius-md); }
|
.banner { padding: 16px; border: 1px solid var(--border-subtle); border-radius: var(--radius-md); }
|
||||||
details { margin-top: 16px; } summary { cursor: pointer; } li { margin: 10px 0; }
|
details { margin-top: 16px; } summary { cursor: pointer; } li { margin: 10px 0; }
|
||||||
@media (max-width: 500px) { .page { padding: 16px; } .panel { padding: 16px; } }
|
@media (max-width: 500px) { .page { padding: 16px; } .panel { padding: 16px; } }
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -5,14 +5,14 @@ import { prepareNativeHost } from '../helpers/native-host.js';
|
|||||||
let fixture: Fixture;
|
let fixture: Fixture;
|
||||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||||
afterEach(async () => { await fixture.cleanup(); });
|
afterEach(async () => { await fixture.cleanup(); });
|
||||||
const settings = { game: 't6', mapSlug: 'town', maximum: 1, mode: 'dry-run' };
|
const settings = { game: 't6', mapSlug: 'town', maximum: 1, mode: 'create-destroy' };
|
||||||
async function setup(mode = 'dry-run') {
|
async function setup(mode = 'create-destroy') {
|
||||||
if (mode.startsWith('create')) await prepareNativeHost();
|
if (mode.startsWith('create')) await prepareNativeHost();
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('key-1','t6','test-key')").run();
|
db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('key-1','t6','test-key')").run();
|
||||||
const input = { ...settings, mode };
|
const input = { ...settings, mode };
|
||||||
const saved = scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token);
|
const saved = scaling.saveScaling(input);
|
||||||
return { db, scaling, saved, input };
|
return { db, scaling, saved, input };
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -22,7 +22,7 @@ it('deletes policy evidence atomically while preserving lobbies, keys, and unrel
|
|||||||
await scaling.runAutoscaling();
|
await scaling.runAutoscaling();
|
||||||
db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('other-key','t5','other-key')").run();
|
db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('other-key','t5','other-key')").run();
|
||||||
const otherSettings = { ...settings, game: 't5', mapSlug: 'theater', mode: 'off' };
|
const otherSettings = { ...settings, game: 't5', mapSlug: 'theater', mode: 'off' };
|
||||||
const other = scaling.saveScaling(otherSettings, undefined, undefined, scaling.previewScaling(otherSettings).token);
|
const other = scaling.saveScaling(otherSettings);
|
||||||
for (const table of ['autoscaling_decisions', 'autoscaling_members', 'autoscaling_samples']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table} WHERE policy_id=?`).get(saved.id)).toEqual({ n: 1 });
|
for (const table of ['autoscaling_decisions', 'autoscaling_members', 'autoscaling_samples']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table} WHERE policy_id=?`).get(saved.id)).toEqual({ n: 1 });
|
||||||
scaling.deleteScaling(saved.id, saved.revision);
|
scaling.deleteScaling(saved.id, saved.revision);
|
||||||
expect(scaling.getScalingPolicy(saved.id)).toBeUndefined();
|
expect(scaling.getScalingPolicy(saved.id)).toBeUndefined();
|
||||||
@@ -31,10 +31,10 @@ it('deletes policy evidence atomically while preserving lobbies, keys, and unrel
|
|||||||
expect(db().prepare('SELECT name FROM servers').all()).toEqual([{ name: 't6_zombie_town_1' }]);
|
expect(db().prepare('SELECT name FROM servers').all()).toEqual([{ name: 't6_zombie_town_1' }]);
|
||||||
expect(db().prepare('SELECT COUNT(*) AS n FROM server_keys').get()).toEqual({ n: 2 });
|
expect(db().prepare('SELECT COUNT(*) AS n FROM server_keys').get()).toEqual({ n: 2 });
|
||||||
expect(db().prepare('PRAGMA foreign_key_check').all()).toEqual([]);
|
expect(db().prepare('PRAGMA foreign_key_check').all()).toEqual([]);
|
||||||
expect(scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token).id).not.toBe(saved.id);
|
expect(scaling.saveScaling({ ...settings, mode: 'off' }).id).not.toBe(saved.id);
|
||||||
});
|
});
|
||||||
|
|
||||||
it.each(['off', 'create-only', 'create-destroy'])('deletes a settled %s policy even after key loss and releases ownership', async mode => {
|
it.each(['off', 'create-destroy'])('deletes a settled %s policy even after key loss and releases ownership', async mode => {
|
||||||
const { db, scaling, saved } = await setup(mode);
|
const { db, scaling, saved } = await setup(mode);
|
||||||
db().prepare('DELETE FROM server_keys').run();
|
db().prepare('DELETE FROM server_keys').run();
|
||||||
scaling.deleteScaling(saved.id, saved.revision);
|
scaling.deleteScaling(saved.id, saved.revision);
|
||||||
@@ -47,20 +47,20 @@ it.each(['off', 'create-only', 'create-destroy'])('deletes a settled %s policy e
|
|||||||
it('rejects stale revisions and preserves the latest settings', async () => {
|
it('rejects stale revisions and preserves the latest settings', async () => {
|
||||||
const { scaling, saved } = await setup();
|
const { scaling, saved } = await setup();
|
||||||
const off = { ...settings, mode: 'off' };
|
const off = { ...settings, mode: 'off' };
|
||||||
const edited = scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token);
|
const edited = scaling.saveScaling(off, saved.id, saved.revision);
|
||||||
expect(() => scaling.deleteScaling(saved.id, saved.revision)).toThrow('changed');
|
expect(() => scaling.deleteScaling(saved.id, saved.revision)).toThrow('changed');
|
||||||
expect(scaling.getScalingPolicy(saved.id)).toEqual(edited);
|
expect(scaling.getScalingPolicy(saved.id)).toEqual(edited);
|
||||||
expect(() => scaling.deleteScaling('missing', 1)).toThrow('not found');
|
expect(() => scaling.deleteScaling('missing', 1)).toThrow('not found');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('blocks deletion with pending intent, then keeps Provisioning history after pause and settlement', async () => {
|
it('blocks deletion with pending intent, then keeps Provisioning history after pause and settlement', async () => {
|
||||||
const { db, scaling, saved } = await setup('create-only');
|
const { db, scaling, saved } = await setup('create-destroy');
|
||||||
await scaling.runAutoscaling();
|
await scaling.runAutoscaling();
|
||||||
const { operation_id: id } = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string };
|
const { operation_id: id } = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string };
|
||||||
expect(() => scaling.deleteScaling(saved.id, saved.revision)).toThrow('outstanding Provisioning');
|
expect(() => scaling.deleteScaling(saved.id, saved.revision)).toThrow('outstanding Provisioning');
|
||||||
expect(scaling.getScalingPolicy(saved.id)?.mode).toBe('create-only');
|
expect(scaling.getScalingPolicy(saved.id)?.mode).toBe('create-destroy');
|
||||||
const off = { ...settings, mode: 'off' };
|
const off = { ...settings, mode: 'off' };
|
||||||
const paused = scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token);
|
const paused = scaling.saveScaling(off, saved.id, saved.revision);
|
||||||
scaling.deleteScaling(saved.id, paused.revision);
|
scaling.deleteScaling(saved.id, paused.revision);
|
||||||
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
|
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
|
||||||
expect(getDeployment(id)).toMatchObject({ deployment: { state: 'cancelled' }, sources: [expect.objectContaining({ kind: 'autoscaling', source_id: saved.id, label: 'Deleted autoscaling policy', href: null })] });
|
expect(getDeployment(id)).toMatchObject({ deployment: { state: 'cancelled' }, sources: [expect.objectContaining({ kind: 'autoscaling', source_id: saved.id, label: 'Deleted autoscaling policy', href: null })] });
|
||||||
|
|||||||
@@ -1,62 +0,0 @@
|
|||||||
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
|
|
||||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
|
||||||
import { routePermission, roleAllows } from '../../src/lib/server/auth/access.js';
|
|
||||||
|
|
||||||
let fixture: Fixture;
|
|
||||||
beforeEach(async () => {
|
|
||||||
vi.resetModules(); fixture = await newFixture([]);
|
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
|
||||||
for (const n of [3, 8, 9, 10]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `secret-${n}`);
|
|
||||||
});
|
|
||||||
afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); });
|
|
||||||
const settings = { game: 't6', mapSlug: 'town', maximum: 4, mode: 'dry-run' };
|
|
||||||
|
|
||||||
it('persists deduplicated dry-run decisions across restart without provisioning or reservations', async () => {
|
|
||||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
|
||||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
|
||||||
const preview = scaling.previewScaling(settings);
|
|
||||||
const saved = scaling.saveScaling(settings, undefined, undefined, preview.token);
|
|
||||||
await scaling.runAutoscaling(); await scaling.runAutoscaling();
|
|
||||||
closeDb();
|
|
||||||
await scaling.runAutoscaling();
|
|
||||||
const view = scaling.scalingReport().policies[0];
|
|
||||||
expect(view.policy.id).toBe(saved.id);
|
|
||||||
expect(view.history).toHaveLength(1);
|
|
||||||
expect(view.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_1' });
|
|
||||||
expect(JSON.stringify(view)).not.toContain('secret');
|
|
||||||
for (const table of ['server_deployments', 'native_creation_intents', 'native_instances']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table}`).get()).toEqual({ n: 0 });
|
|
||||||
});
|
|
||||||
|
|
||||||
it('rejects duplicate policies, stale edits and changed previews', async () => {
|
|
||||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
|
||||||
const preview = scaling.previewScaling(settings);
|
|
||||||
const saved = scaling.saveScaling(settings, undefined, undefined, preview.token);
|
|
||||||
expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('already exists');
|
|
||||||
expect(() => scaling.previewScaling(settings, saved.id, 0)).toThrow('changed');
|
|
||||||
expect(() => scaling.saveScaling({ ...settings, maximum: 5 }, saved.id, saved.revision, preview.token)).toThrow('preview changed');
|
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
|
||||||
const beforeKeyLoss = scaling.previewScaling(settings, saved.id, saved.revision);
|
|
||||||
db().prepare("DELETE FROM server_keys WHERE id='key-10'").run();
|
|
||||||
expect(() => scaling.saveScaling(settings, saved.id, saved.revision, beforeKeyLoss.token)).toThrow('preview changed');
|
|
||||||
expect(scaling.saveScaling(settings, saved.id, saved.revision, scaling.previewScaling(settings, saved.id, saved.revision).token).maximum).toBe(4);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('pauses evaluations and prunes expired observations without deleting current history', async () => {
|
|
||||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
|
||||||
scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token);
|
|
||||||
await scaling.runAutoscaling();
|
|
||||||
scaling.setScalingGlobalPause(true); await scaling.runAutoscaling();
|
|
||||||
expect(scaling.scalingReport().policies[0].decision.reason).toContain('globally paused');
|
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
|
||||||
const id = scaling.scalingReport().policies[0].history[0].id;
|
|
||||||
db().prepare('UPDATE autoscaling_decisions SET last_seen_at=0 WHERE id!=?').run(id);
|
|
||||||
scaling.pruneAutoscaling();
|
|
||||||
expect(scaling.scalingReport().policies[0].history).toHaveLength(1);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('uses fleet authorization for page, reads, and mutations', () => {
|
|
||||||
expect(routePermission('/autoscaling', 'GET')).toBe('viewer');
|
|
||||||
expect(routePermission('/api/autoscaling', 'GET')).toBe('viewer');
|
|
||||||
expect(roleAllows('viewer', routePermission('/api/autoscaling', 'POST'))).toBe(false);
|
|
||||||
expect(roleAllows('operator', routePermission('/api/autoscaling', 'POST'))).toBe(true);
|
|
||||||
});
|
|
||||||
@@ -8,13 +8,13 @@ import { planScaling } from '../../src/lib/server/autoscaling/planner.js';
|
|||||||
let fixture: Fixture;
|
let fixture: Fixture;
|
||||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||||
afterEach(async () => { await fixture.cleanup(); });
|
afterEach(async () => { await fixture.cleanup(); });
|
||||||
const settings = { game: 't6', mapSlug: '*', maximum: 2, mode: 'create-only' };
|
const settings = { game: 't6', mapSlug: '*', maximum: 2, mode: 'create-destroy' };
|
||||||
const maps = zombieMaps('t6');
|
const maps = zombieMaps('t6');
|
||||||
|
|
||||||
it('satisfies a T5 game-wide policy with the ten standard Zombies maps', () => {
|
it('satisfies a T5 game-wide policy with the ten standard Zombies maps', () => {
|
||||||
const now = 2_000_000;
|
const now = 2_000_000;
|
||||||
const policy: ScalingPolicy = { ...scalingSettings({ ...settings, game: 't5' }),
|
const policy: ScalingPolicy = { ...scalingSettings({ ...settings, game: 't5' }),
|
||||||
id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
id: 'policy', revision: 1, pausedReason: null, updatedAt: '' };
|
||||||
const members: ScalingMember[] = zombieMaps('t5').map(map => ({ name: `t5_zombie_${map.slug}_1`, mapSlug: map.slug,
|
const members: ScalingMember[] = zombieMaps('t5').map(map => ({ name: `t5_zombie_${map.slug}_1`, mapSlug: map.slug,
|
||||||
identity: map.slug, state: 'empty', players: 0, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null }));
|
identity: map.slug, state: 'empty', players: 0, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null }));
|
||||||
const snapshot = { members, reservedNames: members.map(m => m.name), keyPool: { total: 20, available: 10 }, blockers: [] };
|
const snapshot = { members, reservedNames: members.map(m => m.name), keyPool: { total: 20, available: 10 }, blockers: [] };
|
||||||
@@ -32,7 +32,7 @@ async function setup() {
|
|||||||
|
|
||||||
it('applies targets and spare deficits independently to every map', () => {
|
it('applies targets and spare deficits independently to every map', () => {
|
||||||
const now = 2_000_000;
|
const now = 2_000_000;
|
||||||
const policy: ScalingPolicy = { ...scalingSettings(settings), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
const policy: ScalingPolicy = { ...scalingSettings(settings), id: 'policy', revision: 1, pausedReason: null, updatedAt: '' };
|
||||||
const members: ScalingMember[] = maps.flatMap(map => [1, 2].map(n => ({ name: `t6_zombie_${map.slug}_${n}`, mapSlug: map.slug,
|
const members: ScalingMember[] = maps.flatMap(map => [1, 2].map(n => ({ name: `t6_zombie_${map.slug}_${n}`, mapSlug: map.slug,
|
||||||
identity: `${map.slug}:${n}`, state: map.slug === 'town' ? 'occupied' : 'empty', players: map.slug === 'town' ? 1 : 0,
|
identity: `${map.slug}:${n}`, state: map.slug === 'town' ? 'occupied' : 'empty', players: map.slug === 'town' ? 1 : 0,
|
||||||
observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null })));
|
observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null })));
|
||||||
@@ -42,7 +42,6 @@ it('applies targets and spare deficits independently to every map', () => {
|
|||||||
snapshot.members = members.filter(m => m.name !== 't6_zombie_town_2');
|
snapshot.members = members.filter(m => m.name !== 't6_zombie_town_2');
|
||||||
snapshot.reservedNames = snapshot.members.map(m => m.name);
|
snapshot.reservedNames = snapshot.members.map(m => m.name);
|
||||||
expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'create', server: 't6_zombie_town_2' });
|
expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'create', server: 't6_zombie_town_2' });
|
||||||
expect(planScaling({ ...policy, nextActionAt: now + 1 }, snapshot, now).action).toBe('wait');
|
|
||||||
snapshot.members[0] = { ...snapshot.members[0], state: 'creating' };
|
snapshot.members[0] = { ...snapshot.members[0], state: 'creating' };
|
||||||
expect(planScaling(policy, snapshot, now).action).toBe('wait');
|
expect(planScaling(policy, snapshot, now).action).toBe('wait');
|
||||||
});
|
});
|
||||||
@@ -50,11 +49,11 @@ it('applies targets and spare deficits independently to every map', () => {
|
|||||||
it('persists a game rule, queues the selected map recipe, and protects every map', async () => {
|
it('persists a game rule, queues the selected map recipe, and protects every map', async () => {
|
||||||
const { db, scaling } = await setup();
|
const { db, scaling } = await setup();
|
||||||
db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('other-key','t5','other-game')").run();
|
db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('other-key','t5','other-game')").run();
|
||||||
const preview = scaling.previewScaling(settings);
|
const evaluation = scaling.evaluateScaling(settings);
|
||||||
expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 1 });
|
expect(evaluation.snapshot.keyPool).toEqual({ total: 1, available: 1 });
|
||||||
const saved = scaling.saveScaling(settings, undefined, undefined, preview.token);
|
const saved = scaling.saveScaling(settings);
|
||||||
expect(() => scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token)).toThrow();
|
expect(() => scaling.saveScaling(settings)).toThrow();
|
||||||
expect(() => scaling.previewScaling({ ...settings, mapSlug: 'town' }, saved.id, saved.revision)).toThrow('cannot change');
|
expect(() => scaling.evaluateScaling({ ...settings, mapSlug: 'town' }, saved.id, saved.revision)).toThrow('cannot change');
|
||||||
expect(await scaling.runAutoscaling()).toBe(true);
|
expect(await scaling.runAutoscaling()).toBe(true);
|
||||||
const job = db().prepare('SELECT deployment_id,map_slug,server FROM server_creation_jobs').get() as { deployment_id: string; map_slug: string; server: string };
|
const job = db().prepare('SELECT deployment_id,map_slug,server FROM server_creation_jobs').get() as { deployment_id: string; map_slug: string; server: string };
|
||||||
expect(job).toMatchObject({ map_slug: 'transit', server: 't6_zombie_transit_1' });
|
expect(job).toMatchObject({ map_slug: 'transit', server: 't6_zombie_transit_1' });
|
||||||
@@ -66,24 +65,54 @@ it('persists a game rule, queues the selected map recipe, and protects every map
|
|||||||
for (const map of maps) expect(scalingOwner('t6', map.slug)).toBe(saved.id);
|
for (const map of maps) expect(scalingOwner('t6', map.slug)).toBe(saved.id);
|
||||||
expect(scalingOwner('t5', 'theater')).toBeNull();
|
expect(scalingOwner('t5', 'theater')).toBeNull();
|
||||||
expect(scalingOwnershipBlocker('t6_zombie_farm_1')).toContain(saved.id);
|
expect(scalingOwnershipBlocker('t6_zombie_farm_1')).toContain(saved.id);
|
||||||
expect(scaling.previewScaling({ ...settings, mapSlug: 'town' }).blockers.join(' ')).toContain('overlapping');
|
expect(scaling.evaluateScaling({ ...settings, mapSlug: 'town' }).blockers.join(' ')).toContain('overlapping');
|
||||||
const off = { ...settings, mode: 'off' };
|
const off = { ...settings, mode: 'off' };
|
||||||
scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token);
|
scaling.saveScaling(off, saved.id, saved.revision);
|
||||||
expect(scalingOwner('t6', 'town')).toBeNull();
|
expect(scalingOwner('t6', 'town')).toBeNull();
|
||||||
const mapSettings = { ...settings, mapSlug: 'town' };
|
const mapSettings = { ...settings, mapSlug: 'town' };
|
||||||
const mapPolicy = scaling.saveScaling(mapSettings, undefined, undefined, scaling.previewScaling(mapSettings).token);
|
const mapPolicy = scaling.saveScaling(mapSettings);
|
||||||
const current = scaling.getScalingPolicy(saved.id)!;
|
const current = scaling.getScalingPolicy(saved.id)!;
|
||||||
expect(scaling.previewScaling(settings, saved.id, current.revision).blockers.join(' ')).toContain(mapPolicy.id);
|
expect(scaling.evaluateScaling(settings, saved.id, current.revision).blockers.join(' ')).toContain(mapPolicy.id);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('holds game-wide growth when its shared pool is exhausted by another map', async () => {
|
it('holds game-wide growth when its shared pool is exhausted by another map', async () => {
|
||||||
const { db, scaling } = await setup();
|
const { db, scaling } = await setup();
|
||||||
db().prepare("INSERT INTO server_key_leases(server_id,key_id) VALUES('t6_zombie_town_2','key-1')").run();
|
db().prepare("INSERT INTO server_key_leases(server_id,key_id) VALUES('t6_zombie_town_2','key-1')").run();
|
||||||
const preview = scaling.previewScaling(settings);
|
const evaluation = scaling.evaluateScaling(settings);
|
||||||
expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 0 });
|
expect(evaluation.snapshot.keyPool).toEqual({ total: 1, available: 0 });
|
||||||
expect(preview.blockers).toEqual([]);
|
expect(evaluation.blockers).toEqual([]);
|
||||||
expect(preview.decision).toMatchObject({ action: 'wait', reason: expect.stringContaining('keys are in use') });
|
expect(evaluation.decision).toMatchObject({ action: 'wait', reason: expect.stringContaining('keys are in use') });
|
||||||
scaling.saveScaling(settings, undefined, undefined, preview.token);
|
scaling.saveScaling(settings);
|
||||||
expect(await scaling.runAutoscaling()).toBe(false);
|
expect(await scaling.runAutoscaling()).toBe(false);
|
||||||
expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 0 });
|
expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 0 });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('moves to the next map as soon as the prior creation settles, without waiting between actions', async () => {
|
||||||
|
const { host, versions } = await prepareNativeHost();
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { config } = await import('../../src/lib/server/config.js');
|
||||||
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
|
for (const n of [1, 2]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `game-key-${n}`);
|
||||||
|
vi.useFakeTimers({ toFake: ['Date'] });
|
||||||
|
try {
|
||||||
|
const at = new Date().toISOString();
|
||||||
|
scaling.saveScaling(settings);
|
||||||
|
expect(await scaling.runAutoscaling()).toBe(true);
|
||||||
|
const job = db().prepare('SELECT deployment_id,server FROM server_creation_jobs').get() as { deployment_id: string; server: string };
|
||||||
|
expect(job.server).toBe('t6_zombie_transit_1');
|
||||||
|
expect(await scaling.runAutoscaling()).toBe(false);
|
||||||
|
expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 1 });
|
||||||
|
|
||||||
|
// Record the first lobby becoming ready without advancing the clock.
|
||||||
|
db().prepare("UPDATE server_deployments SET state='done',finished_at=? WHERE id=?").run(at, job.deployment_id);
|
||||||
|
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at)
|
||||||
|
VALUES('ready-instance',?,?,?,?,28960,?)`).run(job.server, host.id, versions.get('t6'), config.defaultRuntimeImage, at);
|
||||||
|
db().prepare(`INSERT INTO servers(name,game,map,map_slug,container_state,container_checked_at,last_seen_at,last_checked_at,last_players)
|
||||||
|
VALUES(?,'t6','zm_transit','transit','running',?,?,?,0)`).run(job.server, at, at, at);
|
||||||
|
expect(await scaling.runAutoscaling()).toBe(true);
|
||||||
|
expect(db().prepare('SELECT server FROM server_creation_jobs ORDER BY rowid').all()).toEqual([
|
||||||
|
{ server: 't6_zombie_transit_1' }, { server: 't6_zombie_town_1' }
|
||||||
|
]);
|
||||||
|
expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_decisions WHERE resolved=0').get()).toEqual({ n: 1 });
|
||||||
|
} finally { vi.useRealTimers(); }
|
||||||
|
});
|
||||||
|
|||||||
@@ -5,24 +5,24 @@ import { prepareNativeHost } from '../helpers/native-host.js';
|
|||||||
let fixture: Fixture;
|
let fixture: Fixture;
|
||||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||||
afterEach(async () => { await fixture.cleanup(); });
|
afterEach(async () => { await fixture.cleanup(); });
|
||||||
const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'dry-run' };
|
const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'create-destroy' };
|
||||||
async function setup() {
|
async function setup() {
|
||||||
|
await prepareNativeHost();
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
const { saveServerKey, reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
|
const { saveServerKey, reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
|
||||||
return { db, scaling, saveServerKey, reserveServerKey };
|
return { db, scaling, saveServerKey, reserveServerKey };
|
||||||
}
|
}
|
||||||
|
|
||||||
it.each(['off', 'dry-run', 'create-only', 'create-destroy'])('saves a %s policy without preallocating keys for its maximum', async mode => {
|
it.each(['off', 'create-destroy'])('saves a %s policy without preallocating keys for its maximum', async mode => {
|
||||||
if (mode.startsWith('create')) await prepareNativeHost();
|
|
||||||
const { db, scaling } = await setup();
|
const { db, scaling } = await setup();
|
||||||
const input = { ...settings, mode, maximum: 20 };
|
const input = { ...settings, mode, maximum: 20 };
|
||||||
const preview = scaling.previewScaling(input);
|
const evaluation = scaling.evaluateScaling(input);
|
||||||
expect(preview.blockers).toEqual([]);
|
expect(evaluation.blockers).toEqual([]);
|
||||||
expect(preview.decision.action).toBe('wait');
|
expect(evaluation.decision.action).toBe('wait');
|
||||||
if (mode !== 'off') expect(preview.decision.reason).toContain('No Plutonium keys');
|
if (mode !== 'off') expect(evaluation.decision.reason).toContain('No Plutonium keys');
|
||||||
const { POST } = await import('../../src/routes/api/autoscaling/+server.js');
|
const { POST } = await import('../../src/routes/api/autoscaling/+server.js');
|
||||||
const response = await POST({ request: new Request('http://localhost/api/autoscaling', { method: 'POST', body: JSON.stringify({ action: 'save', settings: input, token: preview.token }) }) });
|
const response = await POST({ request: new Request('http://localhost/api/autoscaling', { method: 'POST', body: JSON.stringify({ action: 'save', settings: input }) }) });
|
||||||
expect(response.status).toBe(200);
|
expect(response.status).toBe(200);
|
||||||
expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_policies').get()).toEqual({ n: 1 });
|
expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_policies').get()).toEqual({ n: 1 });
|
||||||
expect(await scaling.runAutoscaling()).toBe(false);
|
expect(await scaling.runAutoscaling()).toBe(false);
|
||||||
@@ -32,46 +32,45 @@ it('uses any free key for the selected game independently of its label or map',
|
|||||||
const { scaling, saveServerKey } = await setup();
|
const { scaling, saveServerKey } = await setup();
|
||||||
saveServerKey('t5', 'other-game-key', 'Town Lobby 1');
|
saveServerKey('t5', 'other-game-key', 'Town Lobby 1');
|
||||||
saveServerKey('t6', 'game-key', 'Farm Lobby 8');
|
saveServerKey('t6', 'game-key', 'Farm Lobby 8');
|
||||||
const preview = scaling.previewScaling(settings);
|
const evaluation = scaling.evaluateScaling(settings);
|
||||||
expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 1 });
|
expect(evaluation.snapshot.keyPool).toEqual({ total: 1, available: 1 });
|
||||||
expect(preview.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_1' });
|
expect(evaluation.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_1' });
|
||||||
expect(scaling.saveScaling(settings, undefined, undefined, preview.token).maximum).toBe(2);
|
expect(scaling.saveScaling(settings).maximum).toBe(2);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('uses one shared pool for every map without requiring a key for each maximum', async () => {
|
it('uses one shared pool for every map without requiring a key for each maximum', async () => {
|
||||||
const { scaling, saveServerKey } = await setup();
|
const { scaling, saveServerKey } = await setup();
|
||||||
saveServerKey('t6', 'game-key');
|
saveServerKey('t6', 'game-key');
|
||||||
const input = { ...settings, mapSlug: '*', maximum: 20 };
|
const input = { ...settings, mapSlug: '*', maximum: 20 };
|
||||||
const preview = scaling.previewScaling(input);
|
const evaluation = scaling.evaluateScaling(input);
|
||||||
expect(preview.blockers).toEqual([]);
|
expect(evaluation.blockers).toEqual([]);
|
||||||
expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 1 });
|
expect(evaluation.snapshot.keyPool).toEqual({ total: 1, available: 1 });
|
||||||
expect(scaling.saveScaling(input, undefined, undefined, preview.token).mapSlug).toBe('*');
|
expect(scaling.saveScaling(input).mapSlug).toBe('*');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('rechecks game pool deletion and reservations after preview', async () => {
|
it('saves against the current key pool without an intermediate confirmation', async () => {
|
||||||
const { db, scaling, saveServerKey, reserveServerKey } = await setup();
|
const { db, scaling, saveServerKey, reserveServerKey } = await setup();
|
||||||
const keyId = saveServerKey('t6', 'game-key');
|
const keyId = saveServerKey('t6', 'game-key');
|
||||||
const preview = scaling.previewScaling(settings);
|
|
||||||
db().prepare('DELETE FROM server_keys WHERE id=?').run(keyId);
|
db().prepare('DELETE FROM server_keys WHERE id=?').run(keyId);
|
||||||
expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('preview changed');
|
const saved = scaling.saveScaling(settings);
|
||||||
|
expect(scaling.scalingReport().policies[0].decision.reason).toContain('No Plutonium keys');
|
||||||
saveServerKey('t6', 'game-key');
|
saveServerKey('t6', 'game-key');
|
||||||
const available = scaling.previewScaling(settings);
|
|
||||||
reserveServerKey('t6_zombie_farm_1', 't6');
|
reserveServerKey('t6_zombie_farm_1', 't6');
|
||||||
expect(() => scaling.saveScaling(settings, undefined, undefined, available.token)).toThrow('preview changed');
|
expect(scaling.saveScaling({ ...settings, maximum: 3 }, saved.id, saved.revision).maximum).toBe(3);
|
||||||
const held = scaling.previewScaling(settings);
|
const view = scaling.scalingReport().policies[0];
|
||||||
expect(held.decision.reason).toContain('keys are in use');
|
expect(view.decision.reason).toContain('keys are in use');
|
||||||
expect(held.snapshot.members).toEqual([]);
|
expect(view.snapshot.members).toEqual([]);
|
||||||
expect(scaling.saveScaling(settings, undefined, undefined, held.token).maximum).toBe(2);
|
expect(await scaling.runAutoscaling()).toBe(false);
|
||||||
|
expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 0 });
|
||||||
});
|
});
|
||||||
|
|
||||||
it('shares capacity between live map policies and admits the job holding the last key', async () => {
|
it('shares capacity between live map policies and admits the job holding the last key', async () => {
|
||||||
await prepareNativeHost();
|
|
||||||
const { db, scaling, saveServerKey } = await setup();
|
const { db, scaling, saveServerKey } = await setup();
|
||||||
saveServerKey('t6', 'game-key');
|
saveServerKey('t6', 'game-key');
|
||||||
saveServerKey('t5', 'other-game-key');
|
saveServerKey('t5', 'other-game-key');
|
||||||
for (const mapSlug of ['town', 'farm']) {
|
for (const mapSlug of ['town', 'farm']) {
|
||||||
const input = { ...settings, mapSlug, mode: 'create-only' };
|
const input = { ...settings, mapSlug, mode: 'create-destroy' };
|
||||||
scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token);
|
scaling.saveScaling(input);
|
||||||
}
|
}
|
||||||
expect(await scaling.runAutoscaling()).toBe(true);
|
expect(await scaling.runAutoscaling()).toBe(true);
|
||||||
const jobs = db().prepare('SELECT deployment_id,server FROM server_creation_jobs').all() as { deployment_id: string; server: string }[];
|
const jobs = db().prepare('SELECT deployment_id,server FROM server_creation_jobs').all() as { deployment_id: string; server: string }[];
|
||||||
@@ -92,7 +91,7 @@ it('reserves a lobby identity for an unsettled failed launch with a retained key
|
|||||||
saveServerKey('t6', 'held-key');
|
saveServerKey('t6', 'held-key');
|
||||||
reserveServerKey('t6_zombie_town_1', 't6');
|
reserveServerKey('t6_zombie_town_1', 't6');
|
||||||
saveServerKey('t6', 'free-key');
|
saveServerKey('t6', 'free-key');
|
||||||
const preview = scaling.previewScaling(settings);
|
const evaluation = scaling.evaluateScaling(settings);
|
||||||
expect(preview.snapshot.reservedNames).toEqual(['t6_zombie_town_1']);
|
expect(evaluation.snapshot.reservedNames).toEqual(['t6_zombie_town_1']);
|
||||||
expect(preview.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_2' });
|
expect(evaluation.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_2' });
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -5,13 +5,13 @@ import { prepareNativeHost } from '../helpers/native-host.js';
|
|||||||
let fixture: Fixture;
|
let fixture: Fixture;
|
||||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||||
afterEach(async () => { await fixture.cleanup(); });
|
afterEach(async () => { await fixture.cleanup(); });
|
||||||
const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'create-only' };
|
const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'create-destroy' };
|
||||||
async function setup() {
|
async function setup() {
|
||||||
await prepareNativeHost();
|
await prepareNativeHost();
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
for (const n of [3, 8]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `key-${n}`);
|
for (const n of [3, 8]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `key-${n}`);
|
||||||
const saved = scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token);
|
const saved = scaling.saveScaling(settings);
|
||||||
return { db, scaling, saved };
|
return { db, scaling, saved };
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -34,15 +34,15 @@ it('withdraws an unsubmitted create and returns its game key when the policy pau
|
|||||||
const { db, scaling, saved } = await setup();
|
const { db, scaling, saved } = await setup();
|
||||||
await scaling.runAutoscaling();
|
await scaling.runAutoscaling();
|
||||||
const off = { ...settings, mode: 'off' };
|
const off = { ...settings, mode: 'off' };
|
||||||
scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token);
|
scaling.saveScaling(off, saved.id, saved.revision);
|
||||||
expect(db().prepare("SELECT state FROM server_deployments WHERE action='create'").get()).toEqual({ state: 'cancelled' });
|
expect(db().prepare("SELECT state FROM server_deployments WHERE action='create'").get()).toEqual({ state: 'cancelled' });
|
||||||
expect(db().prepare('SELECT owns_map FROM autoscaling_policies').get()).toEqual({ owns_map: 0 });
|
expect(db().prepare('SELECT owns_map FROM autoscaling_policies').get()).toEqual({ owns_map: 0 });
|
||||||
expect(db().prepare('SELECT COUNT(*) AS n FROM server_key_leases').get()).toEqual({ n: 0 });
|
expect(db().prepare('SELECT COUNT(*) AS n FROM server_key_leases').get()).toEqual({ n: 0 });
|
||||||
db().prepare('DELETE FROM server_keys').run();
|
db().prepare('DELETE FROM server_keys').run();
|
||||||
const current = scaling.getScalingPolicy(saved.id)!;
|
const current = scaling.getScalingPolicy(saved.id)!;
|
||||||
const preview = scaling.previewScaling(settings, saved.id, current.revision);
|
const evaluation = scaling.evaluateScaling(settings, saved.id, current.revision);
|
||||||
expect(preview.blockers).toEqual([]);
|
expect(evaluation.blockers).toEqual([]);
|
||||||
expect(preview.snapshot.keyPool).toEqual({ total: 0, available: 0 });
|
expect(evaluation.snapshot.keyPool).toEqual({ total: 0, available: 0 });
|
||||||
});
|
});
|
||||||
|
|
||||||
it('blocks manual lifecycle and collection creation, including stale previews', async () => {
|
it('blocks manual lifecycle and collection creation, including stale previews', async () => {
|
||||||
@@ -56,10 +56,10 @@ it('blocks manual lifecycle and collection creation, including stale previews',
|
|||||||
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
|
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||||
const blueprint = saveBlueprint({ name: 'Town', entries: [{ game: 't6', mapSlug: 'town', count: 1 }] });
|
const blueprint = saveBlueprint({ name: 'Town', entries: [{ game: 't6', mapSlug: 'town', count: 1 }] });
|
||||||
expect(previewCollection(blueprint.id).blockers.join(' ')).toContain('Autoscaling');
|
expect(previewCollection(blueprint.id).blockers.join(' ')).toContain('Autoscaling');
|
||||||
const off = { ...settings, mode: 'off' }; scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token);
|
const off = { ...settings, mode: 'off' }; scaling.saveScaling(off, saved.id, saved.revision);
|
||||||
const preview = previewCollection(blueprint.id);
|
const preview = previewCollection(blueprint.id);
|
||||||
const current = scaling.getScalingPolicy(saved.id)!;
|
const current = scaling.getScalingPolicy(saved.id)!;
|
||||||
scaling.saveScaling(settings, saved.id, current.revision, scaling.previewScaling(settings, saved.id, current.revision).token);
|
scaling.saveScaling(settings, saved.id, current.revision);
|
||||||
expect(() => queueCollection({ previewId: preview.id })).toThrow('Autoscaling');
|
expect(() => queueCollection({ previewId: preview.id })).toThrow('Autoscaling');
|
||||||
expect(db().prepare('SELECT COUNT(*) AS n FROM server_deployments').get()).toEqual({ n: 0 });
|
expect(db().prepare('SELECT COUNT(*) AS n FROM server_deployments').get()).toEqual({ n: 0 });
|
||||||
});
|
});
|
||||||
@@ -77,13 +77,16 @@ it('operator cancellation pauses the policy and does not immediately create a re
|
|||||||
|
|
||||||
it('exact matching cannot remove an owned map absent from the collection', async () => {
|
it('exact matching cannot remove an owned map absent from the collection', async () => {
|
||||||
const town = 't6_zombie_town_1';
|
const town = 't6_zombie_town_1';
|
||||||
await prepareNativeHost([town]);
|
const { host } = await prepareNativeHost([town]);
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { autoscalingRemovalCapability } = await import('../../src/lib/server/autoscaling/authorization.js');
|
||||||
|
db().prepare("UPDATE ec_hosts SET inventory=json_set(inventory,'$.capabilities',json(?)) WHERE id=?")
|
||||||
|
.run(JSON.stringify([autoscalingRemovalCapability]), host.id);
|
||||||
const at = new Date().toISOString();
|
const at = new Date().toISOString();
|
||||||
db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,container_checked_at,last_seen_at,last_players) VALUES(?,'t6','zm_transit','town','running',?,?,0)").run(town, at, at);
|
db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,container_checked_at,last_seen_at,last_players) VALUES(?,'t6','zm_transit','town','running',?,?,0)").run(town, at, at);
|
||||||
for (const n of [1, 2]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `key-${n}`);
|
for (const n of [1, 2]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `key-${n}`);
|
||||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token);
|
scaling.saveScaling(settings);
|
||||||
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
||||||
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
|
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||||
const collection = saveBlueprint({ name: 'Moon only', entries: [{ game: 't5', mapSlug: 'moon', count: 1 }] });
|
const collection = saveBlueprint({ name: 'Moon only', entries: [{ game: 't5', mapSlug: 'moon', count: 1 }] });
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import { emptyObservation, planScaling } from '../../src/lib/server/autoscaling/
|
|||||||
import { mapIdentity } from '../../src/lib/server/autoscaling/snapshot.js';
|
import { mapIdentity } from '../../src/lib/server/autoscaling/snapshot.js';
|
||||||
|
|
||||||
const now = 2_000_000;
|
const now = 2_000_000;
|
||||||
const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'dry-run' }), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'create-destroy' }), id: 'policy', revision: 1, pausedReason: null, updatedAt: '' };
|
||||||
const name = (n: number) => `t6_zombie_town_${n}`;
|
const name = (n: number) => `t6_zombie_town_${n}`;
|
||||||
const member = (n: number, state: ScalingMember['state'] = 'empty'): ScalingMember => ({ name: name(n), identity: `instance:${n}`, state, players: state === 'occupied' ? 1 : state === 'empty' ? 0 : null, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null });
|
const member = (n: number, state: ScalingMember['state'] = 'empty'): ScalingMember => ({ name: name(n), identity: `instance:${n}`, state, players: state === 'occupied' ? 1 : state === 'empty' ? 0 : null, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null });
|
||||||
const snapshot = (...members: ScalingMember[]): ScalingSnapshot => ({ members, reservedNames: members.map(m => m.name), keyPool: { total: 4, available: 4 - members.length }, blockers: [] });
|
const snapshot = (...members: ScalingMember[]): ScalingSnapshot => ({ members, reservedNames: members.map(m => m.name), keyPool: { total: 4, available: 4 - members.length }, blockers: [] });
|
||||||
@@ -31,9 +31,8 @@ describe('autoscaling policy planning', () => {
|
|||||||
for (const state of ['creating', 'removing', 'unknown', 'stopped', 'protected'] as const) expect(planScaling(policy, snapshot(member(1, state)), now).action).toBe('wait');
|
for (const state of ['creating', 'removing', 'unknown', 'stopped', 'protected'] as const) expect(planScaling(policy, snapshot(member(1, state)), now).action).toBe('wait');
|
||||||
expect(planScaling(policy, { ...snapshot(), blockers: ['Unresolved instance'] }, now).action).toBe('wait');
|
expect(planScaling(policy, { ...snapshot(), blockers: ['Unresolved instance'] }, now).action).toBe('wait');
|
||||||
});
|
});
|
||||||
it('holds for stale and future samples, cooldown, and disabled policies', () => {
|
it('holds for stale and future samples and disabled policies', () => {
|
||||||
for (const stamp of [null, now - 90_001, now + 1]) expect(planScaling(policy, snapshot({ ...member(1, 'occupied'), observedAt: stamp }), now).action).toBe('wait');
|
for (const stamp of [null, now - 90_001, now + 1]) expect(planScaling(policy, snapshot({ ...member(1, 'occupied'), observedAt: stamp }), now).action).toBe('wait');
|
||||||
expect(planScaling({ ...policy, nextActionAt: now + 1 }, snapshot(), now).action).toBe('wait');
|
|
||||||
expect(planScaling({ ...policy, mode: 'off' }, snapshot(), now).action).toBe('wait');
|
expect(planScaling({ ...policy, mode: 'off' }, snapshot(), now).action).toBe('wait');
|
||||||
});
|
});
|
||||||
it('never sacrifices occupied lobbies or minimum/spare capacity to a maximum', () => {
|
it('never sacrifices occupied lobbies or minimum/spare capacity to a maximum', () => {
|
||||||
|
|||||||
@@ -0,0 +1,158 @@
|
|||||||
|
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
|
||||||
|
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||||
|
import { prepareNativeHost } from '../helpers/native-host.js';
|
||||||
|
import { routePermission, roleAllows } from '../../src/lib/server/auth/access.js';
|
||||||
|
|
||||||
|
let fixture: Fixture;
|
||||||
|
beforeEach(async () => {
|
||||||
|
vi.resetModules(); fixture = await newFixture([]);
|
||||||
|
await prepareNativeHost();
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
for (const n of [3, 8, 9, 10]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `secret-${n}`);
|
||||||
|
});
|
||||||
|
afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); });
|
||||||
|
const settings = { game: 't6', mapSlug: 'town', maximum: 4, mode: 'create-destroy' };
|
||||||
|
|
||||||
|
it('persists deduplicated paused decisions across restart without provisioning or reservations', async () => {
|
||||||
|
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
|
const saved = scaling.saveScaling(settings);
|
||||||
|
scaling.setScalingGlobalPause(true);
|
||||||
|
await scaling.runAutoscaling(); await scaling.runAutoscaling();
|
||||||
|
closeDb();
|
||||||
|
await scaling.runAutoscaling();
|
||||||
|
const view = scaling.scalingReport().policies[0];
|
||||||
|
expect(view.policy.id).toBe(saved.id);
|
||||||
|
expect(view.history).toHaveLength(1);
|
||||||
|
expect(view.decision).toMatchObject({ action: 'wait', server: null });
|
||||||
|
expect(JSON.stringify(view)).not.toContain('secret');
|
||||||
|
for (const table of ['server_deployments', 'native_creation_intents', 'native_instances']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table}`).get()).toEqual({ n: 0 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('saves edits directly while rejecting duplicate policies and stale revisions', async () => {
|
||||||
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
|
const saved = scaling.saveScaling(settings);
|
||||||
|
expect(() => scaling.saveScaling(settings)).toThrow('overlapping policy');
|
||||||
|
expect(() => scaling.saveScaling(settings, saved.id, 0)).toThrow('changed');
|
||||||
|
const edited = scaling.saveScaling({ ...settings, maximum: 5 }, saved.id, saved.revision);
|
||||||
|
expect(edited).toMatchObject({ maximum: 5, revision: saved.revision + 1 });
|
||||||
|
expect(() => scaling.saveScaling(settings, saved.id, saved.revision)).toThrow('changed');
|
||||||
|
expect(scaling.getScalingPolicy(saved.id)).toEqual(edited);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('checks current fleet blockers during save and leaves rejected edits unchanged', async () => {
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
|
const saved = scaling.saveScaling(settings);
|
||||||
|
db().prepare("INSERT INTO servers(name,game,map_slug) VALUES('unknown-map','t6','unknown')").run();
|
||||||
|
expect(() => scaling.saveScaling({ ...settings, maximum: 5 }, saved.id, saved.revision)).toThrow('ambiguous map identity');
|
||||||
|
expect(scaling.getScalingPolicy(saved.id)).toEqual(saved);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('creates and updates policies through the API without a preview or token', async () => {
|
||||||
|
const { POST } = await import('../../src/routes/api/autoscaling/+server.js');
|
||||||
|
const post = (body: object) => POST({ request: new Request('http://localhost/api/autoscaling', {
|
||||||
|
method: 'POST', body: JSON.stringify({ action: 'save', ...body })
|
||||||
|
}) });
|
||||||
|
const response = await post({ settings });
|
||||||
|
expect(response.status).toBe(200);
|
||||||
|
const { policy } = await response.json();
|
||||||
|
const update = { settings: { ...settings, maximum: 3 }, id: policy.id, revision: policy.revision };
|
||||||
|
const changed = await post(update);
|
||||||
|
expect(changed.status).toBe(200);
|
||||||
|
expect((await changed.json()).policy).toMatchObject({ maximum: 3, revision: policy.revision + 1 });
|
||||||
|
expect((await post(update)).status).toBe(409);
|
||||||
|
expect((await post({ action: 'preview', settings })).status).toBe(400);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('pauses evaluations and prunes expired observations without deleting current history', async () => {
|
||||||
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
db().prepare('DELETE FROM server_keys').run();
|
||||||
|
scaling.saveScaling(settings);
|
||||||
|
await scaling.runAutoscaling();
|
||||||
|
scaling.setScalingGlobalPause(true); await scaling.runAutoscaling();
|
||||||
|
expect(scaling.scalingReport().policies[0].decision.reason).toContain('globally paused');
|
||||||
|
const id = scaling.scalingReport().policies[0].history[0].id;
|
||||||
|
db().prepare('UPDATE autoscaling_decisions SET last_seen_at=0 WHERE id!=?').run(id);
|
||||||
|
scaling.pruneAutoscaling();
|
||||||
|
expect(scaling.scalingReport().policies[0].history).toHaveLength(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('uses fleet authorization for page, reads, and mutations', () => {
|
||||||
|
expect(routePermission('/autoscaling', 'GET')).toBe('viewer');
|
||||||
|
expect(routePermission('/api/autoscaling', 'GET')).toBe('viewer');
|
||||||
|
expect(roleAllows('viewer', routePermission('/api/autoscaling', 'POST'))).toBe(false);
|
||||||
|
expect(roleAllows('operator', routePermission('/api/autoscaling', 'POST'))).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each(['dry-run', 'create-only'])('rejects removed %s mode through the API', async mode => {
|
||||||
|
const { POST } = await import('../../src/routes/api/autoscaling/+server.js');
|
||||||
|
for (const action of ['save']) {
|
||||||
|
const response = await POST({ request: new Request('http://localhost/api/autoscaling', {
|
||||||
|
method: 'POST', body: JSON.stringify({ action, settings: { ...settings, mode } })
|
||||||
|
}) });
|
||||||
|
expect(response.status).toBe(400);
|
||||||
|
expect(await response.json()).toMatchObject({ error: 'Choose a valid autoscaling mode.' });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each(['dry-run', 'create-only'])('turns saved %s policies off once on upgrade', async mode => {
|
||||||
|
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
|
const saved = scaling.saveScaling(settings);
|
||||||
|
const otherSettings = { ...settings, mapSlug: 'farm' };
|
||||||
|
const other = scaling.saveScaling(otherSettings);
|
||||||
|
db().prepare('UPDATE autoscaling_policies SET mode=?,settings=json_set(settings,\'$.mode\',?) WHERE id=?').run(mode, mode, saved.id);
|
||||||
|
closeDb();
|
||||||
|
const migrated = scaling.getScalingPolicy(saved.id)!;
|
||||||
|
expect(migrated).toMatchObject({ mode: 'off', revision: saved.revision + 1, maximum: saved.maximum,
|
||||||
|
pausedReason: expect.stringContaining('mode was removed') });
|
||||||
|
expect(db().prepare('SELECT owns_map,json_extract(settings,\'$.mode\') AS mode FROM autoscaling_policies WHERE id=?').get(saved.id))
|
||||||
|
.toEqual({ owns_map: 0, mode: 'off' });
|
||||||
|
expect(scaling.getScalingPolicy(other.id)).toEqual(other);
|
||||||
|
closeDb();
|
||||||
|
expect(scaling.getScalingPolicy(saved.id)).toEqual(migrated);
|
||||||
|
const resumed = scaling.saveScaling(settings, migrated.id, migrated.revision);
|
||||||
|
expect(resumed).toMatchObject({ mode: 'create-destroy', pausedReason: null });
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each([false, true])('reconciles removed-mode creation while preserving submitted work: %s', async submitted => {
|
||||||
|
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
|
const saved = scaling.saveScaling(settings);
|
||||||
|
await scaling.runAutoscaling();
|
||||||
|
const { operation_id: id } = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string };
|
||||||
|
db().prepare("UPDATE autoscaling_policies SET mode='create-only',settings=json_set(settings,'$.mode','create-only') WHERE id=?").run(saved.id);
|
||||||
|
if (submitted) db().prepare('UPDATE server_deployments SET submitted_at=? WHERE id=?').run(new Date().toISOString(), id);
|
||||||
|
closeDb();
|
||||||
|
expect(scaling.getScalingPolicy(saved.id)?.mode).toBe('off');
|
||||||
|
expect(db().prepare('SELECT owns_map FROM autoscaling_policies WHERE id=?').get(saved.id)).toEqual({ owns_map: 1 });
|
||||||
|
expect(await scaling.runAutoscaling()).toBe(false);
|
||||||
|
expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(id)).toEqual({ state: submitted ? 'queued' : 'cancelled' });
|
||||||
|
expect(db().prepare('SELECT owns_map FROM autoscaling_policies WHERE id=?').get(saved.id)).toEqual({ owns_map: submitted ? 1 : 0 });
|
||||||
|
expect(db().prepare('SELECT COUNT(*) AS n FROM server_key_leases').get()).toEqual({ n: submitted ? 1 : 0 });
|
||||||
|
expect(db().prepare("SELECT COUNT(*) AS n FROM server_deployments WHERE action='create'").get()).toEqual({ n: 1 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('removes saved cooldowns on upgrade without changing the empty hold or pausing the policy', async () => {
|
||||||
|
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
|
const input = { ...settings, emptyMinutes: 17 };
|
||||||
|
const saved = scaling.saveScaling(input);
|
||||||
|
db().exec('ALTER TABLE autoscaling_policies ADD COLUMN next_action_at INTEGER NOT NULL DEFAULT 0');
|
||||||
|
db().prepare("UPDATE autoscaling_policies SET next_action_at=?,settings=json_set(settings,'$.cooldownMinutes',60) WHERE id=?")
|
||||||
|
.run(Date.now() + 3600_000, saved.id);
|
||||||
|
closeDb();
|
||||||
|
const restored = scaling.getScalingPolicy(saved.id)!;
|
||||||
|
expect(restored).toEqual(saved);
|
||||||
|
expect(restored).not.toHaveProperty('cooldownMinutes');
|
||||||
|
expect(restored).not.toHaveProperty('nextActionAt');
|
||||||
|
expect(db().pragma('table_info(autoscaling_policies)')).not.toContainEqual(expect.objectContaining({ name: 'next_action_at' }));
|
||||||
|
expect(JSON.parse((db().prepare('SELECT settings FROM autoscaling_policies WHERE id=?').get(saved.id) as { settings: string }).settings))
|
||||||
|
.not.toHaveProperty('cooldownMinutes');
|
||||||
|
closeDb();
|
||||||
|
expect(scaling.getScalingPolicy(saved.id)).toEqual(saved);
|
||||||
|
expect(await scaling.runAutoscaling()).toBe(true);
|
||||||
|
expect(scaling.scalingReport().policies[0].policy.emptyMinutes).toBe(17);
|
||||||
|
});
|
||||||
@@ -8,7 +8,7 @@ let fixture: Fixture;
|
|||||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||||
afterEach(async () => { vi.useRealTimers(); await fixture.cleanup(); vi.restoreAllMocks(); });
|
afterEach(async () => { vi.useRealTimers(); await fixture.cleanup(); vi.restoreAllMocks(); });
|
||||||
const names = ['t6_zombie_town_1', 't6_zombie_town_2'];
|
const names = ['t6_zombie_town_1', 't6_zombie_town_2'];
|
||||||
const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'create-destroy', emptyMinutes: 1, cooldownMinutes: 1 };
|
const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'create-destroy', emptyMinutes: 1 };
|
||||||
|
|
||||||
async function setup(gameWide = false) {
|
async function setup(gameWide = false) {
|
||||||
const servers = gameWide ? [...names, ...zombieMaps('t6').filter(m => m.slug !== 'town').map(m => `t6_zombie_${m.slug}_1`)] : names;
|
const servers = gameWide ? [...names, ...zombieMaps('t6').filter(m => m.slug !== 'town').map(m => `t6_zombie_${m.slug}_1`)] : names;
|
||||||
@@ -33,7 +33,7 @@ async function setup(gameWide = false) {
|
|||||||
db().prepare('UPDATE ec_hosts SET last_heartbeat=?').run(at);
|
db().prepare('UPDATE ec_hosts SET last_heartbeat=?').run(at);
|
||||||
};
|
};
|
||||||
fresh(0);
|
fresh(0);
|
||||||
const saved = scaling.saveScaling(policySettings, undefined, undefined, scaling.previewScaling(policySettings).token);
|
const saved = scaling.saveScaling(policySettings);
|
||||||
for (const offset of [0, 30_000, 60_000]) { fresh(offset); await scaling.runAutoscaling(); }
|
for (const offset of [0, 30_000, 60_000]) { fresh(offset); await scaling.runAutoscaling(); }
|
||||||
const intent = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string };
|
const intent = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string };
|
||||||
expect(intent?.operation_id).toBeTruthy();
|
expect(intent?.operation_id).toBeTruthy();
|
||||||
@@ -84,7 +84,7 @@ it('revokes a delivered removal on policy pause without claiming it was undone',
|
|||||||
const s = await setup(); const command = await s.dispatch();
|
const s = await setup(); const command = await s.dispatch();
|
||||||
s.db().prepare('UPDATE ec_host_operations SET last_delivered_at=? WHERE id=?').run(new Date().toISOString(), command.id);
|
s.db().prepare('UPDATE ec_host_operations SET last_delivered_at=? WHERE id=?').run(new Date().toISOString(), command.id);
|
||||||
const off = { ...settings, mode: 'off' };
|
const off = { ...settings, mode: 'off' };
|
||||||
s.scaling.saveScaling(off, s.policy.id, s.policy.revision, s.scaling.previewScaling(off, s.policy.id, s.policy.revision).token);
|
s.scaling.saveScaling(off, s.policy.id, s.policy.revision);
|
||||||
expect(s.authorization.claimRemovalAuthorization(s.hostId, command.id, command.payload).authorized).toBe(false);
|
expect(s.authorization.claimRemovalAuthorization(s.hostId, command.id, command.payload).authorized).toBe(false);
|
||||||
expect(s.db().prepare('SELECT owns_map FROM autoscaling_policies').get()).toEqual({ owns_map: 1 });
|
expect(s.db().prepare('SELECT owns_map FROM autoscaling_policies').get()).toEqual({ owns_map: 1 });
|
||||||
expect(s.db().prepare('SELECT state FROM server_deployments WHERE id=?').get(s.operationId)).toEqual({ state: 'running' });
|
expect(s.db().prepare('SELECT state FROM server_deployments WHERE id=?').get(s.operationId)).toEqual({ state: 'running' });
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ let fixture: Fixture;
|
|||||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||||
afterEach(async () => { await fixture.cleanup(); });
|
afterEach(async () => { await fixture.cleanup(); });
|
||||||
const town = 't6_zombie_town_1', farm = 't6_zombie_farm_1';
|
const town = 't6_zombie_town_1', farm = 't6_zombie_farm_1';
|
||||||
const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'dry-run' }), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'create-destroy' }), id: 'policy', revision: 1, pausedReason: null, updatedAt: '' };
|
||||||
|
|
||||||
it('separates variants, deduplicates pending identities, and initializes additive schema', async () => {
|
it('separates variants, deduplicates pending identities, and initializes additive schema', async () => {
|
||||||
await prepareNativeHost([town, farm]);
|
await prepareNativeHost([town, farm]);
|
||||||
|
|||||||
Reference in New Issue
Block a user