From f8d0ba393607eea381904af44da21c69d6363b6e Mon Sep 17 00:00:00 2001 From: Josh Wright Date: Mon, 5 Oct 2026 21:02:58 -0400 Subject: [PATCH] Allow revising active betas with fresh tests and history --- docs/beta-test-plans.md | 38 ++- src/lib/beta-revision.ts | 11 + src/lib/server/db/index.ts | 1 + src/lib/server/db/schema.sql | 27 +++ src/lib/server/fleet/beta-revisions.ts | 29 +++ src/lib/server/fleet/beta-tests.ts | 7 +- src/lib/server/fleet/beta.ts | 128 +++++++++- src/lib/server/fleet/staged-files.ts | 25 +- src/lib/server/pipeline/beta-plan-runner.ts | 6 +- src/lib/server/provisioning/native.ts | 11 +- .../ui/components/BetaRevisionDialog.svelte | 100 ++++++++ src/lib/ui/components/BetaTestProgress.svelte | 2 +- src/routes/api/betas/+server.ts | 3 +- src/routes/api/betas/[id]/+server.ts | 4 +- .../api/betas/[id]/revisions/+server.ts | 28 +++ .../[revision]/files/[scriptId]/+server.ts | 16 ++ src/routes/api/betas/[id]/test/+server.ts | 5 +- src/routes/betas/[id]/+page.svelte | 32 ++- tests/unit/beta-plans.test.ts | 24 ++ tests/unit/beta-revisions.test.ts | 222 ++++++++++++++++++ tests/unit/native-provisioning.test.ts | 35 +++ 21 files changed, 713 insertions(+), 41 deletions(-) create mode 100644 src/lib/beta-revision.ts create mode 100644 src/lib/server/fleet/beta-revisions.ts create mode 100644 src/lib/ui/components/BetaRevisionDialog.svelte create mode 100644 src/routes/api/betas/[id]/revisions/+server.ts create mode 100644 src/routes/api/betas/[id]/revisions/[revision]/files/[scriptId]/+server.ts create mode 100644 tests/unit/beta-revisions.test.ts diff --git a/docs/beta-test-plans.md b/docs/beta-test-plans.md index 5d3cb8d..cb2cd91 100644 --- a/docs/beta-test-plans.md +++ b/docs/beta-test-plans.md @@ -197,8 +197,8 @@ Set wave size to 1 for a canary, a larger number for batches, or 0 for all selected servers. Later waves receive files after the previous waves pass. Health-outage budget violations on any tested server block promotion. -The test window starts when the beta opens, including time waiting for empty -servers. Missing evidence at the deadline produces **inconclusive**. +The test window starts when the beta opens or a revision is submitted, including +time waiting for empty servers. Missing evidence at the deadline produces **inconclusive**. Health monitoring continues after a pass until promotion. For failures, select hold as failed, pause for investigation, or automatic @@ -208,7 +208,7 @@ The manual option holds those changes in a rollback campaign until you select **Resume when empty**. Promotion is always manual. Pause holds evaluation and wave advancement. Resume retains completed progress and the -original deadline, but starts new timing intervals. A fresh attempt resets +current revision's deadline, but starts new timing intervals. A fresh attempt resets progress and restarts deployed servers when empty. Recovery cycles repeat gameplay milestones and checklist sign-offs while carrying health outages forward. @@ -217,6 +217,32 @@ recent 200 evidence entries. On upgrading from the old reporter protocol, active progress resets. Unsupported checks in existing plans and saved presets become manual review items; they cannot silently pass. +## Updating an active beta + +Choose **Update beta** on the beta detail page, upload replacements for existing +bundle files, and review the changes and selected servers before choosing +**Deploy revision**. Text files show a diff when both files are small enough; +binary and large files show their checksums. Files must actually change. + +The beta keeps its ID, servers, scope and test policy. Each revision creates new +file versions, clears current test progress and checklist sign-offs, repeats the +configured waves, and starts a fresh testing window at submission. Files install +and servers restart through the normal empty-server and capacity gates. Promotion +requires the new revision to pass. Pause and retry retain the current revision's +deadline; they do not create a revision. + +Revision history retains downloadable bundle files and previous test results, +including the last 200 evidence entries shown at replacement. All underlying +attempts and evidence remain in SQLite, separated by revision. Rejecting the beta +restores the original pre-beta files, even after multiple revisions or while a +new revision is waiting to deploy. Files originally absent are removed. + +Updates are available until promotion or rollback, including paused, failed and +expired tests. An activation already in progress must settle first; interrupted +file writes and deleted participants must be resolved before an update. Pending, +unsubmitted restarts are superseded. Adding/removing files, changing targets or +changing the policy still requires a new beta. + ## API and storage `POST /api/betas` accepts `testPlan`. @@ -228,6 +254,12 @@ recent evidence under `test`. `GET/POST /api/betas/presets` lists/saves reusable plans. Saved presets are copied into a beta; later edits do not change an active plan. +`POST /api/betas/:id/revisions` accepts multipart `expectedRevision`, optional +`notes`, and replacement uploads named `file:`. A stale revision is +rejected. `GET /api/betas/:id` includes the current `revision` and `revisions` +history; `GET /api/betas/:id/revisions/:revision/files/:scriptId` downloads an +archived file (or the original pinned file before its first update). + Internal progress records support attempts and recovery cycles. The reporter does not know about or supply these records. Legacy betas without a test plan also use `game_end` at round 2 or later for their match-completion gate. diff --git a/src/lib/beta-revision.ts b/src/lib/beta-revision.ts new file mode 100644 index 0000000..ac96fa3 --- /dev/null +++ b/src/lib/beta-revision.ts @@ -0,0 +1,11 @@ +import type { BetaTestView } from './beta-plan.js'; + +export interface BetaRevisionFile { + script_id: number; name: string; kind: string; path: string; + version_id: number; version: number; release_version: string | null; sha256: string; +} +export interface BetaRevision { + revision: number; started_at: string; notes: string | null; + files: BetaRevisionFile[]; + results: { state: string; test: BetaTestView | null; servers: Array<{ server: string; stage: string; error: string | null }> } | null; +} diff --git a/src/lib/server/db/index.ts b/src/lib/server/db/index.ts index b0e3f90..7090a0a 100644 --- a/src/lib/server/db/index.ts +++ b/src/lib/server/db/index.ts @@ -347,6 +347,7 @@ export function db(): Database.Database { migrateTable(handle, 'scripts', SCRIPTS_COLUMNS); migrateTable(handle, 'script_versions', SCRIPT_VERSIONS_COLUMNS); migrateTable(handle, 'beta_scripts', BETA_SCRIPTS_COLUMNS); + migrateTable(handle, 'beta_test_runs', [{ name: 'revision', ddl: 'ALTER TABLE beta_test_runs ADD COLUMN revision INTEGER NOT NULL DEFAULT 1' }]); migrateTable(handle, 'deployment_scripts', DEPLOYMENT_SCRIPTS_COLUMNS); migrateTable(handle, 'deployments', DEPLOYMENTS_COLUMNS); migrateTable(handle, 'server_launch_collections', [ diff --git a/src/lib/server/db/schema.sql b/src/lib/server/db/schema.sql index c203068..e59c301 100644 --- a/src/lib/server/db/schema.sql +++ b/src/lib/server/db/schema.sql @@ -180,6 +180,32 @@ CREATE TABLE IF NOT EXISTS betas ( CREATE INDEX IF NOT EXISTS idx_betas_state ON betas(state); CREATE INDEX IF NOT EXISTS idx_betas_script ON betas(script_id); +-- Revisions keep the beta identity and original rollback images. Archived +-- files are independent copies so version cleanup cannot erase this history. +CREATE TABLE IF NOT EXISTS beta_revisions ( + beta_id INTEGER NOT NULL REFERENCES betas(id) ON DELETE CASCADE, + revision INTEGER NOT NULL, + started_at TEXT NOT NULL, + notes TEXT, + results_json TEXT, + PRIMARY KEY (beta_id, revision) +); +CREATE TABLE IF NOT EXISTS beta_revision_files ( + beta_id INTEGER NOT NULL, + revision INTEGER NOT NULL, + script_id INTEGER NOT NULL, + name TEXT NOT NULL, + kind TEXT NOT NULL, + path TEXT NOT NULL, + version_id INTEGER NOT NULL, + version INTEGER NOT NULL, + release_version TEXT, + sha256 TEXT NOT NULL, + content BLOB NOT NULL, + PRIMARY KEY (beta_id, revision, script_id), + FOREIGN KEY (beta_id, revision) REFERENCES beta_revisions(beta_id, revision) ON DELETE CASCADE +); + CREATE TABLE IF NOT EXISTS beta_servers ( beta_id INTEGER NOT NULL REFERENCES betas(id) ON DELETE CASCADE, server TEXT NOT NULL, @@ -208,6 +234,7 @@ CREATE TABLE IF NOT EXISTS beta_test_plans ( CREATE TABLE IF NOT EXISTS beta_test_runs ( id TEXT PRIMARY KEY, beta_id INTEGER NOT NULL REFERENCES betas(id) ON DELETE CASCADE, + revision INTEGER NOT NULL DEFAULT 1, server TEXT NOT NULL, phase INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL, diff --git a/src/lib/server/fleet/beta-revisions.ts b/src/lib/server/fleet/beta-revisions.ts new file mode 100644 index 0000000..90bba00 --- /dev/null +++ b/src/lib/server/fleet/beta-revisions.ts @@ -0,0 +1,29 @@ +import { db } from '../db/index.js'; +import type { BetaRevision } from '../../beta-revision.js'; + +export function currentBetaRevision(id: number): { revision: number; started_at: string } { + return db().prepare(`SELECT revision, started_at FROM beta_revisions WHERE beta_id = ? ORDER BY revision DESC LIMIT 1`).get(id) as + { revision: number; started_at: string } | undefined + ?? { revision: 1, started_at: (db().prepare('SELECT opened_at FROM betas WHERE id = ?').get(id) as { opened_at: string }).opened_at }; +} + +/** Called in the revision transaction, before replacing the pinned bundle. */ +export function archiveBetaRevision(id: number, revision: number, startedAt: string, notes: string | null): void { + db().prepare('INSERT OR IGNORE INTO beta_revisions (beta_id, revision, started_at, notes) VALUES (?, ?, ?, ?)') + .run(id, revision, startedAt, notes); + db().prepare(`INSERT OR IGNORE INTO beta_revision_files + (beta_id, revision, script_id, name, kind, path, version_id, version, release_version, sha256, content) + SELECT bs.beta_id, ?, s.id, s.name, s.kind, s.target_rel_path, v.id, v.version, v.release_version, v.sha256, v.content + FROM beta_scripts bs JOIN scripts s ON s.id = bs.script_id JOIN script_versions v ON v.id = bs.script_version_id + WHERE bs.beta_id = ?`).run(revision, id); +} + +export function betaRevisionHistory(id: number): BetaRevision[] { + const rows = db().prepare('SELECT revision, started_at, notes, results_json FROM beta_revisions WHERE beta_id = ? ORDER BY revision DESC') + .all(id) as Array<{ revision: number; started_at: string; notes: string | null; results_json: string | null }>; + return rows.map(({ results_json, ...row }) => ({ ...row, + results: results_json ? JSON.parse(results_json) : null, + files: db().prepare(`SELECT script_id, name, kind, path, version_id, version, release_version, sha256 + FROM beta_revision_files WHERE beta_id = ? AND revision = ? ORDER BY script_id`).all(id, row.revision) as BetaRevision['files'] + })); +} diff --git a/src/lib/server/fleet/beta-tests.ts b/src/lib/server/fleet/beta-tests.ts index 6fb8939..886a4cf 100644 --- a/src/lib/server/fleet/beta-tests.ts +++ b/src/lib/server/fleet/beta-tests.ts @@ -6,6 +6,7 @@ import { parseBetaPlan, type BetaPlan, type BetaTestView } from '../../beta-plan import type { GameTelemetry } from '../../telemetry.js'; import { applySample, applyGameEvent, emptyMetrics, evaluateRequirements, type TestMetrics } from './beta-evaluation.js'; import { ONLINE_MAX_MS } from '../../server-freshness.js'; +import { currentBetaRevision } from './beta-revisions.js'; export interface TestRun { id: string; beta_id: number; server: string; phase: number; created_at: string; observing_at: string | null; metrics: string } export interface StoredPlan { beta_id: number; plan: string; outcome: string; reason: string | null } @@ -33,8 +34,8 @@ export async function prepareRun(betaId: number, server: string, phase = 0): Pro if (!activeParticipant(betaId, server)) throw new Error('Original test server was deleted; start a new beta'); db().transaction(() => { db().prepare('UPDATE beta_test_runs SET retired_at = ? WHERE beta_id = ? AND server = ? AND retired_at IS NULL').run(nowIso(), betaId, server); - db().prepare('INSERT INTO beta_test_runs (id, beta_id, server, phase, created_at, metrics) VALUES (?, ?, ?, ?, ?, ?)') - .run(randomUUID(), betaId, server, phase, nowIso(), JSON.stringify(emptyMetrics())); + db().prepare('INSERT INTO beta_test_runs (id, beta_id, server, phase, created_at, metrics, revision) VALUES (?, ?, ?, ?, ?, ?, ?)') + .run(randomUUID(), betaId, server, phase, nowIso(), JSON.stringify(emptyMetrics()), currentBetaRevision(betaId).revision); })(); return currentRun(betaId, server)!; } @@ -97,5 +98,5 @@ export function testView(betaId: number): BetaTestView | null { }), evidence: db().prepare(`SELECT e.id, r.server, s.server_deleted_at, e.received_at, e.kind, e.payload FROM beta_test_evidence e JOIN beta_test_runs r ON r.id = e.run_id JOIN beta_servers s ON s.beta_id = r.beta_id AND s.server = r.server - WHERE r.beta_id = ? ORDER BY e.id DESC LIMIT 200`).all(betaId) as BetaTestView['evidence'] }; + WHERE r.beta_id = ? AND r.revision = ? ORDER BY e.id DESC LIMIT 200`).all(betaId, currentBetaRevision(betaId).revision) as BetaTestView['evidence'] }; } diff --git a/src/lib/server/fleet/beta.ts b/src/lib/server/fleet/beta.ts index 7a739c0..506e0b0 100644 --- a/src/lib/server/fleet/beta.ts +++ b/src/lib/server/fleet/beta.ts @@ -1,5 +1,6 @@ import { parseBetaPlan, type BetaPlan } from '../../beta-plan.js'; -import { storedPlan } from './beta-tests.js'; +import { storedPlan, testView, activity, setOutcome } from './beta-tests.js'; +import { archiveBetaRevision, currentBetaRevision } from './beta-revisions.js'; import type { BetaPromotionPreview } from '../../beta-promotion.js'; const operations = new Set(); @@ -28,17 +29,18 @@ import { listVersions, serversMatchingScope, setScope, + uploadNewVersion, type ScopeType, type ScriptRow, type ScriptVersionRow } from './scripts.js'; import { composeServerCfg } from './config-compose.js'; import { getServerScript, removeScript } from './inventory.js'; -import { cancelStagedFiles, isFileApplied, queueFileChange } from './staged-files.js'; +import { cancelStagedFiles, isFileApplied, queueFileChange, stagedFileChanges } from './staged-files.js'; import { emit } from '../pipeline/bus.js'; import { cancelProvisioningJob } from '../provisioning/cancellation.js'; import { reconcileNativeProvisioningReceipt } from '../provisioning/native.js'; -import { requestRestart } from '../provisioning/requests.js'; +import { requestRestart, releaseRestartRequest } from '../provisioning/requests.js'; import type { BetaStage } from '../pipeline/bus.js'; // Whether an artifact writes a dedicated on-disk file (script/plugin) or @@ -497,6 +499,124 @@ export async function openBeta(input: OpenBetaInput): Promise { }; } +export interface BetaRevisionUpload { scriptId: number; content: Buffer; releaseVersion?: string } + +/** Replace existing bundle members; commit files, deadline and evidence reset together. */ +export async function reviseBeta(betaId: number, expectedRevision: number, uploads: BetaRevisionUpload[], notes = ''): Promise<{ revision: number }> { + // Drain a runner already observing the previous revision before resetting it. + const { betaTick } = await import('../pipeline/beta-runner.js'); + await betaTick(); + return exclusive(betaId, async () => { + const beta = getBeta(betaId); + if (!beta) throw new BetaError('Beta not found', 'no-such-beta'); + if (!['running', 'ready-promote'].includes(beta.state)) throw new BetaError('Beta is closed', 'invalid-state'); + const current = currentBetaRevision(betaId); + if (current.revision !== expectedRevision) throw new BetaError('The beta changed. Reload and review the current revision.', 'preview-changed'); + if (!uploads.length || new Set(uploads.map(file => file.scriptId)).size !== uploads.length) { + throw new BetaError('Choose at least one replacement, once per file.', 'invalid-state'); + } + if (notes.length > 1000) throw new BetaError('Change notes must be at most 1000 characters.', 'invalid-state'); + const bundle = listBetaScripts(betaId); + for (const file of uploads) { + const member = bundle.find(row => row.script_id === file.scriptId); + if (!member) throw new BetaError('Updates can only replace files already in this beta.', 'invalid-state'); + if (!file.content.length) throw new BetaError('Choose non-empty replacement files.', 'invalid-state'); + if (getVersion(member.script_version_id)!.sha256 === sha256Buffer(file.content)) { + throw new BetaError(`${getScript(file.scriptId)!.name}: the replacement is identical to the current version.`, 'invalid-state'); + } + } + const servers = listBetaServers(betaId); + if (!servers.length || servers.some(server => !isActiveBetaServer(betaId, server.server))) { + throw new BetaError('An original beta server is missing. Roll back and open a new beta.', 'invalid-state'); + } + ensureHostSchema(db()); + assertNativeActivationSettled(betaId); + if (stagedFileChanges('beta', String(betaId)).some(row => row.state === 'applying')) { + throw new BetaError('A file write was interrupted. Resolve it before updating the beta.', 'conflict'); + } + const jobs = db().prepare(`SELECT DISTINCT d.id, d.state, d.submitted_at, n.operation_id FROM server_deployments d + JOIN provisioning_sources s ON s.operation_id = d.id LEFT JOIN native_provisioning n ON n.deployment_id = d.id + WHERE s.kind = 'beta' AND s.source_id = ?`).all(String(betaId)) as + Array<{ id: string; state: string; submitted_at: string | null; operation_id: string | null }>; + if (jobs.some(job => ['running', 'awaiting-health'].includes(job.state) + || (job.state === 'interrupted' && servers.some(server => server.provisioning_id === job.id)) + || (['waiting', 'queued'].includes(job.state) && (job.submitted_at || job.operation_id)))) { + throw new BetaError('Wait for the current activation and recovery to finish before updating this beta.', 'conflict'); + } + const stored = storedPlan(betaId); + const plan = stored ? parseBetaPlan(JSON.parse(stored.plan)) : null; + const revision = current.revision + 1; + const at = nowIso(); + db().transaction(() => { + archiveBetaRevision(betaId, current.revision, current.started_at, null); + db().prepare('UPDATE beta_revisions SET results_json = ? WHERE beta_id = ? AND revision = ?') + .run(JSON.stringify({ state: beta.state, test: testView(betaId), servers }), betaId, current.revision); + for (const upload of uploads) { + const version = uploadNewVersion(upload.scriptId, upload.content, notes.trim() || undefined, upload.releaseVersion); + db().prepare('UPDATE beta_scripts SET script_version_id = ? WHERE beta_id = ? AND script_id = ?') + .run(version.id, betaId, upload.scriptId); + db().prepare('UPDATE betas SET script_version_id = ? WHERE id = ? AND script_id = ?') + .run(version.id, betaId, upload.scriptId); + } + // Preflight all targets before cancelling work. Errors roll back the + // uploads and history as well as the beta changes. + const changes = new Map(servers.map(server => [server.server, revisionFileChanges(betaId, server.server)])); + // Betas created before staged activation may only have deployed_at + // and snapshots. Record their first writes before replacing staging. + for (const server of servers.filter(server => server.deployed_at)) { + for (const change of changes.get(server.server)!) { + if (isFileApplied('beta', String(betaId), server.server, change.kind, change.relPath) === null) { + db().prepare('INSERT OR IGNORE INTO beta_file_changes (beta_id, server, rel_path) VALUES (?, ?, ?)') + .run(betaId, server.server, change.relPath); + } + } + } + for (const job of jobs.filter(job => ['waiting', 'queued'].includes(job.state))) { + cancelProvisioningJob(job.id, `Superseded by beta #${betaId} revision ${revision}`); + } + for (const server of servers) releaseRestartRequest(server.server, { kind: 'beta', id: String(betaId) }); + // New IDs prevent historical host receipts from resolving to a newer + // revision's desired bytes. Original before images stay intact. + db().prepare("DELETE FROM staged_file_changes WHERE source_kind = 'beta' AND source_id = ?").run(String(betaId)); + db().prepare('UPDATE beta_test_runs SET retired_at = ? WHERE beta_id = ? AND retired_at IS NULL').run(at, betaId); + db().prepare(`UPDATE beta_servers SET stage = 'queued', error = NULL, deployed_at = NULL, + restarted_at = NULL, healthy_at = NULL, match_seen_at = NULL, updated_at = ? WHERE beta_id = ?`).run(at, betaId); + db().prepare("UPDATE betas SET state = 'running' WHERE id = ?").run(betaId); + setOutcome(betaId, 'active', null); + archiveBetaRevision(betaId, revision, at, notes.trim() || null); + // The original first wave follows selection order; later waves are + // ordered by the runner. Preserve that canary selection on revision. + const firstWave = db().prepare('SELECT server FROM beta_servers WHERE beta_id = ? ORDER BY rowid LIMIT ?') + .all(betaId, plan?.waveSize || servers.length) as { server: string }[]; + for (const server of firstWave) { + for (const change of changes.get(server.server)!) queueFileChange(change); + db().prepare("UPDATE beta_servers SET stage = 'awaiting-empty' WHERE beta_id = ? AND server = ?").run(betaId, server.server); + } + activity(betaId, `Revision ${revision} submitted; fresh tests and testing window${notes.trim() ? `: ${notes.trim()}` : ''}`); + })(); + emit({ type: 'beta-state', betaId, state: 'running' }); + return { revision }; + }); +} + +function revisionFileChanges(betaId: number, name: string): Parameters[0][] { + const resolved = listBetaScripts(betaId).map(row => ({ script: getScript(row.script_id)!, version: getVersion(row.script_version_id)! })); + const server = getServer(name)!; + const changes: Parameters[0][] = resolved.filter(row => !isCollapsingKind(row.script.kind)) + .map(({ script, version }) => ({ sourceKind: 'beta', sourceId: String(betaId), server: name, kind: script.kind, + relPath: script.target_rel_path, content: version.content, scriptVersionId: version.id, inventorySource: `beta:${betaId}` })); + if (resolved.some(row => isCollapsingKind(row.script.kind))) { + const composed = composeCfgForServer(name, server.game, server.map, { + inBetaBaseline: resolved.find(row => row.script.kind === 'baseline')?.version, + inBetaOverlays: resolved.filter(row => row.script.kind === 'config') + }); + if (!composed) throw new BetaError(`No baseline available for ${name}.`, 'invalid-state'); + changes.push({ sourceKind: 'beta', sourceId: String(betaId), server: name, kind: 'config', relPath: configFileOf(name), + content: Buffer.from(composed.text), inventorySource: `beta:${betaId}` }); + } + return changes; +} + /** Apply a reserved wave only when its predecessors have passed. */ export async function deployBetaServers(betaId: number, names: string[]): Promise { return exclusive(betaId, () => deployBetaServersInternal(betaId, names)); @@ -620,6 +740,8 @@ async function rollbackBetaInternal(betaId: number, options: { restart?: 'manual const restorations: Array<{ server: string; kind: ArtifactKind; relPath: string; content: Buffer | null; versionId: number | null }> = []; const changed = db().prepare('SELECT 1 FROM beta_file_changes WHERE beta_id = ? AND server = ? AND rel_path = ?'); const applied = (bs: BetaServerRow, kind: ArtifactKind, path: string) => { + // A queued revision may supersede an applied one before its own activation. + if (changed.get(betaId, bs.server, path)) return true; const staged = isFileApplied('beta', String(betaId), bs.server, kind, path); if (staged !== null) return staged; // Existing betas predating staging wrote their files before deployed_at. diff --git a/src/lib/server/fleet/staged-files.ts b/src/lib/server/fleet/staged-files.ts index 5119298..1beb4c0 100644 --- a/src/lib/server/fleet/staged-files.ts +++ b/src/lib/server/fleet/staged-files.ts @@ -80,6 +80,21 @@ function readExisting(path: string): Buffer | null { catch (err) { if ((err as NodeJS.ErrnoException).code === 'ENOENT') return null; throw err; } } +/** The first activation owns the rollback image, including an absent file. */ +export function captureBetaBeforeImage(row: StagedFileChange, before: Buffer | null): void { + if (db().prepare('SELECT 1 FROM beta_file_changes WHERE beta_id = ? AND server = ? AND rel_path = ?') + .get(row.source_id, row.server, row.rel_path)) return; + const prior = getServerScript(row.server, row.rel_path); + if (before !== null) db().prepare(`INSERT OR IGNORE INTO beta_file_snapshots + (beta_id, server, rel_path, prior_content, prior_sha256, prior_size) VALUES (?, ?, ?, ?, ?, ?)`) + .run(row.source_id, row.server, row.rel_path, before, sha256Buffer(before), before.length); + db().prepare(`UPDATE beta_server_scripts SET had_original = ?, prior_version_id = ? WHERE beta_id = ? AND server = ? + AND script_id IN (SELECT id FROM scripts WHERE target_rel_path = ? AND kind = ?)`) + .run(before === null ? 0 : 1, prior?.script_version_id ?? null, row.source_id, row.server, row.rel_path, row.kind); + db().prepare('INSERT INTO beta_file_changes (beta_id, server, rel_path) VALUES (?, ?, ?)') + .run(row.source_id, row.server, row.rel_path); +} + /** Only called after occupancy checks and reservation of the restart slot. * Synchronous file replacement keeps cancellation, new requests, and deletion * from interleaving between the final gates and dispatching the restart. @@ -111,17 +126,9 @@ export function applyStagedFilesForOperation(operationId: string, server: string for (const { row, path } of plans) { // Earlier coalesced campaigns may have just updated the same path. const before = readExisting(path); - const prior = getServerScript(server, row.rel_path); db().transaction(() => { if (row.source_kind === 'beta') { - if (before !== null) db().prepare(`INSERT OR IGNORE INTO beta_file_snapshots - (beta_id, server, rel_path, prior_content, prior_sha256, prior_size) VALUES (?, ?, ?, ?, ?, ?)`) - .run(row.source_id, server, row.rel_path, before, sha256Buffer(before), before.length); - db().prepare(`UPDATE beta_server_scripts SET had_original = ?, prior_version_id = ? WHERE beta_id = ? AND server = ? - AND script_id IN (SELECT id FROM scripts WHERE target_rel_path = ? AND kind = ?)`) - .run(before === null ? 0 : 1, prior?.script_version_id ?? null, row.source_id, server, row.rel_path, row.kind); - db().prepare('INSERT OR IGNORE INTO beta_file_changes (beta_id, server, rel_path) VALUES (?, ?, ?)') - .run(row.source_id, server, row.rel_path); + captureBetaBeforeImage(row, before); } if (row.deployment_file_id && !row.preserve_before_image) db().prepare(`UPDATE deployment_files SET verb = ?, content_before = ?, sha256_before = ? WHERE id = ?`) diff --git a/src/lib/server/pipeline/beta-plan-runner.ts b/src/lib/server/pipeline/beta-plan-runner.ts index 4cd51a7..33bbcf4 100644 --- a/src/lib/server/pipeline/beta-plan-runner.ts +++ b/src/lib/server/pipeline/beta-plan-runner.ts @@ -1,5 +1,6 @@ import { db } from '../db/index.js'; -import { betaOperationInFlight, deployBetaServers, getBeta, isActiveBetaServer, listBetaServers, rollbackBeta } from '../fleet/beta.js'; +import { currentBetaRevision } from '../fleet/beta-revisions.js'; +import { betaOperationInFlight, deployBetaServers, isActiveBetaServer, listBetaServers, rollbackBeta } from '../fleet/beta.js'; import { activity, currentRun, observeRun, prepareRun, readMetrics, saveMetrics, setOutcome, storedPlan, type StoredPlan } from '../fleet/beta-tests.js'; import { coveragePassed, evaluateRequirements } from '../fleet/beta-evaluation.js'; import { parseBetaPlan, type BetaPlan } from '../../beta-plan.js'; @@ -64,7 +65,6 @@ async function tick(): Promise { async function advancePlan(stored: StoredPlan): Promise { const id = stored.beta_id; const plan = parseBetaPlan(JSON.parse(stored.plan)); - const beta = getBeta(id)!; const servers = listBetaServers(id); // Preserve the original coverage and evidence. A replacement with the same // name cannot complete a test that began on a different server. @@ -74,7 +74,7 @@ async function advancePlan(stored: StoredPlan): Promise { setOutcome(id, 'paused', `Test server deleted: ${missing.map(server => server.server).join(', ')}. Roll back this beta and start a new test.`); return; } - if (stored.outcome !== 'passed' && Date.now() - Date.parse(beta.opened_at) >= plan.timeoutHours * 3_600_000) { + if (stored.outcome !== 'passed' && Date.now() - Date.parse(currentBetaRevision(id).started_at) >= plan.timeoutHours * 3_600_000) { betaState(id, false); setOutcome(id, 'inconclusive', 'Deadline reached without enough qualifying evidence'); return; diff --git a/src/lib/server/provisioning/native.ts b/src/lib/server/provisioning/native.ts index 094dec3..2d3e269 100644 --- a/src/lib/server/provisioning/native.ts +++ b/src/lib/server/provisioning/native.ts @@ -22,9 +22,9 @@ import { fleetSettings } from '../settings.js'; import { resolveRecoveredHealthOperation } from '../maintenance/recovery.js'; import { probeServerStatus, probePlayerEvidence } from '../fleet/status-evidence.js'; import { checkRelease, freshRelease, probeVersion } from '../integrations/plutonium.js'; -import { getServerScript, recordScript, removeScript } from '../fleet/inventory.js'; +import { recordScript, removeScript } from '../fleet/inventory.js'; import { sha256Buffer } from '../fleet/files.js'; -import type { StagedFileChange } from '../fleet/staged-files.js'; +import { captureBetaBeforeImage, type StagedFileChange } from '../fleet/staged-files.js'; import type { LaunchPreview } from '../../deployment-plan.js'; import type { ProvisioningAction } from '../../provisioning.js'; import { hostAgentUpdatesEnabled } from '../../hosts.js'; @@ -227,13 +227,8 @@ function applyReceipt(job: Job, result: Record) { const row = db().prepare('SELECT * FROM staged_file_changes WHERE id=?').get(plan.id) as StagedFileChange; if (!row || row.state === 'applied') continue; const bytes = snapshot.contentBase64 === null ? null : Buffer.from(snapshot.contentBase64, 'base64'); - const prior = getServerScript(job.server, row.rel_path); if (row.source_kind === 'beta') { - if (bytes !== null) db().prepare(`INSERT OR IGNORE INTO beta_file_snapshots(beta_id,server,rel_path,prior_content,prior_sha256,prior_size) VALUES(?,?,?,?,?,?)`) - .run(row.source_id, job.server, row.rel_path, bytes, sha256Buffer(bytes), bytes.length); - db().prepare(`UPDATE beta_server_scripts SET had_original=?,prior_version_id=? WHERE beta_id=? AND server=? - AND script_id IN (SELECT id FROM scripts WHERE target_rel_path=? AND kind=?)`).run(bytes === null ? 0 : 1, prior?.script_version_id ?? null, row.source_id, job.server, row.rel_path, row.kind); - db().prepare('INSERT OR IGNORE INTO beta_file_changes(beta_id,server,rel_path) VALUES(?,?,?)').run(row.source_id, job.server, row.rel_path); + captureBetaBeforeImage(row, bytes); db().prepare('UPDATE beta_servers SET deployed_at=?,updated_at=? WHERE beta_id=? AND server=?').run(new Date().toISOString(), new Date().toISOString(), row.source_id, job.server); } if (row.deployment_file_id && !row.preserve_before_image) db().prepare('UPDATE deployment_files SET verb=?,content_before=?,sha256_before=? WHERE id=?') diff --git a/src/lib/ui/components/BetaRevisionDialog.svelte b/src/lib/ui/components/BetaRevisionDialog.svelte new file mode 100644 index 0000000..7ce800f --- /dev/null +++ b/src/lib/ui/components/BetaRevisionDialog.svelte @@ -0,0 +1,100 @@ + + + + {#if error}{/if} + {#if !reviewing} +

Choose replacements for the files you changed.

+
{#each files as entry} + + {/each}
+ + {:else} +

{count} file{count === 1 ? '' : 's'} changed. All {servers.length} selected servers will repeat the test policy, using its existing wave order. Each server installs the bundle and restarts when empty.

+

Previous results move to revision history. Promotion requires this revision to pass. + {#if timeoutHours}A fresh {timeoutHours}-hour test window starts when you deploy, including time waiting for empty servers.{/if} + Rejecting the beta still restores the files from before revision 1.

+
Servers ({servers.length})
    {#each servers as name}
    • {/each}
+ {#each files.filter(entry => replacements[entry.id]) as entry} + {@const replacement = replacements[entry.id]} +

{entry.name}

{entry.target_rel_path} ← {replacement.file.name} ({replacement.file.size.toLocaleString()} bytes)

+

SHA {entry.version?.sha256.slice(0,12)} → {replacement.sha.slice(0,12)}

+ {#if replacement.after !== null} + {:else}

Binary or large file: review the uploaded file and checksum.

{/if} +
+ {/each} + {#if notes}

Change note: {notes}

{/if} + {/if} + {#snippet actions()} + + {#if reviewing} + {:else}{/if} + {/snippet} + + + diff --git a/src/lib/ui/components/BetaTestProgress.svelte b/src/lib/ui/components/BetaTestProgress.svelte index 1011735..6f23174 100644 --- a/src/lib/ui/components/BetaTestProgress.svelte +++ b/src/lib/ui/components/BetaTestProgress.svelte @@ -50,7 +50,7 @@ {#if ['paused','failed'].includes(test.outcome) && !expired}{/if} {/if} {#if retryBlocked && ['paused','failed'].includes(test.outcome)}

{retryBlocked}

{/if} - {#if expired}

The testing window ended. Roll back this beta first, then choose “Reuse files and test plan” to open a new beta with a longer window.

+ {#if expired}

The testing window ended. Use “Update beta” to test changed files in a fresh revision, or roll back and reuse the plan with a longer window.

{:else if test.outcome === 'paused' && !hasDeletedServers}

Resume retains completed milestones; continuous health and in-progress timing intervals start fresh. Pausing does not extend the deadline.

{/if} {/if} {#if error}{/if} diff --git a/src/routes/api/betas/+server.ts b/src/routes/api/betas/+server.ts index 75cf112..d5d8fbb 100644 --- a/src/routes/api/betas/+server.ts +++ b/src/routes/api/betas/+server.ts @@ -10,6 +10,7 @@ import { getScript, getVersion } from '$lib/server/fleet/scripts.js'; import { kick } from '$lib/server/pipeline/worker.js'; import { parseBetaPlan, type BetaPlan } from '$lib/beta-plan.js'; import { storedPlan } from '$lib/server/fleet/beta-tests.js'; +import { currentBetaRevision } from '$lib/server/fleet/beta-revisions.js'; export async function GET({ url }: { url: URL }) { const stateFilter = url.searchParams.get('state') as @@ -38,7 +39,7 @@ export async function GET({ url }: { url: URL }) { ...b, testOutcome: test?.outcome ?? null, testReason: test?.reason ?? null, - deadlineAt: plan ? new Date(Date.parse(b.opened_at) + plan.timeoutHours * 3600000).toISOString() : null, + deadlineAt: plan ? new Date(Date.parse(currentBetaRevision(b.id).started_at) + plan.timeoutHours * 3600000).toISOString() : null, scripts, servers: listBetaServers(b.id) }; diff --git a/src/routes/api/betas/[id]/+server.ts b/src/routes/api/betas/[id]/+server.ts index 3a5c2c8..4833d97 100644 --- a/src/routes/api/betas/[id]/+server.ts +++ b/src/routes/api/betas/[id]/+server.ts @@ -3,6 +3,7 @@ import { db } from '$lib/server/db/index.js'; import { getBeta, listBetaServers, listBetaScripts } from '$lib/server/fleet/beta.js'; import { getScript, getVersion } from '$lib/server/fleet/scripts.js'; import { testView } from '$lib/server/fleet/beta-tests.js'; +import { betaRevisionHistory, currentBetaRevision } from '$lib/server/fleet/beta-revisions.js'; function parseId(raw: string): number | null { const n = Number.parseInt(raw, 10); @@ -37,5 +38,6 @@ export async function GET({ params }: { params: { id: string } }) { failed: (db().prepare("SELECT COUNT(*) n FROM deployment_servers WHERE deployment_id = ? AND server_deleted_at IS NULL AND restart_state = 'failed'").get(campaign.id) as { n: number }).n } : null; const rollback = db().prepare("SELECT id FROM deployments WHERE beta_id = ? AND action = 'rollback' ORDER BY started_at DESC LIMIT 1") .get(id) as { id: string } | undefined; - return json({ beta, scripts, servers, events, test: testView(id), promotion, rollback: rollback ?? null }); + return json({ beta, scripts, servers, events, test: testView(id), promotion, rollback: rollback ?? null, + revision: currentBetaRevision(id), revisions: betaRevisionHistory(id) }); } diff --git a/src/routes/api/betas/[id]/revisions/+server.ts b/src/routes/api/betas/[id]/revisions/+server.ts new file mode 100644 index 0000000..ffe34af --- /dev/null +++ b/src/routes/api/betas/[id]/revisions/+server.ts @@ -0,0 +1,28 @@ +import { json } from '@sveltejs/kit'; +import { BetaError, reviseBeta, type BetaRevisionUpload } from '$lib/server/fleet/beta.js'; +import { ScriptError } from '$lib/server/fleet/scripts.js'; +import { kick } from '$lib/server/pipeline/worker.js'; + +export async function POST({ params, request }: { params: { id: string }; request: Request }) { + const id = Number(params.id); + if (!Number.isSafeInteger(id) || id < 1) return json({ ok: false, error: 'Invalid beta ID' }, { status: 400 }); + try { + const form = await request.formData(); + const expectedRevision = Number(form.get('expectedRevision')); + if (!Number.isSafeInteger(expectedRevision) || expectedRevision < 1) throw new Error('A reviewed revision is required.'); + const uploads: BetaRevisionUpload[] = []; + for (const [key, value] of form) { + if (!key.startsWith('file:')) continue; + const scriptId = Number(key.slice(5)); + if (!Number.isSafeInteger(scriptId) || scriptId < 1 || !(value instanceof File)) throw new Error('Invalid replacement file.'); + uploads.push({ scriptId, content: Buffer.from(await value.arrayBuffer()), + releaseVersion: form.has(`release:${scriptId}`) ? String(form.get(`release:${scriptId}`)) : undefined }); + } + const result = await reviseBeta(id, expectedRevision, uploads, String(form.get('notes') ?? '')); + void kick(); + return json({ ok: true, result }); + } catch (error) { + return json({ ok: false, error: (error as Error).message }, + { status: error instanceof BetaError || error instanceof ScriptError ? 409 : 400 }); + } +} diff --git a/src/routes/api/betas/[id]/revisions/[revision]/files/[scriptId]/+server.ts b/src/routes/api/betas/[id]/revisions/[revision]/files/[scriptId]/+server.ts new file mode 100644 index 0000000..cd9fb84 --- /dev/null +++ b/src/routes/api/betas/[id]/revisions/[revision]/files/[scriptId]/+server.ts @@ -0,0 +1,16 @@ +import { db } from '$lib/server/db/index.js'; + +/** Current files can be reviewed before the first revision is archived. */ +export async function GET({ params }: { params: { id: string; revision: string; scriptId: string } }) { + const values = [params.id, params.revision, params.scriptId].map(Number); + if (values.some(value => !Number.isSafeInteger(value) || value < 1)) return new Response('Invalid file', { status: 400 }); + const [id, revision, scriptId] = values; + const file = db().prepare('SELECT content FROM beta_revision_files WHERE beta_id = ? AND revision = ? AND script_id = ?') + .get(id, revision, scriptId) as { content: Buffer } | undefined + ?? (revision === 1 ? db().prepare(`SELECT v.content FROM beta_scripts bs JOIN script_versions v ON v.id = bs.script_version_id + WHERE bs.beta_id = ? AND bs.script_id = ? AND NOT EXISTS (SELECT 1 FROM beta_revisions WHERE beta_id = ?)`) + .get(id, scriptId, id) as { content: Buffer } | undefined : undefined); + if (!file) return new Response('File not found', { status: 404 }); + return new Response(new Uint8Array(file.content), { headers: { 'Content-Type': 'application/octet-stream', + 'Content-Disposition': 'attachment', 'Cache-Control': 'no-store' } }); +} diff --git a/src/routes/api/betas/[id]/test/+server.ts b/src/routes/api/betas/[id]/test/+server.ts index 8559aaf..5e38d6e 100644 --- a/src/routes/api/betas/[id]/test/+server.ts +++ b/src/routes/api/betas/[id]/test/+server.ts @@ -50,9 +50,8 @@ export async function POST({ params, request }: { params: { id: string }; reques } setOutcome(id, 'active', null); } else if (body.action === 'retry') { - // Fresh attempts keep their predecessors as audit history. Preserve the original deadline: - // operators open another beta if the overall testing window has expired. - if (stored.outcome === 'inconclusive') throw new Error('The testing window ended. Roll back and open a new beta with a longer deadline'); + // Fresh attempts keep history and the current revision's deadline. + if (stored.outcome === 'inconclusive') throw new Error('The testing window ended. Update the files to start a new revision, or roll back and open a new beta with a longer deadline'); if (stagedFileChanges('beta', String(id)).some(change => change.state === 'applying')) { throw new Error('A file write was interrupted. Inspect the server and roll back before opening a fresh beta'); } diff --git a/src/routes/betas/[id]/+page.svelte b/src/routes/betas/[id]/+page.svelte index 785db31..4246cd5 100644 --- a/src/routes/betas/[id]/+page.svelte +++ b/src/routes/betas/[id]/+page.svelte @@ -11,6 +11,8 @@ import type { ActivityEvent } from '$lib/operation-flow.js'; import { pollVisible } from '$ui/poll.js'; import { confirmAction } from '$ui/confirm.js'; + import type { BetaRevision } from '$lib/beta-revision.js'; + import BetaRevisionDialog from '$ui/components/BetaRevisionDialog.svelte'; import BetaTestProgress from '$ui/components/BetaTestProgress.svelte'; import ServerIdentity from '$ui/components/ServerIdentity.svelte'; import HistoricalServerIdentity from '$ui/components/HistoricalServerIdentity.svelte'; @@ -38,10 +40,13 @@ let acting = $state(null), request = 0; let reviewOpen = $state(false), preview = $state(null), reviewError = $state(''); let now = $state(Date.now()); + let revision = $state({ revision: 1, started_at: '' }), revisions = $state([]); + let updateOpen = $state(false), updateRevision = $state(1), updateFiles = $state([]); + function openUpdate() { updateRevision = revision.revision; updateFiles = [...scripts]; updateOpen = true; } const id = $derived(Number($page.params.id)); const closed = $derived(beta?.state === 'promoted' || beta?.state === 'rolled-back'); const hasDeletedServers = $derived(servers.some(server => server.server_deleted_at)); - const deadline = $derived(beta && test ? new Date(Date.parse(beta.opened_at) + test.plan.timeoutHours * 3600000) : null); + const deadline = $derived(beta && test ? new Date(Date.parse(revision.started_at || beta.opened_at) + test.plan.timeoutHours * 3600000) : null); const remaining = $derived(deadline ? Math.max(0, Math.ceil((deadline.getTime() - now) / 60000)) : 0); const deadlineText = $derived(remaining > 0 ? `${Math.floor(remaining / 60)}h ${remaining % 60}m remaining` : 'Testing window ended'); const passed = $derived(test?.servers.filter(server => server.passed).length ?? servers.filter(server => server.stage === 'ready').length); @@ -49,7 +54,7 @@ const waiting = $derived(ranked.filter(server => server.stage !== 'ready')); const nextAction = $derived(closed ? beta?.state === 'promoted' ? 'Follow the campaign to check server activation.' : rollbackCampaign ? 'Follow the rollback campaign to resume held changes or check restoration and restart progress.' : 'Reuse these files and this plan when you are ready to test again.' : hasDeletedServers ? 'A participating server was deleted. Its results remain below. Roll back, then reuse this plan to test the current servers.' - : test?.outcome === 'inconclusive' ? 'The deadline passed without enough evidence. Roll back, then reuse this plan with a longer testing window.' + : test?.outcome === 'inconclusive' ? 'The deadline passed without enough evidence. Update the files to start a new revision, or roll back and reuse this plan.' : test?.outcome === 'paused' ? 'Resume to continue collecting evidence, or start a fresh attempt. The original deadline still applies.' : test?.outcome === 'failed' ? 'Investigate the failed check before a fresh attempt, or roll back.' : beta?.state === 'ready-promote' ? 'All required coverage passed. Review the full promotion impact before deploying to additional servers.' @@ -64,11 +69,12 @@ if (current !== request || requestedId !== id) return; test = body.test; beta = body.beta; scripts = body.scripts; servers = body.servers; events = body.events; promotion = body.promotion ?? null; rollbackCampaign = body.rollback ?? null; + revision = body.revision; revisions = body.revisions ?? []; now = Date.now(); loadError = ''; } catch (error) { if (current === request && requestedId === id) loadError = (error as Error).message; } finally { if (current === request && requestedId === id) loading = false; } } - $effect(() => { id; beta = null; loading = true; loadError = ''; flash = ''; success = ''; reviewOpen = false; void load(); }); + $effect(() => { id; beta = null; loading = true; loadError = ''; flash = ''; success = ''; reviewOpen = false; updateOpen = false; void load(); }); onMount(() => pollVisible(load, 5000, false)); async function reviewPromotion() { reviewOpen = true; preview = null; reviewError = ''; acting = 'preview'; @@ -129,7 +135,7 @@ {:else if !beta && !loadError} {:else if beta}
- + {#snippet crumbs()}{/snippet} {#snippet actions()}{/snippet} @@ -139,12 +145,13 @@

{nextAction}

{#if test?.reason}

{test.reason}

{/if}
Started + {#if revision.revision > 1}Revision {revision.revision} started {$timeFormat.dateTime(revision.started_at)}{/if} {#if deadline && !closed}Ends · {deadlineText}{/if} {#if beta.closed_at}Finished {$timeFormat.dateTime(beta.closed_at)}{/if}
- {#if !closed && test}

Waiting for empty servers uses this window. Pausing or retrying does not extend the deadline.

{/if} + {#if !closed && test}

Waiting for empty servers uses this window. Pausing or retrying does not extend the deadline. Deploying a revision starts a fresh window.

{/if}
- {#if !closed}{/if} + {#if !closed}{/if} {#if closed}{/if} {#if promotion}{/if} {#if rollbackCampaign}{/if} @@ -163,10 +170,22 @@
Deployment progress ({servers.length} servers)
{#each ranked as server}

Files {server.deployed_at ? 'installed' : 'pending'} · Restart {server.restarted_at ? 'submitted' : 'pending'}{#if beta.wait_for_health} · Health {server.healthy_at ? 'verified' : 'pending'}{/if}{#if beta.wait_for_match} · Match {server.match_seen_at ? 'observed' : 'pending'}{/if}

{#if server.provisioning_id}View provisioning{/if}{#if server.error}

{server.error}

{/if}{#if server.stage === 'failed' && !server.server_deleted_at && !test && !closed}{/if}
{/each}
+ {#if revisions.length} +
Revision history ({revisions.length})
+ {#each revisions as item}
Revision {item.revision} · {item.revision === revision.revision ? 'Current' : item.results?.test?.outcome ?? item.results?.state ?? 'Superseded'} · {$timeFormat.dateTime(item.started_at)} + {#if item.notes}

{item.notes}

{/if} +
{#each item.files as file}
{file.name} · {artifactVersionLabel(file)}{file.path}{file.sha256.slice(0,12)}
{/each}
+ {#if item.results?.test} + {:else if item.results}
    {#each item.results.servers as server}
  • {server.server} · {server.stage}{server.error ? ` · ${server.error}` : ''}
    • {/each}
{/if} +
{/each} +
+ {/if}
{/if} + server.server)} timeoutHours={test?.plan.timeoutHours} onUpdate={load} /> + {#if acting === 'preview'}

Preparing promotion impact…

{/if} {#if reviewError}{/if} @@ -182,6 +201,7 @@