Support resumable game folder uploads and remove DLC declarations

This commit is contained in:
2026-10-04 09:51:45 -04:00
parent 95cf7dd36f
commit 840f02b53e
22 changed files with 1121 additions and 102 deletions
+12 -7
View File
@@ -29,11 +29,13 @@ The default Compose data mount preserves the previous
`EC_DATA_DIR` to your preferred persistent directory. Back up that directory,
including SQLite metadata and published game-file storage.
1. **Hosts → Add Host**: configure a Debian 12/13 amd64 machine and run its
1. **Library → Game Files**: select T5 or T6 and upload the complete local game
folder, including its DLC files. Name the library and version, then publish it.
Uploads work before any host is enrolled.
2. **Hosts → Add Host**: configure a Debian 12/13 amd64 machine and run its
generated enrollment command.
2. Prepare a Plutainer runtime from the host page.
3. **Library → Game Files**: import existing T5/T6 files once, then cache a
published version on the destination host.
3. Prepare a Plutainer runtime from the host page and download the published
game-file version to that host from **Game Files**.
4. Add the exact Plutonium server keys in **Keys**, or supply missing keys when
launching.
5. Launch a server or collection with an explicit host, game-file version and
@@ -69,9 +71,12 @@ server ID and retained after removal.
## Libraries and campaigns
Game-file versions include game, version, DLC metadata, manifests, checksums and
size. Actual game files use dedicated filesystem storage, never SQLite blobs.
Downloads resume, report progress and verify checksums before becoming usable.
Game-file versions contain the uploaded T5/T6 folder, with its directory layout
and included DLC preserved. There is no separate DLC declaration. Versions record
the game, label, file manifest, checksums and size. Uploads use bounded chunks,
support pause/resume and verify checksums before publication. Actual game files
use dedicated filesystem storage, never SQLite blobs. Host downloads resume,
report progress and verify checksums before becoming usable.
Multiple instances on one host share read-only base files.
Scripts, plugins, configuration baselines and overlays retain their versioning,
+1 -1
View File
@@ -31,7 +31,7 @@ guessable token. An unset setup token holds setup closed.
| Raw configs, file diffs, logs and telemetry diagnostics | No | Yes | Yes |
| Deploy/start/stop/restart/rebuild/remove servers | No | Yes | Yes |
| Scripts, plugins, configs, campaigns, betas and collections | Read summaries | Manage | Manage |
| Host setup, enrollment, credentials, runtime preparation and game-file import/cache | Read status | Read status | Manage |
| Host setup, enrollment, credentials, runtime preparation and game-file upload/cache | Read status | Read status | Manage |
| Server keys and global settings/maintenance policies | No | No | Manage |
| Create users, change roles, disable accounts and reset passwords | No | No | Yes |
| Own password and sign out | Yes | Yes | Yes |
+1 -1
View File
@@ -131,7 +131,7 @@ the typed key so it cannot accidentally be saved against a different lobby.
Komodo connection settings, target host, RCON configuration, config filename,
lobby availability, and port availability are checked locally. These checks do
not probe the host or verify game/DLC files, images, Docker volumes, firewall
not probe the host or verify game files, images, Docker volumes, firewall
rules, or credentials. Container and RCON verification happen after provisioning.
T6 variants retain their exact catalog slug and launch ruleset; RCON health checks
verify the engine map, not the variant's start location.
+38 -8
View File
@@ -17,9 +17,10 @@ the agent owns local execution and its durable operation journal.
3. The installer validates the platform, installs Python 3 and an absent Docker
installation, verifies the downloaded agent SHA-256 over HTTPS, enrolls it and
enables `electric-cherry-agent.service`. It needs no Git, Node.js or local build.
4. In **Hosts**, prepare a Plutainer runtime image. In **Game Files**, import your
existing `t5_game_files` or `t6_game_files` Docker volume on an enrolled source
host. Name the library and version and record its DLC metadata.
4. In **Hosts**, prepare a Plutainer runtime image. In **Game Files**, select T5 or
T6 and upload your local game folder. Name the library and version and publish
it after the upload finishes. Include the DLC files in that folder; there is
no separate DLC declaration. Uploading also works before enrolling any host.
5. Cache the published version on each destination host. Wait for verification.
Hosts report connection and readiness separately; deployment also checks the
particular library, image, capacity and ports you selected.
@@ -73,8 +74,15 @@ API outline:
- `POST /api/agent/heartbeat`: authenticated inventory, current operation and caches.
- `GET /api/agent/operations`: deliver/reconcile durable operation IDs.
- `POST /api/agent/operations/:id`: authenticated acceptance, progress or result.
- `/api/agent/imports/...` and `/api/agent/libraries/...`: host-scoped uploads and
authenticated game-file downloads.
- `POST /api/game-files`: administrator creates a folder upload with game, name,
version, optional description and `files: [{path, size, lastModified}]`.
- `GET /api/game-files/:id/upload`: durable per-file offsets and publication status.
- `PUT /api/game-files/:id/upload/files/:path`: administrator uploads a raw chunk
with `X-Upload-Offset`, `X-Upload-Length` and `X-Content-SHA256` headers.
- `POST /api/game-files/:id/upload/complete`: administrator starts asynchronous
verification/publication; poll the upload endpoint for completion.
- `/api/agent/libraries/...`: authenticated game-file downloads. Legacy
`/api/agent/imports/...` endpoints remain compatible with already queued imports.
The operation envelope is independent of polling and can be carried over a
persistent transport later. No unrestricted shell operation is available.
@@ -117,15 +125,37 @@ communication loss stops unrelated running containers.
## Libraries and ports
Imports scan the source volume and generate per-file SHA-256 manifests. Uploads
stream to temporary controller storage, are bounded by their manifest sizes and
verified before atomic publication. Published versions cannot be updated;
Choose the actual game folder, not its parent, a ZIP archive or the Plutonium
launcher data folder. The selected folder's name is removed once; nested paths
such as `zone/english/...` and `main/...` stay intact. Plutonium's
[T5 server guide](https://plutonium.pw/docs/server/t5/setting-up-a-server/) and
[T6 server guide](https://plutonium.pw/docs/server/t6/setting-up-a-server/) use a
copy of the game directory, separately from the launcher/runtime. The uploader
preserves the supplied files rather than attempting to infer installed maps from
a manually maintained DLC list.
Folder uploads use chunks of at most 8 MiB, each checked against its SHA-256 before
being appended to temporary controller storage. Memory usage does not grow with
the size of an individual game file. Pause/retry uses the controller's saved file
offsets. After a page reload, choose **Resume upload** and reselect the original,
unchanged folder; relative paths, sizes and modification times must match. Keep
the page open while transferring. Once publication starts, the controller hashes
every complete file and publishes the directory atomically; closing the browser
does not stop that verification. An interrupted controller can retry publication.
Published versions cannot be updated;
changing files requires a new version. Hosts download with HTTP Range resumption,
report progress, verify all checksums and atomically promote the complete cache.
Corrupt or incomplete content never becomes Ready. Cache inventory is reconciled
on heartbeats, including vanished caches. Older immutable versions remain usable
when their exact manifest is verified.
The controller and reverse proxy must accept 8 MiB upload requests and folder
manifests up to 32 MiB. Compose sets `BODY_SIZE_LIMIT=Infinity`; application routes
still enforce these limits, declared file sizes and checksums. Set a proxy body
limit of at least 32 MiB (for example `client_max_body_size 32m` in nginx).
Existing published versions and cached host libraries remain usable after the
upload workflow replaces source-host imports.
Reservations include host ID, local bind address, protocol and port. Wildcard
binds conflict with concrete addresses on the same host. UDP and TCP reservations
for one launch commit together; the same number can be used independently on
+2 -2
View File
@@ -10,10 +10,10 @@ export interface HostLibraryCache {
}
export interface GameLibraryVersion {
id: string; libraryId: string; game: GameLibraryGame; version: string;
description: string; dlc: string[]; state: GameLibraryState; sizeBytes: number;
description: string; state: GameLibraryState; sizeBytes: number;
fileCount: number; manifestSha256: string | null; files: GameFileEntry[];
createdAt: string; publishedAt: string | null; storageLocation: string;
sourceHostId: string; sourceVolume: string; error: string | null; operationId: string | null;
sourceHostId: string | null; sourceVolume: string | null; error: string | null; operationId: string | null;
caches: HostLibraryCache[];
}
export interface GameLibrary {
+2 -1
View File
@@ -7,7 +7,8 @@ import { assertRegularPath, parseByteRange } from './storage.js';
export function gameFileError(cause: unknown): Response {
const status = typeof cause === 'object' && cause !== null && 'status' in cause && typeof cause.status === 'number' ? cause.status : 400;
return json({ error: cause instanceof Error ? cause.message : 'Game file request failed.' }, { status });
const receivedBytes = typeof cause === 'object' && cause !== null && 'receivedBytes' in cause && Number.isSafeInteger(cause.receivedBytes) ? cause.receivedBytes : undefined;
return json({ error: cause instanceof Error ? cause.message : 'Game file request failed.', ...(receivedBytes === undefined ? {} : { receivedBytes }) }, { status });
}
export async function readGameJson(request: Request, maxBytes = 64 * 1024): Promise<Record<string, unknown>> {
if (!request.body) throw new Error('A JSON request body is required.');
+112 -8
View File
@@ -6,6 +6,7 @@ import { config } from '../config.js';
import { getHost, getHostOperation, listHosts, queueHostOperation } from '../hosts/index.js';
import { ensureGameFileSchema } from './schema.js';
import { manifestChecksum, receiveGameFile, validateGamePath, validateManifest, verifyGameFile } from './storage.js';
import { MAX_UPLOAD_CHUNK_BYTES, validateUploadPlan, getUploadFileProgress, receiveUploadChunk, hashUploadedFiles, withUploadLock } from './uploads.js';
import type { GameFileEntry, GameLibrary, GameLibraryGame, GameLibraryVersion, HostLibraryCache, LibraryCacheState } from '../../game-files.js';
function database() { const handle = db(); ensureGameFileSchema(handle); return handle; }
@@ -17,10 +18,10 @@ const cacheFromRow = (row: CacheRow): HostLibraryCache => ({ hostId: row.host_id
function versionFromRow(row: VersionRow): GameLibraryVersion {
const files = JSON.parse(row.manifest) as GameFileEntry[];
return { id: row.id, libraryId: row.library_id, game: row.game, version: row.version,
description: row.description, dlc: JSON.parse(row.dlc), state: row.state, sizeBytes: row.size_bytes,
files, fileCount: files.length, manifestSha256: row.manifest_sha256, createdAt: row.created_at,
publishedAt: row.published_at, storageLocation: row.storage_location, sourceHostId: row.source_host_id,
sourceVolume: row.source_volume, error: row.error, operationId: row.operation_id,
description: row.description, state: row.state, sizeBytes: row.size_bytes,
files, fileCount: files.length || (database().prepare('SELECT COUNT(*) AS count FROM game_file_upload_entries WHERE version_id=?').get(row.id) as { count: number }).count, manifestSha256: row.manifest_sha256, createdAt: row.created_at,
publishedAt: row.published_at, storageLocation: row.storage_location, sourceHostId: row.source_host_id || null,
sourceVolume: row.source_volume || null, error: row.error, operationId: row.operation_id,
caches: (database().prepare('SELECT * FROM host_library_caches WHERE version_id = ?').all(row.id) as CacheRow[]).map(cacheFromRow) };
}
export function getLibraryVersion(id: string): GameLibraryVersion | null {
@@ -60,6 +61,112 @@ function textField(input: unknown, label: string, max: number, required = true):
if (typeof input !== 'string' || input.length > max || (required && !input.trim())) throw new Error(`Provide a valid ${label}.`);
return input.trim();
}
/** Register a folder on the controller; no host or Docker volume is needed. */
export function createLibraryUpload(input: Record<string, unknown>): GameLibraryVersion {
const game = input.game;
if (game !== 't5' && game !== 't6') throw new Error('Choose T5 or T6 game files.');
const name = textField(input.name, 'library name', 160);
const description = textField(input.description ?? '', 'description', 4000, false);
const version = textField(input.version, 'version label', 120);
const files = validateUploadPlan(input.files);
const handle = database();
const id = randomUUID(); const now = new Date().toISOString();
handle.transaction(() => {
let library = handle.prepare('SELECT id,game FROM game_file_libraries WHERE name=?').get(name) as { id: string; game: string } | undefined;
if (library && library.game !== game) throw new Error('This library name is already used for another game.');
if (!library) {
library = { id: randomUUID(), game };
handle.prepare('INSERT INTO game_file_libraries(id,name,game,description,created_at) VALUES(?,?,?,?,?)').run(library.id, name, game, description, now);
}
if (handle.prepare('SELECT 1 FROM game_file_versions WHERE library_id=? AND version=?').get(library.id, version)) throw new Error('This version label already exists. Choose a new version.');
// Keep legacy non-null columns for existing databases and published versions.
handle.prepare(`INSERT INTO game_file_versions(id,library_id,version,description,dlc,state,size_bytes,created_at,storage_location,source_host_id,source_volume)
VALUES(?,?,?,?,'[]','importing',?,?,?,'','')`).run(id, library.id, version, description, files.reduce((sum, file) => sum + file.size, 0), now, join(gameFilesRoot(), game, id));
const insert = handle.prepare('INSERT INTO game_file_upload_entries(version_id,path,size,last_modified) VALUES(?,?,?,?)');
for (const file of files) insert.run(id, file.path, file.size, file.lastModified);
})();
return getLibraryVersion(id)!;
}
type UploadEntry = { path: string; size: number; lastModified: number };
function requireBrowserUpload(id: string): GameLibraryVersion {
const version = getLibraryVersion(id);
if (!version || version.sourceHostId !== null || !database().prepare('SELECT 1 FROM game_file_upload_entries WHERE version_id=? LIMIT 1').get(id)) {
throw Object.assign(new Error('Folder upload not found.'), { status: 404 });
}
return version;
}
function uploadEntries(id: string): UploadEntry[] {
return database().prepare('SELECT path,size,last_modified AS lastModified FROM game_file_upload_entries WHERE version_id=? ORDER BY path').all(id) as UploadEntry[];
}
async function uploadStorageRoot(version: GameLibraryVersion): Promise<string> {
// Recover publication interrupted after the atomic rename but before its DB commit.
try { if ((await stat(version.storageLocation)).isDirectory()) return version.storageLocation; }
catch (cause) { if ((cause as NodeJS.ErrnoException).code !== 'ENOENT') throw cause; }
return stagingRoot(version.id);
}
export async function browserUploadStatus(id: string) {
const version = requireBrowserUpload(id);
const status = version.state === 'ready' ? 'published' as const : publishing.has(id) ? 'verifying' as const : version.state === 'failed' ? 'failed' as const : 'pending' as const;
const root = await uploadStorageRoot(version);
const entries = uploadEntries(id);
const files = [];
// Avoid opening thousands of files concurrently for a large folder.
for (const file of entries) files.push({ ...file, ...(status === 'published' ? { receivedBytes: file.size, complete: true } : await getUploadFileProgress(root, file)) });
const { files: _manifest, ...summary } = version;
return { status, chunkSize: MAX_UPLOAD_CHUNK_BYTES, files, version: summary, error: version.error };
}
export async function uploadBrowserFile(id: string, path: string, chunk: { offset: number; size: number; sha256: string }, body: ReadableStream<Uint8Array> | null) {
validateGamePath(path);
const checkWritable = () => {
const version = requireBrowserUpload(id);
if (version.state === 'ready' || version.manifestSha256 || publishing.has(id)) throw Object.assign(new Error('This upload is no longer writable.'), { status: 409 });
};
checkWritable();
const file = database().prepare('SELECT path,size,last_modified AS lastModified FROM game_file_upload_entries WHERE version_id=? AND path=?').get(id, path) as UploadEntry | undefined;
if (!file) throw new Error('File is not listed in this upload.');
const progress = await receiveUploadChunk(stagingRoot(id), file, chunk, body, checkWritable);
database().prepare("UPDATE game_file_versions SET state='importing',error=NULL WHERE id=? AND state IN ('importing','failed')").run(id);
return progress;
}
export async function publishBrowserUpload(id: string): Promise<GameLibraryVersion> {
const version = requireBrowserUpload(id);
if (version.state === 'ready') return version;
const existing = publishing.get(id); if (existing) return existing;
const task = withUploadLock(stagingRoot(id), async () => {
const root = await uploadStorageRoot(version);
const files = await hashUploadedFiles(root, uploadEntries(id));
const checksum = manifestChecksum(files);
if (version.manifestSha256 && checksum !== version.manifestSha256) throw new Error('Stored files no longer match the verified manifest.');
database().transaction(() => {
database().prepare("UPDATE game_file_versions SET state='importing',manifest=?,manifest_sha256=?,error=NULL WHERE id=? AND state IN ('importing','failed')").run(JSON.stringify(files), checksum, id);
database().prepare('DELETE FROM game_file_entries WHERE version_id=?').run(id);
const insert = database().prepare('INSERT INTO game_file_entries(version_id,path,size,sha256) VALUES(?,?,?,?)');
for (const file of files) insert.run(id, file.path, file.size, file.sha256);
})();
if (root !== version.storageLocation) {
await mkdir(dirname(version.storageLocation), { recursive: true });
await rename(root, version.storageLocation);
}
database().prepare("UPDATE game_file_versions SET state='ready',published_at=?,error=NULL WHERE id=? AND state='importing'").run(new Date().toISOString(), id);
return getLibraryVersion(id)!;
});
publishing.set(id, task);
try { return await task; } finally { publishing.delete(id); }
}
export async function requestBrowserPublication(id: string) {
const version = requireBrowserUpload(id);
if (version.state !== 'ready' && !publishing.has(id)) {
database().prepare("UPDATE game_file_versions SET state='importing',error=NULL WHERE id=? AND state IN ('importing','failed')").run(id);
void publishBrowserUpload(id).catch(cause => {
database().prepare("UPDATE game_file_versions SET state='failed',error=? WHERE id=? AND state='importing'")
.run((cause instanceof Error ? cause.message : 'Folder publication failed.').slice(0, 2000), id);
});
}
return browserUploadStatus(id);
}
export function createLibraryImport(input: Record<string, unknown>): GameLibraryVersion {
const game = input.game;
if (game !== 't5' && game !== 't6') throw new Error('Choose T5 or T6 game files.');
@@ -69,9 +176,6 @@ export function createLibraryImport(input: Record<string, unknown>): GameLibrary
const name = textField(input.name, 'library name', 160);
const description = textField(input.description ?? '', 'description', 4000, false);
const version = textField(input.version, 'version label', 120);
const dlcInput = input.dlc ?? [];
if (!Array.isArray(dlcInput) || dlcInput.length > 64) throw new Error('DLC must be a list of content names.');
const dlc = dlcInput.map(value => textField(value, 'DLC name', 120));
const sourceVolume = game === 't5' ? 't5_game_files' : 't6_game_files';
const handle = database();
let library = handle.prepare('SELECT id,game FROM game_file_libraries WHERE name=?').get(name) as { id: string; game: string } | undefined;
@@ -84,7 +188,7 @@ export function createLibraryImport(input: Record<string, unknown>): GameLibrary
}
if (handle.prepare('SELECT 1 FROM game_file_versions WHERE library_id=? AND version=?').get(library.id, version)) throw new Error('This version label already exists. Choose a new version.');
handle.prepare(`INSERT INTO game_file_versions(id,library_id,version,description,dlc,state,created_at,storage_location,source_host_id,source_volume)
VALUES(?,?,?,?,?,'importing',?,?,?,?)`).run(id, library.id, version, description, JSON.stringify(dlc), now, join(gameFilesRoot(), game, id), hostId, sourceVolume);
VALUES(?,?,?,?,?,'importing',?,?,?,?)`).run(id, library.id, version, description, '[]', now, join(gameFilesRoot(), game, id), hostId, sourceVolume);
})();
try {
const operation = queueHostOperation(hostId, 'library.import', {
+5
View File
@@ -29,6 +29,11 @@ export function ensureGameFileSchema(handle: Database.Database): void {
version_id TEXT NOT NULL REFERENCES game_file_versions(id), path TEXT NOT NULL,
size INTEGER NOT NULL, sha256 TEXT NOT NULL, PRIMARY KEY(version_id,path)
);
CREATE TABLE IF NOT EXISTS game_file_upload_entries (
version_id TEXT NOT NULL REFERENCES game_file_versions(id), path TEXT NOT NULL,
size INTEGER NOT NULL, last_modified INTEGER NOT NULL,
PRIMARY KEY(version_id,path)
);
INSERT OR IGNORE INTO game_file_entries(version_id,path,size,sha256)
SELECT v.id,json_extract(f.value,'$.path'),json_extract(f.value,'$.size'),json_extract(f.value,'$.sha256')
FROM game_file_versions v,json_each(v.manifest) f;
+7 -2
View File
@@ -63,13 +63,18 @@ export async function assertRegularPath(root: string, path: string, allowMissing
return result;
}
export async function verifyGameFile(root: string, file: GameFileEntry): Promise<void> {
/** Compute a file digest without buffering game archives in memory. */
export async function hashGameFile(root: string, file: Pick<GameFileEntry, 'path' | 'size'>): Promise<string> {
const path = await assertRegularPath(root, file.path);
const info = await stat(path);
if (info.size !== file.size) throw new Error(`Size verification failed for ${file.path}.`);
const hash = createHash('sha256');
for await (const chunk of createReadStream(path)) hash.update(chunk);
if (hash.digest('hex') !== file.sha256) throw new Error(`Checksum verification failed for ${file.path}.`);
return hash.digest('hex');
}
export async function verifyGameFile(root: string, file: GameFileEntry): Promise<void> {
if (await hashGameFile(root, file) !== file.sha256) throw new Error(`Checksum verification failed for ${file.path}.`);
}
/** Stream to a private temporary file; publish only after byte count and SHA256 match. */
+155
View File
@@ -0,0 +1,155 @@
import { createHash, randomUUID } from 'node:crypto';
import { constants, createReadStream, createWriteStream } from 'node:fs';
import { lstat, mkdir, open, rm } from 'node:fs/promises';
import { dirname, resolve } from 'node:path';
import { Readable, Transform } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import type { GameFileEntry } from '../../game-files.js';
import { assertRegularPath, hashGameFile, validateManifest } from './storage.js';
export const MAX_UPLOAD_CHUNK_BYTES = 8 * 1024 * 1024;
export interface UploadFilePlan {
path: string;
size: number;
lastModified: number;
}
export interface UploadFileProgress {
receivedBytes: number;
complete: boolean;
}
export class UploadOffsetError extends Error {
readonly status = 409;
readonly receivedBytes: number;
constructor(receivedBytes: number) {
super(`Upload offset does not match the stored file. Resume at byte ${receivedBytes}.`);
this.name = 'UploadOffsetError';
this.receivedBytes = receivedBytes;
}
}
/** Apply the same path, case, size and file/directory collision rules as published manifests. */
export function validateUploadPlan(input: unknown): UploadFilePlan[] {
if (!Array.isArray(input) || !input.length || input.length > 200_000) throw new Error('Provide a folder containing 1–200000 files.');
const modified = new Map<string, number>();
const manifest = validateManifest(input.map(raw => {
if (!raw || typeof raw !== 'object') throw new Error('Invalid upload file.');
const lastModified = raw.lastModified === undefined ? 0 : raw.lastModified;
if (!Number.isSafeInteger(lastModified) || lastModified < 0) throw new Error('Invalid file modification time.');
modified.set(raw.path, lastModified);
return { path: raw.path, size: raw.size, sha256: '0'.repeat(64) };
}));
return manifest.map(file => ({ path: file.path, size: file.size, lastModified: modified.get(file.path)! }));
}
const uploadLocks = new Map<string, Promise<void>>();
/** Serialize transfers and final publication for one import; a rejected operation never poisons the lock. */
export async function withUploadLock<T>(root: string, callback: () => Promise<T> | T): Promise<T> {
const key = resolve(root);
const previous = uploadLocks.get(key) ?? Promise.resolve();
let release!: () => void;
const current = new Promise<void>(done => { release = done; });
uploadLocks.set(key, current);
await previous;
try { return await callback(); }
finally {
release();
if (uploadLocks.get(key) === current) uploadLocks.delete(key);
}
}
/** File size is the resume checkpoint, including after a controller restart during an append. */
export async function getUploadFileProgress(root: string, file: Pick<UploadFilePlan, 'path' | 'size'>): Promise<UploadFileProgress> {
const path = await assertRegularPath(root, file.path, true);
try {
const info = await lstat(path);
if (!info.isFile() || info.isSymbolicLink()) throw new Error('Game storage contains a non-regular path.');
if (info.size > file.size) throw new Error(`Stored upload exceeds the declared size of ${file.path}.`);
return { receivedBytes: info.size, complete: info.size === file.size };
} catch (cause) {
if ((cause as NodeJS.ErrnoException).code === 'ENOENT') return { receivedBytes: 0, complete: false };
throw cause;
}
}
/**
* Verify one bounded request before touching the destination. Only verified bytes may be
* appended, so a process interruption during the append can resume at the actual file size.
* beforeWrite runs under the lock: callers must recheck that the import is still writable.
*/
export async function receiveUploadChunk(
root: string,
file: Pick<UploadFilePlan, 'path' | 'size'>,
chunk: { offset: number; size: number; sha256: string },
body: ReadableStream<Uint8Array> | null,
beforeWrite?: () => Promise<void> | void
): Promise<UploadFileProgress> {
if (!Number.isSafeInteger(chunk.offset) || chunk.offset < 0 || !Number.isSafeInteger(chunk.size)
|| chunk.size < 0 || chunk.size > MAX_UPLOAD_CHUNK_BYTES || chunk.offset + chunk.size > file.size
|| !Number.isSafeInteger(chunk.offset + chunk.size) || !/^[a-f0-9]{64}$/.test(chunk.sha256)
|| (chunk.size === 0 && file.size !== 0)) {
throw new Error('Invalid upload chunk size, offset or SHA256.');
}
if (!body && chunk.size !== 0) throw new Error('A file chunk body is required.');
return withUploadLock(root, async () => {
await beforeWrite?.();
const progress = await getUploadFileProgress(root, file);
if (chunk.offset !== progress.receivedBytes) throw new UploadOffsetError(progress.receivedBytes);
const destination = await assertRegularPath(root, file.path, true);
// Keep transient chunks outside the tree that will become the immutable library.
const chunkRoot = `${resolve(root)}.chunks`;
const temporary = await assertRegularPath(chunkRoot, randomUUID(), true);
await mkdir(dirname(temporary), { recursive: true });
let received = 0;
const hash = createHash('sha256');
const verifier = new Transform({ transform(data, _encoding, done) {
received += data.length;
if (received > chunk.size) return done(new Error('Upload exceeds the declared chunk size.'));
hash.update(data); done(null, data);
} });
try {
await pipeline(body ? Readable.fromWeb(body as never) : Readable.from([]), verifier,
createWriteStream(temporary, { flags: 'wx', mode: 0o600, flush: true }));
if (received !== chunk.size || hash.digest('hex') !== chunk.sha256) throw new Error('Upload chunk checksum or size mismatch.');
await mkdir(dirname(destination), { recursive: true });
await assertRegularPath(root, file.path, true);
const target = await open(destination, constants.O_WRONLY | constants.O_CREAT | constants.O_NOFOLLOW, 0o600);
try {
const info = await target.stat();
if (!info.isFile()) throw new Error('Game storage contains a non-regular path.');
if (info.size !== chunk.offset) throw new UploadOffsetError(info.size);
let position = chunk.offset;
try {
for await (const data of createReadStream(temporary)) {
const bytes = data as Buffer;
let written = 0;
while (written < bytes.length) {
const result = await target.write(bytes, written, bytes.length - written, position);
if (result.bytesWritten === 0) throw new Error('Unable to append the upload chunk.');
written += result.bytesWritten;
position += result.bytesWritten;
}
}
await target.sync();
} catch (cause) {
await target.truncate(chunk.offset);
await target.sync();
throw cause;
}
} finally { await target.close(); }
return { receivedBytes: chunk.offset + chunk.size, complete: chunk.offset + chunk.size === file.size };
} finally { await rm(temporary, { force: true }); }
});
}
/** Caller holds withUploadLock through hashing and publication to prevent concurrent writes. */
export async function hashUploadedFiles(root: string, files: ReadonlyArray<Pick<UploadFilePlan, 'path' | 'size'>>): Promise<GameFileEntry[]> {
const manifest: GameFileEntry[] = [];
for (const file of files) manifest.push({ path: file.path, size: file.size, sha256: await hashGameFile(root, file) });
return validateManifest(manifest);
}
+1 -1
View File
@@ -204,7 +204,7 @@ export function previewCreation(input: { game?: unknown; mapSlug?: unknown; blue
{ label: 'Lobby', state: name ? 'pass' : 'blocked', detail: name ?? 'All 20 lobbies for this map are occupied or reserved.' },
{ label: 'Game port', state: port === null ? 'blocked' : 'pass', detail: port === null ? 'No free game port. Free capacity before launching.' : `${port} / UDP and TCP. Availability is checked again at launch.` },
{ label: 'Promoted configuration', state: 'pass', detail: recipe.configSource },
{ label: 'Game files and host readiness', state: 'info', detail: 'The host must have the selected game and DLC files. This preview does not probe volumes, images, credentials, or network access.' }
{ label: 'Game files and host readiness', state: 'info', detail: 'The host must have the selected game files. This preview does not probe volumes, images, credentials, or network access.' }
];
const id = randomUUID();
const expiresAt = new Date(Date.now() + 30 * 60_000).toISOString();
@@ -30,7 +30,7 @@
{#if !loading && !hosts.some(candidate => candidate.readiness === 'ready')}<p>No deployment-ready hosts. <a href="/hosts" target="_blank" rel="noopener">Set up a host</a>, then refresh.</p>{/if}
{#if host}<p>{host.hostname} · ports {host.portMin}–{host.portMax} · {host.publicAddress || host.bindAddress}</p>{/if}
{#each games as game}<label>{game === 't5' ? 'Black Ops I' : 'Black Ops II'} game-file version<select bind:value={selection.libraryIds[game]} disabled={disabled || loading || !host} onchange={() => onchange?.()}><option value="">Choose an immutable library version</option>{#each versions.filter(version => version.game === game) as version}<option value={version.id} disabled={!cacheReady(version.id)}>{version.name} · {version.version} · {cacheReady(version.id) ? 'Ready on host' : 'Not cached on host'}</option>{/each}</select></label>{/each}
<p><a href="/game-files" target="_blank" rel="noopener">Import or download game files to a host</a>, then refresh. Each launch stays pinned to its selected library version.</p>
<p><a href="/game-files" target="_blank" rel="noopener">Upload a game folder or download a version to a host</a>, then refresh. Each launch stays pinned to its selected library version.</p>
<label>Plutainer runtime image<input required bind:value={selection.image} disabled={disabled} onchange={() => onchange?.()} /></label>
{#if host && !imageReady}<p class="error">Prepare this runtime image from <a href={`/hosts/${host.id}`} target="_blank" rel="noopener">host setup</a> before launch.</p>{/if}
{#if network}<details><summary>Network overrides</summary><label>Local bind address<input bind:value={selection.bindAddress} placeholder={host?.bindAddress ?? 'Host default'} disabled={disabled} onchange={() => onchange?.()} /></label><label>Public / NAT address<input bind:value={selection.publicAddress} placeholder={host?.publicAddress || 'Host default'} disabled={disabled} onchange={() => onchange?.()} /></label><label>Public / NAT port<input type="number" min="1" max="65535" bind:value={selection.publicPort} placeholder="Same as local port" disabled={disabled} onchange={() => onchange?.()} /></label><p>Public mapping does not change the local Docker port reservation.</p></details>{/if}
+38
View File
@@ -0,0 +1,38 @@
export interface SelectedGameFile { path: string; file: File }
export interface SelectedGameFolder { name: string; files: SelectedGameFile[]; size: number }
/** Directory inputs include the selected folder in every relative path. Remove
* that one component and retain the complete game directory below it. */
export function selectGameFolder(selection: Iterable<File>): SelectedGameFolder {
const files: SelectedGameFile[] = [];
const paths = new Set<string>();
let name = '';
let size = 0;
for (const file of selection) {
const [root, ...parts] = file.webkitRelativePath.split('/');
if (!root || root === '.' || root === '..' || !parts.length || name && root !== name) throw new Error('Choose one game folder with the folder picker.');
name = root;
const path = parts.join('/');
if (path.length > 1024 || /[\\\x00-\x1f\x7f:]/.test(path) || parts.some(part => !part || part === '.' || part === '..' || /[. ]$/.test(part))) {
throw new Error(`This file has an unsupported game path: ${file.webkitRelativePath}`);
}
const key = path.toLowerCase();
if (paths.has(key)) throw new Error(`This folder has duplicate game paths: ${path}`);
paths.add(key);
size += file.size;
if (!Number.isSafeInteger(size)) throw new Error('This folder exceeds the supported upload size.');
files.push({ path, file });
if (files.length > 200_000) throw new Error('Choose a folder with no more than 200,000 files.');
}
if (!files.length) throw new Error('The selected folder contains no files.');
for (const { path } of files) {
const parents = path.toLowerCase().split('/');
parents.pop();
while (parents.length) {
if (paths.has(parents.join('/'))) throw new Error(`This folder uses the same game path for a file and a directory: ${path}`);
parents.pop();
}
}
files.sort((a, b) => a.path < b.path ? -1 : a.path > b.path ? 1 : 0);
return { name, files, size };
}
+2 -2
View File
@@ -1,10 +1,10 @@
import { json } from '@sveltejs/kit';
import { createLibraryImport, libraryOverview } from '$lib/server/game-files/index.js';
import { createLibraryUpload, libraryOverview } from '$lib/server/game-files/index.js';
import { gameFileError, readGameJson } from '$lib/server/game-files/http.js';
import { requireManagementOrigin } from '$lib/server/hosts/http.js';
export function GET() { return json(libraryOverview()); }
export async function POST({ request }: { request: Request }) {
try { requireManagementOrigin(request); return json({ version: createLibraryImport(await readGameJson(request)) }, { status: 202 }); }
try { requireManagementOrigin(request); return json({ version: createLibraryUpload(await readGameJson(request, 32 * 1024 * 1024)) }, { status: 202 }); }
catch (cause) { return gameFileError(cause); }
}
@@ -0,0 +1,8 @@
import { json } from '@sveltejs/kit';
import { browserUploadStatus } from '$lib/server/game-files/index.js';
import { gameFileError } from '$lib/server/game-files/http.js';
export async function GET({ params }: { params: { id: string } }) {
try { return json(await browserUploadStatus(params.id), { headers: { 'Cache-Control': 'no-store' } }); }
catch (cause) { return gameFileError(cause); }
}
@@ -0,0 +1,11 @@
import { json } from '@sveltejs/kit';
import { requestBrowserPublication } from '$lib/server/game-files/index.js';
import { gameFileError } from '$lib/server/game-files/http.js';
import { requireManagementOrigin } from '$lib/server/hosts/http.js';
export async function POST({ request, params }: { request: Request; params: { id: string } }) {
try {
requireManagementOrigin(request);
return json(await requestBrowserPublication(params.id), { status: 202 });
} catch (cause) { return gameFileError(cause); }
}
@@ -0,0 +1,17 @@
import { json } from '@sveltejs/kit';
import { uploadBrowserFile } from '$lib/server/game-files/index.js';
import { gameFileError } from '$lib/server/game-files/http.js';
import { requireManagementOrigin } from '$lib/server/hosts/http.js';
function integerHeader(request: Request, name: string): number {
const value = request.headers.get(name);
if (value === null || !/^\d+$/.test(value) || !Number.isSafeInteger(Number(value))) throw new Error(`Provide a valid ${name} header.`);
return Number(value);
}
export async function PUT({ request, params }: { request: Request; params: { id: string; path: string } }) {
try {
requireManagementOrigin(request);
const chunk = { offset: integerHeader(request, 'X-Upload-Offset'), size: integerHeader(request, 'X-Upload-Length'), sha256: request.headers.get('X-Content-SHA256') ?? '' };
return json(await uploadBrowserFile(params.id, params.path, chunk, request.body));
} catch (cause) { return gameFileError(cause); }
}
+215 -58
View File
@@ -1,9 +1,11 @@
<script lang="ts">
import { onMount } from 'svelte';
import { onMount, tick } from 'svelte';
import { page } from '$app/stores';
import { beforeNavigate } from '$app/navigation';
import type { GameLibrary, GameLibraryVersion } from '$lib/game-files.js';
import { formatGameBytes } from '$lib/game-files.js';
import type { Host } from '$lib/hosts.js';
import { selectGameFolder, type SelectedGameFolder } from '$ui/game-folder.js';
import { pollVisible } from '$ui/poll.js';
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
import Button from '$ui/components/primitives/Button.svelte';
@@ -11,110 +13,265 @@
import EmptyState from '$ui/components/primitives/EmptyState.svelte';
type OverviewLibrary = Omit<GameLibrary, 'versions'> & { versions: Omit<GameLibraryVersion, 'files'>[] };
let libraries = $state<OverviewLibrary[]>([]);
let hosts = $state<Host[]>([]);
let loaded = $state(false);
let error = $state(''); let notice = $state(''); let busy = $state(false);
let showImport = $state(false); let game = $state<'t5' | 't6'>('t5');
let name = $state(''); let version = $state(''); let description = $state(''); let dlc = $state(''); let hostId = $state('');
let filterGame = $state('all');
let targets = $state<Record<string, string>>({});
interface UploadFile { path: string; size: number; lastModified: number; receivedBytes: number; complete: boolean }
interface UploadStatus { status: 'pending' | 'verifying' | 'published' | 'failed'; chunkSize: number; files: UploadFile[]; error?: string | null; version: { id: string; state: string } }
type Phase = 'idle' | 'preparing' | 'uploading' | 'paused' | 'verifying' | 'published';
let libraries = $state<OverviewLibrary[]>([]); let hosts = $state<Host[]>([]);
let loaded = $state(false); let error = $state(''); let notice = $state(''); let busy = $state(false);
let showUpload = $state(false); let game = $state<'t5' | 't6'>('t5');
let name = $state(''); let version = $state(''); let description = $state('');
let filterGame = $state('all'); let targets = $state<Record<string, string>>({});
let folder = $state.raw<SelectedGameFolder | null>(null);
let uploadStatus = $state.raw<UploadStatus | null>(null);
let uploadId = $state(''); let uploadBusy = $state(false); let uploadError = $state('');
let phase = $state<Phase>('idle'); let currentPath = $state(''); let pauseRequested = $state(false);
let acceptedBytes = $state(0); let sendingBytes = $state(0); let completedFiles = $state(0);
let folderInput = $state<HTMLInputElement>();
let activeRequest: XMLHttpRequest | null = null; let disposed = false;
const canManage = $derived($page.data.user?.role === 'admin');
const onlineHosts = $derived(hosts.filter(host => host.connected && host.credentialActive && host.mode !== 'disabled'));
const matching = $derived(libraries.filter(library => filterGame === 'all' || library.game === filterGame));
const hostName = (id: string) => hosts.find(host => host.id === id)?.name ?? id;
const totalBytes = $derived(folder?.size ?? uploadStatus?.files.reduce((total, file) => total + file.size, 0) ?? 0);
const totalFiles = $derived(folder?.files.length ?? uploadStatus?.files.length ?? 0);
const progressBytes = $derived(Math.min(totalBytes, acceptedBytes + sendingBytes));
const transferring = $derived(uploadBusy && (phase === 'preparing' || phase === 'uploading'));
const hostName = (id: string | null) => id ? hosts.find(host => host.id === id)?.name ?? id : 'Browser upload';
async function responseBody(response: Response) {
let body;
try { body = await response.json(); } catch { throw new Error('The controller returned an unexpected response. Sign in again if your session expired, then resume the upload.'); }
if (!response.ok) throw new Error(body.error ?? 'The controller could not complete this request.');
return body;
}
async function load() {
try {
const response = await fetch('/api/game-files');
const body = await response.json();
if (!response.ok) throw new Error(body.error ?? 'Could not load game libraries.');
const body = await responseBody(await fetch('/api/game-files'));
libraries = body.libraries; hosts = body.hosts; loaded = true;
} catch (cause) { error = (cause as Error).message; }
}
async function act(url: string, payload: Record<string, unknown>, message: string) {
async function download(versionId: string) {
busy = true; error = ''; notice = '';
try {
const response = await fetch(url, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(payload) });
const result = await response.json();
if (!response.ok) throw new Error(result.error ?? 'Could not start this operation.');
notice = message; await load(); return true;
} catch (cause) { error = (cause as Error).message; return false; }
finally { busy = false; }
await responseBody(await fetch(`/api/game-files/${versionId}/download`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ hostId: targets[versionId] }) }));
notice = 'Library download queued. Progress will update below.'; await load();
} catch (cause) { error = (cause as Error).message; } finally { busy = false; }
}
async function startImport(event: SubmitEvent) {
event.preventDefault();
if (await act('/api/game-files', { game, name, version, description, dlc: dlc.split(',').map(value => value.trim()).filter(Boolean), hostId }, 'Import queued. The source host will scan and upload files, then verify and publish this version.')) showImport = false;
function resetUpload() {
folder = null; uploadStatus = null; uploadId = ''; phase = 'idle'; uploadError = ''; currentPath = '';
acceptedBytes = 0; sendingBytes = 0; completedFiles = 0; pauseRequested = false;
if (folderInput) folderInput.value = '';
}
function newVersion(library: OverviewLibrary) {
showImport = true; game = library.game; name = library.name; description = library.description;
dlc = library.versions[0]?.dlc.join(', ') ?? ''; version = '';
document.getElementById('library-import')?.scrollIntoView({ behavior: 'smooth' });
async function newUpload(library?: OverviewLibrary) {
resetUpload(); showUpload = true; game = library?.game ?? 't5'; name = library?.name ?? '';
description = library?.description ?? ''; version = '';
await tick(); document.getElementById('library-upload')?.scrollIntoView({ behavior: 'smooth' });
}
onMount(() => pollVisible(load, 5000));
function chooseFolder(event: Event) {
uploadError = ''; folder = null;
const input = event.currentTarget as HTMLInputElement;
if (!input.files?.length) return;
try {
const selected = selectGameFolder(Array.from(input.files));
if (uploadStatus) checkSelection(selected, uploadStatus);
folder = selected;
if (!name && !uploadId) name = game === 't5' ? 'Black Ops I game files' : 'Black Ops II game files';
} catch (cause) { uploadError = (cause as Error).message; input.value = ''; }
}
function checkSelection(selected: SelectedGameFolder, status: UploadStatus) {
const originals = new Map(status.files.map(file => [file.path, file]));
if (selected.files.length !== originals.size || selected.files.some(({ path, file }) => {
const original = originals.get(path); return !original || original.size !== file.size || original.lastModified !== file.lastModified;
})) throw new Error('This folder does not match the original upload. Select the same unchanged folder, including its original file names, sizes, and modification dates. Upload changed game files as a new version.');
}
function applyStatus(status: UploadStatus) {
uploadStatus = status;
acceptedBytes = status.files.reduce((total, file) => total + file.receivedBytes, 0);
completedFiles = status.files.filter(file => file.complete).length; sendingBytes = 0;
}
async function getStatus(): Promise<UploadStatus> {
const status = await responseBody(await fetch(`/api/game-files/${uploadId}/upload`)) as UploadStatus;
applyStatus(status); return status;
}
async function followVerification(initial: UploadStatus) {
let status = initial; phase = 'verifying'; currentPath = '';
while (!disposed && status.status === 'verifying') {
await new Promise(resolve => setTimeout(resolve, 1500));
if (disposed) return;
status = await getStatus();
}
if (disposed) return;
if (status.status !== 'published') throw new Error(status.error || 'Verification did not finish. Retry this upload to continue.');
phase = 'published'; folder = null; notice = `${name || 'Game files'} ${version} published. You can download this version to a host below.`;
if (folderInput) folderInput.value = '';
await load();
}
async function resumeUpload(library: OverviewLibrary, item: OverviewLibrary['versions'][number]) {
resetUpload(); showUpload = true; uploadId = item.id; game = library.game; name = library.name; version = item.version; description = item.description;
uploadBusy = true;
try {
const status = await getStatus();
if (status.status === 'verifying' || status.status === 'published') await followVerification(status);
else { phase = 'paused'; uploadError = status.error ?? ''; }
} catch (cause) { uploadError = (cause as Error).message; phase = 'paused'; }
finally { uploadBusy = false; }
await tick(); document.getElementById('library-upload')?.scrollIntoView({ behavior: 'smooth' });
}
function pauseUpload() { pauseRequested = true; activeRequest?.abort(); }
async function sendChunk(path: string, file: File, offset: number, length: number): Promise<number> {
const chunk = file.slice(offset, offset + length);
const hash = await crypto.subtle.digest('SHA-256', await chunk.arrayBuffer());
if (pauseRequested || disposed) throw new Error('Upload paused.');
const sha256 = [...new Uint8Array(hash)].map(value => value.toString(16).padStart(2, '0')).join('');
return new Promise((resolve, reject) => {
const request = new XMLHttpRequest(); activeRequest = request;
request.open('PUT', `/api/game-files/${uploadId}/upload/files/${path.split('/').map(encodeURIComponent).join('/')}`);
request.setRequestHeader('Content-Type', 'application/octet-stream');
request.setRequestHeader('X-Upload-Offset', String(offset)); request.setRequestHeader('X-Upload-Length', String(length));
request.setRequestHeader('X-Content-SHA256', sha256); request.timeout = 300_000;
request.upload.onprogress = event => { sendingBytes = Math.min(length, event.loaded); };
request.onload = () => {
activeRequest = null;
try {
const body = JSON.parse(request.responseText);
if (request.status < 200 || request.status >= 300) throw new Error(body.error ?? `Upload failed (${request.status}).`);
if (body.receivedBytes !== offset + length) throw new Error('The upload position changed. Resume to check the saved progress.');
resolve(body.receivedBytes);
} catch (cause) { reject(cause); }
};
request.onerror = () => { activeRequest = null; reject(new Error('The connection was interrupted. Resume to continue from the last saved chunk.')); };
request.ontimeout = () => { activeRequest = null; reject(new Error('This chunk timed out. Resume to check saved progress and retry.')); };
request.onabort = () => { activeRequest = null; reject(new Error('Upload paused.')); };
request.send(chunk);
});
}
async function startUpload(event: SubmitEvent) {
event.preventDefault(); if (uploadBusy || !folder) return;
if (!globalThis.crypto?.subtle) { uploadError = 'Folder uploads require a secure connection. Open this controller over HTTPS or localhost.'; return; }
uploadBusy = true; pauseRequested = false; uploadError = ''; notice = ''; phase = 'preparing';
try {
if (!uploadId) {
const result = await responseBody(await fetch('/api/game-files', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ game, name, version, description, files: folder.files.map(({ path, file }) => ({ path, size: file.size, lastModified: file.lastModified })) }) }));
uploadId = result.version.id;
}
const status = await getStatus();
if (status.status === 'verifying' || status.status === 'published') { await followVerification(status); return; }
checkSelection(folder, status);
const localFiles = new Map(folder.files.map(item => [item.path, item.file]));
const chunkSize = Math.min(8_388_608, Math.max(1, status.chunkSize));
phase = 'uploading';
for (const remote of status.files) {
if (pauseRequested || disposed) break;
if (remote.complete) continue;
currentPath = remote.path;
const file = localFiles.get(remote.path)!;
do {
if (pauseRequested || disposed) break;
const length = Math.min(chunkSize, remote.size - remote.receivedBytes);
remote.receivedBytes = await sendChunk(remote.path, file, remote.receivedBytes, length);
acceptedBytes += length; sendingBytes = 0;
if (remote.receivedBytes === remote.size) { remote.complete = true; completedFiles++; }
} while (!remote.complete);
}
if (pauseRequested || disposed) { phase = 'paused'; return; }
phase = 'verifying'; currentPath = '';
const verification = await responseBody(await fetch(`/api/game-files/${uploadId}/upload/complete`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{}' })) as UploadStatus;
applyStatus(verification); await followVerification(verification);
} catch (cause) {
if (!disposed) { uploadError = pauseRequested ? '' : (cause as Error).message; phase = 'paused'; }
} finally { uploadBusy = false; sendingBytes = 0; if (!disposed) void load(); }
}
beforeNavigate(navigation => {
if (!transferring) return;
if (navigation.type === 'leave' || !window.confirm('This upload is still transferring files. Leave this page? You can resume later by selecting the original unchanged folder.')) navigation.cancel();
});
onMount(() => {
const stop = pollVisible(load, 5000);
return () => { disposed = true; pauseRequested = true; activeRequest?.abort(); stop(); };
});
</script>
<svelte:head><title>Game Files | Electric Cherry</title></svelte:head>
<div class="page">
<SectionHeader title="Game Files" subtitle="Import immutable base-game versions and distribute verified copies to your hosts.">
{#snippet actions()}{#if canManage}<Button onclick={() => showImport = !showImport}>{showImport ? 'Close import' : 'Import game files'}</Button>{/if}{/snippet}
<SectionHeader title="Game Files" subtitle="Upload your T5 or T6 game folder, then distribute verified versions to your hosts.">
{#snippet actions()}{#if canManage}<Button disabled={uploadBusy} onclick={() => newUpload()}>Upload game folder</Button>{/if}{/snippet}
</SectionHeader>
{#if error}<p class="ui-error" role="alert">{error}</p>{/if}
{#if notice}<p role="status">{notice}</p>{/if}
{#if !canManage}<p class="muted">Administrators import game files and distribute verified copies to hosts.</p>{/if}
{#if canManage && showImport}
<section class="card" id="library-import">
<h2>Import an existing game volume</h2>
<p>The selected host scans its {game}_game_files Docker volume, calculates SHA256 checksums, and streams a copy into controller storage. Publication requires every file to pass verification.</p>
<form onsubmit={startImport}>
<div class="fields">
{#if !canManage}<p class="muted">Administrators upload game files and distribute verified copies to hosts.</p>{/if}
{#if canManage && showUpload}
<section class="card" id="library-upload">
<div class="heading"><h2>{uploadId ? phase === 'published' ? 'Version published' : 'Continue folder upload' : 'Upload a game folder'}</h2><Button size="sm" variant="ghost" disabled={uploadBusy} onclick={() => showUpload = false}>Close</Button></div>
<p>Select your complete T5 or T6 game folder, including its installed DLC. The files upload directly to this controller; a game host is only needed when you are ready to run servers.</p>
{#if uploadError}<p class="ui-error" role="alert">{uploadError}</p>{/if}
<form onsubmit={startUpload}>
<fieldset disabled={uploadBusy || !!uploadId} class="fields">
<label>Game<select bind:value={game}><option value="t5">Black Ops I (T5)</option><option value="t6">Black Ops II (T6)</option></select></label>
<label>Source host<select required bind:value={hostId}><option value="">Select an online host</option>{#each onlineHosts as host}<option value={host.id}>{host.name}</option>{/each}</select></label>
<label>Library name<input required maxlength="160" bind:value={name} placeholder="Black Ops II — Complete Server Files" /></label>
<label>New version label<input required maxlength="120" bind:value={version} placeholder="2026-10-03" /></label>
<label class="wide">Description<textarea maxlength="4000" bind:value={description} rows="2"></textarea></label>
<label class="wide">Content / DLC (comma separated)<input bind:value={dlc} placeholder="Base game, Nuketown, Die Rise, Mob of the Dead, Buried, Origins" /></label>
<label>Library name<input required maxlength="160" bind:value={name} placeholder="Black Ops II game files" /></label>
<label>Version label<input required maxlength="120" bind:value={version} placeholder="2026-10-04" /></label>
<label class="wide">Description <span class="muted">(optional)</span><textarea maxlength="4000" bind:value={description} rows="2"></textarea></label>
</fieldset>
{#if phase !== 'verifying' && phase !== 'published'}
<label class="folder-picker">{uploadId ? 'Select the original unchanged folder' : 'Game folder'}<input bind:this={folderInput} type="file" webkitdirectory multiple onchange={chooseFolder} disabled={uploadBusy} aria-describedby="folder-help" /></label>
<p class="muted" id="folder-help">{uploadId ? 'To resume, file names, sizes, and modification dates must match the original upload.' : 'Choose the game directory itself. Its top-level folder name is removed once; all files and subfolders inside it keep their paths.'}</p>
{/if}
{#if folder}<div class="folder-summary"><strong>{folder.name}</strong><span>{folder.files.length.toLocaleString()} files · {formatGameBytes(folder.size)}</span><details><summary>Preview folder paths</summary><ul class="path-list">{#each folder.files.slice(0, 8) as file}<li>{file.path}</li>{/each}</ul>{#if folder.files.length > 8}<p class="muted">And {(folder.files.length - 8).toLocaleString()} more files.</p>{/if}</details></div>{/if}
{#if phase !== 'idle'}<div class="upload-progress" aria-live="polite">
<div class="heading"><strong>{phase === 'preparing' ? 'Preparing upload…' : phase === 'uploading' ? 'Uploading game files' : phase === 'verifying' ? 'Verifying and publishing…' : phase === 'published' ? 'Published and ready to distribute' : 'Upload paused'}</strong><span>{completedFiles.toLocaleString()} / {totalFiles.toLocaleString()} files</span></div>
<progress max={totalBytes || Math.max(1, totalFiles)} value={totalBytes ? progressBytes : completedFiles} aria-label="Game folder upload progress"></progress>
<p>{formatGameBytes(progressBytes)} / {formatGameBytes(totalBytes)} saved or transferring</p>
{#if currentPath}<p class="current-file">{currentPath}</p>{/if}
{#if phase === 'verifying'}<p class="muted">The controller checks every file and publishes this version when verification finishes. Verification continues if you leave this page.</p>{:else if phase === 'paused'}<p class="muted">Saved chunks are kept on the controller. Resume this upload to check the saved progress and retry.</p>{/if}
</div>{/if}
<div class="upload-actions">
{#if transferring}<Button onclick={pauseUpload} disabled={pauseRequested}>{pauseRequested ? 'Pausing…' : 'Pause upload'}</Button>
{:else if phase !== 'verifying' && phase !== 'published'}<Button type="submit" variant="primary" disabled={uploadBusy || !folder}>{uploadId ? 'Resume upload' : 'Upload and publish'}</Button>{/if}
{#if phase === 'idle' || phase === 'uploading'}<span class="muted">Keep this page open while files transfer. Interrupted uploads can be resumed below.</span>{/if}
</div>
<Button type="submit" disabled={busy || !onlineHosts.length}>Start import</Button>
{#if !onlineHosts.length}<p>Enroll an online source host in <a href="/hosts">Hosts</a> before importing.</p>{/if}
</form>
</section>
{/if}
<div class="filters"><label>Game<select bind:value={filterGame}><option value="all">All games</option><option value="t5">Black Ops I</option><option value="t6">Black Ops II</option></select></label><span class="muted">{libraries.length} libraries · Base-game files and runtime images are versioned separately.</span></div>
<div class="filters"><label>Game<select bind:value={filterGame}><option value="all">All games</option><option value="t5">Black Ops I</option><option value="t6">Black Ops II</option></select></label><span class="muted">{libraries.length} libraries · Game folders and runtime images are versioned separately.</span></div>
{#each matching as library (library.id)}
<section class="card">
<div class="heading"><div><h2>{library.name}</h2><p class="muted">{library.game.toUpperCase()} · {library.description}</p></div>{#if canManage}<Button size="sm" onclick={() => newVersion(library)}>Import new version</Button>{/if}</div>
<div class="heading"><div><h2>{library.name}</h2><p class="muted">{library.game.toUpperCase()} · {library.description}</p></div>{#if canManage}<Button size="sm" disabled={uploadBusy} onclick={() => newUpload(library)}>Upload new version</Button>{/if}</div>
{#each library.versions as item (item.id)}
{@const browserUpload = !item.sourceHostId}
<article class="version">
<div class="heading"><h3>{item.version}</h3><Badge variant={item.state === 'failed' ? 'danger' : item.state === 'ready' ? 'online' : 'info'}>{item.state === 'ready' ? 'Published' : item.state === 'importing' ? 'Importing' : 'Failed'}</Badge></div>
<div class="heading"><h3>{item.version}</h3><Badge variant={item.state === 'failed' ? 'danger' : item.state === 'ready' ? 'online' : 'info'}>{item.state === 'ready' ? 'Published' : item.state === 'importing' ? browserUpload ? item.id === uploadId && phase === 'verifying' ? 'Verifying' : 'Uploading' : 'Importing' : 'Failed'}</Badge></div>
<p>{formatGameBytes(item.sizeBytes)} · {item.fileCount.toLocaleString()} files · Created {new Date(item.createdAt).toLocaleString()}</p>
{#if item.dlc.length}<p>Content: {item.dlc.join(', ')}</p>{/if}
{#if item.error}<p class="ui-error">{item.error}</p>{/if}
<details><summary>Version details</summary><dl><dt>Immutable version ID</dt><dd>{item.id}</dd><dt>Manifest SHA256</dt><dd>{item.manifestSha256 ?? 'Waiting for source scan'}</dd><dt>Controller storage</dt><dd>{item.storageLocation}</dd><dt>Imported from</dt><dd>{hostName(item.sourceHostId)} / {item.sourceVolume}</dd></dl><a href={`/api/game-files/${item.id}`} target="_blank" rel="noreferrer">View complete file manifest</a></details>
<details><summary>Version details</summary><dl><dt>Immutable version ID</dt><dd>{item.id}</dd><dt>Manifest SHA256</dt><dd>{item.manifestSha256 ?? 'Available after verification'}</dd><dt>Uploaded from</dt><dd>{browserUpload ? 'Game folder in browser' : `${hostName(item.sourceHostId)} / ${item.sourceVolume}`}</dd></dl>{#if item.state === 'ready'}<a href={`/api/game-files/${item.id}`} target="_blank" rel="noreferrer">View complete file manifest</a>{/if}</details>
{#if item.state === 'ready'}
{#if canManage}<div class="download"><label>Download to host<select bind:value={targets[item.id]}><option value="">Select host</option>{#each onlineHosts as host}<option value={host.id}>{host.name} · {formatGameBytes(Number(host.inventory.storageAvailable ?? 0))} free</option>{/each}</select></label><Button size="sm" disabled={busy || !targets[item.id]} onclick={() => act(`/api/game-files/${item.id}/download`, { hostId: targets[item.id] }, 'Library download queued. Progress will update below.')}>Download / verify</Button></div>{/if}
{#if canManage}<div class="download"><label>Download to host<select bind:value={targets[item.id]}><option value="">Select host</option>{#each onlineHosts as host}<option value={host.id}>{host.name} · {formatGameBytes(Number(host.inventory.storageAvailable ?? 0))} free</option>{/each}</select></label><Button size="sm" disabled={busy || !targets[item.id]} onclick={() => download(item.id)}>Download / verify</Button></div>{#if !onlineHosts.length}<p class="muted">This version is stored and ready. <a href="/hosts">Connect a host</a> when you want to deploy game servers.</p>{/if}{/if}
<div class="table-wrap"><table><thead><tr><th>Host</th><th>Cache state</th><th>Progress</th><th>Detail</th></tr></thead><tbody>
{#each item.caches as cache}<tr><td><a href={`/hosts/${cache.hostId}`}>{hostName(cache.hostId)}</a></td><td>{cache.status}{!hosts.find(host => host.id === cache.hostId)?.connected ? ' · offline' : ''}</td><td>{#if cache.status === 'downloading' || cache.status === 'verifying'}<progress max={cache.bytesTotal || 1} value={cache.bytesDownloaded}></progress><small>{formatGameBytes(cache.bytesDownloaded)} / {formatGameBytes(cache.bytesTotal)}</small>{:else if cache.status === 'ready' || cache.status === 'outdated'}Verified{cache.status === 'outdated' ? ' · older version' : ''}{:else}—{/if}</td><td>{cache.error ?? ''}</td></tr>{:else}<tr><td colspan="4">Enroll a host to cache this version.</td></tr>{/each}
{#each item.caches as cache}<tr><td><a href={`/hosts/${cache.hostId}`}>{hostName(cache.hostId)}</a></td><td>{cache.status}{!hosts.find(host => host.id === cache.hostId)?.connected ? ' · offline' : ''}</td><td>{#if cache.status === 'downloading' || cache.status === 'verifying'}<progress max={cache.bytesTotal || 1} value={cache.bytesDownloaded} aria-label="Host download progress"></progress><small>{formatGameBytes(cache.bytesDownloaded)} / {formatGameBytes(cache.bytesTotal)}</small>{:else if cache.status === 'ready' || cache.status === 'outdated'}Verified{cache.status === 'outdated' ? ' · older version' : ''}{:else}—{/if}</td><td>{cache.error ?? ''}</td></tr>{:else}<tr><td colspan="4">No host copies yet.</td></tr>{/each}
</tbody></table></div>
{:else if item.state === 'importing'}<p class="muted">The agent operation continues independently of this page. An interrupted import is reconciled when the host reconnects.</p>{/if}
{:else if browserUpload}<p class="muted">Resume with the original unchanged folder. Published versions can be downloaded to any enrolled host.</p>{#if canManage}<div><Button size="sm" disabled={uploadBusy} onclick={() => resumeUpload(library, item)}>Resume upload</Button></div>{/if}
{:else if item.state === 'importing'}<p class="muted">The existing host import continues independently. It is reconciled when the host reconnects.</p>{/if}
</article>
{/each}
</section>
{:else}{#if loaded}<EmptyState title="No game libraries yet" description="Administrators can import T5 or T6 game files from an enrolled host. Each published version can be shared by many instances." />{:else}<p role="status">Loading game libraries…</p>{/if}{/each}
{:else}{#if loaded}<EmptyState title="No game libraries yet" description="Upload a complete T5 or T6 game folder to get started. You can add a host after the files are published." />{:else}<p role="status">Loading game libraries…</p>{/if}{/each}
</div>
<style>
.page, .card, .version, form { display:grid; gap:20px; }
.card { padding: 24px; border: 1px solid var(--border-subtle); background: var(--surface); border-radius: var(--radius-md); }
.card { padding:24px; min-width:0; border:1px solid var(--border-subtle); background:var(--surface); border-radius:var(--radius-md); }
.version { padding-top:20px; border-top:1px solid var(--border-subtle); }
.heading, .filters, .download { display: flex; gap: 16px; align-items: center; justify-content: space-between; flex-wrap: wrap; }
.download { justify-content: flex-start; align-items: end; }
.fields { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 16px; }
.heading, .filters, .download, .upload-actions { display:flex; gap:16px; align-items:center; justify-content:space-between; flex-wrap:wrap; }
.download, .upload-actions { justify-content:flex-start; align-items:end; }
.fields { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:16px; border:0; padding:0; margin:0; min-width:0; }
.wide { grid-column:1/-1; } label { display:grid; gap:6px; font-size:var(--text-sm); }
input, select, textarea { width:100%; min-width:0; box-sizing:border-box; padding:10px; background:var(--surface); color:inherit; border:1px solid var(--border-strong); border-radius:var(--radius-sm); }
input:disabled, select:disabled, textarea:disabled { opacity:.7; } input[type='file'] { padding:16px; cursor:pointer; }
.folder-summary,.upload-progress { padding:20px; display:grid; gap:12px; background:var(--surface-sunken); border:1px solid var(--border-subtle); border-radius:var(--radius-md); min-width:0; }
.folder-summary>span,.upload-progress span,.upload-actions>span { color:var(--text-secondary); font-size:var(--text-sm); }
.path-list { padding-left:20px; font-family:var(--font-mono); font-size:var(--text-xs); overflow-wrap:anywhere; }
.current-file { font-family:var(--font-mono); font-size:var(--text-xs); overflow-wrap:anywhere; } progress { width:100%; accent-color:var(--accent-primary); }
h2,h3,p { margin:0; } h2 { font-size:var(--text-md); } h3 { font-size:var(--text-sm); }
p,td,th,details { font-size:var(--text-sm); line-height:1.6; } .muted,small { color:var(--text-secondary); }
small { display:block; } a { color:var(--accent-primary-hover); } summary { cursor:pointer; }
dl { display:grid; grid-template-columns:auto 1fr; gap:6px 16px; } dd { margin:0; overflow-wrap:anywhere; }
.table-wrap { overflow-x:auto; } table { width:100%; border-collapse:collapse; text-align:left; } th,td { padding:10px; border-bottom:1px solid var(--border-subtle); }
@media(max-width: 640px) { .fields { grid-template-columns: 1fr; } .card { padding: 16px; } dl { grid-template-columns: 1fr; } }
@media(max-width:640px) { .fields { grid-template-columns:1fr; } .card,.folder-summary,.upload-progress { padding:16px; } dl { grid-template-columns:1fr; } .heading>div { min-width:0; } }
</style>
+2 -1
View File
@@ -49,7 +49,8 @@ describe('management route permissions', () => {
['/api/hosts', 'POST'], ['/api/hosts/[id]', 'PATCH'], ['/api/hosts/[id]/enrollment', 'POST'],
['/api/hosts/[id]/credentials', 'DELETE'], ['/api/hosts/[id]/credentials', 'POST'],
['/api/hosts/[id]/operations', 'POST'], ['/api/settings', 'PUT'], ['/api/game-files', 'POST'],
['/api/game-files/[id]/download', 'POST'], ['/api/maintenance', 'POST'], ['/api/watchdog', 'POST'], ['/api/plutonium', 'POST']
['/api/game-files/[id]/download', 'POST'], ['/api/game-files/[id]/upload/files/[...path]', 'PUT'],
['/api/game-files/[id]/upload/complete', 'POST'], ['/api/maintenance', 'POST'], ['/api/watchdog', 'POST'], ['/api/plutonium', 'POST']
])('protects host, game-file, and global policy mutations at %s', (route, method) => {
expect(routePermission(route, method)).toBe('admin');
expect(allows('operator', route, method)).toBe(false);
+227
View File
@@ -0,0 +1,227 @@
import Database from 'better-sqlite3';
import { createHash } from 'node:crypto';
import { mkdir, mkdtemp, readFile, rename, rm, stat } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { dirname, join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
const state = vi.hoisted(() => ({ handle: null as Database.Database | null }));
vi.mock('../../src/lib/server/db/index.js', () => ({ db: () => state.handle }));
import {
browserUploadStatus, createLibraryImport, createLibraryUpload, getHostLibraryCache,
getLibraryVersion, listLibraries, publishBrowserUpload, queueLibraryDownload,
requestBrowserPublication, requireReadyLibrary, updateHostLibraryCaches, uploadBrowserFile
} from '../../src/lib/server/game-files/index.js';
import { createEnrollment, createHost, enrollHost, listHosts } from '../../src/lib/server/hosts/index.js';
import { manifestChecksum } from '../../src/lib/server/game-files/storage.js';
import { serveGameFile } from '../../src/lib/server/game-files/http.js';
import { GET as manifestRoute } from '../../src/routes/api/agent/libraries/[id]/manifest/+server.js';
let directory: string;
const modified = 1_700_000_000_000;
const bytes = Buffer.from('verified Plutonium game content');
const sha = (value: Uint8Array) => createHash('sha256').update(value).digest('hex');
const file = (path = 'zone/base.ff', content = bytes) => ({ path, size: content.length, lastModified: modified });
const upload = (id: string, path: string, content: Buffer, offset = 0, checksum = sha(content)) =>
uploadBrowserFile(id, path, { offset, size: content.length, sha256: checksum }, new Response(new Uint8Array(content)).body);
const begin = (game: 't5' | 't6' = 't6', files = [file()], version = '1') =>
createLibraryUpload({ game, name: `Plutonium ${game.toUpperCase()}`, version, description: 'Uploaded game folder', files });
beforeEach(async () => {
directory = await mkdtemp(join(tmpdir(), 'ec-browser-library-'));
state.handle = new Database(join(directory, 'controller.sqlite'));
state.handle.pragma('foreign_keys=ON');
vi.stubEnv('GAME_FILES_ROOT', join(directory, 'libraries'));
});
afterEach(async () => {
state.handle?.close(); state.handle = null;
await rm(directory, { recursive: true, force: true });
vi.unstubAllEnvs();
});
function host() {
const created = createHost({ name: 'Fresh download target' });
const { credential } = enrollHost(createEnrollment(created.id).token, {
protocolVersion: 1, dockerRunning: true, directoriesWritable: true, storageAvailable: 100 * 1024 ** 3
});
return { id: created.id, credential };
}
describe('browser folder uploads', () => {
it.each(['t5', 't6'] as const)('publishes %s from a browser without an enrolled host or DLC declarations', async game => {
expect(listHosts()).toEqual([]);
const initial = begin(game);
expect(initial).toMatchObject({ state: 'importing', operationId: null, sourceHostId: null, sourceVolume: null });
expect(initial).not.toHaveProperty('dlc');
expect(await browserUploadStatus(initial.id)).toMatchObject({
status: 'pending', files: [{ ...file(), receivedBytes: 0, complete: false }]
});
await upload(initial.id, file().path, bytes);
const accepted = await requestBrowserPublication(initial.id);
expect(['verifying', 'published']).toContain(accepted.status);
const ready = await publishBrowserUpload(initial.id);
expect(ready).toMatchObject({ state: 'ready', sizeBytes: bytes.length, fileCount: 1 });
expect(ready.files).toEqual([{ path: file().path, size: bytes.length, sha256: sha(bytes) }]);
expect(ready.manifestSha256).toBe(manifestChecksum(ready.files));
expect(await readFile(join(ready.storageLocation, file().path))).toEqual(bytes);
expect(await browserUploadStatus(initial.id)).toMatchObject({ status: 'published' });
expect(listLibraries()[0].versions[0]).not.toHaveProperty('dlc');
expect(listHosts()).toEqual([]);
});
it('preserves nested relative contents and derives a canonical manifest in path order', async () => {
const selected = [file('zone/english/a.ff'), file('main/iw_00.iwd'), file('empty.txt', Buffer.alloc(0))];
const initial = begin('t6', selected);
for (const item of selected) await upload(initial.id, item.path, item.size ? bytes : Buffer.alloc(0));
const ready = await publishBrowserUpload(initial.id);
expect(ready.files.map(item => item.path)).toEqual(['empty.txt', 'main/iw_00.iwd', 'zone/english/a.ff']);
expect(ready.files[0].sha256).toBe(sha(Buffer.alloc(0)));
expect(ready.manifestSha256).toBe(manifestChecksum(ready.files));
expect(await readFile(join(ready.storageLocation, 'zone/english/a.ff'))).toEqual(bytes);
});
it('rejects incomplete publication and allows the remaining files to finish the same version', async () => {
const initial = begin('t5', [file('zone/first.ff'), file('zone/second.ff')]);
await upload(initial.id, 'zone/first.ff', bytes);
await expect(publishBrowserUpload(initial.id)).rejects.toThrow();
expect(getLibraryVersion(initial.id)?.state).not.toBe('ready');
const hidden = await serveGameFile(new Request('https://ec.test/files'), initial.id, 'zone/first.ff');
expect(hidden.status).toBe(404);
await upload(initial.id, 'zone/second.ff', bytes);
const ready = await publishBrowserUpload(initial.id);
expect(ready.state).toBe('ready');
expect(ready.id).toBe(initial.id);
});
it('does not acknowledge corrupt chunks and permits a verified retry', async () => {
const initial = begin();
await expect(upload(initial.id, file().path, bytes, 0, '0'.repeat(64))).rejects.toThrow();
expect((await browserUploadStatus(initial.id)).files[0]).toMatchObject({ receivedBytes: 0, complete: false });
await expect(uploadBrowserFile(initial.id, file().path, {
offset: 0, size: bytes.length, sha256: sha(bytes)
}, new Response(bytes.subarray(0, -1)).body)).rejects.toThrow();
expect((await browserUploadStatus(initial.id)).files[0].receivedBytes).toBe(0);
await upload(initial.id, file().path, bytes);
expect((await browserUploadStatus(initial.id)).files[0]).toMatchObject({ receivedBytes: bytes.length, complete: true });
expect((await publishBrowserUpload(initial.id)).state).toBe('ready');
});
it('surfaces asynchronous incomplete publication as retryable failure', async () => {
const initial = begin();
await requestBrowserPublication(initial.id);
await vi.waitFor(async () => expect((await browserUploadStatus(initial.id)).status).toBe('failed'));
expect((await browserUploadStatus(initial.id)).error).toBeTruthy();
await upload(initial.id, file().path, bytes);
expect(await browserUploadStatus(initial.id)).toMatchObject({ status: 'pending', error: null });
await requestBrowserPublication(initial.id);
expect((await publishBrowserUpload(initial.id)).state).toBe('ready');
});
it('resumes from the verified chunk boundary and safely retries a lost acknowledgement', async () => {
const sizeProbe = begin();
const chunkSize = (await browserUploadStatus(sizeProbe.id)).chunkSize;
expect(chunkSize).toBeGreaterThan(0);
const large = Buffer.alloc(chunkSize + 17, 0x41);
const initial = begin('t6', [file('main/archive.iwd', large)], 'large');
const first = large.subarray(0, chunkSize);
await upload(initial.id, 'main/archive.iwd', first);
expect((await browserUploadStatus(initial.id)).files[0]).toMatchObject({ receivedBytes: chunkSize, complete: false });
state.handle!.close();
state.handle = new Database(join(directory, 'controller.sqlite'));
state.handle.pragma('foreign_keys=ON');
await expect(upload(initial.id, 'main/archive.iwd', first)).rejects.toMatchObject({ receivedBytes: chunkSize });
expect((await browserUploadStatus(initial.id)).files[0].receivedBytes).toBe(chunkSize);
await upload(initial.id, 'main/archive.iwd', large.subarray(chunkSize), chunkSize);
const ready = await publishBrowserUpload(initial.id);
expect(ready.files[0].sha256).toBe(sha(large));
expect((await readFile(join(ready.storageLocation, 'main/archive.iwd'))).equals(large)).toBe(true);
});
it('rejects unknown file paths, traversal, and chunks outside the declared file', async () => {
const initial = begin();
await expect(upload(initial.id, 'zone/unselected.ff', bytes)).rejects.toThrow();
await expect(upload(initial.id, '../escape.ff', bytes)).rejects.toThrow();
await expect(upload(initial.id, file().path, bytes, bytes.length + 1)).rejects.toThrow();
await expect(upload(initial.id, file().path, Buffer.concat([bytes, bytes]))).rejects.toThrow();
expect((await browserUploadStatus(initial.id)).files[0].receivedBytes).toBe(0);
});
it('keeps ready bytes and version metadata immutable and requires a new version label', async () => {
const initial = begin(); await upload(initial.id, file().path, bytes);
const ready = await publishBrowserUpload(initial.id);
await expect(upload(initial.id, file().path, bytes)).rejects.toThrow();
expect(() => state.handle!.prepare('UPDATE game_file_versions SET description=? WHERE id=?').run('changed', ready.id)).toThrow(/immutable/);
expect(() => begin()).toThrow(/already exists/);
expect((await publishBrowserUpload(initial.id)).manifestSha256).toBe(ready.manifestSha256);
expect(begin('t6', [file()], '2').id).not.toBe(ready.id);
expect(await readFile(join(ready.storageLocation, file().path))).toEqual(bytes);
});
it('recovers publication after the verified directory was renamed but the ready commit was interrupted', async () => {
const initial = begin(); await upload(initial.id, file().path, bytes);
const files = [{ path: file().path, size: bytes.length, sha256: sha(bytes) }];
const checksum = manifestChecksum(files);
state.handle!.transaction(() => {
state.handle!.prepare('UPDATE game_file_versions SET manifest=?,manifest_sha256=? WHERE id=?')
.run(JSON.stringify(files), checksum, initial.id);
state.handle!.prepare('INSERT INTO game_file_entries(version_id,path,size,sha256) VALUES(?,?,?,?)')
.run(initial.id, file().path, bytes.length, sha(bytes));
})();
await mkdir(dirname(initial.storageLocation), { recursive: true });
await rename(join(directory, 'libraries', '.imports', initial.id), initial.storageLocation);
state.handle!.close(); state.handle = new Database(join(directory, 'controller.sqlite'));
state.handle.pragma('foreign_keys=ON');
expect(await browserUploadStatus(initial.id)).toMatchObject({ status: 'pending', files: [{ complete: true }] });
const ready = await publishBrowserUpload(initial.id);
expect(ready).toMatchObject({ id: initial.id, state: 'ready', manifestSha256: checksum, files });
expect(await readFile(join(ready.storageLocation, file().path))).toEqual(bytes);
});
it('rejects chunks racing with final publication without recreating staging files', async () => {
const initial = begin(); await upload(initial.id, file().path, bytes);
const publishing = publishBrowserUpload(initial.id);
await expect(upload(initial.id, file().path, bytes)).rejects.toMatchObject({ status: 409 });
const ready = await publishing;
expect(ready.state).toBe('ready');
expect(await readFile(join(ready.storageLocation, file().path))).toEqual(bytes);
await expect(stat(join(directory, 'libraries', '.imports', initial.id))).rejects.toMatchObject({ code: 'ENOENT' });
});
it('rejects browser operations against an existing host-sourced import', async () => {
const source = host();
const legacy = createLibraryImport({ hostId: source.id, game: 't6', name: 'Legacy import', version: '1' });
expect(legacy.sourceHostId).toBe(source.id);
await expect(Promise.resolve().then(() => browserUploadStatus(legacy.id))).rejects.toThrow();
await expect(upload(legacy.id, file().path, bytes)).rejects.toThrow();
await expect(publishBrowserUpload(legacy.id)).rejects.toThrow();
await expect(Promise.resolve().then(() => requestBrowserPublication(legacy.id))).rejects.toThrow();
expect(getLibraryVersion(legacy.id)?.state).toBe('importing');
});
it('distributes uploaded versions through the unchanged agent manifest, ranges, and verified cache contract', async () => {
const initial = begin('t5'); await upload(initial.id, file().path, bytes);
const ready = await publishBrowserUpload(initial.id);
const target = host();
const operation = queueLibraryDownload(target.id, ready.id);
expect(operation).toMatchObject({ kind: 'library.download', payload: {
libraryVersionId: ready.id, game: 't5', manifestSha256: ready.manifestSha256,
manifestUrl: `/api/agent/libraries/${ready.id}/manifest`, filesBaseUrl: `/api/agent/libraries/${ready.id}/files`
} });
const manifest = manifestRoute({ request: new Request(`https://ec.test/api/agent/libraries/${ready.id}/manifest`, {
headers: { Authorization: `Bearer ${target.credential}` }
}), params: { id: ready.id } });
expect(manifest.status).toBe(200);
expect(await manifest.json()).toEqual({ id: ready.id, game: 't5', files: ready.files, sizeBytes: bytes.length, manifestSha256: ready.manifestSha256 });
const range = await serveGameFile(new Request('https://ec.test/file', { headers: { Range: 'bytes=9-' } }), ready.id, file().path);
expect(range.status).toBe(206);
expect(Buffer.from(await range.arrayBuffer())).toEqual(bytes.subarray(9));
expect(() => requireReadyLibrary(target.id, ready.id)).toThrow(/finish downloading/);
updateHostLibraryCaches(target.id, [{ libraryVersionId: ready.id, status: 'ready', manifestSha256: '0'.repeat(64) }]);
expect(getHostLibraryCache(target.id, ready.id)?.status).toBe('failed');
updateHostLibraryCaches(target.id, [{ libraryVersionId: ready.id, status: 'ready', manifestSha256: ready.manifestSha256 }]);
expect(requireReadyLibrary(target.id, ready.id).id).toBe(ready.id);
const replacement = begin('t5', [file()], '2'); await upload(replacement.id, file().path, bytes);
await publishBrowserUpload(replacement.id);
expect(requireReadyLibrary(target.id, ready.id).manifestSha256).toBe(ready.manifestSha256);
});
});
+180
View File
@@ -0,0 +1,180 @@
import { createHash } from 'node:crypto';
import { mkdir, mkdtemp, readFile, readdir, rm, symlink, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import {
getUploadFileProgress, hashUploadedFiles, MAX_UPLOAD_CHUNK_BYTES,
receiveUploadChunk, UploadOffsetError, validateUploadPlan, withUploadLock
} from '../../src/lib/server/game-files/uploads.js';
import { manifestChecksum, verifyGameFile } from '../../src/lib/server/game-files/storage.js';
const sha = (value: string | Uint8Array) => createHash('sha256').update(value).digest('hex');
const stream = (value: string) => new Response(value).body!;
const file = { path: 'zone/english/map.ff', size: 10, lastModified: 1234 };
const chunk = (value: string, offset = 0) => ({ offset, size: Buffer.byteLength(value), sha256: sha(value) });
let directory: string;
let root: string;
beforeEach(async () => {
directory = await mkdtemp(join(tmpdir(), 'ec-folder-upload-'));
root = join(directory, 'import');
});
afterEach(async () => { await rm(directory, { recursive: true, force: true }); });
describe('browser folder upload plans', () => {
it('preserves folder structure and source identity while sorting canonical paths', () => {
expect(validateUploadPlan([{ path: 'zone/map.ff', size: 10, lastModified: 100 }, { path: 'main/data.iwd', size: 0 }])).toEqual([
{ path: 'main/data.iwd', size: 0, lastModified: 0 },
{ path: 'zone/map.ff', size: 10, lastModified: 100 }
]);
});
it.each([
[], [{ path: '../zone.ff', size: 1 }], [{ path: 'zone\\map.ff', size: 1 }],
[{ path: 'Zone/a', size: 1 }, { path: 'zone/A', size: 1 }],
[{ path: 'ZONE', size: 1 }, { path: 'zone/a', size: 1 }],
[{ path: 'zone/a', size: -1 }], [{ path: 'zone/a', size: 1.5 }],
[{ path: 'zone/a', size: Number.MAX_SAFE_INTEGER }, { path: 'zone/b', size: 1 }],
[{ path: 'zone/a', size: 1, lastModified: -1 }], [{ path: 'zone/a', size: 1, lastModified: 1.5 }],
[{ path: 'zone/a', size: 1, lastModified: null }]
].map(plan => ({ plan })))('rejects ambiguous or unsafe plan $plan', ({ plan }) => {
expect(() => validateUploadPlan(plan)).toThrow();
});
});
describe('bounded resumable file transfer', () => {
it('resumes arbitrary chunks by filesystem offset and produces a distribution-compatible manifest', async () => {
expect(await getUploadFileProgress(root, file)).toEqual({ receivedBytes: 0, complete: false });
expect(await receiveUploadChunk(root, file, chunk('0123'), stream('0123'))).toEqual({ receivedBytes: 4, complete: false });
// A new request discovers its checkpoint from disk; no process-local receipt is required.
expect(await getUploadFileProgress(root, file)).toEqual({ receivedBytes: 4, complete: false });
expect(await receiveUploadChunk(root, file, chunk('456789', 4), stream('456789'))).toEqual({ receivedBytes: 10, complete: true });
expect(await readFile(join(root, file.path), 'utf8')).toBe('0123456789');
const manifest = await withUploadLock(root, () => hashUploadedFiles(root, [file]));
expect(manifest).toEqual([{ path: file.path, size: 10, sha256: sha('0123456789') }]);
expect(manifestChecksum(manifest)).toBe(sha(JSON.stringify(manifest)));
await verifyGameFile(root, manifest[0]);
expect(await readdir(root)).toEqual(['zone']);
expect(await readdir(`${root}.chunks`)).toEqual([]);
});
it('reports committed progress after reply loss instead of appending duplicate bytes', async () => {
await receiveUploadChunk(root, file, chunk('0123'), stream('0123'));
await expect(receiveUploadChunk(root, file, chunk('0123'), stream('0123'))).rejects.toMatchObject({
name: 'UploadOffsetError', receivedBytes: 4, status: 409
});
await expect(receiveUploadChunk(root, file, chunk('89', 8), stream('89'))).rejects.toBeInstanceOf(UploadOffsetError);
expect(await readFile(join(root, file.path), 'utf8')).toBe('0123');
await receiveUploadChunk(root, file, chunk('456789', 4), stream('456789'));
});
it('can resume a verified append interrupted by process termination at an unaligned byte', async () => {
await mkdir(join(root, 'zone/english'), { recursive: true });
await writeFile(join(root, file.path), '0123456');
expect(await getUploadFileProgress(root, file)).toEqual({ receivedBytes: 7, complete: false });
await receiveUploadChunk(root, file, chunk('789', 7), stream('789'));
expect(await readFile(join(root, file.path), 'utf8')).toBe('0123456789');
});
it('distinguishes an absent empty file and creates it using the empty digest', async () => {
const empty = { path: 'main/empty.dat', size: 0 };
expect(await getUploadFileProgress(root, empty)).toEqual({ receivedBytes: 0, complete: false });
await expect(hashUploadedFiles(root, [empty])).rejects.toThrow();
expect(await receiveUploadChunk(root, empty, chunk(''), null)).toEqual({ receivedBytes: 0, complete: true });
expect(await getUploadFileProgress(root, empty)).toEqual({ receivedBytes: 0, complete: true });
expect(await hashUploadedFiles(root, [empty])).toEqual([{ ...empty, sha256: sha('') }]);
});
it('does not advance progress for wrong checksums, truncated requests, oversized streams or disconnects', async () => {
await receiveUploadChunk(root, file, chunk('0123'), stream('0123'));
await expect(receiveUploadChunk(root, file, chunk('456789', 4), stream('xxxxxx'))).rejects.toThrow(/checksum/);
await expect(receiveUploadChunk(root, file, chunk('456789', 4), stream('45'))).rejects.toThrow(/size mismatch/);
await expect(receiveUploadChunk(root, file, chunk('456789', 4), stream('4567890'))).rejects.toThrow(/exceeds/);
let reads = 0;
const disconnect = new ReadableStream<Uint8Array>({ pull(controller) {
if (reads++ === 0) controller.enqueue(new TextEncoder().encode('45'));
else controller.error(new Error('Browser connection ended.'));
} });
await expect(receiveUploadChunk(root, file, chunk('456789', 4), disconnect)).rejects.toThrow(/connection ended/);
expect(await getUploadFileProgress(root, file)).toEqual({ receivedBytes: 4, complete: false });
expect(await readFile(join(root, file.path), 'utf8')).toBe('0123');
expect(await readdir(`${root}.chunks`)).toEqual([]);
await receiveUploadChunk(root, file, chunk('456789', 4), stream('456789'));
});
it('bounds request size before reading any body and rejects invalid metadata', async () => {
const tooLarge = MAX_UPLOAD_CHUNK_BYTES + 1;
await expect(receiveUploadChunk(root, { path: 'large.iwd', size: tooLarge }, {
offset: 0, size: tooLarge, sha256: sha('')
}, null)).rejects.toThrow(/Invalid upload chunk/);
for (const metadata of [
{ ...chunk('a'), offset: -1 }, { ...chunk('a'), offset: 0.5 }, { ...chunk('a'), size: 1.5 },
{ ...chunk('a'), sha256: 'wrong' }, chunk('', 0), chunk('01234567890', 0)
]) await expect(receiveUploadChunk(root, file, metadata, stream('a'))).rejects.toThrow(/Invalid upload chunk/);
await expect(receiveUploadChunk(root, file, chunk('a'), null)).rejects.toThrow(/body is required/);
});
it('streams a maximum-size chunk supplied in small pieces', async () => {
const piece = new Uint8Array(64 * 1024).fill(97);
let remaining = MAX_UPLOAD_CHUNK_BYTES;
const digest = createHash('sha256');
for (let i = 0; i < MAX_UPLOAD_CHUNK_BYTES / piece.length; i++) digest.update(piece);
const body = new ReadableStream<Uint8Array>({ pull(controller) {
if (remaining === 0) controller.close();
else { remaining -= piece.length; controller.enqueue(piece); }
} });
const large = { path: 'main/large.iwd', size: MAX_UPLOAD_CHUNK_BYTES };
const sha256 = digest.digest('hex');
await receiveUploadChunk(root, large, { offset: 0, size: large.size, sha256 }, body);
expect(await hashUploadedFiles(root, [large])).toEqual([{ ...large, sha256 }]);
});
it('rejects missing or incorrectly sized files before publication', async () => {
await receiveUploadChunk(root, file, chunk('0123'), stream('0123'));
await expect(hashUploadedFiles(root, [file])).rejects.toThrow(/Size verification/);
await writeFile(join(root, file.path), '01234567890');
await expect(getUploadFileProgress(root, file)).rejects.toThrow(/exceeds/);
});
it('refuses traversal and symlinks in destination and temporary storage', async () => {
await mkdir(root);
const outside = join(directory, 'outside');
await mkdir(outside);
await expect(receiveUploadChunk(root, { path: '../escape', size: 1 }, chunk('a'), stream('a'))).rejects.toThrow(/relative/);
await symlink(outside, join(root, 'zone'));
await expect(receiveUploadChunk(root, file, chunk('0123'), stream('0123'))).rejects.toThrow(/non-regular/);
await rm(join(root, 'zone'));
await symlink(outside, `${root}.chunks`);
await expect(receiveUploadChunk(root, file, chunk('0123'), stream('0123'))).rejects.toThrow(/non-regular/);
expect(await readdir(outside)).toEqual([]);
});
});
describe('upload and publication coordination', () => {
it('serializes simultaneous writes with the same offset', async () => {
const responses = await Promise.allSettled([
receiveUploadChunk(root, file, chunk('0123'), stream('0123')),
receiveUploadChunk(root, file, chunk('0123'), stream('0123'))
]);
expect(responses.map(response => response.status)).toEqual(['fulfilled', 'rejected']);
expect((responses[1] as PromiseRejectedResult).reason).toMatchObject({ name: 'UploadOffsetError', receivedBytes: 4 });
expect(await readFile(join(root, file.path), 'utf8')).toBe('0123');
});
it('rechecks writable state inside the lock after queued publication and releases failed locks', async () => {
let finishPublication!: () => void;
const gate = new Promise<void>(resolve => { finishPublication = resolve; });
let published = false;
const publication = withUploadLock(root, async () => { await gate; published = true; });
const transfer = receiveUploadChunk(root, file, chunk('0123'), stream('0123'), () => {
if (published) throw new Error('Library is immutable.');
});
finishPublication();
await publication;
await expect(transfer).rejects.toThrow(/immutable/);
expect(await getUploadFileProgress(root, file)).toEqual({ receivedBytes: 0, complete: false });
await expect(withUploadLock(root, () => { throw new Error('Failed work'); })).rejects.toThrow(/Failed work/);
expect(await withUploadLock(root, () => 'next operation')).toBe('next operation');
});
});
+75
View File
@@ -0,0 +1,75 @@
import { describe, expect, it, vi } from 'vitest';
import { selectGameFolder } from '../../src/lib/ui/game-folder.js';
function selectedFile(path: string, size = 10) {
const read = vi.fn(() => { throw new Error('Folder selection must not read file bodies.'); });
const file = {
name: path.split('/').at(-1), webkitRelativePath: path, size, lastModified: 1_700_000_000_000,
arrayBuffer: read, text: read, stream: read, slice: read
} as unknown as File;
return { file, read };
}
describe('game folder selection', () => {
it('removes the selected root exactly once while preserving nested game paths', () => {
const first = selectedFile('T6/T6/zone/english/base.ff', 20);
const second = selectedFile('T6/main/iw_00.iwd', 30);
const result = selectGameFolder([first.file, second.file]);
expect(result.name).toBe('T6');
expect(result.size).toBe(50);
expect(result.files).toEqual([
{ path: 'T6/zone/english/base.ff', file: first.file },
{ path: 'main/iw_00.iwd', file: second.file }
]);
expect(first.read).not.toHaveBeenCalled();
expect(second.read).not.toHaveBeenCalled();
});
it('prepares multi-gigabyte and empty files using metadata alone', () => {
const large = selectedFile('Game/zone/base.ff', 7 * 1024 ** 3);
const empty = selectedFile('Game/empty.txt', 0);
const result = selectGameFolder([large.file, empty.file]);
expect(result.size).toBe(7 * 1024 ** 3);
expect(result.files.map(item => item.path)).toEqual(['empty.txt', 'zone/base.ff']);
expect(result.files[1].file).toBe(large.file);
expect(large.read).not.toHaveBeenCalled();
expect(empty.read).not.toHaveBeenCalled();
});
it('rejects case-insensitive duplicate file names', () => {
expect(() => selectGameFolder([
selectedFile('Game/Zone/base.ff').file,
selectedFile('Game/zone/BASE.ff').file
])).toThrow('duplicate game paths');
});
it('rejects a path used as both a file and a directory regardless of case', () => {
expect(() => selectGameFolder([
selectedFile('Game/zone/english/base.ff').file,
selectedFile('Game/ZONE').file
])).toThrow('same game path for a file and a directory');
});
it.each(['../base.ff', './base.ff', '/Game/base.ff', 'base.ff'])(
'rejects an absent or parent root in %s', path => {
expect(() => selectGameFolder([selectedFile(path).file])).toThrow('one game folder');
}
);
it.each(['Game/../base.ff', 'Game/zone/./base.ff', 'Game/zone//base.ff', 'Game/zone\\base.ff'])(
'rejects traversal and non-normalized content paths in %s', path => {
expect(() => selectGameFolder([selectedFile(path).file])).toThrow('unsupported game path');
}
);
it('rejects files from multiple selected roots', () => {
expect(() => selectGameFolder([
selectedFile('T5/main/base.ff').file,
selectedFile('T6/main/base.ff').file
])).toThrow('one game folder');
});
it('rejects empty selections', () => {
expect(() => selectGameFolder([])).toThrow('contains no files');
});
});