Initial Mist scaffold

Successor to the Josh Steam prototypes. Single-VM Docker Compose stack with the load-bearing core/ logic ported from JoshSteam CDN with bug fixes. Contents: - backend/ FastAPI + Celery (same image, two entrypoints) core/ hdiff, librsync, chain_replay, manifest, compression, discord, steam, unrealpak, paths api/ auth, catalog, admin, builds (skeletons) + downloads (real) worker/ Celery factory replacing the missing prototype Tasks/__init__.py db/ SQLAlchemy models + Alembic initial migration - admin-web/ SvelteKit + Tailwind skeleton - client/ Tauri 2 + Svelte skeleton (Mist placeholder UI) - mistpipe/ click-based admin CLI with subcommand stubs - docs/ ARCHITECTURE, DECISIONS (9 ADRs), RUNBOOK - docker-compose.yml + dev overlay + .github/workflows Bugs fixed during port: - Routes/download.py:2 stray backslash on import line - Utils/celery.py inspect.reserved() missing parens + double active() typo - Hardcoded OneDrive/Desktop paths replaced with pydantic-settings config - Discord webhook URL + RabbitMQ password moved to env vars - Missing Tasks/__init__.py reconstructed as worker/__init__.py Out of scope for this commit: route bodies, UI screens, mistpipe subcommand bodies, real image builds.
2026-06-07 19:39:25 -04:00
commit bfd6771a9a
76 changed files with 3890 additions and 0 deletions
@@ -0,0 +1,140 @@
+# Mist — Architecture
+
+## Purpose
+
+A private Steam-clone for ~5–10 friends. Distributes games and updates with bandwidth-efficient delta patching. Sized to the actual problem — the complexity is in the delta-patching system, not in the deployment.
+
+## Topology
+
+Friends use a regular web/desktop client over the open internet. Public DNS resolves to a border server. Nginx Proxy Manager terminates TLS (Let's Encrypt) and reverse-proxies to the backend VM over Tailscale. The backend VM itself has no public exposure.
+
+```
+            ┌──────────────────────────────┐
+   Friend   │ Tauri client (Svelte UI      │
+   (open    │  inside Rust shell)          │
+  internet) └──────────────┬───────────────┘
+                           │ HTTPS, public domain
+                           │ store.mist.example
+                           │ admin.mist.example
+                           │ dl.mist.example
+                           ▼
+            ┌─────────────────────────────────┐
+            │ Border server (public IP)       │
+            │ Nginx Proxy Manager + Let's     │
+            │ Encrypt TLS termination         │
+            └────────────────┬────────────────┘
+                             │ Tailscale (private backhaul)
+                             ▼
+       ┌─────────────────────────────────────────────┐
+       │ Proxmox VM "mist"                           │
+       │   Docker Compose stack:                     │
+       │                                             │
+       │   ┌──────────────┐    ┌──────────────────┐ │
+       │   │ api          │    │ admin-web        │ │
+       │   │ FastAPI:     │    │ static Svelte/   │ │
+       │   │ /auth        │    │ TS served by     │ │
+       │   │ /catalog     │    │ tiny nginx       │ │
+       │   │ /admin       │    │ (calls api with  │ │
+       │   │ /downloads   │    │  admin JWT)      │ │
+       │   │ /builds      │    └──────────────────┘ │
+       │   └──────┬───────┘                         │
+       │          │                                 │
+       │          ▼                                 │
+       │   ┌──────────────┐                         │
+       │   │ worker       │                         │
+       │   │ Celery       │   same image,           │
+       │   │ delta-gen,   │   different entrypoint  │
+       │   │ archive prep,│                         │
+       │   │ notifications│                         │
+       │   └──────┬───────┘                         │
+       │          │                                 │
+       │          ▼                                 │
+       │   ┌──────────┐ ┌────────┐ ┌─────────────┐ │
+       │   │ postgres │ │ redis  │ │ rabbitmq    │ │
+       │   └──────────┘ └────────┘ └─────────────┘ │
+       │                                            │
+       │   Volumes:                                 │
+       │     - hot cache (.tar.zst archives)        │
+       │     - postgres data                        │
+       │     - redis data                           │
+       │     - rabbitmq data                        │
+       │     - /mnt/nas → NFS to NAS (games)        │
+       └────────────────────────────────────────────┘
+```
+
+## Containers
+
+| Container | Stack | Owns |
+|---|---|---|
+| **api** | FastAPI + SQLAlchemy + Postgres + Redis | Single web app with internally-modular code: `auth/`, `catalog/`, `admin/`, `downloads/`, `builds/`. Issues JWTs. Serves resumable downloads. Receives `mistpipe` uploads. Queues background work into RabbitMQ. |
+| **worker** | Celery (same image as `api`, different entrypoint) | Consumes RabbitMQ. Runs the heavy stuff: hdiff delta generation, librsync indirect-delta generation, `chain_replay` cold reconstruction, `.tar.zst` archive packing, Discord notifications. |
+| **admin-web** | SvelteKit, built static + tiny nginx | Admin UI. Calls `api/admin/*` with admin JWT. |
+| **postgres** | postgres:16 | Catalog, users, build job state. |
+| **redis** | redis:7 | Celery result backend, cache, ephemeral session data. |
+| **rabbitmq** | rabbitmq:3.13-management | Celery broker, event bus for `notification.*` events. |
+
+## Non-container artifacts
+
+| Artifact | Stack | Notes |
+|---|---|---|
+| **client** | Tauri 2 (Rust shell + Svelte UI) | Friend-facing app. Distributed as a per-platform installer. Embeds (or spawns) the patch-application logic. |
+| **mistpipe** | Python + click | Admin CLI. `login`, `new-game`, `push`, `ls`, `rm`, `resync-steam`. JWT stored in OS keychain. |
+
+## Storage
+
+**NAS** (mounted at `/mnt/nas` inside the VM via NFS) is the **source of truth** for game files:
+
+```
+/mnt/nas/mist/games/<Title>/
+  base_version.7z              ← immutable original
+  depot/                       ← current latest version's files
+  manifests/
+    <Title>.json               ← ordered linear version list (legacy; will migrate to Postgres)
+    <version>.json             ← per-version SHA-256 manifest
+  deltas/<version>/
+    delta_manifest.json
+    new_files/...
+    *.patch                    ← hdiff direct patches
+```
+
+**Docker volumes** on the VM hold the hot path:
+
+- `cache-vol` — `.tar.zst` archives ready to serve, reconstructed historical versions
+- `tmp-vol` — in-flight delta-gen working dirs
+- `postgres-vol`, `redis-vol`, `rabbitmq-vol` — service data
+
+## Update modes
+
+Two delta strategies, decided at request time:
+
+- **Direct update** = consecutive versions (`1.0.0.2` → `1.0.0.3`). Deltas were pre-generated by `hdiff` at push time. Serve from cache or zip on demand.
+- **Indirect update** = arbitrary version jumps (`1.0.0.0` → `1.0.0.3`). Server tells client which files changed. Client generates `librsync` signatures of its local files, POSTs them. Worker generates `rdiff` deltas against the server's copy and packs them into a `.tar.zst`. Client applies with `rdiff patch`.
+
+For arbitrary historical reconstructions the worker runs `chain_replay` — starts from the base or closest cached version and walks forward applying hdiff patches at each step, caching results for next time.
+
+## Auth
+
+- Username + password, you provision accounts via the admin portal
+- Passwords hashed with argon2id
+- JWT issued on login, scope claim distinguishes `user` from `admin`
+- Admin scope required for `/admin/*` and `/builds/*`
+- Per-game `is_private` boolean flag; non-admin users only see public games + games they've been explicitly granted (future)
+
+## Catalog metadata
+
+Steam appdetails pull-through: when an admin adds a game with an `app_id`, the catalog service fetches `https://store.steampowered.com/api/appdetails` and stores `short_description` + `header_image`. Admin can override either via hand-edited fields.
+
+## Out of scope (for MVP)
+
+- Branches (stable/beta/internal)
+- Save sync, achievements, friend lists, in-app chat
+- Payments
+- Multi-tenancy (one store)
+- Public self-serve signup
+- Per-user entitlements beyond `is_private`
+- Client-side delta-gen in `mistpipe` (server does it for MVP)
+- High availability — single VM is fine at this scale; backup + restore covers failure
+
+## Why this shape
+
+The original "Josh Steam" prototypes proved out the hard parts: two-mode delta-patching, chain-replay for cold reconstruction, resumable downloads. The 2-year-later rebuild focuses on **finishing the product**, not re-exploring the design space. Microservices and k8s were considered and rejected for the actual scale — see `DECISIONS.md`.