goddard
bfd6771a9a
Successor to the Josh Steam prototypes. Single-VM Docker Compose stack with
the load-bearing core/ logic ported from JoshSteam CDN with bug fixes.
Contents:
- backend/ FastAPI + Celery (same image, two entrypoints)
core/ hdiff, librsync, chain_replay, manifest, compression,
discord, steam, unrealpak, paths
api/ auth, catalog, admin, builds (skeletons) + downloads (real)
worker/ Celery factory replacing the missing prototype Tasks/__init__.py
db/ SQLAlchemy models + Alembic initial migration
- admin-web/ SvelteKit + Tailwind skeleton
- client/ Tauri 2 + Svelte skeleton (Mist placeholder UI)
- mistpipe/ click-based admin CLI with subcommand stubs
- docs/ ARCHITECTURE, DECISIONS (9 ADRs), RUNBOOK
- docker-compose.yml + dev overlay + .github/workflows
Bugs fixed during port:
- Routes/download.py:2 stray backslash on import line
- Utils/celery.py inspect.reserved() missing parens + double active() typo
- Hardcoded OneDrive/Desktop paths replaced with pydantic-settings config
- Discord webhook URL + RabbitMQ password moved to env vars
- Missing Tasks/__init__.py reconstructed as worker/__init__.py
Out of scope for this commit: route bodies, UI screens, mistpipe subcommand
bodies, real image builds.
141 lines
8.2 KiB
Markdown
141 lines
8.2 KiB
Markdown
# Mist — Architecture
|
||
|
||
## Purpose
|
||
|
||
A private Steam-clone for ~5–10 friends. Distributes games and updates with bandwidth-efficient delta patching. Sized to the actual problem — the complexity is in the delta-patching system, not in the deployment.
|
||
|
||
## Topology
|
||
|
||
Friends use a regular web/desktop client over the open internet. Public DNS resolves to a border server. Nginx Proxy Manager terminates TLS (Let's Encrypt) and reverse-proxies to the backend VM over Tailscale. The backend VM itself has no public exposure.
|
||
|
||
```
|
||
┌──────────────────────────────┐
|
||
Friend │ Tauri client (Svelte UI │
|
||
(open │ inside Rust shell) │
|
||
internet) └──────────────┬───────────────┘
|
||
│ HTTPS, public domain
|
||
│ store.mist.example
|
||
│ admin.mist.example
|
||
│ dl.mist.example
|
||
▼
|
||
┌─────────────────────────────────┐
|
||
│ Border server (public IP) │
|
||
│ Nginx Proxy Manager + Let's │
|
||
│ Encrypt TLS termination │
|
||
└────────────────┬────────────────┘
|
||
│ Tailscale (private backhaul)
|
||
▼
|
||
┌─────────────────────────────────────────────┐
|
||
│ Proxmox VM "mist" │
|
||
│ Docker Compose stack: │
|
||
│ │
|
||
│ ┌──────────────┐ ┌──────────────────┐ │
|
||
│ │ api │ │ admin-web │ │
|
||
│ │ FastAPI: │ │ static Svelte/ │ │
|
||
│ │ /auth │ │ TS served by │ │
|
||
│ │ /catalog │ │ tiny nginx │ │
|
||
│ │ /admin │ │ (calls api with │ │
|
||
│ │ /downloads │ │ admin JWT) │ │
|
||
│ │ /builds │ └──────────────────┘ │
|
||
│ └──────┬───────┘ │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌──────────────┐ │
|
||
│ │ worker │ │
|
||
│ │ Celery │ same image, │
|
||
│ │ delta-gen, │ different entrypoint │
|
||
│ │ archive prep,│ │
|
||
│ │ notifications│ │
|
||
│ └──────┬───────┘ │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌──────────┐ ┌────────┐ ┌─────────────┐ │
|
||
│ │ postgres │ │ redis │ │ rabbitmq │ │
|
||
│ └──────────┘ └────────┘ └─────────────┘ │
|
||
│ │
|
||
│ Volumes: │
|
||
│ - hot cache (.tar.zst archives) │
|
||
│ - postgres data │
|
||
│ - redis data │
|
||
│ - rabbitmq data │
|
||
│ - /mnt/nas → NFS to NAS (games) │
|
||
└────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Containers
|
||
|
||
| Container | Stack | Owns |
|
||
|---|---|---|
|
||
| **api** | FastAPI + SQLAlchemy + Postgres + Redis | Single web app with internally-modular code: `auth/`, `catalog/`, `admin/`, `downloads/`, `builds/`. Issues JWTs. Serves resumable downloads. Receives `mistpipe` uploads. Queues background work into RabbitMQ. |
|
||
| **worker** | Celery (same image as `api`, different entrypoint) | Consumes RabbitMQ. Runs the heavy stuff: hdiff delta generation, librsync indirect-delta generation, `chain_replay` cold reconstruction, `.tar.zst` archive packing, Discord notifications. |
|
||
| **admin-web** | SvelteKit, built static + tiny nginx | Admin UI. Calls `api/admin/*` with admin JWT. |
|
||
| **postgres** | postgres:16 | Catalog, users, build job state. |
|
||
| **redis** | redis:7 | Celery result backend, cache, ephemeral session data. |
|
||
| **rabbitmq** | rabbitmq:3.13-management | Celery broker, event bus for `notification.*` events. |
|
||
|
||
## Non-container artifacts
|
||
|
||
| Artifact | Stack | Notes |
|
||
|---|---|---|
|
||
| **client** | Tauri 2 (Rust shell + Svelte UI) | Friend-facing app. Distributed as a per-platform installer. Embeds (or spawns) the patch-application logic. |
|
||
| **mistpipe** | Python + click | Admin CLI. `login`, `new-game`, `push`, `ls`, `rm`, `resync-steam`. JWT stored in OS keychain. |
|
||
|
||
## Storage
|
||
|
||
**NAS** (mounted at `/mnt/nas` inside the VM via NFS) is the **source of truth** for game files:
|
||
|
||
```
|
||
/mnt/nas/mist/games/<Title>/
|
||
base_version.7z ← immutable original
|
||
depot/ ← current latest version's files
|
||
manifests/
|
||
<Title>.json ← ordered linear version list (legacy; will migrate to Postgres)
|
||
<version>.json ← per-version SHA-256 manifest
|
||
deltas/<version>/
|
||
delta_manifest.json
|
||
new_files/...
|
||
*.patch ← hdiff direct patches
|
||
```
|
||
|
||
**Docker volumes** on the VM hold the hot path:
|
||
|
||
- `cache-vol` — `.tar.zst` archives ready to serve, reconstructed historical versions
|
||
- `tmp-vol` — in-flight delta-gen working dirs
|
||
- `postgres-vol`, `redis-vol`, `rabbitmq-vol` — service data
|
||
|
||
## Update modes
|
||
|
||
Two delta strategies, decided at request time:
|
||
|
||
- **Direct update** = consecutive versions (`1.0.0.2` → `1.0.0.3`). Deltas were pre-generated by `hdiff` at push time. Serve from cache or zip on demand.
|
||
- **Indirect update** = arbitrary version jumps (`1.0.0.0` → `1.0.0.3`). Server tells client which files changed. Client generates `librsync` signatures of its local files, POSTs them. Worker generates `rdiff` deltas against the server's copy and packs them into a `.tar.zst`. Client applies with `rdiff patch`.
|
||
|
||
For arbitrary historical reconstructions the worker runs `chain_replay` — starts from the base or closest cached version and walks forward applying hdiff patches at each step, caching results for next time.
|
||
|
||
## Auth
|
||
|
||
- Username + password, you provision accounts via the admin portal
|
||
- Passwords hashed with argon2id
|
||
- JWT issued on login, scope claim distinguishes `user` from `admin`
|
||
- Admin scope required for `/admin/*` and `/builds/*`
|
||
- Per-game `is_private` boolean flag; non-admin users only see public games + games they've been explicitly granted (future)
|
||
|
||
## Catalog metadata
|
||
|
||
Steam appdetails pull-through: when an admin adds a game with an `app_id`, the catalog service fetches `https://store.steampowered.com/api/appdetails` and stores `short_description` + `header_image`. Admin can override either via hand-edited fields.
|
||
|
||
## Out of scope (for MVP)
|
||
|
||
- Branches (stable/beta/internal)
|
||
- Save sync, achievements, friend lists, in-app chat
|
||
- Payments
|
||
- Multi-tenancy (one store)
|
||
- Public self-serve signup
|
||
- Per-user entitlements beyond `is_private`
|
||
- Client-side delta-gen in `mistpipe` (server does it for MVP)
|
||
- High availability — single VM is fine at this scale; backup + restore covers failure
|
||
|
||
## Why this shape
|
||
|
||
The original "Josh Steam" prototypes proved out the hard parts: two-mode delta-patching, chain-replay for cold reconstruction, resumable downloads. The 2-year-later rebuild focuses on **finishing the product**, not re-exploring the design space. Microservices and k8s were considered and rejected for the actual scale — see `DECISIONS.md`.
|