Mist/docs/ARCHITECTURE.md

Mist — Architecture

Purpose

A private Steam-clone for ~5–10 friends. Distributes games and updates with bandwidth-efficient delta patching. Sized to the actual problem — the complexity is in the delta-patching system, not in the deployment.

Topology

Friends use a regular web/desktop client over the open internet. Public DNS resolves to a border server. Nginx Proxy Manager terminates TLS (Let's Encrypt) and reverse-proxies to the backend VM over Tailscale. The backend VM itself has no public exposure.

            ┌──────────────────────────────┐
   Friend   │ Tauri client (Svelte UI      │
   (open    │  inside Rust shell)          │
  internet) └──────────────┬───────────────┘
                           │ HTTPS, public domain
                           │ store.mist.example
                           │ admin.mist.example
                           │ dl.mist.example
                           ▼
            ┌─────────────────────────────────┐
            │ Border server (public IP)       │
            │ Nginx Proxy Manager + Let's     │
            │ Encrypt TLS termination         │
            └────────────────┬────────────────┘
                             │ Tailscale (private backhaul)
                             ▼
       ┌─────────────────────────────────────────────┐
       │ Proxmox VM "mist"                           │
       │   Docker Compose stack:                     │
       │                                             │
       │   ┌──────────────┐    ┌──────────────────┐ │
       │   │ api          │    │ admin-web        │ │
       │   │ FastAPI:     │    │ static Svelte/   │ │
       │   │ /auth        │    │ TS served by     │ │
       │   │ /catalog     │    │ tiny nginx       │ │
       │   │ /admin       │    │ (calls api with  │ │
       │   │ /downloads   │    │  admin JWT)      │ │
       │   │ /builds      │    └──────────────────┘ │
       │   └──────┬───────┘                         │
       │          │                                 │
       │          ▼                                 │
       │   ┌──────────────┐                         │
       │   │ worker       │                         │
       │   │ Celery       │   same image,           │
       │   │ delta-gen,   │   different entrypoint  │
       │   │ archive prep,│                         │
       │   │ notifications│                         │
       │   └──────┬───────┘                         │
       │          │                                 │
       │          ▼                                 │
       │   ┌──────────┐ ┌────────┐ ┌─────────────┐ │
       │   │ postgres │ │ redis  │ │ rabbitmq    │ │
       │   └──────────┘ └────────┘ └─────────────┘ │
       │                                            │
       │   Volumes:                                 │
       │     - hot cache (.tar.zst archives)        │
       │     - postgres data                        │
       │     - redis data                           │
       │     - rabbitmq data                        │
       │     - /mnt/nas → NFS to NAS (games)        │
       └────────────────────────────────────────────┘

Containers

Container	Stack	Owns
api	FastAPI + SQLAlchemy + Postgres + Redis	Single web app with internally-modular code: auth/, catalog/, admin/, downloads/, builds/. Issues JWTs. Serves resumable downloads. Receives mistpipe uploads. Queues background work into RabbitMQ.
worker	Celery (same image as api, different entrypoint)	Consumes RabbitMQ. Runs the heavy stuff: hdiff delta generation, librsync indirect-delta generation, chain_replay cold reconstruction, .tar.zst archive packing, Discord notifications.
admin-web	SvelteKit, built static + tiny nginx	Admin UI. Calls api/admin/* with admin JWT.
postgres	postgres:16	Catalog, users, build job state.
redis	redis:7	Celery result backend, cache, ephemeral session data.
rabbitmq	rabbitmq:3.13-management	Celery broker, event bus for notification.* events.

Non-container artifacts

Artifact	Stack	Notes
client	Tauri 2 (Rust shell + Svelte UI)	Friend-facing app. Distributed as a per-platform installer. Embeds (or spawns) the patch-application logic.
mistpipe	Python + click	Admin CLI. login, new-game, push, ls, rm, resync-steam. JWT stored in OS keychain.

Storage

NAS (mounted at /mnt/nas inside the VM via NFS) is the source of truth for game files:

/mnt/nas/mist/games/<Title>/
  base_version.7z              ← immutable original
  depot/                       ← current latest version's files
  manifests/
    <Title>.json               ← ordered linear version list (legacy; will migrate to Postgres)
    <version>.json             ← per-version SHA-256 manifest
  deltas/<version>/
    delta_manifest.json
    new_files/...
    *.patch                    ← hdiff direct patches

Docker volumes on the VM hold the hot path:

• cache-vol — .tar.zst archives ready to serve, reconstructed historical versions
• tmp-vol — in-flight delta-gen working dirs
• postgres-vol, redis-vol, rabbitmq-vol — service data

Update modes

Two delta strategies, decided at request time:

• Direct update = consecutive versions (1.0.0.2 → 1.0.0.3). Deltas were pre-generated by hdiff at push time. Serve from cache or zip on demand.
• Indirect update = arbitrary version jumps (1.0.0.0 → 1.0.0.3). Server tells client which files changed. Client generates librsync signatures of its local files, POSTs them. Worker generates rdiff deltas against the server's copy and packs them into a .tar.zst. Client applies with rdiff patch.

For arbitrary historical reconstructions the worker runs chain_replay — starts from the base or closest cached version and walks forward applying hdiff patches at each step, caching results for next time.

Auth

• Username + password, you provision accounts via the admin portal
• Passwords hashed with argon2id
• JWT issued on login, scope claim distinguishes user from admin
• Admin scope required for /admin/* and /builds/*
• Per-game is_private boolean flag; non-admin users only see public games + games they've been explicitly granted (future)

Catalog metadata

Steam appdetails pull-through: when an admin adds a game with an app_id, the catalog service fetches https://store.steampowered.com/api/appdetails and stores short_description + header_image. Admin can override either via hand-edited fields.

Out of scope (for MVP)

• Branches (stable/beta/internal)
• Save sync, achievements, friend lists, in-app chat
• Payments
• Multi-tenancy (one store)
• Public self-serve signup
• Per-user entitlements beyond is_private
• Client-side delta-gen in mistpipe (server does it for MVP)
• High availability — single VM is fine at this scale; backup + restore covers failure

Why this shape

The original "Josh Steam" prototypes proved out the hard parts: two-mode delta-patching, chain-replay for cold reconstruction, resumable downloads. The 2-year-later rebuild focuses on finishing the product, not re-exploring the design space. Microservices and k8s were considered and rejected for the actual scale — see DECISIONS.md.