The agent's API is carefully not a remote shell, but ec_agent_update.py:276,437-446 installs whatever ec-agent.py the controller serves, checks it only against a checksum from that same controller, and runs it as root about every 5 minutes. Code execution on the internet-facing controller therefore means root across the whole fleet.
Fix
Sign agent releases with a key that never lives on the controller
Verify the signature on the host before installing. Debian always ships gpgv (apt depends on it), so the agent can verify without leaving the standard library.
The agent's API is carefully not a remote shell, but `ec_agent_update.py:276,437-446` installs whatever `ec-agent.py` the controller serves, checks it only against a checksum from that same controller, and runs it as root about every 5 minutes. Code execution on the internet-facing controller therefore means root across the whole fleet.
**Fix**
- [x] Sign agent releases with a key that never lives on the controller
- [x] Verify the signature on the host before installing. Debian always ships `gpgv` (apt depends on it), so the agent can verify without leaving the standard library.
Reviewed 6f71822d41, a93ddb5ef9, 44e755a166. Agent releases are now signed offline and every host verifies the signature against a key from the repository, so the controller alone can no longer push code to hosts. Both fix items are done, so I'm closing this. Two deployment steps in the review notes still decide whether the fleet is actually protected.
Checklist
Sign releases with a key that never lives on the controller.sign_release.py makes a detached signature of the exact manifest bytes from a GnuPG home that must sit outside the checkout and release directory (sign_release.py:32). Builds never sign, and packaging drops a signature whose manifest changed (package_release.py:42). .dockerignore:13 keeps key material out of the image. The manifest now also covers the updater and installer checksums. The published signature (static/agent/manifest.json.sig) verifies with gpgv against agent/release-signing.gpg (ed25519, fingerprint A9A1C6FA5417D0CEF74887D1C5E9F75236788DFA). The manifest rebuilds byte for byte from the agent sources, so the packaging step in npm run build keeps that signature.
Verify the signature on the host before installing. The updater fetches the keyring from a fixed repository URL before every check (ec_agent_update.py:31, 319), with no redirects and no cached fallback. It then runs gpgv with an isolated home and only that keyring (ec_agent_update.py:200). The exact downloaded bytes are verified before the JSON is parsed (ec_agent_update.py:326-328), and the agent's checksum comes from that signed manifest. The installer does the same before installing anything (install.sh:170-232), and also checks the updater's checksum (install.sh:222). The enrollment command now downloads install.sh from the repository instead of the controller (http.ts:70-74). The key and installer served at those URLs match HEAD byte for byte.
Review notes
Good extra catches: the updater still refuses older releases, so a replayed signed manifest can't downgrade a host (ec_agent_update.py:488). Removing a key from the repository keyring takes effect on every host's next check, and a test covers that rotation. The docs spell out the trust model and its limits (native-hosts.md:46-85).
I also checked the published release against tampering. Changing its checksum or its version in the manifest makes the signature check fail. The real-GnuPG tests cover a manifest signed by an unknown key.
The agent tests pass locally (323, including the real-GnuPG signing tests), and so do the 3 affected TypeScript test files (119 tests). CI run 845 on this commit is green.
Important, not blocking: hosts that are already enrolled stay exposed until the installer is rerun on each one (native-hosts.md:219-222). Their old protocol 1 updater defers on this protocol 2 release and asks for a rerun, but a compromised controller can still serve it a protocol 1 manifest with any checksum.
Important, not blocking: the trust now rests on the repository's main branch, and GET /branches/main reports "protected": false. Anyone with write access can replace the key or the installer, and that includes this Goddard account ("user_can_push": true). native-hosts.md:70-73 already says these must stay outside the controller's reach. Protecting main so that only you can push would also keep a leaked Goddard or CI credential from becoming root on hosts. I couldn't check the protection rules themselves, because /branch_protections returns 403 for Goddard.
Minor and non-blocking: the installer doesn't compare the signed release with the agent that's already installed (install.sh:186-232). Rerunning it against a compromised controller could install an older signed release. Only 1.8.7 is signed so far, so this has no effect yet.
Reviewed 6f71822d41dd2b1c5e35bf3dbb841dd85556c71b, a93ddb5ef9fb281fda21ffdf24f4c82f0b5bf96e, 44e755a166df6a02b084dcdbb5d5427ad9ee1453. Agent releases are now signed offline and every host verifies the signature against a key from the repository, so the controller alone can no longer push code to hosts. Both fix items are done, so I'm closing this. Two deployment steps in the review notes still decide whether the fleet is actually protected.
**Checklist**
- [x] **Sign releases with a key that never lives on the controller.** `sign_release.py` makes a detached signature of the exact manifest bytes from a GnuPG home that must sit outside the checkout and release directory (`sign_release.py:32`). Builds never sign, and packaging drops a signature whose manifest changed (`package_release.py:42`). `.dockerignore:13` keeps key material out of the image. The manifest now also covers the updater and installer checksums. The published signature (`static/agent/manifest.json.sig`) verifies with `gpgv` against `agent/release-signing.gpg` (ed25519, fingerprint `A9A1C6FA5417D0CEF74887D1C5E9F75236788DFA`). The manifest rebuilds byte for byte from the agent sources, so the packaging step in `npm run build` keeps that signature.
- [x] **Verify the signature on the host before installing.** The updater fetches the keyring from a fixed repository URL before every check (`ec_agent_update.py:31`, `319`), with no redirects and no cached fallback. It then runs `gpgv` with an isolated home and only that keyring (`ec_agent_update.py:200`). The exact downloaded bytes are verified before the JSON is parsed (`ec_agent_update.py:326-328`), and the agent's checksum comes from that signed manifest. The installer does the same before installing anything (`install.sh:170-232`), and also checks the updater's checksum (`install.sh:222`). The enrollment command now downloads `install.sh` from the repository instead of the controller (`http.ts:70-74`). The key and installer served at those URLs match `HEAD` byte for byte.
**Review notes**
- Good extra catches: the updater still refuses older releases, so a replayed signed manifest can't downgrade a host (`ec_agent_update.py:488`). Removing a key from the repository keyring takes effect on every host's next check, and a test covers that rotation. The docs spell out the trust model and its limits (`native-hosts.md:46-85`).
- I also checked the published release against tampering. Changing its checksum or its version in the manifest makes the signature check fail. The real-GnuPG tests cover a manifest signed by an unknown key.
- The agent tests pass locally (323, including the real-GnuPG signing tests), and so do the 3 affected TypeScript test files (119 tests). CI [run 845](https://gitea.thewrightserver.net/josh/electric-cherry/actions/runs/845) on this commit is green.
- Important, not blocking: hosts that are already enrolled stay exposed until the installer is rerun on each one (`native-hosts.md:219-222`). Their old protocol 1 updater defers on this protocol 2 release and asks for a rerun, but a compromised controller can still serve it a protocol 1 manifest with any checksum.
- Important, not blocking: the trust now rests on the repository's `main` branch, and `GET /branches/main` reports `"protected": false`. Anyone with write access can replace the key or the installer, and that includes this Goddard account (`"user_can_push": true`). `native-hosts.md:70-73` already says these must stay outside the controller's reach. Protecting `main` so that only you can push would also keep a leaked Goddard or CI credential from becoming root on hosts. I couldn't check the protection rules themselves, because `/branch_protections` returns 403 for Goddard.
- Minor and non-blocking: the installer doesn't compare the signed release with the agent that's already installed (`install.sh:186-232`). Rerunning it against a compromised controller could install an older signed release. Only 1.8.7 is signed so far, so this has no effect yet.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The agent's API is carefully not a remote shell, but
ec_agent_update.py:276,437-446installs whateverec-agent.pythe controller serves, checks it only against a checksum from that same controller, and runs it as root about every 5 minutes. Code execution on the internet-facing controller therefore means root across the whole fleet.Fix
gpgv(apt depends on it), so the agent can verify without leaving the standard library.Reviewed
6f71822d41,a93ddb5ef9,44e755a166. Agent releases are now signed offline and every host verifies the signature against a key from the repository, so the controller alone can no longer push code to hosts. Both fix items are done, so I'm closing this. Two deployment steps in the review notes still decide whether the fleet is actually protected.Checklist
sign_release.pymakes a detached signature of the exact manifest bytes from a GnuPG home that must sit outside the checkout and release directory (sign_release.py:32). Builds never sign, and packaging drops a signature whose manifest changed (package_release.py:42)..dockerignore:13keeps key material out of the image. The manifest now also covers the updater and installer checksums. The published signature (static/agent/manifest.json.sig) verifies withgpgvagainstagent/release-signing.gpg(ed25519, fingerprintA9A1C6FA5417D0CEF74887D1C5E9F75236788DFA). The manifest rebuilds byte for byte from the agent sources, so the packaging step innpm run buildkeeps that signature.ec_agent_update.py:31,319), with no redirects and no cached fallback. It then runsgpgvwith an isolated home and only that keyring (ec_agent_update.py:200). The exact downloaded bytes are verified before the JSON is parsed (ec_agent_update.py:326-328), and the agent's checksum comes from that signed manifest. The installer does the same before installing anything (install.sh:170-232), and also checks the updater's checksum (install.sh:222). The enrollment command now downloadsinstall.shfrom the repository instead of the controller (http.ts:70-74). The key and installer served at those URLs matchHEADbyte for byte.Review notes
ec_agent_update.py:488). Removing a key from the repository keyring takes effect on every host's next check, and a test covers that rotation. The docs spell out the trust model and its limits (native-hosts.md:46-85).native-hosts.md:219-222). Their old protocol 1 updater defers on this protocol 2 release and asks for a rerun, but a compromised controller can still serve it a protocol 1 manifest with any checksum.mainbranch, andGET /branches/mainreports"protected": false. Anyone with write access can replace the key or the installer, and that includes this Goddard account ("user_can_push": true).native-hosts.md:70-73already says these must stay outside the controller's reach. Protectingmainso that only you can push would also keep a leaked Goddard or CI credential from becoming root on hosts. I couldn't check the protection rules themselves, because/branch_protectionsreturns 403 for Goddard.install.sh:186-232). Rerunning it against a compromised controller could install an older signed release. Only 1.8.7 is signed so far, so this has no effect yet.