alice.pds.demo.boltless.dev
This repository has no description
1# spindle microVM engine
2
3This document describes the architecture of the microvm engine for spindle. In
4short it allows the spindle to spin up microvm guests, and implements a guest
5[agent protocol](../../agentproto) for communicating with those guests (via the
6[shuttle](../../../shuttle) implementation of that proto). It implements some
7fairly simple resource budgeting and optionally sets up cgroups for better
8enforcing resource limits, and hardens the VM network access. It has Nix cache
9integration for any paths built in the VM, those will get pushed to a Nix cache
10by the spindle (if one is configured). The runner is abstracted behind an
11interface; right now only the QEMU microVM impl is supported, but others (e.g.
12firecracker) can slot in later.
13
14Currently two kinds of images are supported:
15
16- NixOS images: these allow configuration such as `dependencies`, `services`,
17 `virtualisation`, `registry`, `caches` in the workflow file itself. The guest
18 agent will build (or if it's cached, spindle will send the store path for
19 realization) and activate it before any workflow steps are ran.
20- Non-NixOS: this is mainly just Alpine for now, but can be anything else.
21 Workflow-level configuration like NixOS aren't supported while using these. If
22 Nix exists inside the image (like in our Alpine image) it will still be able
23 to make use of the spindle cache.
24
25(For testing, you can run `bash spindle/engines/microvm/test-spindle-microvm.sh`
26from repo root. These test the Alpine & NixOS, and features like if Docker
27works, public internet is reachable, and so on.)
28
29## Image builds
30
31Image builds right now are done via Nix:
32
33- For NixOS, we use [microvm.nix](https://github.com/microvm-nix/microvm.nix),
34 and layer our own configs on-top, see [here](../../../nix/microvm).
35- For Alpine we have a small-ish Nix definition that includes fetching the
36 kernel, initrd, kernel modules; setting up the init script that configures the
37 VM proper; copying dependencies (like `nix` or `git`) into a rootfs and
38 creating a squashfs from it.
39
40This does not mean it *has* to be done via Nix, as long as your images are what
41spindle expects, they should work. That is:
42- a guest agent is present inside of the image and when that image boots it will
43 get started,
44- the `spindle-workflow` user exists, is unprivileged (non-zero uid/gid), and has
45 a usable login shell and home dir set in the image's passwd: workflow steps run
46 as this user, and the debug shell (see below) launches its passwd shell as a
47 login shell in its home dir. an unset or `nologin`/`false` shell breaks debug
48 ssh,
49- and the work directory is configured (`/workspace`, with `/workspace/repo` as
50 the per-step working dir).
51
52## Image discovery
53
54Each built image ships with a `spec.json` next to its artifacts. This spec
55describes everything needed to run the image: the kernel, initrd and read-only
56store disk paths, boot args, memory/vCPU sizing, the shell used for workflow
57steps, writable volumes, network interfaces, and runner-specific config (machine
58type, CPU, extra args for QEMU). NixOS images also carry a `baseConfigHash`
59identifying the base configuration baked into the image.
60
61An image lives in the configured image directory either as a directory
62containing a `spec.json` (alongside the kernel/initrd/store-disk artifacts) or,
63for a self-contained spec, as a flat `<name>.json` file. An operator keeping
64multiple arches side by side can name them `<name>-<arch>` (eg. `nixos-x86_64`,
65`alpine-aarch64`); that arch suffix is just part of the name, not something
66resolution infers.
67
68A workflow names an image with the `image` key at top-level (falling back to
69`SPINDLE_MICROVM_PIPELINES_DEFAULT_IMAGE` if unset). The name is matched
70literally: we look for `<name>` (a directory with a `spec.json`) then
71`<name>.json`. Resolution depends only on the name and what is on disk, never on
72the host, so the same workflow resolves identically on every spindle. If for
73example an operator wants `nixos` to work, they can symlink `nixos` to
74`nixos-x86_64`.
75
76The spec is validated at resolve time (required fields, positive sizes etc.),
77and right before launch we also check the referenced files actually exist on
78disk and that the host has the commands we need: `mkfs.ext4` for volume
79formatting, plus whatever the selected runner requires. For QEMU that's the QEMU
80binary for the spec's arch, `/dev/vhost-vsock` (for guest vsock configuration),
81`/dev/vsock` (for host listener sockets), `/dev/kvm` (if KVM is enabled), `/dev/net/tun`
82(for guest networking), and the `ip`, `mount`, `slirp4netns`, `unshare` toolchain when the
83image has network interfaces.
84
85## microVM lifecycle
86
87```mermaid
88flowchart LR
89 Init["InitWorkflow<br/><small>parse manifest, resolve image, build steps</small>"]
90 Acquire["AcquireWorkflowSlot<br/><small>queue until resources fit budget</small>"]
91 Setup["SetupWorkflow<br/><small>proxies, VM, agent handshake</small>"]
92 Run["RunStep ×N<br/><small>exec via agent</small>"]
93 Destroy["DestroyWorkflow<br/><small>drain cache, poweroff, cleanup</small>"]
94
95 Init --> Acquire --> Setup --> Run --> Destroy
96```
97
98While a workflow is running, things look like this (everything inside the cgroup
99box is what gets resource-limited):
100
101```mermaid
102flowchart LR
103 subgraph Host["spindle host"]
104 Hub["agent hub"]
105 ReadProxy["read cache proxy"]
106 UploadProxy["upload cache proxy"]
107 subgraph Cgroup["per-workflow cgroup"]
108 QEMU["qemu"]
109 Slirp["slirp4netns"]
110 end
111 end
112
113 subgraph Guest["guest"]
114 Agent["guest agent"]
115 end
116
117 Agent -->|"vsock"| Hub
118 Agent -->|substitutions| ReadProxy
119 Agent -->|built paths| UploadProxy
120 QEMU --- Guest
121 Slirp -->|outbound only| Internet["the internet"]
122 ReadProxy --> Substituters["upstream caches"]
123 UploadProxy --> NixCache["spindle nix cache"]
124```
125
126`InitWorkflow` parses the workflow manifest, resolves the image, and assembles
127the step list: the clone step first, then (for NixOS images with a workflow
128config) a "NixOS config activation" system step, then the user steps. Before any
129of this actually runs the workflow has to acquire a slot from the resource
130scheduler, each image declares its memory/vCPUs/disk and workflows queue until
131their request fits within the configured budget. The scheduler is
132work-conserving with aging and per-user fairness, so one user submitting a pile
133of jobs won't starve everyone else, and slots don't sit idle while there's
134queued work that fits in the budget.
135
136### Configuration
137
138Setup allocates a random vsock CID for the guest and registers it with the agent
139hub, which listens on a single host vsock port. Incoming agent connections are
140matched to workflows by CID, anything with an unknown CID is dropped. It then
141creates a per-workflow work directory and starts three host-side proxies the guest
142reaches over vsock: a read cache proxy (fronting the configured Nix substituters
143plus any workflow-level `caches`) and an upload cache proxy (for pushing paths
144built in the guest to the spindle's cache), plus a DNS proxy that resolves
145through the host's resolver and filters private/special-purpose address answers.
146
147Then the VM itself. Writable volumes from the spec are created as sparse files
148and formatted ext4, the store disk is attached read-only. QEMU runs with
149`-sandbox on`, `-nodefaults`, no display/monitor, etc., serial output to a log
150file, and a QMP socket for control.
151
152For network hardening: if the image has network interfaces, QEMU doesn't run in
153the host network namespace at all. We `unshare` into fresh user/net/mount
154namespaces, and a small wrapper script inside the namespace bind-mounts a
155resolv.conf that disables qemu's slirp DNS and adds blackhole routes for every
156special-use IPv4/IPv6 range (RFC 6890, so private networks, link-local,
157loopback, CGNAT, multicast, ULAs and so on) before exec'ing QEMU. `slirp4netns`
158(with `--disable-host-loopback`, sandbox and seccomp enabled) then provides
159outbound connectivity for the namespace. The guest's `/etc/resolv.conf` points
160at shuttle on localhost; shuttle forwards DNS packets over vsock to the
161host-side DNS proxy. The guest sits behind a second layer of QEMU user-mode
162networking inside that namespace, so guest traffic can only ever reach the
163outside world, never the host or anything on its local networks.
164
165Optionally the whole thing (QEMU and slirp4netns) is placed in a per-workflow
166cgroup with memory, swap and pids limits, so the budget above is actually
167enforced and not just bookkeeping. That also allows us to, for example, if the
168cgroup OOM-kills the VM we can detect that and report it as such instead of a
169generic crash. The spindle supervisor itself also gets a cgroup with a
170protected `memory.min`, so under host memory pressure it's the workflows that
171get OOM-killed first, not spindle.
172
173### Boot - run - death
174
175Once QEMU is up we poll the QMP socket until it accepts a connection and reports
176the guest as running, then wait for the guest agent to send handshake message
177over vsock from the expected CID. It reports its protocol and versions, and
178spindle sends it the job id, trusted cache public keys, and the cache/DNS proxy
179ports.
180
181First the activation step is ran (if on a NixOS image and the workflow is
182configured with anything), spindle sends the user config (or a cached toplevel
183store path, if we've built this exact base + config combo before) and the agent
184builds and activates it before the user steps run. Afterwards, each step is sent
185as an exec request (`$shell -lc <command>` as an unprivileged workflow user in
186`/workspace/repo`, with workflow/step environment and unlocked secrets), and
187stdout/stderr stream back as messages until an exit message arrives. Timeouts
188are cooperative: we derive a deadline from the workflow timeout and ship it to
189the guest, with a little grace on the host side so the guest gets to report the
190timeout itself. While a step runs we also watch for the VM crashing, if it does
191we tail the serial (and qemu) logs into the step's stderr so you get something
192more useful than "guest agent connection lost: EOF".
193
194Teardown is same whether the workflow succeeded, failed or timed out: drain the
195guest's pending Nix cache uploads, ask the agent to power off and wait for QEMU
196to exit (falling back to QMP `system_powerdown` and finally a kill if it
197doesn't), then close the proxies and remove the work directory. For non-HTTP
198upload targets the host-side import already happened synchronously when the
199guest committed each narinfo, so there is no second host-side cache drain step
200at teardown.
201
202### Nix cache
203
204The two host-side proxies are how the guest talks to spindle's Nix cache without
205ever needing credentials or direct network access; like the agent they reach the
206host over vsock.
207
208The read proxy fronts the configured substituters plus any workflow-level
209`caches`. When the guest needs to realize a store path it asks the proxy, which
210queries the read caches concurrently and returns the first successful response,
211with a 404 only winning if every upstream returns 404.
212
213The upload proxy goes the other way: paths built inside the guest are pushed to
214spindle's configured upload cache (if any) so the next workflow that needs them
215doesn't rebuild. Paths already present on any configured read cache are skipped.
216
217For `http://` and `https://` upload targets the proxy just reverse-proxies the
218guest's binary-cache upload traffic to the configured remote cache, while still
219answering narinfo existence checks across the upload target plus the read
220caches.
221
222For `ssh://`, `ssh-ng://`, `daemon`, and `local` targets spindle implements the
223small HTTP binary-cache upload surface itself. It stages uploaded `nar/` objects
224and narinfos under the workflow workdir, validates the narinfo, then treats the
225narinfo upload as the commit point: once `<hash>.narinfo` is written spindle
226runs:
227
228```bash
229nix copy \
230 --from file://<staging-dir> \
231 --to <target-store> \
232 --no-check-sigs \
233 --substitute-on-destination \
234 <store-path>
235```
236
237That copy is synchronous. If it fails, spindle removes the staged narinfo again
238so future `GET`/`HEAD <hash>.narinfo` requests do not falsely dedupe a path that
239never made it to the destination store. The guest still only ever sees the same
240HTTP binary-cache upload protocol over vsock; it never gets direct access to
241SSH credentials or the destination store itself.
242
243### Debug ssh
244
245When a workflow fails, spindle can keep its microVM alive for a configured grace
246window (`MicroVMPipelines.SSH`) and print an `ssh` invocation so you can poke at
247the failed VM interactively. Spindle terminates the ssh connection itself and
248bridges a pty into the live guest over the agent's vsock; the guest stays
249keyless and never runs an ssh daemon.
250
251Access mirrors a git push: the ssh username is the job id, and the offered
252public key is sent to the job's repo knot (`sh.tangled.repo.checkPushAllowed`).
253The session is accepted only if that key is allowed to push to the job's repo.
254
255The shell is deliberately not configurable from either end. It always:
256- runs as the `spindle-workflow` user (the ssh username selects the *job*, not a
257 unix user),
258- uses that user's login shell from the image's passwd, launched as a login
259 shell (`-l`), and
260- starts in the dir where the repo was cloned to.
261
262The only things the client influences are the terminal type and window size
263(forwarded from the ssh pty request, and on resize). This relies on the image
264configuring `spindle-workflow` properly per the expectations above; in
265particular a missing or `nologin`/`false` won't work of course.