alice.pds.demo.boltless.dev
This repository has no description
1---
2atroot: true
3template:
4slug: spindle-microvm
5title: Spindle's new microVM engine
6subtitle: How we built the new QEMU-based microVM engine
7date: 2026-06-16
8image: https://assets.tangled.network/blog/microvm.png
9authors:
10 - name: dawn
11 email: dawn@tangled.org
12 handle: ptr.pet
13---
14
15Spindle gains a second engine: `microvm`. Each workflow gets its own little
16virtual machine, a whole real environment you can do anything inside. It's an
17upgrade from the Nixery engine while staying fully compatible with it, so if you
18already have a working Nixery workflow, just change `nixery` to `microvm` and it
19will work!
20
21The interesting part is NixOS images: you configure the machine directly from
22the workflow file. A few things you can do:
23
24You can bring services up:
25
26```yaml
27services:
28 postgresql:
29 enable: true
30 ensureDatabases: ["spindle-workflow"]
31 ensureUsers:
32 - name: spindle-workflow
33 ensureDBOwnership: true
34```
35
36You can build Docker containers:
37
38```yaml
39virtualisation:
40 docker: true
41steps:
42 - name: "do the thing!"
43 command: docker build ...
44```
45
46And you can use non-NixOS images too:
47
48```yaml
49image: alpine
50steps:
51 - name: install golang
52 command: apk add go
53```
54
55It's quick on the second run, too, because it caches aggressively: your
56dependencies, your services, and any other Nix derivation built inside the
57microVM get pushed to spindle's Nix cache, so the next workflow that needs them
58doesn't rebuild those. More on that [below](#the-nix-cache-both-ways).
59
60And like everything else in tangled, the whole thing is self-hostable, so you
61can run your own spindle with the microVM engine on your own hardware (see the
62[self-hosting
63guide](https://docs.tangled.org/spindles.html#self-hosting-guide)). If you want
64fuller examples, there are [recipes in the
65docs](https://docs.tangled.org/spindles.html#recipes) too.
66
67## What's in a microVM
68
69A microVM is just a VM with most of the boring parts removed. There's no BIOS,
70no PCI bus to probe, no emulated graphics card, none of the slow legacy stuff a
71normal QEMU machine drags along. You get virtio devices and not much
72else, which means it boots very quickly and uses very little memory. Right now
73QEMU is the only runner we support, but the engine is written so that other
74runners (firecracker for example) can slot in later.
75
76Inside the guest there's a small piece of software we call the agent. Spindle
77never SSHes in or runs commands "from the outside"; instead the agent dials back
78to spindle over vsock the moment it boots, says hello, and from then on every
79step of your workflow is sent to it as a message. The agent runs the command as
80an unprivileged user, streams stdout and stderr back, and reports the exit code.
81The host side of this lives in
82[`spindle`](https://tangled.org/tangled.org/core/tree/master/spindle/engines/microvm/agent.go)
83and the guest side is a little Rust binary called
84[`shuttle`](https://tangled.org/tangled.org/core/tree/master/shuttle).
85(`shuttle` implements
86[`agentproto`](https://tangled.org/tangled.org/core/tree/master/spindle/) which
87is the protocol used by `spindle`. Technically speaking anyone could implement
88this and, assuming side effects hold, you could have your own agent!)
89
90
91
92## Two kinds of images
93
94There are two "flavours" of image you can boot, and they're aimed at fairly
95different people.
96
97The first is **NixOS images**. These are the interesting ones: because the whole
98guest is built with Nix, you can configure it from your workflow file directly.
99Things like `dependencies`, `services`, `virtualisation` (e.g. Docker),
100`registry` and `caches` are all written right there in the YAML, and the guest
101agent builds and activates that config before any of your steps run. If we've
102built that exact base plus config before, spindle can just hand the guest a
103store path to realize (fetching from whatever cache `spindle` has configured)
104instead of rebuilding it, so the second run is quick.
105
106The second is **non-NixOS images**, which today just means Alpine, but can be
107anything. You don't get the workflow-level NixOS config here (there's no NixOS
108to configure), but if Nix happens to exist inside the image, like it does in our
109Alpine one, it can still talk to the spindle Nix cache just fine.
110
111## An example NixOS workflow
112
113If you've used spindle before, this will look familiar: it's the same manifest
114you already know, just with a few extra keys that the NixOS image understands.
115Here's a workflow that needs Postgres to test against and Docker to build an
116image:
117
118```yaml
119# .tangled/workflows/test.yaml
120engine: microvm
121
122when:
123 - event: ["push", "pull_request"]
124 branch: ["master"]
125
126image: nixos
127
128dependencies:
129 - go
130 - github:nixos/nixpkgs#hello
131
132registry:
133 nixpkgs: github:nixos/nixpkgs/nixos-unstable
134
135caches:
136 https://nix-community.cachix.org: "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
137
138services:
139 postgresql:
140 enable: true
141 ensureDatabases: ["spindle-workflow"]
142 ensureUsers:
143 - name: spindle-workflow
144 ensureDBOwnership: true
145
146virtualisation:
147 docker: true
148
149steps:
150 - name: run tests
151 environment:
152 PGHOST: /run/postgresql
153 command: |
154 docker build -t app .
155 psql -c "select 1"
156 go test ./...
157```
158
159The new keys each do one job:
160
161- **`dependencies`** are the packages your steps get to use. They go into a
162 `mkShellNoCC` devshell that every step sources before it runs, so you get the
163 whole stdenv environment (setup hooks like `pkg-config` wiring up
164 `PKG_CONFIG_PATH`, etc.) and not just the bare binaries. That means you can
165 use a dependency like `openssl` and compile the `openssl-sys` Rust crate
166 without pain! A bare name like `go` is looked up in nixpkgs (same as Nixery),
167 but you can also point at any flake with the `flakeref#attr` syntax, so
168 `github:nixos/nixpkgs#hello` pulls `hello` straight out of that flake.
169- **`registry`** is how you remap the global refs. Here we pin `nixpkgs` to
170 `nixos-unstable`, so now the bare `go` above resolves from unstable. You can
171 alias your own flakes the same way (`myflake: github:me/x`, then
172 `myflake#tool` in `dependencies`).
173- **`caches`** is a map of binary cache URL to its trusted public key. They get
174 wired into the read proxy (more on that just below), so the guest can
175 substitute prebuilt paths from them instead of building everything from
176 scratch.
177
178`services` and `virtualisation` are the interesting parts: they're passed
179straight through to NixOS, so anything you could write in a NixOS config you can
180write here. `services.postgresql.enable` brings Postgres up before any of your
181steps run.
182
183Since steps run as the `spindle-workflow` user, naming a database after that
184user with `ensureDBOwnership` is the easy path to a working DB -- Postgres peer
185auth maps the unix user straight to the matching role, so `psql` connects over
186the socket with no password and no extra setup (this name-matching is a NixOS
187requirement for `ensureDBOwnership`, if you want a differently named DB you'd
188grant access yourself).
189
190`virtualisation.docker: true` is shorthand for `virtualisation.docker.enable =
191true`, which gets you a real Docker daemon inside the VM. By the time your first
192step runs, Postgres is listening and the Docker socket is there, no sidecar
193dance, it's just part of the machine.
194
195(`true` works as shorthand for `.enable = true` anywhere an `enable` option
196exists, so most "just turn this on" services are a one-liner!)
197
198## The architecture
199
200
201
202### Nix cache, both ways
203
204Spindle talks to its Nix cache through two proxies that run on the host, so the
205guest never needs credentials or direct network access to reach it. Like the
206agent, they use vsock to talk to spindle.
207
208The read proxy fans out to the configured substituters plus any caches you
209listed in your workflow, so when the guest needs to realize a store path it asks
210the proxy and the proxy fetches it. The request is sent concurrently to the read
211caches, so the one that answers it first wins.
212
213The upload proxy goes the other way: any path built inside the guest gets pushed
214back out to spindle's Nix cache (if one is configured), so the next workflow
215that needs it doesn't have to build it again. Any paths that already exist on
216any of the configured read caches won't be uploaded. As the agent reports
217built paths, they're queued and uploaded in the background while the rest of the
218workflow keeps running, so uploads overlap with work instead of blocking it. If
219any are still in flight when we reach VM teardown, the workflow waits until
220everything has drained.
221
222Spindle can be configured to use `http`, `ssh-ng` or `ssh` URLs as a binary
223cache to upload to, so for example, `ssh-ng://localhost` would just upload to
224the local Nix store on the machine that the spindle runs on! `ssh-ng` and `ssh`
225require Nix to be present in PATH so that the spindle can use `nix copy` to
226upload to them, but if you are using a binary cache that supports `http` (for
227example, [ncps](https://github.com/kalbasit/ncps)) Nix does not need to be
228present.
229
230### Building the images
231
232Image builds are done with Nix. For NixOS we lean on
233[microvm.nix](https://github.com/microvm-nix/microvm.nix) and layer our own bits
234on top (stripping down kernel modules, configuring users, etc.). For Alpine
235there's a smallish Nix definition that fetches the kernel, the initrd and the
236kernel modules, sets up an init script that configures the machine on boot,
237copies in the dependencies we want (`nix`, `git`, etc.) and compresses the whole
238rootfs into a squashfs.
239
240None of this *has* to be Nix, though. As far as spindle is concerned an image is
241valid as long as a few things hold: a guest agent (that implements `agentproto`)
242is present and gets started on boot, a `spindle-workflow` user exists, and the
243work directory is set up at `/workspace`. That can be built however you like.
244
245### Finding an image
246
247Every built image ships a `spec.json` next to its artifacts. The spec is the
248whole contract: where the kernel and initrd and read-only store disk live, the
249boot args, how much memory and how many vCPUs to give it, the shell to run steps
250in, the writable volumes, the network interfaces, and the runner-specific knobs
251(machine type, CPU, extra QEMU args). NixOS images also carry a `baseConfigHash`
252identifying the base config baked in (this is the hash of
253`nixosSystem.config.system.build.toplevel.outPath`).
254
255A workflow picks an image with the `image` key at the top level. The name is
256matched literally against what's on disk, we look for a directory called
257`<name>` with a `spec.json` in it, then fall back to a flat `<name>.json`. The
258nice property here is that resolution depends *only* on the name and what's on
259disk, never on the host doing the resolving, so the same workflow resolves to
260the same image on every spindle. If an operator keeps multiple arches side by
261side they can name them `nixos-x86_64`, `alpine-aarch64` and so on (that suffix
262is just part of the name, it's not handled specially). If you want, for example,
263`nixos` to work, you can just symlink `nixos` to `nixos-x86_64`.
264
265Right before launch we double-check the referenced files actually exist
266and that the host has the tools we need: `mkfs.ext4` for the volumes, the
267QEMU binary for the spec's arch, `/dev/kvm` and `/dev/vhost-vsock`, plus
268the `ip` / `mount` / `slirp4netns` / `unshare` toolchain if the image
269wants networking.
270
271### The life of a workflow
272
273A workflow moves through a handful of stages: it gets parsed and its
274image resolved, it waits for a slot, it gets set up, its steps run, and
275then everything is torn down.
276
277The waiting bit matters a lot. Each image declares how much memory, how many
278vCPUs and how much disk it needs, and a workflow has to acquire a slot from a
279resource scheduler before anything boots. The scheduler is work-conserving with
280aging and per-user fairness, so one person submitting a hundred jobs won't
281starve everyone else, and slots don't sit idle if there's work that fits in the
282budget.
283
284Once a slot is acquired, we do the setup. Spindle allocates a random vsock CID
285for the guest and registers it with the agent hub. It creates the per-workflow
286work directory, starts the two cache proxies (described earlier), a DNS proxy
287that resolves through the host and filters out private/special-use addresses,
288then creates the VM: writable volumes become sparse files formatted ext4, the
289store disk is attached read-only, and QEMU is started with `-sandbox on`,
290`-nodefaults`, no display, no monitor, etc. with serial (on boot) /
291`virtio_console` output to a log file and a QMP socket for control.
292
293Then we wait for the machine. We poll QMP until QEMU says the guest is running,
294then wait for the agent's handshake to arrive over vsock from the CID we expect.
295The agent tells us its protocol and versions, and spindle sends back the job id,
296the trusted cache public keys, and the cache and DNS proxy ports. From there
297steps run one at a time as `$shell -lc <command>`, as the unprivileged workflow
298user in `/workspace/repo`, with the right environment and any unlocked secrets.
299If the workflow activates a NixOS config and we've already built that exact base
300plus config, the activation step can realize a cached toplevel store path instead
301of rebuilding. Either way, whether it's building the config fresh or pulling a
302cached toplevel down, that output streams straight into the activation step's log
303as it happens, so you can watch the closure come in instead of staring at a blank
304screen wondering if anything's happening.
305
306Timeouts are cooperative: we work out a deadline from the workflow timeout
307and send it to the guest, with a little grace on our side so the guest
308gets a chance to report the timeout itself rather than us just yanking the
309machine out from under it. And if the VM crashes mid-step we tail the
310serial and QEMU logs into the step's stderr, because "guest agent
311connection lost: EOF" is a genuinely useless thing to read at 2am...
312
313Teardown is the same whether the workflow passed, failed or timed out:
314drain any pending Nix cache uploads, ask the agent to power off, wait for
315QEMU to exit (falling back to a QMP `system_powerdown`, and finally a
316kill if it's being stubborn), then close the proxies and remove the work
317directory.
318
319### Locking down the network
320
321A VM that can reach the host's local network is a VM that can reach things it
322has no business reaching. So QEMU doesn't run in the host's network namespace at
323all. We `unshare` into fresh user, net and mount namespaces first. Inside that
324namespace a small wrapper bind-mounts a resolv.conf pointing at `127.0.0.1` so
325that QEMU's built-in slirp DNS isn't used, then installs blackhole routes for
326every special-use IP range (RFC 6890, so private networks, link-local, loopback,
327etc.) before it execs QEMU. `slirp4netns` then provides the namespace's outbound
328internet connection, with `--disable-host-loopback`, sandbox and seccomp all on.
329QEMU runs *inside* that namespace, and the guest's network card is attached to
330QEMU's own built-in user-mode networking. So every packet from the guest takes
331two hops: guest → QEMU's slirp → the namespace's `slirp4netns` → the internet.
332The guest never sees the host's network and the host's network never sees the
333guest. All of this is done without needing any privileges!
334
335Guest DNS doesn't use either slirp layer. The guest's `/etc/resolv.conf` points
336at shuttle on `127.0.0.1:53`, and shuttle forwards DNS packets over vsock to
337the host-side DNS proxy. That proxy resolves through the host's real resolver
338and strips any answers that point at private or special-use addresses, so guest
339traffic can only ever reach the outside world, never the host or anything on its
340local networks.
341
342### Budgets and cgroups
343
344The scheduler's budget is bookkeeping on its own, it tracks what it's handed
345out, and the runner (QEMU) will ensure that a workflow only gets those. But
346optionally the whole thing (QEMU and slirp4netns both) gets placed in a
347per-workflow cgroup with memory, swap etc. limits, which is an extra enforcement
348layer on top, considering QEMU and slirp4netns themselves also use resources. A
349nice side effect is that when the cgroup OOM-kills the VM we can see that it was
350an OOM and report it as such, instead of surfacing it as a generic crash and
351leaving you guessing.
352
353The spindle itself also gets a cgroup with `memory.min` set, which means that in
354a host OOM situation, it should be the workflows that die first, not the spindle
355itself.
356
357## On the roadmap
358
359A few things that are coming next:
360
361- [firecracker](https://github.com/firecracker-microvm/firecracker) runner
362 support. QEMU microVMs are good and all, but firecracker VMs are more
363 efficient to run concurrently and are leaner overall.
364- ssh-on-fail: when a workflow fails, you should be able to ssh in to debug why.
365 This can be really useful in situations where you need just *a little* bit
366 more info if something unexpected fails so you don't sit around there running
367 the workflow 10 times over.
368
369Feel free to come and ask any questions you might have on https://chat.tangled.sh!