alice.pds.demo.boltless.dev
This repository has no description
1# spindle mill
2
3This document describes the architecture of the mill, spindle's distributed
4job placement layer. A mill is a spindle that doesn't run jobs itself: it
5places them on remote executors, which run the real engines (microvm, nixery,
6dummy) exactly as they would standalone.
7
8## The engine mirroring model
9
10Neither side's execution path changes between local and remote runs.
11
12On the mill host, engines are registered under the real engine names
13("microvm", "nixery", "dummy"), but each is a stand-in that places jobs
14remotely instead of running them. `InitWorkflow` on the stand-in parses
15nothing, it stashes the raw pipeline and workflow and builds a synthetic
16one-step workflow, so everything upstream (trigger handling, pending status,
17`TANGLED_*` env) behaves exactly like a local run. The real `InitWorkflow`
18runs exactly once, on the executor.
19
20On the executor, the reserve handler wraps the real engine in a
21`reservedEngine`: the slot acquired when the reservation is accepted is the
22same slot `StartWorkflows` later runs on. Nothing acquires twice, and from the
23engine's point of view the job is indistinguishable from a local one.
24
25The same mirroring applies to output: the executor's jobs write status rows
26and log lines exactly like a standalone spindle, and the mill re-authors them
27into its own event stream and log store, so appview and the log endpoints see
28no difference between a local and a remote job.
29
30The only real divergence: secrets are withheld from untrusted pipelines at the
31mill, and they cross the wire exactly once, inside `CommitLease`, to the one
32node that won the bid. Losing bidders never see them.
33
34## Placement: labels, seats, and rejection
35
36Placement decides which executors can run a workflow right now, using one
37kind of fact:
38
39- **labels** are operator-defined strings on the executor's token, checked
40 against the workflow's `runs_on`. they're for coarse fleet partitioning
41 ("this pool is for trusted jobs", "this box has a gpu"), nothing more
42
43Matching is exact intersection: candidate sessions whose snapshot has the
44engine available, a free seat, and labels covering `runs_on`. Among the
45eligible candidates the mill ranks least-loaded first and bids the top-K
46concurrently with `ReserveSeat`. The best-ranked accept wins, losers get
47`ReleaseLease` so they free their held seats immediately. If nobody
48accepts, the job waits for a change: a new executor connecting, a snapshot
49flipping availability, or a lease finishing somewhere.
50
51Image and arch compatibility is deliberately *not* a mill concern. The mill
52never parses an image name or an arch string; the executor re-validates
53everything at reserve time against its own disk (engine exists, workflow
54parses, the image spec validates and is natively runnable, the runner is
55usable) and rejects what it can't run with `incompatible`. A reject rotates
56the bid to the next candidate, and if every candidate rejects, the user
57gets the collected reasons as the placement error. Multi-arch falls out of
58this: an arm64 box simply cannot accept an x86_64 image, so it never holds
59one. If you *want* to pin an arch or an image explicitly, label the nodes
60and use `runs_on`.
61
62### Example: an alpine microvm job
63
64Say a workflow asks for the microvm engine with `image: alpine`, no
65`runs_on`, and the fleet looks like this:
66
67| node | arch | labels | engines | load |
68|---------|---------|---------|-------------------|--------|
69| ci-1 | x86_64 | [linux] | microvm | busy |
70| ci-2 | aarch64 | [linux] | microvm | idle |
71| ci-3 | x86_64 | [linux] | microvm, nixery | idle |
72| ci-4 | x86_64 | [gpu] | microvm | idle |
73
74The walkthrough:
75
76```mermaid
77flowchart TD
78 W["workflow<br/><small>engine: microvm, image: alpine</small>"] --> F{"candidate filter"}
79 F -->|"all pass runs_on (empty)"| L{"has microvm<br/>available?"}
80 L -->|all four| R{"rank by load"}
81 R -->|"ci-1 busy"| X2["ci-1 ranked last"]
82 R --> B["bid top-K: ci-2, ci-4, ci-1"]
83```
84
85Say ci-2's alpine image is actually x86_64-only: the mill offers the bid
86anyway, ci-2's reserve-time validation rejects it as incompatible, and the
87bid rotates to ci-4. The mill learns "ci-2 can't run alpine" from the
88rejection, not from any advertisement, and the reason reaches the user if
89every candidate fails the same way.
90
91The bid then runs concurrently:
92
93```mermaid
94sequenceDiagram
95 participant M as mill
96 participant C2 as ci-2
97 participant C4 as ci-4
98 participant C1 as ci-1
99
100 par bids
101 M->>C2: ReserveSeat (raw pipeline+workflow)
102 M->>C4: ReserveSeat
103 M->>C1: ReserveSeat
104 end
105 C2-->>M: accept (idle)
106 C4-->>M: accept
107 C1-->>M: reject (transient, seats full)
108 Note over M: ci-2 ranked above ci-4,<br/>ci-4 gets ReleaseLease
109 M->>C2: CommitLease (secrets)
110 C2-->>M: Committed
111 M->>C4: ReleaseLease
112```
113
114Both idle nodes accepted, so rank breaks the tie: ci-2 keeps the seat, ci-4
115frees its immediately, and only ci-2 ever sees the secrets. From here ci-2
116runs the job exactly like a standalone spindle would, booting the alpine
117image under QEMU, while the mill blocks on the terminal event.
118
119## The protocol's durability model
120
121Executor to mill is a single websocket per node, and everything the executor
122reports (status, logs, terminal results) travels as sequenced `Event`s. Two
123identifiers keep it all consistent:
124
125- the **epoch** names one lifetime of the executor process. a restarted
126 executor connects with a fresh epoch, and anything arriving for an old one
127 is invalid. leases are bound to node+epoch, so a zombie from a previous
128 process can't act on them
129- the **seqno** is a dense per-epoch counter on events. the executor persists
130 events in a local outbox before sending and trims it only when the mill
131 acks. on reconnect it resumes from the mill's acked position and replays.
132 the mill applies idempotently: replays drop, gaps kill the session
133
134The mill is equally restartable: leases, acked positions and the canonical
135log all live in its db. Restored leases start as orphans and must be
136reclaimed by the executor's first snapshot, or a sweep fails them after one
137grace window. The invariant throughout: at any moment, for any lease, exactly
138one of {executor outbox, mill db} holds the newest state, and the seqno/epoch
139pair says who.
140
141## Leases
142
143A lease is the mill-side handle for one placed job:
144
145```mermaid
146stateDiagram-v2
147 [*] --> reserved: bid won
148 reserved --> committing: CommitLease sent
149 committing --> running: Committed
150 running --> done: terminal arrived
151 reserved --> done: released / expired
152 committing --> done: released / expired
153```
154
155Commit retries ride reconnects: a reservation outlives one disconnect, so a
156lost session means wait and retry, not a failed job. What ends a job is the
157job timeout, a terminal event, or the executor being declared dead after
158reconnect grace expires, at which point every lease the node held fails
159(preserving a pending cancellation as the reason).