alice.pds.demo.boltless.dev
This repository has no description
1---
2title: Tangled docs
3author: The Tangled Contributors
4date: 21 Sun, Dec 2025
5abstract: |
6 Tangled is a decentralized code hosting and collaboration
7 platform. Every component of Tangled is open-source and
8 self-hostable. [tangled.org](https://tangled.org) also
9 provides hosting and CI services that are free to use.
10
11 There are several models for decentralized code
12 collaboration platforms, ranging from ActivityPub’s
13 (Forgejo) federated model, to Radicle’s entirely P2P model.
14 Our approach attempts to be the best of both worlds by
15 adopting the AT Protocol—a protocol for building decentralized
16 social applications with a central identity
17
18 Our approach to this is the idea of “knots”. Knots are
19 lightweight, headless servers that enable users to host Git
20 repositories with ease. Knots are designed for either single
21 or multi-tenant use which is perfect for self-hosting on a
22 Raspberry Pi at home, or larger “community” servers. By
23 default, Tangled provides managed knots where you can host
24 your repositories for free.
25
26 The appview at tangled.org acts as a consolidated "view"
27 into the whole network, allowing users to access, clone and
28 contribute to repositories hosted across different knots
29 seamlessly.
30---
31
32# Quick start guide
33
34## Login or sign up
35
36You can [login](https://tangled.org) by using your AT Protocol
37account. If you are unclear on what that means, simply head
38to the [signup](https://tangled.org/signup) page and create
39an account. By doing so, you will be choosing Tangled as
40your account provider (you will be granted a handle of the
41form `user.tngl.sh`).
42
43In the AT Protocol network, users are free to choose their account
44provider (known as a "Personal Data Service", or PDS), and
45login to applications that support AT accounts.
46
47You can think of it as "one account for all of the atmosphere"!
48
49If you already have an AT account (you may have one if you
50signed up to Bluesky, for example), you can login with the
51same handle on Tangled (so just use `user.bsky.social` on
52the login page).
53
54## Add an SSH key
55
56Once you are logged in, you can start creating repositories
57and pushing code. Tangled supports pushing git repositories
58over SSH.
59
60First, you'll need to generate an SSH key if you don't
61already have one:
62
63```bash
64ssh-keygen -t ed25519 -C "foo@bar.com"
65```
66
67When prompted, save the key to the default location
68(`~/.ssh/id_ed25519`) and optionally set a passphrase.
69
70Copy your public key to your clipboard:
71
72```bash
73# on X11
74cat ~/.ssh/id_ed25519.pub | xclip -sel c
75
76# on wayland
77cat ~/.ssh/id_ed25519.pub | wl-copy
78
79# on macos
80cat ~/.ssh/id_ed25519.pub | pbcopy
81```
82
83Now, navigate to 'Settings' -> 'Keys' and hit 'Add Key',
84paste your public key, give it a descriptive name, and hit
85save.
86
87## Create a repository
88
89Once your SSH key is added, create your first repository:
90
911. Hit the green `+` icon on the topbar, and select
92 repository
932. Enter a repository name
943. Add a description
954. Choose a knotserver to host this repository on
965. Hit create
97
98Knots are self-hostable, lightweight Git servers that can
99host your repository. Unlike traditional code forges, your
100code can live on any server. Read the [Knots](TODO) section
101for more.
102
103## Configure SSH
104
105To ensure Git uses the correct SSH key and connects smoothly
106to Tangled, add this configuration to your `~/.ssh/config`
107file:
108
109```
110Host tangled.org
111 Hostname tangled.org
112 User git
113 IdentityFile ~/.ssh/id_ed25519
114 AddressFamily inet
115```
116
117This tells SSH to use your specific key when connecting to
118Tangled and prevents authentication issues if you have
119multiple SSH keys.
120
121Note that this configuration only works for knotservers that
122are hosted by tangled.org. If you use a custom knot, refer
123to the [Knots](TODO) section.
124
125## Push your first repository
126
127Initialize a new Git repository:
128
129```bash
130mkdir my-project
131cd my-project
132
133git init
134echo "# My Project" > README.md
135```
136
137Add some content and push!
138
139```bash
140git add README.md
141git commit -m "Initial commit"
142git remote add origin git@tangled.org:user.tngl.sh/my-project
143git push -u origin main
144```
145
146That's it! Your code is now hosted on Tangled.
147
148## Migrating an existing repository
149
150Moving your repositories from GitHub, GitLab, Bitbucket, or
151any other Git forge to Tangled is straightforward. You'll
152simply change your repository's remote URL. At the moment,
153Tangled does not have any tooling to migrate data such as
154GitHub issues or pull requests.
155
156First, create a new repository on tangled.org as described
157in the [Quick Start Guide](#create-a-repository).
158
159Navigate to your existing local repository:
160
161```bash
162cd /path/to/your/existing/repo
163```
164
165You can inspect your existing Git remote like so:
166
167```bash
168git remote -v
169```
170
171You'll see something like:
172
173```bash
174origin git@github.com:username/my-project.git (fetch)
175origin git@github.com:username/my-project.git (push)
176```
177
178Update the remote URL to point to tangled:
179
180```bash
181git remote set-url origin git@tangled.org:user.tngl.sh/my-project
182```
183
184Verify the change:
185
186```bash
187git remote -v
188```
189
190You should now see:
191
192```bash
193origin git@tangled.org:user.tngl.sh/my-project (fetch)
194origin git@tangled.org:user.tngl.sh/my-project (push)
195```
196
197Push all your branches and tags to Tangled:
198
199```bash
200git push -u origin --all
201git push -u origin --tags
202```
203
204Your repository is now migrated to Tangled! All commit
205history, branches, and tags have been preserved.
206
207## Mirroring a repository to Tangled
208
209If you want to maintain your repository on multiple forges
210simultaneously, for example, keeping your primary repository
211on GitHub while mirroring to Tangled for backup or
212redundancy, you can do so by adding [multiple remotes](https://git-scm.com/docs/git-push#_remotes).
213
214You can configure your local repository to push to both
215Tangled and, say, GitHub. You may already have the following
216setup:
217
218```bash
219$ git remote -v
220origin git@github.com:username/my-project.git (fetch)
221origin git@github.com:username/my-project.git (push)
222```
223
224Now add Tangled as an additional push URL to the same
225remote:
226
227```bash
228git remote set-url --add --push origin git@tangled.org:user.tngl.sh/my-project
229```
230
231You also need to re-add the original URL as a push
232destination (Git will now use the original URL to fetch only):
233
234```bash
235git remote set-url --add --push origin git@github.com:username/my-project.git
236```
237
238Verify your configuration:
239
240```bash
241$ git remote -v
242origin git@github.com:username/my-project.git (fetch)
243origin git@tangled.org:user.tngl.sh/my-project (push)
244origin git@github.com:username/my-project.git (push)
245```
246
247Notice that there's one fetch URL (the primary remote) and
248two push URLs. Now, whenever you push, Git will
249automatically push to both remotes:
250
251```bash
252git push origin main
253```
254
255This single command pushes your `main` branch to both GitHub
256and Tangled simultaneously.
257
258To push all branches and tags:
259
260```bash
261git push origin --all
262git push origin --tags
263```
264
265If you prefer more control over which remote you push to,
266you can maintain separate remotes:
267
268```bash
269git remote add github git@github.com:username/my-project.git
270git remote add tangled git@tangled.org:user.tngl.sh/my-project
271```
272
273Then push to each explicitly:
274
275```bash
276git push github main
277git push tangled main
278```
279
280# Hosting websites on Tangled
281
282You can serve static websites directly from your git repositories on
283Tangled. If you've used GitHub Pages or Codeberg Pages, this should feel
284familiar.
285
286## Overview
287
288Every user gets a sites domain. If you signed up through Tangled's own
289PDS (`tngl.sh`), your sites domain is automatically
290`<your-handle>.tngl.sh` no setup needed. Otherwise, you can claim a
291`<subdomain>.tngl.io` domain from your settings.
292
293You can serve multiple sites per domain:
294
295- One **index site** served at the root of your domain (e.g.
296 `alice.tngl.sh`)
297- Any number of **sub-path sites** served under the repository name
298 (e.g. `alice.tngl.sh/my-project`)
299
300## Claiming a domain
301
302If you don't have a `tngl.sh` handle, you need to claim a domain before
303publishing sites:
304
3051. Go to **Settings → Sites**
3062. Enter a subdomain (e.g. `alice` to claim `alice.tngl.io`)
3073. Click **claim**
308
309You can only hold one domain at a time. Releasing a domain puts it in a
31030-day cooldown before anyone else can claim it.
311
312## Configuring a site for a repository
313
3141. Navigate to your repository
3152. Go to **Settings → Sites**
3163. Choose a **branch** to deploy from
3174. Set the **deploy directory** — the path within the repository
318 containing your `index.html`. Use `/` for the root, or a subdirectory
319 like `/docs` or `/public`
3205. Choose the **site type**:
321 - **Index site** — served at the root of your domain (e.g.
322 `alice.tngl.sh`)
323 - **Sub-path site** — served under the repository name (e.g.
324 `alice.tngl.sh/my-project`)
3256. Click **save**
326
327The site will be deployed automatically. You can see the status of your
328previous deploys in the **Recent Deploys** section at the bottom of the
329page.
330
331Sites are redeployed automatically on every push to the configured
332branch.
333
334## Custom domains
335
336Tangled currently doesn't support custom domains for sites. This will be
337added in a future update.
338
339## Deploy directory
340
341The deploy directory is the path within your repository that Tangled
342serves as the site root. It must contain an `index.html`.
343
344| Deploy directory | Result |
345|---|---|
346| `/` | Serves the repository root |
347| `/docs` | Serves the `docs/` subdirectory |
348| `/public` | Serves the `public/` subdirectory |
349
350Directories are served with automatic `index.html` resolution -- a
351request to `/about` will serve `/about/index.html` if it exists.
352
353## Site types
354
355| Type | URL |
356|---|---|
357| Index site | `alice.tngl.sh` |
358| Sub-path site | `alice.tngl.sh/my-project` |
359
360Only one repository can be the index site for a given domain at a time.
361If another repository already holds the index site, you will see a
362notice in the settings and only the sub-path option will be available.
363
364## Deploy triggers
365
366A deployment is triggered automatically when:
367
368- You push to the configured branch
369- You change the site configuration (branch, deploy directory, or site
370 type)
371
372## Disabling a site
373
374To stop serving a site, go to **Settings → Sites** in your repository
375and click **Disable**. This removes the site configuration and stops
376serving the site. The deployed files are also deleted from storage.
377
378Releasing your domain from **Settings → Sites** at the account level
379will disable all sites associated with it and delete their files.
380
381
382# Knot self-hosting guide
383
384So you want to run your own knot server? Great! Here are a few prerequisites:
385
3861. A server of some kind (a VPS, a Raspberry Pi, etc.). Preferably running a Linux distribution of some kind.
3872. A (sub)domain name. People generally use `knot.example.com`.
3883. A valid SSL certificate for your domain.
389
390## NixOS
391
392Refer to the [knot
393module](https://tangled.org/tangled.org/core/blob/master/nix/modules/knot.nix)
394for a full list of options. Sample configurations:
395
396- [The test VM](https://tangled.org/tangled.org/core/blob/master/nix/vm.nix#L85)
397- [@pyrox.dev/nix](https://tangled.org/pyrox.dev/nix/blob/c2b644c214d278af12523618de952ee2eab1af3d/hosts/marvin/services/tangled.nix#L15-26)
398
399## Docker
400
401Refer to
402[@tangled.org/knot-docker](https://tangled.org/@tangled.org/knot-docker).
403Note that this is community maintained.
404
405## Manual setup
406
407First, clone this repository:
408
409```
410git clone https://tangled.org/@tangled.org/core
411```
412
413Then, build the `knot` CLI. This is the knot administration
414and operation tool. For the purpose of this guide, we're
415only concerned with these subcommands:
416
417- `knot server`: the main knot server process, typically
418 run as a supervised service
419- `knot guard`: handles role-based access control for git
420 over SSH (you'll never have to run this yourself)
421- `knot keys`: fetches SSH keys associated with your knot;
422 we'll use this to generate the SSH
423 `AuthorizedKeysCommand`
424
425```
426cd core
427export CGO_ENABLED=1
428go build -o knot ./cmd/knot
429```
430
431Next, move the `knot` binary to a location owned by `root` --
432`/usr/local/bin/` is a good choice. Make sure the binary itself is also owned by `root`:
433
434```
435sudo mv knot /usr/local/bin/knot
436sudo chown root:root /usr/local/bin/knot
437```
438
439This is necessary because SSH `AuthorizedKeysCommand` requires [really
440specific permissions](https://stackoverflow.com/a/27638306). The
441`AuthorizedKeysCommand` specifies a command that is run by `sshd` to
442retrieve a user's public SSH keys dynamically for authentication. Let's
443set that up.
444
445```
446sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF
447Match User git
448 AuthorizedKeysCommand /usr/local/bin/knot keys -o authorized-keys
449 AuthorizedKeysCommandUser nobody
450EOF
451```
452
453Then, reload `sshd`:
454
455```
456sudo systemctl reload ssh
457```
458
459Next, create the `git` user. We'll use the `git` user's home directory
460to store repositories:
461
462```
463sudo adduser git
464```
465
466Create `/home/git/.knot.env` with the following, updating the values as
467necessary. The `KNOT_SERVER_OWNER` should be set to your
468DID, you can find your DID in the [Settings](https://tangled.sh/settings) page.
469
470```
471KNOT_REPO_SCAN_PATH=/home/git
472KNOT_SERVER_HOSTNAME=knot.example.com
473APPVIEW_ENDPOINT=https://tangled.org
474KNOT_SERVER_OWNER=did:plc:foobar
475KNOT_SERVER_INTERNAL_LISTEN_ADDR=127.0.0.1:5444
476KNOT_SERVER_LISTEN_ADDR=127.0.0.1:5555
477```
478
479If you run a Linux distribution that uses systemd, you can
480use the provided service file to run the server. Copy
481[`knotserver.service`](https://tangled.org/tangled.org/core/blob/master/systemd/knotserver.service)
482to `/etc/systemd/system/`. Then, run:
483
484```
485systemctl enable knotserver
486systemctl start knotserver
487```
488
489The last step is to configure a reverse proxy like Nginx or Caddy to front your
490knot. Here's an example configuration for Nginx:
491
492```
493server {
494 listen 80;
495 listen [::]:80;
496 server_name knot.example.com;
497
498 location / {
499 proxy_pass http://localhost:5555;
500 proxy_set_header Host $host;
501 proxy_set_header X-Real-IP $remote_addr;
502 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
503 proxy_set_header X-Forwarded-Proto $scheme;
504 }
505
506 # wss endpoint for git events
507 location /events {
508 proxy_set_header X-Forwarded-For $remote_addr;
509 proxy_set_header Host $http_host;
510 proxy_set_header Upgrade websocket;
511 proxy_set_header Connection Upgrade;
512 proxy_pass http://localhost:5555;
513 }
514 # additional config for SSL/TLS go here.
515}
516
517```
518
519Remember to use Let's Encrypt or similar to procure a certificate for your
520knot domain.
521
522You should now have a running knot server! You can finalize
523your registration by hitting the `verify` button on the
524[/settings/knots](https://tangled.org/settings/knots) page. This simply creates
525a record on your PDS to announce the existence of the knot.
526
527### Custom paths
528
529(This section applies to manual setup only. Docker users should edit the mounts
530in `docker-compose.yml` instead.)
531
532Right now, the database and repositories of your knot lives in `/home/git`. You
533can move these paths if you'd like to store them in another folder. Be careful
534when adjusting these paths:
535
536- Stop your knot when moving data (e.g. `systemctl stop knotserver`) to prevent
537 any possible side effects. Remember to restart it once you're done.
538- Make backups before moving in case something goes wrong.
539- Make sure the `git` user can read and write from the new paths.
540
541#### Database
542
543As an example, let's say the current database is at `/home/git/knotserver.db`,
544and we want to move it to `/home/git/database/knotserver.db`.
545
546Copy the current database to the new location. Make sure to copy the `.db-shm`
547and `.db-wal` files if they exist.
548
549```
550mkdir /home/git/database
551cp /home/git/knotserver.db* /home/git/database
552```
553
554In the environment (e.g. `/home/git/.knot.env`), set `KNOT_SERVER_DB_PATH` to
555the new file path (_not_ the directory):
556
557```
558KNOT_SERVER_DB_PATH=/home/git/database/knotserver.db
559```
560
561#### Repositories
562
563As an example, let's say the repositories are currently in `/home/git`, and we
564want to move them into `/home/git/repositories`.
565
566Create the new folder, then move the existing repositories (if there are any):
567
568```
569mkdir /home/git/repositories
570# move all DIDs into the new folder; these will vary for you!
571mv /home/git/did:plc:wshs7t2adsemcrrd4snkeqli /home/git/repositories
572```
573
574In the environment (e.g. `/home/git/.knot.env`), update `KNOT_REPO_SCAN_PATH`
575to the new directory:
576
577```
578KNOT_REPO_SCAN_PATH=/home/git/repositories
579```
580
581Similarly, update your `sshd` `AuthorizedKeysCommand` to use the updated
582repository path:
583
584```
585sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF
586Match User git
587 AuthorizedKeysCommand /usr/local/bin/knot keys -o authorized-keys -git-dir /home/git/repositories
588 AuthorizedKeysCommandUser nobody
589EOF
590```
591
592Make sure to restart your SSH server!
593
594#### MOTD (message of the day)
595
596To configure the MOTD used ("Welcome to this knot!" by default), edit the
597`/home/git/motd` file:
598
599```
600printf "Hi from this knot!\n" > /home/git/motd
601```
602
603Note that you should add a newline at the end if setting a non-empty message
604since the knot won't do this for you.
605
606## Secure Mode
607
608Secure Mode isolates each `git` subprocess to the repository it is
609operating on, using two mechanisms:
610
611- **Linux Landlock** restricts the filesystem paths the subprocess
612 can access -- it can only read/write its own repository and the
613 system directories it needs to run.
614- **UID isolation** runs each subprocess as a virtual UID assigned
615 to the repository owner, so that repositories belonging to
616 different owners are isolated from each other at the OS level
617 even if Landlock were somehow bypassed.
618
619Secure Mode requires:
620
621- Linux kernel >= 5.19 (Landlock V2). This is the minimum needed
622 for `git push` to work, because receive-pack's quarantine
623 migration uses cross-directory rename which requires the
624 Landlock `REFER` access right (added in V2). Kernels 5.13-5.18
625 support Landlock V1 and clones will work, but pushes will fail
626 with cross-device link errors. On kernels without any Landlock
627 support (< 5.13), the sandbox call is a no-op: UID isolation
628 still applies but no filesystem restriction is enforced.
629- `CAP_SETUID`, `CAP_SETGID`, and `CAP_CHOWN` available to the
630 knot process. The NixOS module grants these automatically; for
631 manual setups see the `setcap` step below.
632
633### NixOS
634
635Add `server.secureMode = true;` to your knot module configuration:
636
637```nix
638services.tangled.knot = {
639 server.secureMode = true;
640 # ... other options
641};
642```
643
644The NixOS module handles everything else automatically:
645
646- Grants the required capabilities to the knot service via
647 `AmbientCapabilities` in the systemd unit.
648- Installs a capability-bearing wrapper at
649 `/run/wrappers/bin/knot` via `security.wrappers`, so that
650 SSH-invoked git operations (pushes) also run under the correct
651 UID without requiring the service to run as root.
652- Runs `knot migrate-isolation` at service start to chown
653 existing repositories to their virtual UIDs.
654
655### Manual setup
656
657**Step 1.** Grant the required capabilities to the knot binary.
658This allows the knot process to switch to virtual UIDs at runtime
659without running as root. You will need to repeat this step
660whenever the binary is updated.
661
662```
663sudo setcap cap_setuid,cap_setgid,cap_chown+eip /usr/local/bin/knot
664```
665
666**Step 2.** Run the migration tool to assign virtual UIDs to all
667existing repositories and set their filesystem permissions. This
668must be run as root:
669
670```
671sudo knot migrate-isolation \
672 --git-dir /home/git \
673 --db /home/git/knotserver.db \
674 --internal-api 127.0.0.1:5444
675```
676
677You can re-run this at any time with `--force` to reapply
678permissions (e.g. after a manual repair or after updating the
679binary).
680
681**Step 2a.** Ensure the home directory is traversable by
682non-group users. Git subprocesses run as virtual UIDs that are
683not in the git group, and they need to resolve
684`$HOME/.config/git/config` to load the global config:
685
686```
687sudo chmod o+x /home/git
688```
689
690This adds only the execute bit, not read -- the virtual UIDs can
691traverse to known paths but cannot list directory contents.
692
693**Step 3.** Enable Secure Mode in your environment file:
694
695```
696KNOT_SERVER_SECURE_MODE=true
697```
698
699Or pass it as a flag:
700
701```
702knot server --secure-mode
703```
704
705**Step 4.** Regenerate the `AuthorizedKeysCommand` with the
706`-secure-mode` flag. This causes `knot keys` to emit guard
707command lines that include `-secure-mode`, so SSH pushes also
708get UID isolation:
709
710```
711sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF
712Match User git
713 AuthorizedKeysCommand /usr/local/bin/knot keys \
714 -o authorized-keys -secure-mode
715 AuthorizedKeysCommandUser nobody
716EOF
717```
718
719Reload `sshd` after making this change.
720
721> **Note:** the server will refuse to start in Secure Mode if any
722> repositories have not yet been isolation-migrated. Re-run
723> `migrate-isolation` if you see this error.
724
725## Troubleshooting
726
727If you run your own knot, you may run into some of these
728common issues. You can always join the
729[IRC](https://web.libera.chat/#tangled) or
730[Discord](https://chat.tangled.org/) if this section does
731not help.
732
733### Unable to push
734
735If you are unable to push to your knot or repository:
736
7371. First, ensure that you have added your SSH public key to
738 your account
7392. Check to see that your knot has synced the key by running
740 `knot keys`
7413. Check to see if git is supplying the correct private key
742 when pushing: `GIT_SSH_COMMAND="ssh -v" git push ...`
7434. Check to see if `sshd` on the knot is rejecting the push
744 for some reason: `journalctl -xeu ssh` (or `sshd`,
745 depending on your machine). These logs are unavailable if
746 using docker.
7475. Check to see if the knot itself is rejecting the push,
748 depending on your setup, the logs might be in one of the
749 following paths:
750 - `/tmp/knotguard.log`
751 - `/home/git/log`
752 - `/home/git/guard.log`
753
754# Spindles
755
756## Pipelines
757
758Spindle workflows allow you to write CI/CD pipelines in a
759simple format. They're located in the `.tangled/workflows`
760directory at the root of your repository, and are defined
761using YAML.
762
763A workflow has a set of common fields that apply no matter
764which engine you pick:
765
766- [Trigger](#trigger): A **required** field that defines
767 when a workflow should be triggered.
768- [Engine](#engine): A **required** field that defines which
769 engine a workflow should run on.
770- [Clone options](#clone-options): An **optional** field
771 that defines how the repository should be cloned.
772- [Environment](#environment): An **optional** field that
773 allows you to define environment variables.
774- [Steps](#steps): An **optional** field that allows you to
775 define what steps should run in the workflow.
776
777On top of these, each engine has its own options for things
778like dependencies and images. See [Engines](#engines) for
779the per-engine fields.
780
781### Trigger
782
783The first thing to add to a workflow is the trigger, which
784defines when a workflow runs. This is defined using a `when`
785field, which takes in a list of conditions. Each condition
786has the following fields:
787
788- `event`: This is a **required** field that defines when
789 your workflow should run. It's a list that can take one or
790 more of the following values:
791 - `push`: The workflow should run every time a commit is
792 pushed to the repository.
793 - `pull_request`: The workflow should run every time a
794 pull request is made or updated.
795 - `manual`: The workflow can be triggered manually.
796- `branch`: Defines which branches the workflow should run
797 for. If used with the `push` event, commits to the
798 branch(es) listed here will trigger the workflow. If used
799 with the `pull_request` event, updates to pull requests
800 targeting the branch(es) listed here will trigger the
801 workflow. This field has no effect with the `manual`
802 event. Supports glob patterns using `*` and `**` (e.g.,
803 `main`, `develop`, `release-*`). Either `branch` or `tag`
804 (or both) must be specified for `push` events.
805- `tag`: Defines which tags the workflow should run for.
806 Only used with the `push` event - when tags matching the
807 pattern(s) listed here are pushed, the workflow will
808 trigger. This field has no effect with `pull_request` or
809 `manual` events. Supports glob patterns using `*` and `**`
810 (e.g., `v*`, `v1.*`, `release-**`). Either `branch` or
811 `tag` (or both) must be specified for `push` events.
812
813For example, if you'd like to define a workflow that runs
814when commits are pushed to the `main` and `develop`
815branches, or when pull requests that target the `main`
816branch are updated, or manually, you can do so with:
817
818```yaml
819when:
820 - event: ["push", "manual"]
821 branch: ["main", "develop"]
822 - event: ["pull_request"]
823 branch: ["main"]
824```
825
826You can also trigger workflows on tag pushes. For instance,
827to run a deployment workflow when tags matching `v*` are
828pushed:
829
830```yaml
831when:
832 - event: ["push"]
833 tag: ["v*"]
834```
835
836You can even combine branch and tag patterns in a single
837constraint (the workflow triggers if either matches):
838
839```yaml
840when:
841 - event: ["push"]
842 branch: ["main", "release-*"]
843 tag: ["v*", "stable"]
844```
845
846To skip CI for a push, pass a Git push option:
847
848```sh
849git push -o skip-ci
850```
851
852`ci-skip` is also accepted.
853
854### Engine
855
856Next is the engine on which the workflow should run, defined
857using the **required** `engine` field. The currently
858supported engines are:
859
860- `nixery`: This uses an instance of
861 [Nixery](https://nixery.dev) to run steps, which allows
862 you to add [dependencies](#dependencies) from
863 Nixpkgs (https://github.com/NixOS/nixpkgs). You can
864 search for packages on https://search.nixos.org, and
865 there's a pretty good chance the package(s) you're looking
866 for will be there.
867 See [Nixery engine](#nixery-engine).
868- `microvm`: Runs the whole workflow inside its own
869 microVM. Has configuration features for NixOS images
870 that will let you enable services, do Docker-in-VM, etc.
871 See [microVM engine](#microvm-engine).
872
873Example:
874
875```yaml
876engine: "nixery"
877```
878
879Each engine also adds its own workflow fields (dependencies,
880images, services, and so on). These are documented under
881[Engines](#engines).
882
883### Clone options
884
885When a workflow starts, the first step is to clone the
886repository. You can customize this behavior using the
887**optional** `clone` field. It has the following fields:
888
889- `skip`: Setting this to `true` will skip cloning the
890 repository. This can be useful if your workflow is doing
891 something that doesn't require anything from the
892 repository itself. This is `false` by default.
893- `depth`: This sets the number of commits, or the "clone
894 depth", to fetch from the repository. For example, if you
895 set this to 2, the last 2 commits will be fetched. By
896 default, the depth is set to 1, meaning only the most
897 recent commit will be fetched, which is the commit that
898 triggered the workflow.
899- `submodules`: If you use Git submodules
900 (https://git-scm.com/book/en/v2/Git-Tools-Submodules)
901 in your repository, setting this field to `true` will
902 recursively fetch all submodules. This is `false` by
903 default.
904
905The default settings are:
906
907```yaml
908clone:
909 skip: false
910 depth: 1
911 submodules: false
912```
913
914### Environment
915
916The `environment` field allows you define environment
917variables that will be available throughout the entire
918workflow. **Do not put secrets here, these environment
919variables are visible to anyone viewing the repository. You
920can add secrets for pipelines in your repository's
921settings.**
922
923Example:
924
925```yaml
926environment:
927 GOOS: "linux"
928 GOARCH: "arm64"
929 NODE_ENV: "production"
930 MY_ENV_VAR: "MY_ENV_VALUE"
931```
932
933By default, the following environment variables are set:
934
935- `CI` - Always set to `true` to indicate a CI environment
936- `TANGLED_PIPELINE_ID` - The AT URI of the current pipeline
937- `TANGLED_PIPELINE_KIND` - One of `push`, `pull_request` or
938 `manual`
939- `TANGLED_REPO_KNOT` - The repository's knot hostname
940- `TANGLED_REPO_DID` - The DID of the repository owner
941- `TANGLED_REPO_NAME` - The name of the repository
942- `TANGLED_REPO_DEFAULT_BRANCH` - The default branch of the
943 repository
944- `TANGLED_REPO_URL` - The full URL to the repository
945
946These variables are only available when the pipeline is
947triggered by a push:
948
949- `TANGLED_REF` - The full git reference (e.g.,
950 `refs/heads/main` or `refs/tags/v1.0.0`)
951- `TANGLED_REF_NAME` - The short name of the reference
952 (e.g., `main` or `v1.0.0`)
953- `TANGLED_REF_TYPE` - The type of reference, either
954 `branch` or `tag`
955- `TANGLED_SHA` - The commit SHA that triggered the pipeline
956- `TANGLED_COMMIT_SHA` - Alias for `TANGLED_SHA`
957
958These variables are only available when the pipeline is
959triggered by a pull request:
960
961- `TANGLED_PR_SOURCE_BRANCH` - The source branch of the pull
962 request
963- `TANGLED_PR_TARGET_BRANCH` - The target branch of the pull
964 request
965- `TANGLED_PR_SOURCE_SHA` - The commit SHA of the source
966 branch
967
968### Steps
969
970The `steps` field allows you to define what steps should run
971in the workflow. It's a list of step objects, each with the
972following fields:
973
974- `name`: This field allows you to give your step a name.
975 This name is visible in your workflow runs, and is used to
976 describe what the step is doing.
977- `command`: This field allows you to define a command to
978 run in that step. The step is run in a Bash shell, and the
979 logs from the command will be visible in the pipelines
980 page on the Tangled website. Any dependencies you added in
981 your engine's section (see [Engines](#engines)) will be
982 available to use here.
983- `environment`: Similar to the global
984 [environment](#environment) config, this **optional**
985 field is a key-value map that allows you to set
986 environment variables for the step. **Do not put secrets
987 here, these environment variables are visible to anyone
988 viewing the repository. You can add secrets for pipelines
989 in your repository's settings.**
990
991Example:
992
993```yaml
994steps:
995 - name: "Build backend"
996 command: "go build"
997 environment:
998 GOOS: "darwin"
999 GOARCH: "arm64"
1000 - name: "Build frontend"
1001 command: "npm run build"
1002 environment:
1003 NODE_ENV: "production"
1004```
1005
1006## Engines
1007
1008The common fields above apply to every workflow. Each engine
1009then adds its own fields on top. Pick an engine with the
1010[`engine`](#engine) field and use the matching section below.
1011
1012### Nixery engine
1013
1014#### Dependencies
1015
1016When you're running a workflow you'll usually need additional
1017dependencies. The `dependencies` field lets you define which
1018dependencies to get, and from where. It's a key-value map,
1019with the key being the registry to fetch dependencies from,
1020and the value being the list of dependencies to fetch.
1021
1022The registry URL syntax can be found [on the nix
1023manual](https://nix.dev/manual/nix/2.18/command-ref/new-cli/nix3-registry-add).
1024
1025Say you want to fetch Node.js and Go from `nixpkgs`, and a
1026package called `my_pkg` you've made from your own registry
1027at your repository at
1028`https://tangled.org/@example.com/my_pkg`. You can define
1029those dependencies like so:
1030
1031```yaml
1032dependencies:
1033 # nixpkgs
1034 nixpkgs:
1035 - nodejs
1036 - go
1037 # unstable
1038 nixpkgs/nixpkgs-unstable:
1039 - bun
1040 # custom registry
1041 git+https://tangled.org/@example.com/my_pkg:
1042 - my_pkg
1043```
1044
1045Now these dependencies are available to use in your
1046workflow!
1047
1048#### Complete nixery workflow
1049
1050```yaml
1051# .tangled/workflows/build.yml
1052
1053when:
1054 - event: ["push", "manual"]
1055 branch: ["main", "develop"]
1056 - event: ["pull_request"]
1057 branch: ["main"]
1058
1059engine: "nixery"
1060
1061# using the default values
1062clone:
1063 skip: false
1064 depth: 1
1065 submodules: false
1066
1067dependencies:
1068 # nixpkgs
1069 nixpkgs:
1070 - nodejs
1071 - go
1072 # custom registry
1073 git+https://tangled.org/@example.com/my_pkg:
1074 - my_pkg
1075
1076environment:
1077 GOOS: "linux"
1078 GOARCH: "arm64"
1079 NODE_ENV: "production"
1080 MY_ENV_VAR: "MY_ENV_VALUE"
1081
1082steps:
1083 - name: "Build backend"
1084 command: "go build"
1085 environment:
1086 GOOS: "darwin"
1087 GOARCH: "arm64"
1088 - name: "Build frontend"
1089 command: "npm run build"
1090 environment:
1091 NODE_ENV: "production"
1092```
1093
1094If you want another example of a workflow, you can look at
1095the one [Tangled uses to build the
1096project](https://tangled.org/@tangled.org/core/blob/master/.tangled/workflows/build.yml).
1097
1098### microVM engine
1099
1100#### Image
1101
1102A workflow picks the image to boot with the top-level `image`
1103field:
1104
1105```yaml
1106engine: microvm
1107image: nixos
1108```
1109
1110There are two flavours of images:
1111
1112- **NixOS images** (e.g. `nixos`): the whole guest is built
1113 with Nix, so you can configure it from the workflow file
1114 itself. The `dependencies`, `services`, `virtualisation`,
1115 `registry` and `caches` fields below are all understood
1116 here, and the guest builds and activates that configuration
1117 before any of your steps run.
1118- **Non-NixOS images** (e.g. `alpine`): there's no NixOS to
1119 configure, so the workflow-level config fields above have
1120 no effect. You still get a full machine to run steps in.
1121
1122The available image names depend on what the spindle operator
1123has installed. `nixos` and `alpine` are examples. If `image`
1124is omitted, the spindle's configured default image is used.
1125
1126#### Dependencies
1127
1128On the microVM engine, `dependencies` is a flat list of
1129packages that are made available to every step. This field
1130only applies to **NixOS images**; for other images you can
1131use the package manager included in a step.
1132
1133The guest builds a [`nix develop`](https://nix.dev/manual/nix/2.18/command-ref/new-cli/nix3-develop)-style
1134devshell from your dependencies and uses it for each step,
1135so you can, for example, add `pkg-config` and `openssl` and
1136have the `openssl-sys` crate while compiling a Rust project
1137just work.
1138
1139A bare name like `go` is looked up in nixpkgs. You can also
1140point at any flake with the `flakeref#attr` syntax, so
1141`github:nixos/nixpkgs#hello` pulls `hello` straight out of
1142that flake.
1143
1144```yaml
1145dependencies:
1146 - go
1147 - github:nixos/nixpkgs#hello
1148```
1149
1150#### Registry
1151
1152The `registry` field remaps flake references, the same way
1153`nix registry` does. This lets you pin or alias the flakes
1154used by `dependencies`.
1155
1156For example, pin `nixpkgs` to `nixos-unstable` so that the
1157bare `go` above resolves from unstable, and alias your own
1158flake so you can use `myflake#tool` in `dependencies`:
1159
1160```yaml
1161registry:
1162 nixpkgs: github:nixos/nixpkgs/nixos-unstable
1163 myflake: github:me/x
1164```
1165
1166#### Caches
1167
1168The `caches` field is a map of Nix binary cache URL to its
1169trusted public key. These are fed into the spindle's read
1170proxy, so the guest can substitute prebuilt paths from them
1171instead of building everything from scratch.
1172
1173```yaml
1174caches:
1175 https://nix-community.cachix.org: "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
1176```
1177
1178#### Services and virtualisation
1179
1180The `services` and `virtualisation` fields are passed straight
1181through to NixOS. Anything you could write under
1182`services.*` or `virtualisation.*` in a NixOS configuration,
1183you can write here, and it's brought up before any of your
1184steps run.
1185
1186As a convenience, `true` works as shorthand for
1187`.enable = true` anywhere an `enable` option exists (e.g.
1188`virtualisation.docker: true`).
1189
1190```yaml
1191services:
1192 postgresql:
1193 enable: true
1194 ensureDatabases: ["spindle-workflow"]
1195 ensureUsers:
1196 - name: spindle-workflow
1197 ensureDBOwnership: true
1198
1199virtualisation:
1200 docker: true
1201```
1202
1203#### Recipes
1204
1205##### Lint, test and build a Node project
1206
1207```yaml
1208when:
1209 - event: ["push", "pull_request"]
1210 branch: ["main"]
1211
1212engine: microvm
1213image: nixos
1214
1215dependencies:
1216 - pnpm
1217
1218steps:
1219 - name: "Install dependencies"
1220 command: pnpm install --frozen-lockfile
1221 - name: "Lint and test"
1222 command: |
1223 pnpm run lint
1224 pnpm test
1225 - name: "Build"
1226 command: pnpm run build
1227```
1228
1229##### Check formatting
1230
1231```yaml
1232when:
1233 - event: ["push", "pull_request"]
1234 branch: ["main"]
1235
1236engine: microvm
1237image: alpine # slimmer image for checking the formatting
1238
1239steps:
1240 - name: "Install go"
1241 command: apk add go
1242 - name: "Check formatting"
1243 command: test -z $(gofmt -l .)
1244```
1245
1246##### Build a Rust project that links OpenSSL
1247
1248```yaml
1249when:
1250 - event: ["push", "pull_request"]
1251 branch: ["main"]
1252
1253engine: microvm
1254image: nixos
1255
1256dependencies:
1257 - gcc
1258 - cargo
1259 - rustc
1260 - clippy
1261 - rustfmt
1262 - pkg-config # exports PKG_CONFIG_PATH for the libraries below
1263 - openssl # the C library + headers openssl-sys links against
1264
1265steps:
1266 - name: "Check formatting"
1267 command: cargo fmt --check
1268 - name: "Clippy"
1269 command: cargo clippy --all-targets -- -D warnings
1270 - name: "Test"
1271 command: cargo test --all
1272 - name: "Release build"
1273 command: cargo build --release
1274```
1275
1276##### Run migrations and integration tests against PostgreSQL
1277
1278```yaml
1279when:
1280 - event: ["push", "pull_request"]
1281 branch: ["main"]
1282
1283engine: microvm
1284image: nixos
1285
1286environment:
1287 DATABASE_URL: "postgresql:///spindle-workflow?host=/run/postgresql"
1288
1289dependencies:
1290 - gcc
1291 - cargo
1292 - rustc
1293 - pkg-config
1294 - openssl
1295 - sqlx-cli
1296
1297services:
1298 postgresql:
1299 enable: true
1300 # has to be same name as the user for peer auth to work automatically
1301 ensureDatabases: ["spindle-workflow"]
1302 ensureUsers:
1303 - name: spindle-workflow
1304 ensureDBOwnership: true
1305
1306steps:
1307 - name: "Run migrations"
1308 command: sqlx migrate run
1309 - name: "Integration tests"
1310 command: cargo test --all
1311```
1312
1313##### Build and push a Docker image on tag
1314
1315```yaml
1316when:
1317 - event: ["push"]
1318 tag: ["v*"]
1319
1320engine: microvm
1321image: nixos
1322
1323virtualisation:
1324 docker: true
1325
1326steps:
1327 - name: "Build and push to ghcr.io"
1328 command: |
1329 set -euo pipefail
1330
1331 echo "$REGISTRY_TOKEN" | docker login ghcr.io -u "$REGISTRY_USER" --password-stdin
1332 image="ghcr.io/$REGISTRY_USER/myapp:$TANGLED_REF_NAME"
1333
1334 docker build -t "$image" -t "ghcr.io/$REGISTRY_USER/myapp:latest" .
1335 docker push "$image"
1336 docker push "ghcr.io/$REGISTRY_USER/myapp:latest"
1337```
1338
1339##### Deploy to Cloudflare Workers on tag
1340
1341```yaml
1342# .tangled/workflows/deploy.yml
1343when:
1344 - event: ["push"]
1345 tag: ["v*"]
1346
1347engine: microvm
1348image: nixos
1349
1350dependencies:
1351 - pnpm
1352
1353steps:
1354 - name: "Install dependencies"
1355 command: pnpm install --frozen-lockfile
1356 - name: "Deploy worker"
1357 # `wrangler` picks up `CLOUDFLARE_API_TOKEN` from the env.
1358 # set it under **Settings → Secrets**.
1359 command: pnpm exec wrangler deploy
1360```
1361
1362##### Publish a release artifact
1363
1364```yaml
1365when:
1366 - event: ["push"]
1367 tag: ["v*"] # trigger on versions
1368
1369engine: microvm
1370image: nixos
1371
1372dependencies:
1373 - go
1374
1375steps:
1376 - name: "Build release binary"
1377 command: |
1378 mkdir -p dist
1379 CGO_ENABLED=0 go build -trimpath -ldflags "-s -w" -o dist/myapp ./cmd/myapp
1380
1381 - name: "Publish artifact record"
1382 command: |
1383 set -euo pipefail
1384 # change this if you're not on `tngl.sh`
1385 PDS="https://tngl.sh"
1386 # also update this to your handle or did
1387 ATP_IDENTIFIER="user.tngl.sh"
1388 ARTIFACT_PATH="dist/myapp"
1389 ARTIFACT_NAME="myapp"
1390
1391 # set `ATP_APP_PASSWORD` under **Settings → Secrets**
1392 session=$(curl -fsS -X POST "$PDS/xrpc/com.atproto.server.createSession" \
1393 -H "Content-Type: application/json" \
1394 -d "{\"identifier\":\"$ATP_IDENTIFIER\",\"password\":\"$ATP_APP_PASSWORD\"}")
1395 jwt=$(echo "$session" | jq -r .accessJwt)
1396 did=$(echo "$session" | jq -r .did)
1397
1398 # upload the binary as a blob
1399 blob=$(curl -fsS -X POST "$PDS/xrpc/com.atproto.repo.uploadBlob" \
1400 -H "Authorization: Bearer $jwt" \
1401 -H "Content-Type: application/octet-stream" \
1402 --data-binary @"$ARTIFACT_PATH")
1403
1404 # note that this requires an annotated tag (`git tag -a v1.0.0 -m ...`)
1405 tag_hash=$(git rev-parse "$TANGLED_REF_NAME^{tag}")
1406 tag_bytes=$(printf '%s' "$tag_hash" | xxd -r -p | base64 | tr -d '=')
1407
1408 # the sh.tangled.repo.artifact record for your artifact
1409 record=$(jq -n \
1410 --arg did "$did" \
1411 --arg tag "$tag_bytes" \
1412 --arg name "$ARTIFACT_NAME" \
1413 --arg repo "$TANGLED_REPO_URL" \
1414 --arg created "$(date -Iseconds)" \
1415 --argjson blob "$(echo "$blob" | jq .blob)" '{
1416 repo: $did,
1417 collection: "sh.tangled.repo.artifact",
1418 validate: false,
1419 record: {
1420 "$type": "sh.tangled.repo.artifact",
1421 tag: {"$bytes": $tag},
1422 name: $name,
1423 repo: $repo,
1424 artifact: $blob,
1425 createdAt: $created
1426 }
1427 }')
1428
1429 # create the record on the PDS
1430 curl -fsS -X POST "$PDS/xrpc/com.atproto.repo.createRecord" \
1431 -H "Authorization: Bearer $jwt" \
1432 -H "Content-Type: application/json" \
1433 -d "$record"
1434```
1435
1436## Self-hosting guide
1437
1438### Prerequisites
1439
1440- Go
1441- For the **nixery** engine: Docker (or Podman with Docker
1442 compatibility enabled).
1443- For the **microVM** engine: a Linux host with KVM, plus the
1444 microVM host dependencies described in [Running microVM
1445 workflows](#running-microvm-workflows).
1446
1447### Configuration
1448
1449Spindle is configured using environment variables. The following environment variables are available:
1450
1451- `SPINDLE_SERVER_LISTEN_ADDR`: The address the server listens on (default: `"0.0.0.0:6555"`).
1452- `SPINDLE_SERVER_DB_PATH`: The path to the SQLite database file (default: `"spindle.db"`).
1453- `SPINDLE_SERVER_HOSTNAME`: The hostname of the server (required).
1454- `SPINDLE_SERVER_JETSTREAM_ENDPOINT`: The endpoint of the Jetstream server (default: `"wss://jetstream1.us-west.bsky.network/subscribe"`).
1455- `SPINDLE_SERVER_DEV`: A boolean indicating whether the server is running in development mode (default: `false`).
1456- `SPINDLE_SERVER_INVITE_ONLY`: A boolean indicating whether only members of this spindle may register repos. When `false`, the spindle accepts repos from anyone on the network (default: `true`).
1457- `SPINDLE_SERVER_LOG_DIR`: The directory to store workflow logs (default: `"/var/log/spindle"`).
1458- `SPINDLE_SERVER_DOCKER_SOCKET`: Path to Docker socket to expose to invoked Spindle containers (default: `""`).
1459- `SPINDLE_NIXERY_PIPELINES_NIXERY`: The Nixery URL (default: `"nixery.tangled.sh"`).
1460- `SPINDLE_NIXERY_PIPELINES_WORKFLOW_TIMEOUT`: The default workflow timeout (default: `"5m"`).
1461
1462For the microVM engine, the following are also available
1463(prefix `SPINDLE_MICROVM_PIPELINES_`):
1464
1465- `SPINDLE_MICROVM_PIPELINES_IMAGE_DIR`: Directory containing
1466 microVM images (**required** to use the engine). See
1467 [Running microVM workflows](#running-microvm-workflows).
1468- `SPINDLE_MICROVM_PIPELINES_DEFAULT_IMAGE`: Image used when a
1469 workflow doesn't set `image` (default: `"nixos-x86_64"`).
1470- `SPINDLE_MICROVM_PIPELINES_OVERLAY_DIR`: Where per-workflow
1471 temporary disks are created (default: the system temp dir).
1472- `SPINDLE_MICROVM_PIPELINES_ENABLE_KVM`: Use KVM hardware
1473 acceleration (default: `true`). Without KVM, guests fall
1474 back to slow software emulation.
1475- `SPINDLE_MICROVM_PIPELINES_WORKFLOW_TIMEOUT`: Default
1476 workflow timeout (default: `"5m"`).
1477
1478Optional resource limits (a value of `0` disables that
1479limit). The limits cap usage across all running microVM
1480workflows:
1481
1482- `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_MEMORY_MIB`
1483- `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_VCPUS`
1484- `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_DISK_MIB`
1485
1486Optional cgroup enforcement:
1487
1488- `SPINDLE_MICROVM_PIPELINES_ENABLE_CGROUPS`: Place each
1489 workflow's QEMU and slirp4netns in a per-workflow cgroup=
1490 (default: `false`).
1491- `SPINDLE_MICROVM_PIPELINES_CGROUP_PARENT`: Parent cgroup;
1492 `self` resolves the spindle service's own cgroup (default:
1493 `"self"`).
1494- `SPINDLE_MICROVM_PIPELINES_CGROUP_PIDS_MAX`: Max processes
1495 per workflow cgroup (default: `4096`).
1496- `SPINDLE_MICROVM_PIPELINES_CGROUP_SWAP_MAX_MIB`: Max swap
1497 per workflow cgroup (default: `0`, no swap).
1498- `SPINDLE_MICROVM_PIPELINES_CGROUP_SUPERVISOR_MEMORY_MIN_MIB`:
1499 Memory protected for spindle itself so it isn't OOM-killed
1500 before the workflows (default: `512`).
1501
1502To push paths built inside microVMs back to a shared Nix
1503cache (and read from it), configure the cache (prefix
1504`SPINDLE_NIX_CACHE_`):
1505
1506- `SPINDLE_NIX_CACHE_READ_URLS`: Comma-separated binary cache
1507 URLs the guest reads from.
1508- `SPINDLE_NIX_CACHE_TRUSTED_PUBLIC_KEYS`: Comma-separated
1509 trusted public keys for those caches.
1510- `SPINDLE_NIX_CACHE_UPLOAD_URL`: Cache URL that paths built
1511 in the guest are uploaded to.
1512
1513### Running spindle
1514
15151. **Set the environment variables.** For example:
1516
1517 ```shell
1518 export SPINDLE_SERVER_HOSTNAME="your-hostname"
1519 ```
1520
15212. **Build the Spindle binary.**
1522
1523 ```shell
1524 cd core
1525 go mod download
1526 go build -o cmd/spindle/spindle cmd/spindle/main.go
1527 ```
1528
15293. **Create the log directory.**
1530
1531 ```shell
1532 sudo mkdir -p /var/log/spindle
1533 sudo chown $USER:$USER -R /var/log/spindle
1534 ```
1535
15364. **Run the Spindle binary.**
1537
1538 ```shell
1539 ./cmd/spindle/spindle
1540 ```
1541
1542Spindle will now start, connect to the Jetstream server, and begin processing pipelines.
1543
1544### Running microVM workflows
1545
1546The microVM engine needs a few extra things on the host, and
1547it needs images to boot.
1548
1549#### Host dependencies
1550
1551microVM workflows depend on a handful of host tools and
1552devices. spindle checks for the ones an image needs right
1553before it launches, so a missing dependency surfaces as a
1554clear error. You'll need:
1555
1556- `qemu`: the runner. The QEMU binary for the image's arch
1557 must be present (e.g. `qemu-system-x86_64`).
1558- `mkfs.ext4` (from `e2fsprogs`): to format the per-workflow
1559 writable volumes.
1560- [`slirp4netns`](https://github.com/rootless-containers/slirp4netns#install),
1561 `ip` (from `iproute2`), `mount` and `unshare` (from `util-linux`):
1562 used to sandbox guest networking.
1563- `/dev/kvm`: for hardware acceleration (unless you disable
1564 KVM with `SPINDLE_MICROVM_PIPELINES_ENABLE_KVM=false`).
1565- `/dev/vhost-vsock`: used by QEMU to enable guest-to-host vsock
1566 communication.
1567- `/dev/vsock`: used by spindle on the host to listen on vsock ports
1568 and accept guest agent connections.
1569- `/dev/net/tun`: required by `slirp4netns` to set up tap devices
1570 for sandboxed guest networking.
1571
1572On NixOS, the [spindle
1573module](https://tangled.org/tangled.org/core/blob/master/nix/modules/spindle.nix)
1574puts `qemu`, `e2fsprogs`, `slirp4netns`, `iproute2` and
1575`util-linux` on the service's `PATH` for you.
1576
1577#### Container virtualization
1578
1579Running `spindle` inside a container (e.g., Docker, Podman, or LXD) requires passing host device nodes into the container and granting the runtime additional privileges. Because spindle uses nested namespaces (`unshare`) and helper tools (`slirp4netns`), the container configuration will require:
1580
1581- `/dev/vsock`, `/dev/vhost-vsock`, `/dev/kvm`, and `/dev/net/tun` mapped from the host.
1582- `NET_ADMIN` and `SYS_ADMIN` capabilities to manage network namespaces and mounts inside the container.
1583- Relaxed seccomp filters (e.g., `seccomp=unconfined`) and SELinux/AppArmor containment if they restrict namespace creation or device access.
1584
1585#### Building images
1586
1587Images are built with Nix. The flake exposes packages for the
1588two stock images (use the `-tarball` prefixed ones for a gzipped
1589tarball you can copy to another host):
1590
1591```shell
1592# a NixOS image
1593nix build .#spindle-nixos-image
1594# an Alpine image
1595nix build .#spindle-alpine-image
1596```
1597
1598#### Installing images
1599
1600Spindle looks for images in
1601`SPINDLE_MICROVM_PIPELINES_IMAGE_DIR`. An image is resolved by
1602the name a workflow puts in its `image` field, matched
1603literally against what's on disk:
1604
16051. a directory `<name>/` containing a `spec.json` (next to the
1606 kernel/initrd/store-disk), or
16072. a flat `<name>.json` self-contained spec.
1608
1609Resolution depends only on the name and what's on disk, never
1610on the host doing the resolving, so the same workflow resolves
1611to the same image on every spindle. If you keep multiple
1612arches side by side, you can name them `<name>-<arch>` (e.g.
1613`nixos-x86_64`, `alpine-aarch64`); the suffix is just part of
1614the name. To make a name like `nixos` work if you are hosting
1615multiple arches, you can use symlinks.
1616
1617On NixOS, you'll most likely want to use `systemd.tmpfiles.rules`
1618to set these up declaratively.
1619
1620## Architecture
1621
1622Spindle is a small CI runner service. Here's a high-level overview of how it operates:
1623
1624- Listens for [`sh.tangled.spindle.member`](/lexicons/spindle/member.json) and
1625 [`sh.tangled.repo`](/lexicons/repo.json) records on the Jetstream.
1626- When a new repo record comes through (typically when you add a spindle to a
1627 repo from the settings), spindle then resolves the underlying knot and
1628 subscribes to repo events (see:
1629 [`sh.tangled.pipeline`](/lexicons/pipeline.json)).
1630- The spindle engine then handles execution of the pipeline, with results and
1631 logs beamed on the spindle event stream over WebSocket
1632
1633### The engines
1634
1635Spindle has two execution backends, picked per-workflow with
1636the [`engine`](#engine) field:
1637
1638- **nixery**: executes each step in a fresh Docker container
1639 (Podman works too, if Docker compatibility is enabled so
1640 that `/run/docker.sock` is created), with state persisted
1641 across steps within the `/tangled/workspace` directory. The
1642 base image for the container is constructed on the fly using
1643 [Nixery](https://nixery.dev), which is/rhandy for caching
1644 layers for frequently used packages.
1645- **microvm**: runs the whole workflow inside its own
1646 microVM, supporting different images, with extra
1647 configuration for NixOS images (e.g. services in workflow file)
1648 See the [engine
1649 README](https://tangled.org/tangled.org/core/blob/master/spindle/engines/microvm/README.md)
1650 for the architecture in depth.
1651
1652The pipeline manifest is [specified here](https://docs.tangled.org/spindles.html#pipelines).
1653
1654## Secrets with openbao
1655
1656This document covers setting up spindle to use OpenBao for secrets
1657management via OpenBao Proxy instead of the default SQLite backend.
1658
1659### Overview
1660
1661Spindle now uses OpenBao Proxy for secrets management. The proxy handles
1662authentication automatically using AppRole credentials, while spindle
1663connects to the local proxy instead of directly to the OpenBao server.
1664
1665This approach provides better security, automatic token renewal, and
1666simplified application code.
1667
1668### Installation
1669
1670Install OpenBao from Nixpkgs:
1671
1672```bash
1673nix shell nixpkgs#openbao # for a local server
1674```
1675
1676### Setup
1677
1678The setup process can is documented for both local development and production.
1679
1680#### Local development
1681
1682Start OpenBao in dev mode:
1683
1684```bash
1685bao server -dev -dev-root-token-id="root" -dev-listen-address=127.0.0.1:8201
1686```
1687
1688This starts OpenBao on `http://localhost:8201` with a root token.
1689
1690Set up environment for bao CLI:
1691
1692```bash
1693export BAO_ADDR=http://localhost:8200
1694export BAO_TOKEN=root
1695```
1696
1697#### Production
1698
1699You would typically use a systemd service with a
1700configuration file. Refer to
1701[@tangled.org/infra](https://tangled.org/@tangled.org/infra)
1702for how this can be achieved using Nix.
1703
1704Then, initialize the bao server:
1705
1706```bash
1707bao operator init -key-shares=1 -key-threshold=1
1708```
1709
1710This will print out an unseal key and a root key. Save them
1711somewhere (like a password manager). Then unseal the vault
1712to begin setting it up:
1713
1714```bash
1715bao operator unseal <unseal_key>
1716```
1717
1718All steps below remain the same across both dev and
1719production setups.
1720
1721#### Configure openbao server
1722
1723Create the spindle KV mount:
1724
1725```bash
1726bao secrets enable -path=spindle -version=2 kv
1727```
1728
1729Set up AppRole authentication and policy:
1730
1731Create a policy file `spindle-policy.hcl`:
1732
1733```hcl
1734# Full access to spindle KV v2 data
1735path "spindle/data/*" {
1736 capabilities = ["create", "read", "update", "delete"]
1737}
1738
1739# Access to metadata for listing and management
1740path "spindle/metadata/*" {
1741 capabilities = ["list", "read", "delete", "update"]
1742}
1743
1744# Allow listing at root level
1745path "spindle/" {
1746 capabilities = ["list"]
1747}
1748
1749# Required for connection testing and health checks
1750path "auth/token/lookup-self" {
1751 capabilities = ["read"]
1752}
1753```
1754
1755Apply the policy and create an AppRole:
1756
1757```bash
1758bao policy write spindle-policy spindle-policy.hcl
1759bao auth enable approle
1760bao write auth/approle/role/spindle \
1761 token_policies="spindle-policy" \
1762 token_ttl=1h \
1763 token_max_ttl=4h \
1764 bind_secret_id=true \
1765 secret_id_ttl=0 \
1766 secret_id_num_uses=0
1767```
1768
1769Get the credentials:
1770
1771```bash
1772# Get role ID (static)
1773ROLE_ID=$(bao read -field=role_id auth/approle/role/spindle/role-id)
1774
1775# Generate secret ID
1776SECRET_ID=$(bao write -f -field=secret_id auth/approle/role/spindle/secret-id)
1777
1778echo "Role ID: $ROLE_ID"
1779echo "Secret ID: $SECRET_ID"
1780```
1781
1782#### Create proxy configuration
1783
1784Create the credential files:
1785
1786```bash
1787# Create directory for OpenBao files
1788mkdir -p /tmp/openbao
1789
1790# Save credentials
1791echo "$ROLE_ID" > /tmp/openbao/role-id
1792echo "$SECRET_ID" > /tmp/openbao/secret-id
1793chmod 600 /tmp/openbao/role-id /tmp/openbao/secret-id
1794```
1795
1796Create a proxy configuration file `/tmp/openbao/proxy.hcl`:
1797
1798```hcl
1799# OpenBao server connection
1800vault {
1801 address = "http://localhost:8200"
1802}
1803
1804# Auto-Auth using AppRole
1805auto_auth {
1806 method "approle" {
1807 mount_path = "auth/approle"
1808 config = {
1809 role_id_file_path = "/tmp/openbao/role-id"
1810 secret_id_file_path = "/tmp/openbao/secret-id"
1811 }
1812 }
1813
1814 # Optional: write token to file for debugging
1815 sink "file" {
1816 config = {
1817 path = "/tmp/openbao/token"
1818 mode = 0640
1819 }
1820 }
1821}
1822
1823# Proxy listener for spindle
1824listener "tcp" {
1825 address = "127.0.0.1:8201"
1826 tls_disable = true
1827}
1828
1829# Enable API proxy with auto-auth token
1830api_proxy {
1831 use_auto_auth_token = true
1832}
1833
1834# Enable response caching
1835cache {
1836 use_auto_auth_token = true
1837}
1838
1839# Logging
1840log_level = "info"
1841```
1842
1843#### Start the proxy
1844
1845Start OpenBao Proxy:
1846
1847```bash
1848bao proxy -config=/tmp/openbao/proxy.hcl
1849```
1850
1851The proxy will authenticate with OpenBao and start listening on
1852`127.0.0.1:8201`.
1853
1854#### Configure spindle
1855
1856Set these environment variables for spindle:
1857
1858```bash
1859export SPINDLE_SERVER_SECRETS_PROVIDER=openbao
1860export SPINDLE_SERVER_SECRETS_OPENBAO_PROXY_ADDR=http://127.0.0.1:8201
1861export SPINDLE_SERVER_SECRETS_OPENBAO_MOUNT=spindle
1862```
1863
1864On startup, spindle will now connect to the local proxy,
1865which handles all authentication automatically.
1866
1867### Production setup for proxy
1868
1869For production, you'll want to run the proxy as a service:
1870
1871Place your production configuration in
1872`/etc/openbao/proxy.hcl` with proper TLS settings for the
1873vault connection.
1874
1875### Verifying setup
1876
1877Test the proxy directly:
1878
1879```bash
1880# Check proxy health
1881curl -H "X-Vault-Request: true" http://127.0.0.1:8201/v1/sys/health
1882
1883# Test token lookup through proxy
1884curl -H "X-Vault-Request: true" http://127.0.0.1:8201/v1/auth/token/lookup-self
1885```
1886
1887Test OpenBao operations through the server:
1888
1889```bash
1890# List all secrets
1891bao kv list spindle/
1892
1893# Add a test secret via the spindle API, then check it exists
1894bao kv list spindle/repos/
1895
1896# Get a specific secret
1897bao kv get spindle/repos/your_repo_path/SECRET_NAME
1898```
1899
1900### How it works
1901
1902- Spindle connects to OpenBao Proxy on localhost (typically
1903 port 8200 or 8201)
1904- The proxy authenticates with OpenBao using AppRole
1905 credentials
1906- All spindle requests go through the proxy, which injects
1907 authentication tokens
1908- Secrets are stored at
1909 `spindle/repos/{sanitized_repo_path}/{secret_key}`
1910- Repository paths like `did:plc:alice/myrepo` become
1911 `did_plc_alice_myrepo`
1912- The proxy handles all token renewal automatically
1913- Spindle no longer manages tokens or authentication
1914 directly
1915
1916### Troubleshooting
1917
1918**Connection refused**: Check that the OpenBao Proxy is
1919running and listening on the configured address.
1920
1921**403 errors**: Verify the AppRole credentials are correct
1922and the policy has the necessary permissions.
1923
1924**404 route errors**: The spindle KV mount probably doesn't
1925exist—run the mount creation step again.
1926
1927**Proxy authentication failures**: Check the proxy logs and
1928verify the role-id and secret-id files are readable and
1929contain valid credentials.
1930
1931**Secret not found after writing**: This can indicate policy
1932permission issues. Verify the policy includes both
1933`spindle/data/*` and `spindle/metadata/*` paths with
1934appropriate capabilities.
1935
1936Check proxy logs:
1937
1938```bash
1939# If running as systemd service
1940journalctl -u openbao-proxy -f
1941
1942# If running directly, check the console output
1943```
1944
1945Test AppRole authentication manually:
1946
1947```bash
1948bao write auth/approle/login \
1949 role_id="$(cat /tmp/openbao/role-id)" \
1950 secret_id="$(cat /tmp/openbao/secret-id)"
1951```
1952
1953# Webhooks
1954
1955Webhooks allow you to receive HTTP POST notifications when events occur in your repositories. This enables you to integrate Tangled with external services, trigger CI/CD pipelines, send notifications, or automate workflows.
1956
1957## Overview
1958
1959Webhooks send HTTP POST requests to URLs you configure whenever specific events happen. Currently, Tangled supports push, repository rename, and pull request events, with more event types coming soon.
1960
1961## Configuring webhooks
1962
1963To set up a webhook for your repository:
1964
19651. Navigate to your repository
19662. Go to **Settings → Hooks**
19673. Click **new webhook**
19684. Configure your webhook:
1969 - **Payload URL**: The endpoint that will receive the webhook POST requests
1970 - **Secret**: An optional secret key for verifying webhook authenticity (leave blank to send unsigned webhooks)
1971 - **Events**: Select which events trigger the webhook
1972 - **Active**: Toggle whether the webhook is enabled
1973
1974## Webhook payload
1975
1976### Push
1977
1978When a push event occurs, Tangled sends a POST request with a JSON payload of the format:
1979
1980```json
1981{
1982 "after": "7b320e5cbee2734071e4310c1d9ae401d8f6cab5",
1983 "before": "c04ddf64eddc90e4e2a9846ba3b43e67a0e2865e",
1984 "pusher": {
1985 "did": "did:plc:hwevmowznbiukdf6uk5dwrrq"
1986 },
1987 "ref": "refs/heads/main",
1988 "repository": {
1989 "clone_url": "https://tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo",
1990 "created_at": "2025-09-15T08:57:23Z",
1991 "description": "an example repository",
1992 "fork": false,
1993 "full_name": "did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo",
1994 "html_url": "https://tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo",
1995 "name": "some-repo",
1996 "open_issues_count": 5,
1997 "owner": {
1998 "did": "did:plc:hwevmowznbiukdf6uk5dwrrq"
1999 },
2000 "ssh_url": "ssh://git@tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo",
2001 "stars_count": 1,
2002 "updated_at": "2025-09-15T08:57:23Z"
2003 }
2004}
2005```
2006
2007### Pull request
2008
2009Pull request events are sent as separate event types, so you can subscribe to
2010exactly the transitions you care about:
2011
2012- `pull_request:created` — a pull request was opened
2013- `pull_request:resubmitted` — a new round (revision) was pushed to a pull request
2014- `pull_request:merged` — a pull request was merged
2015- `pull_request:closed` — a pull request was closed
2016- `pull_request:reopened` — a closed pull request was reopened
2017
2018All pull request events share the same payload format:
2019
2020```json
2021{
2022 "action": "created",
2023 "pull_request": {
2024 "number": 4,
2025 "title": "add dark mode",
2026 "body": "implements dark mode as discussed in #2",
2027 "state": "open",
2028 "target_branch": "main",
2029 "source": {
2030 "branch": "dark-mode",
2031 "sha": "7b320e5cbee2734071e4310c1d9ae401d8f6cab5"
2032 },
2033 "round_number": 0,
2034 "owner": {
2035 "did": "did:plc:hwevmowznbiukdf6uk5dwrrq"
2036 },
2037 "html_url": "https://tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo/pulls/4",
2038 "patch_url": "https://tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo/pulls/4/round/0.patch",
2039 "created_at": "2025-09-15T08:57:23Z"
2040 },
2041 "repository": { ... },
2042 "sender": {
2043 "did": "did:plc:hwevmowznbiukdf6uk5dwrrq"
2044 }
2045}
2046```
2047
2048Notes:
2049
2050- `action` mirrors the event type suffix (`created`, `resubmitted`, `merged`, `closed`, `reopened`).
2051- `repository` has the same format as in the push payload.
2052- The patch itself is not embedded in the payload (patches can be large); fetch it from `patch_url` instead. `round_number` identifies the latest round, and `patch_url` always points at that round's patch.
2053- `source` is only present for branch-based and fork-based pull requests; it is omitted for patch-based pulls. For fork-based pulls, `source.repo` contains the DID of the source repository.
2054- `sender` is the user who performed the action.
2055
2056## HTTP headers
2057
2058Each webhook request includes the following headers:
2059
2060- `Content-Type: application/json`
2061- `User-Agent: Tangled-Hook/<short-sha>` — User agent with short SHA of the commit (push events); `Tangled-Hook/pull_request` for pull request events
2062- `X-Tangled-Event: push` — The full event type (e.g. `push`, `pull_request:merged`)
2063- `X-Tangled-Hook-ID: <webhook-id>` — The webhook ID
2064- `X-Tangled-Delivery: <uuid>` — Unique delivery ID
2065- `X-Tangled-Signature-256: sha256=<hmac>` — HMAC-SHA256 signature (if secret configured)
2066
2067## Verifying webhook signatures
2068
2069If you configured a secret, you should verify the webhook signature to ensure requests are authentic. For example, in Go:
2070
2071```go
2072package main
2073
2074import (
2075 "crypto/hmac"
2076 "crypto/sha256"
2077 "encoding/hex"
2078 "io"
2079 "net/http"
2080 "strings"
2081)
2082
2083func verifySignature(payload []byte, signatureHeader, secret string) bool {
2084 // Remove 'sha256=' prefix from signature header
2085 signature := strings.TrimPrefix(signatureHeader, "sha256=")
2086
2087 // Compute expected signature
2088 mac := hmac.New(sha256.New, []byte(secret))
2089 mac.Write(payload)
2090 expected := hex.EncodeToString(mac.Sum(nil))
2091
2092 // Use constant-time comparison to prevent timing attacks
2093 return hmac.Equal([]byte(signature), []byte(expected))
2094}
2095
2096func webhookHandler(w http.ResponseWriter, r *http.Request) {
2097 // Read the request body
2098 payload, err := io.ReadAll(r.Body)
2099 if err != nil {
2100 http.Error(w, "Bad request", http.StatusBadRequest)
2101 return
2102 }
2103
2104 // Get signature from header
2105 signatureHeader := r.Header.Get("X-Tangled-Signature-256")
2106
2107 // Verify signature
2108 if signatureHeader != "" && verifySignature(payload, signatureHeader, yourSecret) {
2109 // Webhook is authentic, process it
2110 processWebhook(payload)
2111 w.WriteHeader(http.StatusOK)
2112 } else {
2113 http.Error(w, "Invalid signature", http.StatusUnauthorized)
2114 }
2115}
2116```
2117
2118## Delivery retries
2119
2120Webhooks are automatically retried on failure:
2121
2122- **3 total attempts** (1 initial + 2 retries)
2123- **Exponential backoff** starting at 1 second, max 10 seconds
2124- **Retried on**:
2125 - Network errors
2126 - HTTP 5xx server errors
2127- **Not retried on**:
2128 - HTTP 4xx client errors (bad request, unauthorized, etc.)
2129
2130### Timeouts
2131
2132Webhook requests timeout after 30 seconds. If your endpoint needs more time:
2133
21341. Respond with 200 OK immediately
21352. Process the webhook asynchronously in the background
2136
2137## Example integrations
2138
2139### Discord notifications
2140
2141```javascript
2142app.post("/webhook", (req, res) => {
2143 const payload = req.body;
2144
2145 fetch("https://discord.com/api/webhooks/...", {
2146 method: "POST",
2147 headers: { "Content-Type": "application/json" },
2148 body: JSON.stringify({
2149 content: `New push to ${payload.repository.full_name}`,
2150 embeds: [
2151 {
2152 title: `${payload.pusher.did} pushed to ${payload.ref}`,
2153 url: payload.repository.html_url,
2154 color: 0x00ff00,
2155 },
2156 ],
2157 }),
2158 });
2159
2160 res.status(200).send("OK");
2161});
2162```
2163
2164# Migrating knots and spindles
2165
2166Sometimes, non-backwards compatible changes are made to the
2167knot/spindle XRPC APIs. If you host a knot or a spindle, you
2168will need to follow this guide to upgrade. Typically, this
2169only requires you to deploy the newest version.
2170
2171This document is laid out in reverse-chronological order.
2172Newer migration guides are listed first, and older guides
2173are further down the page.
2174
2175## Upgrading to v1.16.0-alpha
2176
2177Starting with v1.16.0-alpha, spindles own CI pipeline data
2178directly. The appview no longer stores pipeline runs or follows
2179spindle event streams for pipeline history. Instead, it asks the
2180configured spindle for pipeline lists, single pipeline details,
2181workflow logs, retries, and cancellations over XRPC.
2182
2183This means that existing pipeline logs / runs won't appear after
2184you upgrade. Existing pipeline history from the appview cannot be
2185automatically migrated. If you want to migrate your data, you can
2186reach out to us and we will send you an SQL file that'll add the data
2187into your spindle.
2188
2189- Upgrade to the latest tag (v1.16.0 or above)
2190- Head to the [spindle
2191 dashboard](https://tangled.org/settings/spindles) and hit the
2192 "retry" button to verify your spindle
2193
2194## Upgrading to v1.15.0-alpha
2195
2196With v1.15.0-alpha, a knot itself owns its members and
2197per-repo collaborators directly. Previously this data was sourced from
2198PDS records (`sh.tangled.knot.member` and `sh.tangled.repo.collaborator`)
2199that the appview and the knot both read off the firehose.
2200The knot is now the source of truth and serves them over XRPC instead:
2201
2202- `sh.tangled.knot.addMember`, `sh.tangled.knot.removeMember`, `sh.tangled.knot.listMembers`
2203- `sh.tangled.repo.addCollaborator`, `sh.tangled.repo.removeCollaborator`, `sh.tangled.repo.listCollaborators`
2204
2205Until your knot is upgraded, the appview keeps reading its
2206members and collaborators from the old firehose-sourced records.
2207Upgrade to move your knot onto knot-owned access control.
2208
2209- Upgrade to the latest tag (v1.15.0 or above)
2210- Head to the [knot dashboard](https://tangled.org/settings/knots) and
2211 hit the "retry" button to verify your knot
2212
2213## Upgrading to v1.14.0-alpha
2214
2215Starting with v1.14.0-alpha, the fully knot uses the repoDID as its
2216canonical handle for repositories. This unlocks repository
2217renames from the appview UI and changes the wire format for
2218the following lexicons (`sh.tangled.repo.pull`, `sh.tangled.repo.collaborator`,
2219`sh.tangled.repo.issue`, `sh.tangled.git.refUpdate`).
2220
2221Knots that have not been upgraded may silently drop new push
2222events, pull requests, issues, and collaborator invites for
2223repositories they host until upgraded. So upgrade please!!!
2224
2225- Upgrade to the latest tag (v1.14.0 or above)
2226- Head to the [knot dashboard](https://tangled.org/settings/knots) and
2227 hit the "retry" button to verify your knot
2228
2229## Upgrading to v1.13.0-alpha
2230
2231Starting with v1.13.0-alpha, every repository on a knot is
2232assigned a DID. This makes repositories stable across
2233renames and transfers.
2234
2235When you upgrade your knot to this version, the server will
2236automatically mint DIDs for all existing repositories on
2237startup. This is a one-time process and you may see
2238additional log output during the first boot as DIDs are
2239assigned.
2240
2241- Upgrade to the latest tag (v1.13.0 or above)
2242- Head to the [knot dashboard](https://tangled.org/settings/knots) and
2243 hit the "retry" button to verify your knot
2244
2245## Upgrading from v1.8.x
2246
2247After v1.8.2, the HTTP API for knots and spindles has been
2248deprecated and replaced with XRPC. Repositories on outdated
2249knots will not be viewable from the appview. Upgrading is
2250straightforward however.
2251
2252For knots:
2253
2254- Upgrade to the latest tag (v1.9.0 or above)
2255- Head to the [knot dashboard](https://tangled.org/settings/knots) and
2256 hit the "retry" button to verify your knot
2257
2258For spindles:
2259
2260- Upgrade to the latest tag (v1.9.0 or above)
2261- Head to the [spindle
2262 dashboard](https://tangled.org/settings/spindles) and hit the
2263 "retry" button to verify your spindle
2264
2265## Upgrading from v1.7.x
2266
2267After v1.7.0, knot secrets have been deprecated. You no
2268longer need a secret from the appview to run a knot. All
2269authorized commands to knots are managed via [Inter-Service
2270Authentication](https://atproto.com/specs/xrpc#inter-service-authentication-jwt).
2271Knots will be read-only until upgraded.
2272
2273Upgrading is quite easy, in essence:
2274
2275- `KNOT_SERVER_SECRET` is no more, you can remove this
2276 environment variable entirely
2277- `KNOT_SERVER_OWNER` is now required on boot, set this to
2278 your DID. You can find your DID in the
2279 [settings](https://tangled.org/settings) page.
2280- Restart your knot once you have replaced the environment
2281 variable
2282- Head to the [knot dashboard](https://tangled.org/settings/knots) and
2283 hit the "retry" button to verify your knot. This simply
2284 writes a `sh.tangled.knot` record to your PDS.
2285
2286If you use the nix module, simply bump the flake to the
2287latest revision, and change your config block like so:
2288
2289```diff
2290 services.tangled.knot = {
2291 enable = true;
2292 server = {
2293- secretFile = /path/to/secret;
2294+ owner = "did:plc:foo";
2295 };
2296 };
2297```
2298
2299# Bobbin
2300
2301Bobbin is an API appview for Tangled records. It serves XRPC
2302endpoints for `sh.tangled.*`, with it you can get repos,
2303issues, pulls, comments, follows, stars, labels, pipelines,
2304and profiles. It is read-only, there is no auth, since that
2305should all be handled direct-to-PDS and knot respectively.
2306
2307**Bobbin has no permanent storage**.
2308
2309It is only a glorified edge index, in the graph theory
2310sense. Additionally it has a record cache, re-filled on
2311demand. All other data that Bobbin serves comes live from
2312PDSes & knots.
2313
2314## What Bobbin needs
2315
2316The way that Bobbin is able to pull off being
2317so stateless is by moving state upstream.
2318Primarily it depends on an instance of
2319[Hydrant](https://tangled.org/did:plc:6v3ul2ptnqctyxwkz5ti4amn)
2320, which is the service that gives an event stream
2321for Bobbin to quickly backfill from on every restart.
2322Backfilling ought to take less than a couple of minutes
2323maximum. If the upstream instance of Hydrant fails
2324while Bobbin is live, its list/count endpoints stop
2325advancing and report a stale cursor. Single-lookups
2326will continue working, due to the second dependency:
2327[Slingshot](https://tangled.org/did:plc:c7mc2fn47ihdihul4vjwsuy3/tree/main/slingshot).
2328Slingshot fetches individual records & resolves identities.
2329If the upstream instance of Slingshot fails, single-lookups
2330will fail with a `502` error. There are some aggregation
2331endpoints that use Slingshot for hydrating, which will also
2332fail.
2333
2334A soft dependency that ought to exist for Bobbin to operate
2335correctly is simply the plethora of knots that are out
2336there, that Bobbin talks to directly for git data and, for
2337knots at v1.15+, members & collaborators.
2338
2339## Building Bobbin
2340
2341Bobbin is under [Tangled's core monorepo, under bobbin/](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/tree/master/bobbin).
2342Here's an easy local debug-build:
2343
2344```sh
2345cargo build -p bobbin
2346```
2347
2348Bobbin loves being in a container. When using
2349`bobbin/containerfiles/bobbin.Containerfile`, it runs `cargo
2350build --release --bin bobbin --package bobbin` within a
2351little Debian runtime, exposing port 8090.
2352
2353## Configuration
2354
2355The best way to configure Bobbin is via a toml config file.
2356There's an `example.toml` in [Bobbin's subdir](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/blob/master/bobbin/example.toml).
2357Every value is overridable by a `BOBBIN_*` env var.
2358The load order is env, then `--config <path>`, then
2359`/etc/bobbin/config.toml`, then built-in defaults.
2360
2361Load and check a config without starting the server:
2362
2363```sh
2364bobbin --config config.toml validate
2365```
2366
2367Minimal config is the two upstream URLs. The hydrant URL
2368takes `ws://` or `wss://`. An `http://` or `https://`
2369URL is rewritten to the matching websocket scheme at
2370connection-time.
2371
2372```toml
2373[server]
2374binds = ["127.0.0.1:8090"]
2375
2376# Loopback-only & can leave empty to disable debug introspection.
2377debug_bind = "127.0.0.1:8091"
2378
2379[hydrant]
2380url = "https://hydrant.example.com"
2381
2382[slingshot]
2383url = "https://slingshot.example.com"
2384```
2385
2386> 🦪 Lewis
2387>
2388> At time of writing, we (Tangled) don't host public
2389> instances of Hydrant or Slingshot. You will have to
2390> find public instances or spin these up yourself! :P
2391
2392Take a gander in the project's example.toml for an
2393exhaustive list of things to configure.
2394
2395You will discover fun things such as a configurable adaptive
2396loop that watches the cgroup memory limit & throttles heavy
2397requests under pressure. It only works if it detects a
2398cgroup limit is present. The config for that is in the
2399`[backpressure]` block of the config template.
2400
2401## Running Bobbin
2402
2403Start the server using a config toml:
2404
2405```bash
2406bobbin --config config.toml
2407```
2408Bobbin wakes up in a cold sweat and immediately gets to
2409work:
24101. It binds its listeners, connects to the Hydrant stream
2411 in the background.
24122. It serves requests from the first
2413 moment it's alive, even before the Hydrant stream connects
2414 or finishes catching up. Having a cold Hydrant itself
2415 costs only latency and approximate counts.
2416
2417## The API
2418
2419**Single lookups** take a record's AT-URI.
2420
2421- `getRepo` takes the repo URI:
2422
2423```sh
2424curl "$BOBBIN/xrpc/sh.tangled.repo.getRepo?repo=at://did:plc:boltless/sh.tangled.repo/squid"
2425```
2426```json
2427{
2428 "uri": "at://did:plc:boltless/sh.tangled.repo/squid",
2429 "cid": "bafyrei...",
2430 "value": { "$type": "sh.tangled.repo", "knot": "knot1.tangled.sh", "description": "...", "createdAt": "..." }
2431}
2432```
2433
2434- `getProfile` takes the full profile record URI, so a bare
2435 handle or DID will not resolve:
2436
2437```sh
2438curl "$BOBBIN/xrpc/sh.tangled.actor.getProfile?actor=at://did:plc:boltless/sh.tangled.actor.profile/self"
2439```
2440
2441- If Slingshot cannot serve the record, the response is `502`:
2442
2443```json
2444{ "error": "UpstreamFailed", "message": "upstream unavailable: ..." }
2445```
2446
2447**Aggregation** endpoints come in `list*` and `count*` pairs,
2448each with a `*By` sibling, and require a `subject` query param.
2449
2450- `listRepos` and `countRepos` key on the owner DID:
2451
2452```sh
2453curl "$BOBBIN/xrpc/sh.tangled.repo.countRepos?subject=did:plc:boltless"
2454```
2455```json
2456{ "count": 7, "distinctAuthors": 1 }
2457```
2458
2459```sh
2460curl "$BOBBIN/xrpc/sh.tangled.repo.listRepos?subject=did:plc:boltless&limit=3"
2461```
2462```json
2463{ "items": [ { "uri": "at://did:plc:boltless/sh.tangled.repo/squid", "cid": "bafyrei...", "value": { } } ], "cursor": null }
2464```
2465
2466- Bobbin validates the subject per collection. Here a repo URI
2467 is passed where a bare DID is required, so the call returns a
2468 `400`:
2469
2470```sh
2471curl "$BOBBIN/xrpc/sh.tangled.graph.listFollows?subject=at://did:plc:boltless/sh.tangled.repo/squid"
2472```
2473```json
2474{ "error": "InvalidRequest", "message": "invalid request: subject must be a bare did, got at-uri with collection sh.tangled.repo" }
2475```
2476
2477**Search** is a single endpoint over an in-mem full-text
2478index:
2479
2480```sh
2481curl "$BOBBIN/xrpc/sh.tangled.search.query?q=tangled&limit=2"
2482```
2483```json
2484{ "hits": [ { "uri": "at://...", "cid": "...", "nsid": "sh.tangled.repo", "score": 27.1, "value": { } } ], "cursor": null }
2485```
2486
2487**Git data** such as blob, tree, diff, log, and archive proxies
2488straight to the repo's knot, streamed back without caching.
2489
2490## Coverage and warm-up
2491
2492- While the edge index is catching up from Hydrant,
2493 the aggregation count is a lower bound & may still climb.
2494- One endpoint reports how far along the backfill it is:
2495
2496```sh
2497curl "$BOBBIN/xrpc/sh.tangled.bobbin.getCoverage"
2498```
2499
2500While warming up:
2501
2502```json
2503{ "ready": false, "eventsProcessed": 45588, "lastCursor": 51658 }
2504```
2505
2506Once caught up, Bobbin flips to ready:
2507
2508```json
2509{ "ready": true, "eventsProcessed": 106085, "lastCursor": 116527 }
2510```
2511
2512If starting up Hydrant for the first time, Hydrant itself
2513will take a decent while (a couple of hours) to backfill
2514from PDSes. Hydrant stores its backfill on disk. Bobbin
2515restart reaches `ready` in minutes by replaying event from
2516an already-populated Hydrant. If your Hydrant is new, expect
2517Bobbin to backfill in that same couple of hours that Hydrant
2518takes.
2519
2520## Loose ends and not-gonna-impl
2521
2522- **No coverage signal for per-knot rosters yet.**
2523 Coverage tracks the hydrant stream only. A v1.15 knot
2524 that is unreachable serves a stale or empty member set
2525 with nothing to flag it.
2526- **Knot eventstream fan-out isn't pooled.**
2527 Bobbin opens one websocket per v1.15
2528 knot on top of the hydrant subscription. A network with
2529 thousands of knots wants pooling or a shared subscription.
2530- **No sequential issue or PR numbers.** bobbin returns rkeys,
2531 not `#42` style ids like the web appview. A client
2532 deriving a display number does it from creation order. But
2533 why bother? rkeys are the IDs.
2534
2535# Hacking on Tangled
2536
2537We highly recommend [installing
2538Nix](https://nixos.org/download/) (the package manager)
2539before working on the codebase. The Nix flake provides a lot
2540of helpers to get started and most importantly, builds and
2541dev shells are entirely deterministic.
2542
2543To set up your dev environment:
2544
2545```bash
2546nix develop
2547```
2548
2549Non-Nix users can look at the `devShell` attribute in the
2550`flake.nix` file to determine necessary dependencies.
2551
2552## Running the appview
2553
2554The appview requires Redis and OAuth JWKs. Start these
2555first, before launching the appview itself.
2556
2557```bash
2558# OAuth JWKs should already be set up by the Nix devshell:
2559echo $TANGLED_OAUTH_CLIENT_SECRET
2560z42ty4RT1ovnTopY8B8ekz9NuziF2CuMkZ7rbRFpAR9jBqMc
2561
2562echo $TANGLED_OAUTH_CLIENT_KID
25631761667908
2564
2565# if not, you can set it up yourself:
2566goat key generate -t P-256
2567Key Type: P-256 / secp256r1 / ES256 private key
2568Secret Key (Multibase Syntax): save this securely (eg, add to password manager)
2569 z42tuPDKRfM2mz2Kv953ARen2jmrPA8S9LX9tRq4RVcUMwwL
2570Public Key (DID Key Syntax): share or publish this (eg, in DID document)
2571 did:key:zDnaeUBxtG6Xuv3ATJE4GaWeyXM3jyamJsZw3bSPpxx4bNXDR
2572
2573# the secret key from above
2574export TANGLED_OAUTH_CLIENT_SECRET="z42tuP..."
2575
2576# Run Redis in a new shell to store OAuth sessions
2577redis-server
2578```
2579
2580The Nix flake exposes a few `app` attributes (run `nix
2581flake show` to see a full list of what the flake provides),
2582one of the apps runs the appview with the `air`
2583live-reloader:
2584
2585```bash
2586TANGLED_DEV=true nix run .#watch-appview
2587
2588# TANGLED_DB_PATH might be of interest to point to
2589# different sqlite DBs
2590
2591# in a separate shell, you can live-reload tailwind
2592nix run .#watch-tailwind
2593```
2594
2595## Running knots and spindles
2596
2597An end-to-end knot setup requires setting up a machine with
2598`sshd`, `AuthorizedKeysCommand`, and a Git user, which is
2599quite cumbersome. So the Nix flake provides a
2600`nixosConfiguration` to do so.
2601
2602<details>
2603 <summary><strong>macOS users will have to set up a Nix Builder first</strong></summary>
2604
2605In order to build Tangled's dev VM on macOS, you will
2606first need to set up a Linux Nix builder. The recommended
2607way to do so is to run a [`darwin.linux-builder`
2608VM](https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder)
2609and to register it in `nix.conf` as a builder for Linux
2610with the same architecture as your Mac (`linux-aarch64` if
2611you are using Apple Silicon).
2612
2613If you're on nix-darwin, you can simply add
2614
2615```
2616nix.linux-builder.enable = true;
2617```
2618
2619to your host's `configuration.nix`.
2620
2621Alternatively, you can use any other method to set up a
2622Linux machine with Nix installed that you can `sudo ssh`
2623into (in other words, root user on your Mac has to be able
2624to ssh into the Linux machine without entering a password)
2625and that has the same architecture as your Mac. See
2626[remote builder
2627instructions](https://nix.dev/manual/nix/2.28/advanced-topics/distributed-builds.html#requirements)
2628for how to register such a builder in `nix.conf`.
2629
2630> WARNING: If you'd like to use
2631> [`nixos-lima`](https://github.com/nixos-lima/nixos-lima) or
2632> [Orbstack](https://orbstack.dev/), note that setting them up so that `sudo
2633ssh` works can be tricky. It seems to be [possible with
2634> Orbstack](https://github.com/orgs/orbstack/discussions/1669).
2635
2636</details>
2637
2638To begin, grab your DID from http://localhost:3000/settings.
2639Then, set `TANGLED_VM_KNOT_OWNER` to your DID. You can now start a
2640lightweight NixOS VM like so:
2641
2642```bash
2643nix run --impure .#vm
2644
2645# type `poweroff` at the shell to exit the VM
2646```
2647
2648This starts a knot on port 6444, a spindle on port 6555
2649with `ssh` exposed on port 2222.
2650
2651Once the services are running, head to
2652http://localhost:3000/settings/knots and hit "Verify". It should
2653verify the ownership of the services instantly if everything
2654went smoothly.
2655
2656You can push repositories to this VM with this ssh config
2657block on your main machine:
2658
2659```bash
2660Host nixos-shell
2661 Hostname localhost
2662 Port 2222
2663 User git
2664 IdentityFile ~/.ssh/my_tangled_key
2665```
2666
2667Set up a remote called `local-dev` on a git repo:
2668
2669```bash
2670git remote add local-dev git@nixos-shell:user/repo
2671git push local-dev main
2672```
2673
2674The above VM should already be running a spindle on
2675`localhost:6555`. Head to http://localhost:3000/settings/spindles and
2676hit "Verify". You can then configure each repository to use
2677this spindle and run CI jobs.
2678
2679Of interest when debugging spindles:
2680
2681```
2682# Service logs from journald:
2683journalctl -xeu spindle
2684
2685# CI job logs from disk:
2686ls /var/log/spindle
2687
2688# Debugging spindle database:
2689sqlite3 /var/lib/spindle/spindle.db
2690
2691# litecli has a nicer REPL interface:
2692litecli /var/lib/spindle/spindle.db
2693```
2694
2695If for any reason you wish to disable either one of the
2696services in the VM, modify [nix/vm.nix](/nix/vm.nix) and set
2697`services.tangled.spindle.enable` (or
2698`services.tangled.knot.enable`) to `false`.
2699
2700# Contribution guide
2701
2702## Commit guidelines
2703
2704We follow a commit style similar to the Go project. Please keep commits:
2705
2706- **atomic**: each commit should represent one logical change
2707- **descriptive**: the commit message should clearly describe what the
2708 change does and why it's needed
2709
2710### Message format
2711
2712```
2713<service/top-level directory>/<affected package/directory>: <short summary of change>
2714
2715Optional longer description can go here, if necessary. Explain what the
2716change does and why, especially if not obvious. Reference relevant
2717issues or PRs when applicable. These can be links for now since we don't
2718auto-link issues/PRs yet.
2719```
2720
2721Here are some examples:
2722
2723```
2724appview/state: fix token expiry check in middleware
2725
2726The previous check did not account for clock drift, leading to premature
2727token invalidation.
2728```
2729
2730```
2731knotserver/git/service: improve error checking in upload-pack
2732```
2733
2734### General notes
2735
2736- PRs get merged "as-is" (fast-forward)—like applying a patch-series
2737 using `git am`. At present, there is no squashing—so please author
2738 your commits as they would appear on `master`, following the above
2739 guidelines.
2740- If there is a lot of nesting, for example "appview:
2741 pages/templates/repo/fragments: ...", these can be truncated down to
2742 just "appview: repo/fragments: ...". If the change affects a lot of
2743 subdirectories, you may abbreviate to just the top-level names, e.g.
2744 "appview: ..." or "knotserver: ...".
2745- Keep commits lowercased with no trailing period.
2746- Use the imperative mood in the summary line (e.g., "fix bug" not
2747 "fixed bug" or "fixes bug").
2748- Try to keep the summary line under 72 characters, but we aren't too
2749 fussed about this.
2750- Follow the same formatting for PR titles if filled manually.
2751- Don't include unrelated changes in the same commit.
2752- Avoid noisy commit messages like "wip" or "final fix"—rewrite history
2753 before submitting if necessary.
2754
2755## Code formatting
2756
2757We use a variety of tools to format our code, and multiplex them with
2758[`treefmt`](https://treefmt.com). All you need to do to format your changes
2759is run `nix run .#fmt` (or just `treefmt` if you're in the devshell).
2760
2761## Proposals for bigger changes
2762
2763Small fixes like typos, minor bugs, or trivial refactors can be
2764submitted directly as PRs.
2765
2766For larger changes—especially those introducing new features, significant
2767refactoring, or altering system behavior—please open a proposal first. This
2768helps us evaluate the scope, design, and potential impact before implementation.
2769
2770Create a new issue titled:
2771
2772```
2773proposal: <affected scope>: <summary of change>
2774```
2775
2776In the description, explain:
2777
2778- What the change is
2779- Why it's needed
2780- How you plan to implement it (roughly)
2781- Any open questions or tradeoffs
2782
2783We'll use the issue thread to discuss and refine the idea before moving
2784forward.
2785
2786## Developer Certificate of Origin (DCO)
2787
2788We require all contributors to certify that they have the right to
2789submit the code they're contributing. To do this, we follow the
2790[Developer Certificate of Origin
2791(DCO)](https://developercertificate.org/).
2792
2793By signing your commits, you're stating that the contribution is your
2794own work, or that you have the right to submit it under the project's
2795license. This helps us keep things clean and legally sound.
2796
2797To sign your commit, just add the `-s` flag when committing:
2798
2799```sh
2800git commit -s -m "your commit message"
2801```
2802
2803This appends a line like:
2804
2805```
2806Signed-off-by: Your Name <your.email@example.com>
2807```
2808
2809We won't merge commits if they aren't signed off. If you forget, you can
2810amend the last commit like this:
2811
2812```sh
2813git commit --amend -s
2814```
2815
2816If you're submitting a PR with multiple commits, make sure each one is
2817signed.
2818
2819For [jj](https://jj-vcs.github.io/jj/latest/) users, you can run the following command
2820to make it sign off commits in the tangled repo:
2821
2822```shell
2823# Safety check, should say "No matching config key..."
2824jj config list templates.commit_trailers
2825# The command below may need to be adjusted if the command above returned something.
2826jj config set --repo templates.commit_trailers "format_signed_off_by_trailer(self)"
2827```
2828
2829Refer to the [jujutsu
2830documentation](https://jj-vcs.github.io/jj/latest/config/#commit-trailers)
2831for more information.
2832
2833# Troubleshooting guide
2834
2835## Login issues
2836
2837Owing to the distributed nature of OAuth on AT Protocol, you
2838may run into issues with logging in. If you run a
2839self-hosted PDS:
2840
2841- You may need to ensure that your PDS is timesynced using
2842 NTP:
2843 - Enable the `ntpd` service
2844 - Run `ntpd -qg` to synchronize your clock
2845- You may need to increase the default request timeout:
2846 `NODE_OPTIONS="--network-family-autoselection-attempt-timeout=500"`
2847
2848## Empty punchcard
2849
2850For Tangled to register commits that you make across the
2851network, you need to setup one of following:
2852
2853- The committer email should be a verified email associated
2854 to your account. You can add and verify emails on the
2855 settings page.
2856- Or, the committer email should be set to your account's
2857 DID: `git config user.email "did:plc:foobar"`. You can find
2858 your account's DID on the settings page
2859
2860## Commit is not marked as verified
2861
2862Tangled only supports SSH commit signatures. Ensure the SSH public key you use
2863for signing is uploaded to Tangled.
2864
2865To sign commits using an SSH key with git:
2866
2867```
2868git config --global gpg.format ssh
2869git config --global user.signingkey ~/.ssh/tangled-key
2870```
2871
2872To sign commits using an SSH key with jj, add this to your
2873config:
2874
2875```
2876[signing]
2877behavior = "own"
2878backend = "ssh"
2879key = "~/.ssh/tangled-key"
2880```
2881
2882## Self-hosted knot issues
2883
2884If you need help troubleshooting a self-hosted knot, check
2885out the [knot troubleshooting
2886guide](/knot-self-hosting-guide.html#troubleshooting).