alice.pds.demo.boltless.dev / core
This repository has no description
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

core / docs / DOCS.md
83 kB 2812 lines
1--- 2title: Tangled docs 3author: The Tangled Contributors 4date: 21 Sun, Dec 2025 5abstract: | 6 Tangled is a decentralized code hosting and collaboration 7 platform. Every component of Tangled is open-source and 8 self-hostable. [tangled.org](https://tangled.org) also 9 provides hosting and CI services that are free to use. 10 11 There are several models for decentralized code 12 collaboration platforms, ranging from ActivityPub’s 13 (Forgejo) federated model, to Radicle’s entirely P2P model. 14 Our approach attempts to be the best of both worlds by 15 adopting the AT Protocol—a protocol for building decentralized 16 social applications with a central identity 17 18 Our approach to this is the idea of “knots”. Knots are 19 lightweight, headless servers that enable users to host Git 20 repositories with ease. Knots are designed for either single 21 or multi-tenant use which is perfect for self-hosting on a 22 Raspberry Pi at home, or larger “community” servers. By 23 default, Tangled provides managed knots where you can host 24 your repositories for free. 25 26 The appview at tangled.org acts as a consolidated "view" 27 into the whole network, allowing users to access, clone and 28 contribute to repositories hosted across different knots 29 seamlessly. 30--- 31 32# Quick start guide 33 34## Login or sign up 35 36You can [login](https://tangled.org) by using your AT Protocol 37account. If you are unclear on what that means, simply head 38to the [signup](https://tangled.org/signup) page and create 39an account. By doing so, you will be choosing Tangled as 40your account provider (you will be granted a handle of the 41form `user.tngl.sh`). 42 43In the AT Protocol network, users are free to choose their account 44provider (known as a "Personal Data Service", or PDS), and 45login to applications that support AT accounts. 46 47You can think of it as "one account for all of the atmosphere"! 48 49If you already have an AT account (you may have one if you 50signed up to Bluesky, for example), you can login with the 51same handle on Tangled (so just use `user.bsky.social` on 52the login page). 53 54## Add an SSH key 55 56Once you are logged in, you can start creating repositories 57and pushing code. Tangled supports pushing git repositories 58over SSH. 59 60First, you'll need to generate an SSH key if you don't 61already have one: 62 63```bash 64ssh-keygen -t ed25519 -C "foo@bar.com" 65``` 66 67When prompted, save the key to the default location 68(`~/.ssh/id_ed25519`) and optionally set a passphrase. 69 70Copy your public key to your clipboard: 71 72```bash 73# on X11 74cat ~/.ssh/id_ed25519.pub | xclip -sel c 75 76# on wayland 77cat ~/.ssh/id_ed25519.pub | wl-copy 78 79# on macos 80cat ~/.ssh/id_ed25519.pub | pbcopy 81``` 82 83Now, navigate to 'Settings' -> 'Keys' and hit 'Add Key', 84paste your public key, give it a descriptive name, and hit 85save. 86 87## Create a repository 88 89Once your SSH key is added, create your first repository: 90 911. Hit the green `+` icon on the topbar, and select 92 repository 932. Enter a repository name 943. Add a description 954. Choose a knotserver to host this repository on 965. Hit create 97 98Knots are self-hostable, lightweight Git servers that can 99host your repository. Unlike traditional code forges, your 100code can live on any server. Read the [Knots](TODO) section 101for more. 102 103## Configure SSH 104 105To ensure Git uses the correct SSH key and connects smoothly 106to Tangled, add this configuration to your `~/.ssh/config` 107file: 108 109``` 110Host tangled.org 111 Hostname tangled.org 112 User git 113 IdentityFile ~/.ssh/id_ed25519 114 AddressFamily inet 115``` 116 117This tells SSH to use your specific key when connecting to 118Tangled and prevents authentication issues if you have 119multiple SSH keys. 120 121Note that this configuration only works for knotservers that 122are hosted by tangled.org. If you use a custom knot, refer 123to the [Knots](TODO) section. 124 125## Push your first repository 126 127Initialize a new Git repository: 128 129```bash 130mkdir my-project 131cd my-project 132 133git init 134echo "# My Project" > README.md 135``` 136 137Add some content and push! 138 139```bash 140git add README.md 141git commit -m "Initial commit" 142git remote add origin git@tangled.org:user.tngl.sh/my-project 143git push -u origin main 144``` 145 146That's it! Your code is now hosted on Tangled. 147 148## Migrating an existing repository 149 150Moving your repositories from GitHub, GitLab, Bitbucket, or 151any other Git forge to Tangled is straightforward. You'll 152simply change your repository's remote URL. At the moment, 153Tangled does not have any tooling to migrate data such as 154GitHub issues or pull requests. 155 156First, create a new repository on tangled.org as described 157in the [Quick Start Guide](#create-a-repository). 158 159Navigate to your existing local repository: 160 161```bash 162cd /path/to/your/existing/repo 163``` 164 165You can inspect your existing Git remote like so: 166 167```bash 168git remote -v 169``` 170 171You'll see something like: 172 173```bash 174origin git@github.com:username/my-project.git (fetch) 175origin git@github.com:username/my-project.git (push) 176``` 177 178Update the remote URL to point to tangled: 179 180```bash 181git remote set-url origin git@tangled.org:user.tngl.sh/my-project 182``` 183 184Verify the change: 185 186```bash 187git remote -v 188``` 189 190You should now see: 191 192```bash 193origin git@tangled.org:user.tngl.sh/my-project (fetch) 194origin git@tangled.org:user.tngl.sh/my-project (push) 195``` 196 197Push all your branches and tags to Tangled: 198 199```bash 200git push -u origin --all 201git push -u origin --tags 202``` 203 204Your repository is now migrated to Tangled! All commit 205history, branches, and tags have been preserved. 206 207## Mirroring a repository to Tangled 208 209If you want to maintain your repository on multiple forges 210simultaneously, for example, keeping your primary repository 211on GitHub while mirroring to Tangled for backup or 212redundancy, you can do so by adding [multiple remotes](https://git-scm.com/docs/git-push#_remotes). 213 214You can configure your local repository to push to both 215Tangled and, say, GitHub. You may already have the following 216setup: 217 218```bash 219$ git remote -v 220origin git@github.com:username/my-project.git (fetch) 221origin git@github.com:username/my-project.git (push) 222``` 223 224Now add Tangled as an additional push URL to the same 225remote: 226 227```bash 228git remote set-url --add --push origin git@tangled.org:user.tngl.sh/my-project 229``` 230 231You also need to re-add the original URL as a push 232destination (Git will now use the original URL to fetch only): 233 234```bash 235git remote set-url --add --push origin git@github.com:username/my-project.git 236``` 237 238Verify your configuration: 239 240```bash 241$ git remote -v 242origin git@github.com:username/my-project.git (fetch) 243origin git@tangled.org:user.tngl.sh/my-project (push) 244origin git@github.com:username/my-project.git (push) 245``` 246 247Notice that there's one fetch URL (the primary remote) and 248two push URLs. Now, whenever you push, Git will 249automatically push to both remotes: 250 251```bash 252git push origin main 253``` 254 255This single command pushes your `main` branch to both GitHub 256and Tangled simultaneously. 257 258To push all branches and tags: 259 260```bash 261git push origin --all 262git push origin --tags 263``` 264 265If you prefer more control over which remote you push to, 266you can maintain separate remotes: 267 268```bash 269git remote add github git@github.com:username/my-project.git 270git remote add tangled git@tangled.org:user.tngl.sh/my-project 271``` 272 273Then push to each explicitly: 274 275```bash 276git push github main 277git push tangled main 278``` 279 280# Hosting websites on Tangled 281 282You can serve static websites directly from your git repositories on 283Tangled. If you've used GitHub Pages or Codeberg Pages, this should feel 284familiar. 285 286## Overview 287 288Every user gets a sites domain. If you signed up through Tangled's own 289PDS (`tngl.sh`), your sites domain is automatically 290`<your-handle>.tngl.sh` no setup needed. Otherwise, you can claim a 291`<subdomain>.tngl.io` domain from your settings. 292 293You can serve multiple sites per domain: 294 295- One **index site** served at the root of your domain (e.g. 296 `alice.tngl.sh`) 297- Any number of **sub-path sites** served under the repository name 298 (e.g. `alice.tngl.sh/my-project`) 299 300## Claiming a domain 301 302If you don't have a `tngl.sh` handle, you need to claim a domain before 303publishing sites: 304 3051. Go to **Settings → Sites** 3062. Enter a subdomain (e.g. `alice` to claim `alice.tngl.io`) 3073. Click **claim** 308 309You can only hold one domain at a time. Releasing a domain puts it in a 31030-day cooldown before anyone else can claim it. 311 312## Configuring a site for a repository 313 3141. Navigate to your repository 3152. Go to **Settings → Sites** 3163. Choose a **branch** to deploy from 3174. Set the **deploy directory** — the path within the repository 318 containing your `index.html`. Use `/` for the root, or a subdirectory 319 like `/docs` or `/public` 3205. Choose the **site type**: 321 - **Index site** — served at the root of your domain (e.g. 322 `alice.tngl.sh`) 323 - **Sub-path site** — served under the repository name (e.g. 324 `alice.tngl.sh/my-project`) 3256. Click **save** 326 327The site will be deployed automatically. You can see the status of your 328previous deploys in the **Recent Deploys** section at the bottom of the 329page. 330 331Sites are redeployed automatically on every push to the configured 332branch. 333 334## Custom domains 335 336Tangled currently doesn't support custom domains for sites. This will be 337added in a future update. 338 339## Deploy directory 340 341The deploy directory is the path within your repository that Tangled 342serves as the site root. It must contain an `index.html`. 343 344| Deploy directory | Result | 345|---|---| 346| `/` | Serves the repository root | 347| `/docs` | Serves the `docs/` subdirectory | 348| `/public` | Serves the `public/` subdirectory | 349 350Directories are served with automatic `index.html` resolution -- a 351request to `/about` will serve `/about/index.html` if it exists. 352 353## Site types 354 355| Type | URL | 356|---|---| 357| Index site | `alice.tngl.sh` | 358| Sub-path site | `alice.tngl.sh/my-project` | 359 360Only one repository can be the index site for a given domain at a time. 361If another repository already holds the index site, you will see a 362notice in the settings and only the sub-path option will be available. 363 364## Deploy triggers 365 366A deployment is triggered automatically when: 367 368- You push to the configured branch 369- You change the site configuration (branch, deploy directory, or site 370 type) 371 372## Disabling a site 373 374To stop serving a site, go to **Settings → Sites** in your repository 375and click **Disable**. This removes the site configuration and stops 376serving the site. The deployed files are also deleted from storage. 377 378Releasing your domain from **Settings → Sites** at the account level 379will disable all sites associated with it and delete their files. 380 381 382# Knot self-hosting guide 383 384So you want to run your own knot server? Great! Here are a few prerequisites: 385 3861. A server of some kind (a VPS, a Raspberry Pi, etc.). Preferably running a Linux distribution of some kind. 3872. A (sub)domain name. People generally use `knot.example.com`. 3883. A valid SSL certificate for your domain. 389 390## NixOS 391 392Refer to the [knot 393module](https://tangled.org/tangled.org/core/blob/master/nix/modules/knot.nix) 394for a full list of options. Sample configurations: 395 396- [The test VM](https://tangled.org/tangled.org/core/blob/master/nix/vm.nix#L85) 397- [@pyrox.dev/nix](https://tangled.org/pyrox.dev/nix/blob/c2b644c214d278af12523618de952ee2eab1af3d/hosts/marvin/services/tangled.nix#L15-26) 398 399## Docker 400 401Refer to 402[@tangled.org/knot-docker](https://tangled.org/@tangled.org/knot-docker). 403Note that this is community maintained. 404 405## Manual setup 406 407First, clone this repository: 408 409``` 410git clone https://tangled.org/@tangled.org/core 411``` 412 413Then, build the `knot` CLI. This is the knot administration 414and operation tool. For the purpose of this guide, we're 415only concerned with these subcommands: 416 417- `knot server`: the main knot server process, typically 418 run as a supervised service 419- `knot guard`: handles role-based access control for git 420 over SSH (you'll never have to run this yourself) 421- `knot keys`: fetches SSH keys associated with your knot; 422 we'll use this to generate the SSH 423 `AuthorizedKeysCommand` 424 425``` 426cd core 427export CGO_ENABLED=1 428go build -o knot ./cmd/knot 429``` 430 431Next, move the `knot` binary to a location owned by `root` -- 432`/usr/local/bin/` is a good choice. Make sure the binary itself is also owned by `root`: 433 434``` 435sudo mv knot /usr/local/bin/knot 436sudo chown root:root /usr/local/bin/knot 437``` 438 439This is necessary because SSH `AuthorizedKeysCommand` requires [really 440specific permissions](https://stackoverflow.com/a/27638306). The 441`AuthorizedKeysCommand` specifies a command that is run by `sshd` to 442retrieve a user's public SSH keys dynamically for authentication. Let's 443set that up. 444 445``` 446sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF 447Match User git 448 AuthorizedKeysCommand /usr/local/bin/knot keys -o authorized-keys 449 AuthorizedKeysCommandUser nobody 450EOF 451``` 452 453Then, reload `sshd`: 454 455``` 456sudo systemctl reload ssh 457``` 458 459Next, create the `git` user. We'll use the `git` user's home directory 460to store repositories: 461 462``` 463sudo adduser git 464``` 465 466Create `/home/git/.knot.env` with the following, updating the values as 467necessary. The `KNOT_SERVER_OWNER` should be set to your 468DID, you can find your DID in the [Settings](https://tangled.sh/settings) page. 469 470``` 471KNOT_REPO_SCAN_PATH=/home/git 472KNOT_SERVER_HOSTNAME=knot.example.com 473APPVIEW_ENDPOINT=https://tangled.org 474KNOT_SERVER_OWNER=did:plc:foobar 475KNOT_SERVER_INTERNAL_LISTEN_ADDR=127.0.0.1:5444 476KNOT_SERVER_LISTEN_ADDR=127.0.0.1:5555 477``` 478 479If you run a Linux distribution that uses systemd, you can 480use the provided service file to run the server. Copy 481[`knotserver.service`](https://tangled.org/tangled.org/core/blob/master/systemd/knotserver.service) 482to `/etc/systemd/system/`. Then, run: 483 484``` 485systemctl enable knotserver 486systemctl start knotserver 487``` 488 489The last step is to configure a reverse proxy like Nginx or Caddy to front your 490knot. Here's an example configuration for Nginx: 491 492``` 493server { 494 listen 80; 495 listen [::]:80; 496 server_name knot.example.com; 497 498 location / { 499 proxy_pass http://localhost:5555; 500 proxy_set_header Host $host; 501 proxy_set_header X-Real-IP $remote_addr; 502 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 503 proxy_set_header X-Forwarded-Proto $scheme; 504 } 505 506 # wss endpoint for git events 507 location /events { 508 proxy_set_header X-Forwarded-For $remote_addr; 509 proxy_set_header Host $http_host; 510 proxy_set_header Upgrade websocket; 511 proxy_set_header Connection Upgrade; 512 proxy_pass http://localhost:5555; 513 } 514 # additional config for SSL/TLS go here. 515} 516 517``` 518 519Remember to use Let's Encrypt or similar to procure a certificate for your 520knot domain. 521 522You should now have a running knot server! You can finalize 523your registration by hitting the `verify` button on the 524[/settings/knots](https://tangled.org/settings/knots) page. This simply creates 525a record on your PDS to announce the existence of the knot. 526 527### Custom paths 528 529(This section applies to manual setup only. Docker users should edit the mounts 530in `docker-compose.yml` instead.) 531 532Right now, the database and repositories of your knot lives in `/home/git`. You 533can move these paths if you'd like to store them in another folder. Be careful 534when adjusting these paths: 535 536- Stop your knot when moving data (e.g. `systemctl stop knotserver`) to prevent 537 any possible side effects. Remember to restart it once you're done. 538- Make backups before moving in case something goes wrong. 539- Make sure the `git` user can read and write from the new paths. 540 541#### Database 542 543As an example, let's say the current database is at `/home/git/knotserver.db`, 544and we want to move it to `/home/git/database/knotserver.db`. 545 546Copy the current database to the new location. Make sure to copy the `.db-shm` 547and `.db-wal` files if they exist. 548 549``` 550mkdir /home/git/database 551cp /home/git/knotserver.db* /home/git/database 552``` 553 554In the environment (e.g. `/home/git/.knot.env`), set `KNOT_SERVER_DB_PATH` to 555the new file path (_not_ the directory): 556 557``` 558KNOT_SERVER_DB_PATH=/home/git/database/knotserver.db 559``` 560 561#### Repositories 562 563As an example, let's say the repositories are currently in `/home/git`, and we 564want to move them into `/home/git/repositories`. 565 566Create the new folder, then move the existing repositories (if there are any): 567 568``` 569mkdir /home/git/repositories 570# move all DIDs into the new folder; these will vary for you! 571mv /home/git/did:plc:wshs7t2adsemcrrd4snkeqli /home/git/repositories 572``` 573 574In the environment (e.g. `/home/git/.knot.env`), update `KNOT_REPO_SCAN_PATH` 575to the new directory: 576 577``` 578KNOT_REPO_SCAN_PATH=/home/git/repositories 579``` 580 581Similarly, update your `sshd` `AuthorizedKeysCommand` to use the updated 582repository path: 583 584``` 585sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF 586Match User git 587 AuthorizedKeysCommand /usr/local/bin/knot keys -o authorized-keys -git-dir /home/git/repositories 588 AuthorizedKeysCommandUser nobody 589EOF 590``` 591 592Make sure to restart your SSH server! 593 594#### MOTD (message of the day) 595 596To configure the MOTD used ("Welcome to this knot!" by default), edit the 597`/home/git/motd` file: 598 599``` 600printf "Hi from this knot!\n" > /home/git/motd 601``` 602 603Note that you should add a newline at the end if setting a non-empty message 604since the knot won't do this for you. 605 606## Secure Mode 607 608Secure Mode isolates each `git` subprocess to the repository it is 609operating on, using two mechanisms: 610 611- **Linux Landlock** restricts the filesystem paths the subprocess 612 can access -- it can only read/write its own repository and the 613 system directories it needs to run. 614- **UID isolation** runs each subprocess as a virtual UID assigned 615 to the repository owner, so that repositories belonging to 616 different owners are isolated from each other at the OS level 617 even if Landlock were somehow bypassed. 618 619Secure Mode requires: 620 621- Linux kernel >= 5.19 (Landlock V2). This is the minimum needed 622 for `git push` to work, because receive-pack's quarantine 623 migration uses cross-directory rename which requires the 624 Landlock `REFER` access right (added in V2). Kernels 5.13-5.18 625 support Landlock V1 and clones will work, but pushes will fail 626 with cross-device link errors. On kernels without any Landlock 627 support (< 5.13), the sandbox call is a no-op: UID isolation 628 still applies but no filesystem restriction is enforced. 629- `CAP_SETUID`, `CAP_SETGID`, and `CAP_CHOWN` available to the 630 knot process. The NixOS module grants these automatically; for 631 manual setups see the `setcap` step below. 632 633### NixOS 634 635Add `server.secureMode = true;` to your knot module configuration: 636 637```nix 638services.tangled.knot = { 639 server.secureMode = true; 640 # ... other options 641}; 642``` 643 644The NixOS module handles everything else automatically: 645 646- Grants the required capabilities to the knot service via 647 `AmbientCapabilities` in the systemd unit. 648- Installs a capability-bearing wrapper at 649 `/run/wrappers/bin/knot` via `security.wrappers`, so that 650 SSH-invoked git operations (pushes) also run under the correct 651 UID without requiring the service to run as root. 652- Runs `knot migrate-isolation` at service start to chown 653 existing repositories to their virtual UIDs. 654 655### Manual setup 656 657**Step 1.** Grant the required capabilities to the knot binary. 658This allows the knot process to switch to virtual UIDs at runtime 659without running as root. You will need to repeat this step 660whenever the binary is updated. 661 662``` 663sudo setcap cap_setuid,cap_setgid,cap_chown+eip /usr/local/bin/knot 664``` 665 666**Step 2.** Run the migration tool to assign virtual UIDs to all 667existing repositories and set their filesystem permissions. This 668must be run as root: 669 670``` 671sudo knot migrate-isolation \ 672 --git-dir /home/git \ 673 --db /home/git/knotserver.db \ 674 --internal-api 127.0.0.1:5444 675``` 676 677You can re-run this at any time with `--force` to reapply 678permissions (e.g. after a manual repair or after updating the 679binary). 680 681**Step 2a.** Ensure the home directory is traversable by 682non-group users. Git subprocesses run as virtual UIDs that are 683not in the git group, and they need to resolve 684`$HOME/.config/git/config` to load the global config: 685 686``` 687sudo chmod o+x /home/git 688``` 689 690This adds only the execute bit, not read -- the virtual UIDs can 691traverse to known paths but cannot list directory contents. 692 693**Step 3.** Enable Secure Mode in your environment file: 694 695``` 696KNOT_SERVER_SECURE_MODE=true 697``` 698 699Or pass it as a flag: 700 701``` 702knot server --secure-mode 703``` 704 705**Step 4.** Regenerate the `AuthorizedKeysCommand` with the 706`-secure-mode` flag. This causes `knot keys` to emit guard 707command lines that include `-secure-mode`, so SSH pushes also 708get UID isolation: 709 710``` 711sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF 712Match User git 713 AuthorizedKeysCommand /usr/local/bin/knot keys \ 714 -o authorized-keys -secure-mode 715 AuthorizedKeysCommandUser nobody 716EOF 717``` 718 719Reload `sshd` after making this change. 720 721> **Note:** the server will refuse to start in Secure Mode if any 722> repositories have not yet been isolation-migrated. Re-run 723> `migrate-isolation` if you see this error. 724 725## Troubleshooting 726 727If you run your own knot, you may run into some of these 728common issues. You can always join the 729[IRC](https://web.libera.chat/#tangled) or 730[Discord](https://chat.tangled.org/) if this section does 731not help. 732 733### Unable to push 734 735If you are unable to push to your knot or repository: 736 7371. First, ensure that you have added your SSH public key to 738 your account 7392. Check to see that your knot has synced the key by running 740 `knot keys` 7413. Check to see if git is supplying the correct private key 742 when pushing: `GIT_SSH_COMMAND="ssh -v" git push ...` 7434. Check to see if `sshd` on the knot is rejecting the push 744 for some reason: `journalctl -xeu ssh` (or `sshd`, 745 depending on your machine). These logs are unavailable if 746 using docker. 7475. Check to see if the knot itself is rejecting the push, 748 depending on your setup, the logs might be in one of the 749 following paths: 750 - `/tmp/knotguard.log` 751 - `/home/git/log` 752 - `/home/git/guard.log` 753 754# Spindles 755 756## Pipelines 757 758Spindle workflows allow you to write CI/CD pipelines in a 759simple format. They're located in the `.tangled/workflows` 760directory at the root of your repository, and are defined 761using YAML. 762 763A workflow has a set of common fields that apply no matter 764which engine you pick: 765 766- [Trigger](#trigger): A **required** field that defines 767 when a workflow should be triggered. 768- [Engine](#engine): A **required** field that defines which 769 engine a workflow should run on. 770- [Clone options](#clone-options): An **optional** field 771 that defines how the repository should be cloned. 772- [Environment](#environment): An **optional** field that 773 allows you to define environment variables. 774- [Steps](#steps): An **optional** field that allows you to 775 define what steps should run in the workflow. 776 777On top of these, each engine has its own options for things 778like dependencies and images. See [Engines](#engines) for 779the per-engine fields. 780 781### Trigger 782 783The first thing to add to a workflow is the trigger, which 784defines when a workflow runs. This is defined using a `when` 785field, which takes in a list of conditions. Each condition 786has the following fields: 787 788- `event`: This is a **required** field that defines when 789 your workflow should run. It's a list that can take one or 790 more of the following values: 791 - `push`: The workflow should run every time a commit is 792 pushed to the repository. 793 - `pull_request`: The workflow should run every time a 794 pull request is made or updated. 795 - `manual`: The workflow can be triggered manually. 796- `branch`: Defines which branches the workflow should run 797 for. If used with the `push` event, commits to the 798 branch(es) listed here will trigger the workflow. If used 799 with the `pull_request` event, updates to pull requests 800 targeting the branch(es) listed here will trigger the 801 workflow. This field has no effect with the `manual` 802 event. Supports glob patterns using `*` and `**` (e.g., 803 `main`, `develop`, `release-*`). Either `branch` or `tag` 804 (or both) must be specified for `push` events. 805- `tag`: Defines which tags the workflow should run for. 806 Only used with the `push` event - when tags matching the 807 pattern(s) listed here are pushed, the workflow will 808 trigger. This field has no effect with `pull_request` or 809 `manual` events. Supports glob patterns using `*` and `**` 810 (e.g., `v*`, `v1.*`, `release-**`). Either `branch` or 811 `tag` (or both) must be specified for `push` events. 812 813For example, if you'd like to define a workflow that runs 814when commits are pushed to the `main` and `develop` 815branches, or when pull requests that target the `main` 816branch are updated, or manually, you can do so with: 817 818```yaml 819when: 820 - event: ["push", "manual"] 821 branch: ["main", "develop"] 822 - event: ["pull_request"] 823 branch: ["main"] 824``` 825 826You can also trigger workflows on tag pushes. For instance, 827to run a deployment workflow when tags matching `v*` are 828pushed: 829 830```yaml 831when: 832 - event: ["push"] 833 tag: ["v*"] 834``` 835 836You can even combine branch and tag patterns in a single 837constraint (the workflow triggers if either matches): 838 839```yaml 840when: 841 - event: ["push"] 842 branch: ["main", "release-*"] 843 tag: ["v*", "stable"] 844``` 845 846### Engine 847 848Next is the engine on which the workflow should run, defined 849using the **required** `engine` field. The currently 850supported engines are: 851 852- `nixery`: This uses an instance of 853 [Nixery](https://nixery.dev) to run steps, which allows 854 you to add [dependencies](#dependencies) from 855 Nixpkgs (https://github.com/NixOS/nixpkgs). You can 856 search for packages on https://search.nixos.org, and 857 there's a pretty good chance the package(s) you're looking 858 for will be there. 859 See [Nixery engine](#nixery-engine). 860- `microvm`: Runs the whole workflow inside its own 861 microVM. Has configuration features for NixOS images 862 that will let you enable services, do Docker-in-VM, etc. 863 See [microVM engine](#microvm-engine). 864 865Example: 866 867```yaml 868engine: "nixery" 869``` 870 871Each engine also adds its own workflow fields (dependencies, 872images, services, and so on). These are documented under 873[Engines](#engines). 874 875### Clone options 876 877When a workflow starts, the first step is to clone the 878repository. You can customize this behavior using the 879**optional** `clone` field. It has the following fields: 880 881- `skip`: Setting this to `true` will skip cloning the 882 repository. This can be useful if your workflow is doing 883 something that doesn't require anything from the 884 repository itself. This is `false` by default. 885- `depth`: This sets the number of commits, or the "clone 886 depth", to fetch from the repository. For example, if you 887 set this to 2, the last 2 commits will be fetched. By 888 default, the depth is set to 1, meaning only the most 889 recent commit will be fetched, which is the commit that 890 triggered the workflow. 891- `submodules`: If you use Git submodules 892 (https://git-scm.com/book/en/v2/Git-Tools-Submodules) 893 in your repository, setting this field to `true` will 894 recursively fetch all submodules. This is `false` by 895 default. 896 897The default settings are: 898 899```yaml 900clone: 901 skip: false 902 depth: 1 903 submodules: false 904``` 905 906### Environment 907 908The `environment` field allows you define environment 909variables that will be available throughout the entire 910workflow. **Do not put secrets here, these environment 911variables are visible to anyone viewing the repository. You 912can add secrets for pipelines in your repository's 913settings.** 914 915Example: 916 917```yaml 918environment: 919 GOOS: "linux" 920 GOARCH: "arm64" 921 NODE_ENV: "production" 922 MY_ENV_VAR: "MY_ENV_VALUE" 923``` 924 925By default, the following environment variables are set: 926 927- `CI` - Always set to `true` to indicate a CI environment 928- `TANGLED_PIPELINE_ID` - The AT URI of the current pipeline 929- `TANGLED_PIPELINE_KIND` - One of `push`, `pull_request` or 930 `manual` 931- `TANGLED_REPO_KNOT` - The repository's knot hostname 932- `TANGLED_REPO_DID` - The DID of the repository owner 933- `TANGLED_REPO_NAME` - The name of the repository 934- `TANGLED_REPO_DEFAULT_BRANCH` - The default branch of the 935 repository 936- `TANGLED_REPO_URL` - The full URL to the repository 937 938These variables are only available when the pipeline is 939triggered by a push: 940 941- `TANGLED_REF` - The full git reference (e.g., 942 `refs/heads/main` or `refs/tags/v1.0.0`) 943- `TANGLED_REF_NAME` - The short name of the reference 944 (e.g., `main` or `v1.0.0`) 945- `TANGLED_REF_TYPE` - The type of reference, either 946 `branch` or `tag` 947- `TANGLED_SHA` - The commit SHA that triggered the pipeline 948- `TANGLED_COMMIT_SHA` - Alias for `TANGLED_SHA` 949 950These variables are only available when the pipeline is 951triggered by a pull request: 952 953- `TANGLED_PR_SOURCE_BRANCH` - The source branch of the pull 954 request 955- `TANGLED_PR_TARGET_BRANCH` - The target branch of the pull 956 request 957- `TANGLED_PR_SOURCE_SHA` - The commit SHA of the source 958 branch 959 960### Steps 961 962The `steps` field allows you to define what steps should run 963in the workflow. It's a list of step objects, each with the 964following fields: 965 966- `name`: This field allows you to give your step a name. 967 This name is visible in your workflow runs, and is used to 968 describe what the step is doing. 969- `command`: This field allows you to define a command to 970 run in that step. The step is run in a Bash shell, and the 971 logs from the command will be visible in the pipelines 972 page on the Tangled website. Any dependencies you added in 973 your engine's section (see [Engines](#engines)) will be 974 available to use here. 975- `environment`: Similar to the global 976 [environment](#environment) config, this **optional** 977 field is a key-value map that allows you to set 978 environment variables for the step. **Do not put secrets 979 here, these environment variables are visible to anyone 980 viewing the repository. You can add secrets for pipelines 981 in your repository's settings.** 982 983Example: 984 985```yaml 986steps: 987 - name: "Build backend" 988 command: "go build" 989 environment: 990 GOOS: "darwin" 991 GOARCH: "arm64" 992 - name: "Build frontend" 993 command: "npm run build" 994 environment: 995 NODE_ENV: "production" 996``` 997 998## Engines 999 1000The common fields above apply to every workflow. Each engine 1001then adds its own fields on top. Pick an engine with the 1002[`engine`](#engine) field and use the matching section below. 1003 1004### Nixery engine 1005 1006#### Dependencies 1007 1008When you're running a workflow you'll usually need additional 1009dependencies. The `dependencies` field lets you define which 1010dependencies to get, and from where. It's a key-value map, 1011with the key being the registry to fetch dependencies from, 1012and the value being the list of dependencies to fetch. 1013 1014The registry URL syntax can be found [on the nix 1015manual](https://nix.dev/manual/nix/2.18/command-ref/new-cli/nix3-registry-add). 1016 1017Say you want to fetch Node.js and Go from `nixpkgs`, and a 1018package called `my_pkg` you've made from your own registry 1019at your repository at 1020`https://tangled.org/@example.com/my_pkg`. You can define 1021those dependencies like so: 1022 1023```yaml 1024dependencies: 1025 # nixpkgs 1026 nixpkgs: 1027 - nodejs 1028 - go 1029 # unstable 1030 nixpkgs/nixpkgs-unstable: 1031 - bun 1032 # custom registry 1033 git+https://tangled.org/@example.com/my_pkg: 1034 - my_pkg 1035``` 1036 1037Now these dependencies are available to use in your 1038workflow! 1039 1040#### Complete nixery workflow 1041 1042```yaml 1043# .tangled/workflows/build.yml 1044 1045when: 1046 - event: ["push", "manual"] 1047 branch: ["main", "develop"] 1048 - event: ["pull_request"] 1049 branch: ["main"] 1050 1051engine: "nixery" 1052 1053# using the default values 1054clone: 1055 skip: false 1056 depth: 1 1057 submodules: false 1058 1059dependencies: 1060 # nixpkgs 1061 nixpkgs: 1062 - nodejs 1063 - go 1064 # custom registry 1065 git+https://tangled.org/@example.com/my_pkg: 1066 - my_pkg 1067 1068environment: 1069 GOOS: "linux" 1070 GOARCH: "arm64" 1071 NODE_ENV: "production" 1072 MY_ENV_VAR: "MY_ENV_VALUE" 1073 1074steps: 1075 - name: "Build backend" 1076 command: "go build" 1077 environment: 1078 GOOS: "darwin" 1079 GOARCH: "arm64" 1080 - name: "Build frontend" 1081 command: "npm run build" 1082 environment: 1083 NODE_ENV: "production" 1084``` 1085 1086If you want another example of a workflow, you can look at 1087the one [Tangled uses to build the 1088project](https://tangled.org/@tangled.org/core/blob/master/.tangled/workflows/build.yml). 1089 1090### microVM engine 1091 1092#### Image 1093 1094A workflow picks the image to boot with the top-level `image` 1095field: 1096 1097```yaml 1098engine: microvm 1099image: nixos 1100``` 1101 1102There are two flavours of images: 1103 1104- **NixOS images** (e.g. `nixos`): the whole guest is built 1105 with Nix, so you can configure it from the workflow file 1106 itself. The `dependencies`, `services`, `virtualisation`, 1107 `registry` and `caches` fields below are all understood 1108 here, and the guest builds and activates that configuration 1109 before any of your steps run. 1110- **Non-NixOS images** (e.g. `alpine`): there's no NixOS to 1111 configure, so the workflow-level config fields above have 1112 no effect. You still get a full machine to run steps in. 1113 1114The available image names depend on what the spindle operator 1115has installed. `nixos` and `alpine` are examples. If `image` 1116is omitted, the spindle's configured default image is used. 1117 1118#### Dependencies 1119 1120On the microVM engine, `dependencies` is a flat list of 1121packages that are made available to every step. This field 1122only applies to **NixOS images**; for other images you can 1123use the package manager included in a step. 1124 1125The guest builds a [`nix develop`](https://nix.dev/manual/nix/2.18/command-ref/new-cli/nix3-develop)-style 1126devshell from your dependencies and uses it for each step, 1127so you can, for example, add `pkg-config` and `openssl` and 1128have the `openssl-sys` crate while compiling a Rust project 1129just work. 1130 1131A bare name like `go` is looked up in nixpkgs. You can also 1132point at any flake with the `flakeref#attr` syntax, so 1133`github:nixos/nixpkgs#hello` pulls `hello` straight out of 1134that flake. 1135 1136```yaml 1137dependencies: 1138 - go 1139 - github:nixos/nixpkgs#hello 1140``` 1141 1142#### Registry 1143 1144The `registry` field remaps flake references, the same way 1145`nix registry` does. This lets you pin or alias the flakes 1146used by `dependencies`. 1147 1148For example, pin `nixpkgs` to `nixos-unstable` so that the 1149bare `go` above resolves from unstable, and alias your own 1150flake so you can use `myflake#tool` in `dependencies`: 1151 1152```yaml 1153registry: 1154 nixpkgs: github:nixos/nixpkgs/nixos-unstable 1155 myflake: github:me/x 1156``` 1157 1158#### Caches 1159 1160The `caches` field is a map of Nix binary cache URL to its 1161trusted public key. These are fed into the spindle's read 1162proxy, so the guest can substitute prebuilt paths from them 1163instead of building everything from scratch. 1164 1165```yaml 1166caches: 1167 https://nix-community.cachix.org: "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=" 1168``` 1169 1170#### Services and virtualisation 1171 1172The `services` and `virtualisation` fields are passed straight 1173through to NixOS. Anything you could write under 1174`services.*` or `virtualisation.*` in a NixOS configuration, 1175you can write here, and it's brought up before any of your 1176steps run. 1177 1178As a convenience, `true` works as shorthand for 1179`.enable = true` anywhere an `enable` option exists (e.g. 1180`virtualisation.docker: true`). 1181 1182```yaml 1183services: 1184 postgresql: 1185 enable: true 1186 ensureDatabases: ["spindle-workflow"] 1187 ensureUsers: 1188 - name: spindle-workflow 1189 ensureDBOwnership: true 1190 1191virtualisation: 1192 docker: true 1193``` 1194 1195#### Recipes 1196 1197##### Lint, test and build a Node project 1198 1199```yaml 1200when: 1201 - event: ["push", "pull_request"] 1202 branch: ["main"] 1203 1204engine: microvm 1205image: nixos 1206 1207dependencies: 1208 - pnpm 1209 1210steps: 1211 - name: "Install dependencies" 1212 command: pnpm install --frozen-lockfile 1213 - name: "Lint and test" 1214 command: | 1215 pnpm run lint 1216 pnpm test 1217 - name: "Build" 1218 command: pnpm run build 1219``` 1220 1221##### Check formatting 1222 1223```yaml 1224when: 1225 - event: ["push", "pull_request"] 1226 branch: ["main"] 1227 1228engine: microvm 1229image: alpine # slimmer image for checking the formatting 1230 1231steps: 1232 - name: "Install go" 1233 command: apk add go 1234 - name: "Check formatting" 1235 command: test -z $(gofmt -l .) 1236``` 1237 1238##### Build a Rust project that links OpenSSL 1239 1240```yaml 1241when: 1242 - event: ["push", "pull_request"] 1243 branch: ["main"] 1244 1245engine: microvm 1246image: nixos 1247 1248dependencies: 1249 - gcc 1250 - cargo 1251 - rustc 1252 - clippy 1253 - rustfmt 1254 - pkg-config # exports PKG_CONFIG_PATH for the libraries below 1255 - openssl # the C library + headers openssl-sys links against 1256 1257steps: 1258 - name: "Check formatting" 1259 command: cargo fmt --check 1260 - name: "Clippy" 1261 command: cargo clippy --all-targets -- -D warnings 1262 - name: "Test" 1263 command: cargo test --all 1264 - name: "Release build" 1265 command: cargo build --release 1266``` 1267 1268##### Run migrations and integration tests against PostgreSQL 1269 1270```yaml 1271when: 1272 - event: ["push", "pull_request"] 1273 branch: ["main"] 1274 1275engine: microvm 1276image: nixos 1277 1278environment: 1279 DATABASE_URL: "postgresql:///spindle-workflow?host=/run/postgresql" 1280 1281dependencies: 1282 - gcc 1283 - cargo 1284 - rustc 1285 - pkg-config 1286 - openssl 1287 - sqlx-cli 1288 1289services: 1290 postgresql: 1291 enable: true 1292 # has to be same name as the user for peer auth to work automatically 1293 ensureDatabases: ["spindle-workflow"] 1294 ensureUsers: 1295 - name: spindle-workflow 1296 ensureDBOwnership: true 1297 1298steps: 1299 - name: "Run migrations" 1300 command: sqlx migrate run 1301 - name: "Integration tests" 1302 command: cargo test --all 1303``` 1304 1305##### Build and push a Docker image on tag 1306 1307```yaml 1308when: 1309 - event: ["push"] 1310 tag: ["v*"] 1311 1312engine: microvm 1313image: nixos 1314 1315virtualisation: 1316 docker: true 1317 1318steps: 1319 - name: "Build and push to ghcr.io" 1320 command: | 1321 set -euo pipefail 1322 1323 echo "$REGISTRY_TOKEN" | docker login ghcr.io -u "$REGISTRY_USER" --password-stdin 1324 image="ghcr.io/$REGISTRY_USER/myapp:$TANGLED_REF_NAME" 1325 1326 docker build -t "$image" -t "ghcr.io/$REGISTRY_USER/myapp:latest" . 1327 docker push "$image" 1328 docker push "ghcr.io/$REGISTRY_USER/myapp:latest" 1329``` 1330 1331##### Deploy to Cloudflare Workers on tag 1332 1333```yaml 1334# .tangled/workflows/deploy.yml 1335when: 1336 - event: ["push"] 1337 tag: ["v*"] 1338 1339engine: microvm 1340image: nixos 1341 1342dependencies: 1343 - pnpm 1344 1345steps: 1346 - name: "Install dependencies" 1347 command: pnpm install --frozen-lockfile 1348 - name: "Deploy worker" 1349 # `wrangler` picks up `CLOUDFLARE_API_TOKEN` from the env. 1350 # set it under **Settings → Secrets**. 1351 command: pnpm exec wrangler deploy 1352``` 1353 1354##### Publish a release artifact 1355 1356```yaml 1357when: 1358 - event: ["push"] 1359 tag: ["v*"] # trigger on versions 1360 1361engine: microvm 1362image: nixos 1363 1364dependencies: 1365 - go 1366 1367steps: 1368 - name: "Build release binary" 1369 command: | 1370 mkdir -p dist 1371 CGO_ENABLED=0 go build -trimpath -ldflags "-s -w" -o dist/myapp ./cmd/myapp 1372 1373 - name: "Publish artifact record" 1374 command: | 1375 set -euo pipefail 1376 # change this if you're not on `tngl.sh` 1377 PDS="https://tngl.sh" 1378 # also update this to your handle or did 1379 ATP_IDENTIFIER="user.tngl.sh" 1380 ARTIFACT_PATH="dist/myapp" 1381 ARTIFACT_NAME="myapp" 1382 1383 # set `ATP_APP_PASSWORD` under **Settings → Secrets** 1384 session=$(curl -fsS -X POST "$PDS/xrpc/com.atproto.server.createSession" \ 1385 -H "Content-Type: application/json" \ 1386 -d "{\"identifier\":\"$ATP_IDENTIFIER\",\"password\":\"$ATP_APP_PASSWORD\"}") 1387 jwt=$(echo "$session" | jq -r .accessJwt) 1388 did=$(echo "$session" | jq -r .did) 1389 1390 # upload the binary as a blob 1391 blob=$(curl -fsS -X POST "$PDS/xrpc/com.atproto.repo.uploadBlob" \ 1392 -H "Authorization: Bearer $jwt" \ 1393 -H "Content-Type: application/octet-stream" \ 1394 --data-binary @"$ARTIFACT_PATH") 1395 1396 # note that this requires an annotated tag (`git tag -a v1.0.0 -m ...`) 1397 tag_hash=$(git rev-parse "$TANGLED_REF_NAME^{tag}") 1398 tag_bytes=$(printf '%s' "$tag_hash" | xxd -r -p | base64 | tr -d '=') 1399 1400 # the sh.tangled.repo.artifact record for your artifact 1401 record=$(jq -n \ 1402 --arg did "$did" \ 1403 --arg tag "$tag_bytes" \ 1404 --arg name "$ARTIFACT_NAME" \ 1405 --arg repo "$TANGLED_REPO_URL" \ 1406 --arg created "$(date -Iseconds)" \ 1407 --argjson blob "$(echo "$blob" | jq .blob)" '{ 1408 repo: $did, 1409 collection: "sh.tangled.repo.artifact", 1410 validate: false, 1411 record: { 1412 "$type": "sh.tangled.repo.artifact", 1413 tag: {"$bytes": $tag}, 1414 name: $name, 1415 repo: $repo, 1416 artifact: $blob, 1417 createdAt: $created 1418 } 1419 }') 1420 1421 # create the record on the PDS 1422 curl -fsS -X POST "$PDS/xrpc/com.atproto.repo.createRecord" \ 1423 -H "Authorization: Bearer $jwt" \ 1424 -H "Content-Type: application/json" \ 1425 -d "$record" 1426``` 1427 1428## Self-hosting guide 1429 1430### Prerequisites 1431 1432- Go 1433- For the **nixery** engine: Docker (or Podman with Docker 1434 compatibility enabled). 1435- For the **microVM** engine: a Linux host with KVM, plus the 1436 microVM host dependencies described in [Running microVM 1437 workflows](#running-microvm-workflows). 1438 1439### Configuration 1440 1441Spindle is configured using environment variables. The following environment variables are available: 1442 1443- `SPINDLE_SERVER_LISTEN_ADDR`: The address the server listens on (default: `"0.0.0.0:6555"`). 1444- `SPINDLE_SERVER_DB_PATH`: The path to the SQLite database file (default: `"spindle.db"`). 1445- `SPINDLE_SERVER_HOSTNAME`: The hostname of the server (required). 1446- `SPINDLE_SERVER_JETSTREAM_ENDPOINT`: The endpoint of the Jetstream server (default: `"wss://jetstream1.us-west.bsky.network/subscribe"`). 1447- `SPINDLE_SERVER_DEV`: A boolean indicating whether the server is running in development mode (default: `false`). 1448- `SPINDLE_SERVER_OWNER`: The DID of the owner (required). 1449- `SPINDLE_SERVER_LOG_DIR`: The directory to store workflow logs (default: `"/var/log/spindle"`). 1450- `SPINDLE_SERVER_DOCKER_SOCKET`: Path to Docker socket to expose to invoked Spindle containers (default: `""`). 1451- `SPINDLE_PIPELINES_NIXERY`: The Nixery URL (default: `"nixery.tangled.sh"`). 1452- `SPINDLE_PIPELINES_WORKFLOW_TIMEOUT`: The default workflow timeout (default: `"5m"`). 1453 1454For the microVM engine, the following are also available 1455(prefix `SPINDLE_MICROVM_PIPELINES_`): 1456 1457- `SPINDLE_MICROVM_PIPELINES_IMAGE_DIR`: Directory containing 1458 microVM images (**required** to use the engine). See 1459 [Running microVM workflows](#running-microvm-workflows). 1460- `SPINDLE_MICROVM_PIPELINES_DEFAULT_IMAGE`: Image used when a 1461 workflow doesn't set `image` (default: `"nixos-x86_64"`). 1462- `SPINDLE_MICROVM_PIPELINES_OVERLAY_DIR`: Where per-workflow 1463 temporary disks are created (default: the system temp dir). 1464- `SPINDLE_MICROVM_PIPELINES_ENABLE_KVM`: Use KVM hardware 1465 acceleration (default: `true`). Without KVM, guests fall 1466 back to slow software emulation. 1467- `SPINDLE_MICROVM_PIPELINES_WORKFLOW_TIMEOUT`: Default 1468 workflow timeout (default: `"5m"`). 1469 1470Optional resource limits (a value of `0` disables that 1471limit). The limits cap usage across all running microVM 1472workflows: 1473 1474- `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_MEMORY_MIB` 1475- `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_VCPUS` 1476- `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_DISK_MIB` 1477 1478Optional cgroup enforcement: 1479 1480- `SPINDLE_MICROVM_PIPELINES_ENABLE_CGROUPS`: Place each 1481 workflow's QEMU and slirp4netns in a per-workflow cgroup= 1482 (default: `false`). 1483- `SPINDLE_MICROVM_PIPELINES_CGROUP_PARENT`: Parent cgroup; 1484 `self` resolves the spindle service's own cgroup (default: 1485 `"self"`). 1486- `SPINDLE_MICROVM_PIPELINES_CGROUP_PIDS_MAX`: Max processes 1487 per workflow cgroup (default: `4096`). 1488- `SPINDLE_MICROVM_PIPELINES_CGROUP_SWAP_MAX_MIB`: Max swap 1489 per workflow cgroup (default: `0`, no swap). 1490- `SPINDLE_MICROVM_PIPELINES_CGROUP_SUPERVISOR_MEMORY_MIN_MIB`: 1491 Memory protected for spindle itself so it isn't OOM-killed 1492 before the workflows (default: `512`). 1493 1494To push paths built inside microVMs back to a shared Nix 1495cache (and read from it), configure the cache (prefix 1496`SPINDLE_NIX_CACHE_`): 1497 1498- `SPINDLE_NIX_CACHE_READ_URLS`: Comma-separated binary cache 1499 URLs the guest reads from. 1500- `SPINDLE_NIX_CACHE_TRUSTED_PUBLIC_KEYS`: Comma-separated 1501 trusted public keys for those caches. 1502- `SPINDLE_NIX_CACHE_UPLOAD_URL`: Cache URL that paths built 1503 in the guest are uploaded to. 1504 1505### Running spindle 1506 15071. **Set the environment variables.** For example: 1508 1509 ```shell 1510 export SPINDLE_SERVER_HOSTNAME="your-hostname" 1511 export SPINDLE_SERVER_OWNER="your-did" 1512 ``` 1513 15142. **Build the Spindle binary.** 1515 1516 ```shell 1517 cd core 1518 go mod download 1519 go build -o cmd/spindle/spindle cmd/spindle/main.go 1520 ``` 1521 15223. **Create the log directory.** 1523 1524 ```shell 1525 sudo mkdir -p /var/log/spindle 1526 sudo chown $USER:$USER -R /var/log/spindle 1527 ``` 1528 15294. **Run the Spindle binary.** 1530 1531 ```shell 1532 ./cmd/spindle/spindle 1533 ``` 1534 1535Spindle will now start, connect to the Jetstream server, and begin processing pipelines. 1536 1537### Running microVM workflows 1538 1539The microVM engine needs a few extra things on the host, and 1540it needs images to boot. 1541 1542#### Host dependencies 1543 1544microVM workflows depend on a handful of host tools and 1545devices. spindle checks for the ones an image needs right 1546before it launches, so a missing dependency surfaces as a 1547clear error. You'll need: 1548 1549- `qemu`: the runner. The QEMU binary for the image's arch 1550 must be present (e.g. `qemu-system-x86_64`). 1551- `mkfs.ext4` (from `e2fsprogs`): to format the per-workflow 1552 writable volumes. 1553- [`slirp4netns`](https://github.com/rootless-containers/slirp4netns#install), 1554 `ip` (from `iproute2`), `mount` and `unshare` (from `util-linux`): 1555 used to sandbox guest networking. 1556- `/dev/kvm`: for hardware acceleration (unless you disable 1557 KVM with `SPINDLE_MICROVM_PIPELINES_ENABLE_KVM=false`). 1558- `/dev/vhost-vsock`: used by QEMU to enable guest-to-host vsock 1559 communication. 1560- `/dev/vsock`: used by spindle on the host to listen on vsock ports 1561 and accept guest agent connections. 1562- `/dev/net/tun`: required by `slirp4netns` to set up tap devices 1563 for sandboxed guest networking. 1564 1565On NixOS, the [spindle 1566module](https://tangled.org/tangled.org/core/blob/master/nix/modules/spindle.nix) 1567puts `qemu`, `e2fsprogs`, `slirp4netns`, `iproute2` and 1568`util-linux` on the service's `PATH` for you. 1569 1570#### Container virtualization 1571 1572Running `spindle` inside a container (e.g., Docker, Podman, or LXD) requires passing host device nodes into the container and granting the runtime additional privileges. Because spindle uses nested namespaces (`unshare`) and helper tools (`slirp4netns`), the container configuration will require: 1573 1574- `/dev/vsock`, `/dev/vhost-vsock`, `/dev/kvm`, and `/dev/net/tun` mapped from the host. 1575- `NET_ADMIN` and `SYS_ADMIN` capabilities to manage network namespaces and mounts inside the container. 1576- Relaxed seccomp filters (e.g., `seccomp=unconfined`) and SELinux/AppArmor containment if they restrict namespace creation or device access. 1577 1578#### Building images 1579 1580Images are built with Nix. The flake exposes packages for the 1581two stock images (use the `-tarball` prefixed ones for a gzipped 1582tarball you can copy to another host): 1583 1584```shell 1585# a NixOS image 1586nix build .#spindle-nixos-image 1587# an Alpine image 1588nix build .#spindle-alpine-image 1589``` 1590 1591#### Installing images 1592 1593Spindle looks for images in 1594`SPINDLE_MICROVM_PIPELINES_IMAGE_DIR`. An image is resolved by 1595the name a workflow puts in its `image` field, matched 1596literally against what's on disk: 1597 15981. a directory `<name>/` containing a `spec.json` (next to the 1599 kernel/initrd/store-disk), or 16002. a flat `<name>.json` self-contained spec. 1601 1602Resolution depends only on the name and what's on disk, never 1603on the host doing the resolving, so the same workflow resolves 1604to the same image on every spindle. If you keep multiple 1605arches side by side, you can name them `<name>-<arch>` (e.g. 1606`nixos-x86_64`, `alpine-aarch64`); the suffix is just part of 1607the name. To make a name like `nixos` work if you are hosting 1608multiple arches, you can use symlinks. 1609 1610On NixOS, you'll most likely want to use `systemd.tmpfiles.rules` 1611to set these up declaratively. 1612 1613## Architecture 1614 1615Spindle is a small CI runner service. Here's a high-level overview of how it operates: 1616 1617- Listens for [`sh.tangled.spindle.member`](/lexicons/spindle/member.json) and 1618 [`sh.tangled.repo`](/lexicons/repo.json) records on the Jetstream. 1619- When a new repo record comes through (typically when you add a spindle to a 1620 repo from the settings), spindle then resolves the underlying knot and 1621 subscribes to repo events (see: 1622 [`sh.tangled.pipeline`](/lexicons/pipeline.json)). 1623- The spindle engine then handles execution of the pipeline, with results and 1624 logs beamed on the spindle event stream over WebSocket 1625 1626### The engines 1627 1628Spindle has two execution backends, picked per-workflow with 1629the [`engine`](#engine) field: 1630 1631- **nixery**: executes each step in a fresh Docker container 1632 (Podman works too, if Docker compatibility is enabled so 1633 that `/run/docker.sock` is created), with state persisted 1634 across steps within the `/tangled/workspace` directory. The 1635 base image for the container is constructed on the fly using 1636 [Nixery](https://nixery.dev), which is/rhandy for caching 1637 layers for frequently used packages. 1638- **microvm**: runs the whole workflow inside its own 1639 microVM, supporting different images, with extra 1640 configuration for NixOS images (e.g. services in workflow file) 1641 See the [engine 1642 README](https://tangled.org/tangled.org/core/blob/master/spindle/engines/microvm/README.md) 1643 for the architecture in depth. 1644 1645The pipeline manifest is [specified here](https://docs.tangled.org/spindles.html#pipelines). 1646 1647## Secrets with openbao 1648 1649This document covers setting up spindle to use OpenBao for secrets 1650management via OpenBao Proxy instead of the default SQLite backend. 1651 1652### Overview 1653 1654Spindle now uses OpenBao Proxy for secrets management. The proxy handles 1655authentication automatically using AppRole credentials, while spindle 1656connects to the local proxy instead of directly to the OpenBao server. 1657 1658This approach provides better security, automatic token renewal, and 1659simplified application code. 1660 1661### Installation 1662 1663Install OpenBao from Nixpkgs: 1664 1665```bash 1666nix shell nixpkgs#openbao # for a local server 1667``` 1668 1669### Setup 1670 1671The setup process can is documented for both local development and production. 1672 1673#### Local development 1674 1675Start OpenBao in dev mode: 1676 1677```bash 1678bao server -dev -dev-root-token-id="root" -dev-listen-address=127.0.0.1:8201 1679``` 1680 1681This starts OpenBao on `http://localhost:8201` with a root token. 1682 1683Set up environment for bao CLI: 1684 1685```bash 1686export BAO_ADDR=http://localhost:8200 1687export BAO_TOKEN=root 1688``` 1689 1690#### Production 1691 1692You would typically use a systemd service with a 1693configuration file. Refer to 1694[@tangled.org/infra](https://tangled.org/@tangled.org/infra) 1695for how this can be achieved using Nix. 1696 1697Then, initialize the bao server: 1698 1699```bash 1700bao operator init -key-shares=1 -key-threshold=1 1701``` 1702 1703This will print out an unseal key and a root key. Save them 1704somewhere (like a password manager). Then unseal the vault 1705to begin setting it up: 1706 1707```bash 1708bao operator unseal <unseal_key> 1709``` 1710 1711All steps below remain the same across both dev and 1712production setups. 1713 1714#### Configure openbao server 1715 1716Create the spindle KV mount: 1717 1718```bash 1719bao secrets enable -path=spindle -version=2 kv 1720``` 1721 1722Set up AppRole authentication and policy: 1723 1724Create a policy file `spindle-policy.hcl`: 1725 1726```hcl 1727# Full access to spindle KV v2 data 1728path "spindle/data/*" { 1729 capabilities = ["create", "read", "update", "delete"] 1730} 1731 1732# Access to metadata for listing and management 1733path "spindle/metadata/*" { 1734 capabilities = ["list", "read", "delete", "update"] 1735} 1736 1737# Allow listing at root level 1738path "spindle/" { 1739 capabilities = ["list"] 1740} 1741 1742# Required for connection testing and health checks 1743path "auth/token/lookup-self" { 1744 capabilities = ["read"] 1745} 1746``` 1747 1748Apply the policy and create an AppRole: 1749 1750```bash 1751bao policy write spindle-policy spindle-policy.hcl 1752bao auth enable approle 1753bao write auth/approle/role/spindle \ 1754 token_policies="spindle-policy" \ 1755 token_ttl=1h \ 1756 token_max_ttl=4h \ 1757 bind_secret_id=true \ 1758 secret_id_ttl=0 \ 1759 secret_id_num_uses=0 1760``` 1761 1762Get the credentials: 1763 1764```bash 1765# Get role ID (static) 1766ROLE_ID=$(bao read -field=role_id auth/approle/role/spindle/role-id) 1767 1768# Generate secret ID 1769SECRET_ID=$(bao write -f -field=secret_id auth/approle/role/spindle/secret-id) 1770 1771echo "Role ID: $ROLE_ID" 1772echo "Secret ID: $SECRET_ID" 1773``` 1774 1775#### Create proxy configuration 1776 1777Create the credential files: 1778 1779```bash 1780# Create directory for OpenBao files 1781mkdir -p /tmp/openbao 1782 1783# Save credentials 1784echo "$ROLE_ID" > /tmp/openbao/role-id 1785echo "$SECRET_ID" > /tmp/openbao/secret-id 1786chmod 600 /tmp/openbao/role-id /tmp/openbao/secret-id 1787``` 1788 1789Create a proxy configuration file `/tmp/openbao/proxy.hcl`: 1790 1791```hcl 1792# OpenBao server connection 1793vault { 1794 address = "http://localhost:8200" 1795} 1796 1797# Auto-Auth using AppRole 1798auto_auth { 1799 method "approle" { 1800 mount_path = "auth/approle" 1801 config = { 1802 role_id_file_path = "/tmp/openbao/role-id" 1803 secret_id_file_path = "/tmp/openbao/secret-id" 1804 } 1805 } 1806 1807 # Optional: write token to file for debugging 1808 sink "file" { 1809 config = { 1810 path = "/tmp/openbao/token" 1811 mode = 0640 1812 } 1813 } 1814} 1815 1816# Proxy listener for spindle 1817listener "tcp" { 1818 address = "127.0.0.1:8201" 1819 tls_disable = true 1820} 1821 1822# Enable API proxy with auto-auth token 1823api_proxy { 1824 use_auto_auth_token = true 1825} 1826 1827# Enable response caching 1828cache { 1829 use_auto_auth_token = true 1830} 1831 1832# Logging 1833log_level = "info" 1834``` 1835 1836#### Start the proxy 1837 1838Start OpenBao Proxy: 1839 1840```bash 1841bao proxy -config=/tmp/openbao/proxy.hcl 1842``` 1843 1844The proxy will authenticate with OpenBao and start listening on 1845`127.0.0.1:8201`. 1846 1847#### Configure spindle 1848 1849Set these environment variables for spindle: 1850 1851```bash 1852export SPINDLE_SERVER_SECRETS_PROVIDER=openbao 1853export SPINDLE_SERVER_SECRETS_OPENBAO_PROXY_ADDR=http://127.0.0.1:8201 1854export SPINDLE_SERVER_SECRETS_OPENBAO_MOUNT=spindle 1855``` 1856 1857On startup, spindle will now connect to the local proxy, 1858which handles all authentication automatically. 1859 1860### Production setup for proxy 1861 1862For production, you'll want to run the proxy as a service: 1863 1864Place your production configuration in 1865`/etc/openbao/proxy.hcl` with proper TLS settings for the 1866vault connection. 1867 1868### Verifying setup 1869 1870Test the proxy directly: 1871 1872```bash 1873# Check proxy health 1874curl -H "X-Vault-Request: true" http://127.0.0.1:8201/v1/sys/health 1875 1876# Test token lookup through proxy 1877curl -H "X-Vault-Request: true" http://127.0.0.1:8201/v1/auth/token/lookup-self 1878``` 1879 1880Test OpenBao operations through the server: 1881 1882```bash 1883# List all secrets 1884bao kv list spindle/ 1885 1886# Add a test secret via the spindle API, then check it exists 1887bao kv list spindle/repos/ 1888 1889# Get a specific secret 1890bao kv get spindle/repos/your_repo_path/SECRET_NAME 1891``` 1892 1893### How it works 1894 1895- Spindle connects to OpenBao Proxy on localhost (typically 1896 port 8200 or 8201) 1897- The proxy authenticates with OpenBao using AppRole 1898 credentials 1899- All spindle requests go through the proxy, which injects 1900 authentication tokens 1901- Secrets are stored at 1902 `spindle/repos/{sanitized_repo_path}/{secret_key}` 1903- Repository paths like `did:plc:alice/myrepo` become 1904 `did_plc_alice_myrepo` 1905- The proxy handles all token renewal automatically 1906- Spindle no longer manages tokens or authentication 1907 directly 1908 1909### Troubleshooting 1910 1911**Connection refused**: Check that the OpenBao Proxy is 1912running and listening on the configured address. 1913 1914**403 errors**: Verify the AppRole credentials are correct 1915and the policy has the necessary permissions. 1916 1917**404 route errors**: The spindle KV mount probably doesn't 1918exist—run the mount creation step again. 1919 1920**Proxy authentication failures**: Check the proxy logs and 1921verify the role-id and secret-id files are readable and 1922contain valid credentials. 1923 1924**Secret not found after writing**: This can indicate policy 1925permission issues. Verify the policy includes both 1926`spindle/data/*` and `spindle/metadata/*` paths with 1927appropriate capabilities. 1928 1929Check proxy logs: 1930 1931```bash 1932# If running as systemd service 1933journalctl -u openbao-proxy -f 1934 1935# If running directly, check the console output 1936``` 1937 1938Test AppRole authentication manually: 1939 1940```bash 1941bao write auth/approle/login \ 1942 role_id="$(cat /tmp/openbao/role-id)" \ 1943 secret_id="$(cat /tmp/openbao/secret-id)" 1944``` 1945 1946# Webhooks 1947 1948Webhooks allow you to receive HTTP POST notifications when events occur in your repositories. This enables you to integrate Tangled with external services, trigger CI/CD pipelines, send notifications, or automate workflows. 1949 1950## Overview 1951 1952Webhooks send HTTP POST requests to URLs you configure whenever specific events happen. Currently, Tangled supports push events, with more event types coming soon. 1953 1954## Configuring webhooks 1955 1956To set up a webhook for your repository: 1957 19581. Navigate to your repository 19592. Go to **Settings → Hooks** 19603. Click **new webhook** 19614. Configure your webhook: 1962 - **Payload URL**: The endpoint that will receive the webhook POST requests 1963 - **Secret**: An optional secret key for verifying webhook authenticity (leave blank to send unsigned webhooks) 1964 - **Events**: Select which events trigger the webhook (currently only push events) 1965 - **Active**: Toggle whether the webhook is enabled 1966 1967## Webhook payload 1968 1969### Push 1970 1971When a push event occurs, Tangled sends a POST request with a JSON payload of the format: 1972 1973```json 1974{ 1975 "after": "7b320e5cbee2734071e4310c1d9ae401d8f6cab5", 1976 "before": "c04ddf64eddc90e4e2a9846ba3b43e67a0e2865e", 1977 "pusher": { 1978 "did": "did:plc:hwevmowznbiukdf6uk5dwrrq" 1979 }, 1980 "ref": "refs/heads/main", 1981 "repository": { 1982 "clone_url": "https://tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo", 1983 "created_at": "2025-09-15T08:57:23Z", 1984 "description": "an example repository", 1985 "fork": false, 1986 "full_name": "did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo", 1987 "html_url": "https://tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo", 1988 "name": "some-repo", 1989 "open_issues_count": 5, 1990 "owner": { 1991 "did": "did:plc:hwevmowznbiukdf6uk5dwrrq" 1992 }, 1993 "ssh_url": "ssh://git@tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo", 1994 "stars_count": 1, 1995 "updated_at": "2025-09-15T08:57:23Z" 1996 } 1997} 1998``` 1999 2000## HTTP headers 2001 2002Each webhook request includes the following headers: 2003 2004- `Content-Type: application/json` 2005- `User-Agent: Tangled-Hook/<short-sha>` — User agent with short SHA of the commit 2006- `X-Tangled-Event: push` — The event type 2007- `X-Tangled-Hook-ID: <webhook-id>` — The webhook ID 2008- `X-Tangled-Delivery: <uuid>` — Unique delivery ID 2009- `X-Tangled-Signature-256: sha256=<hmac>` — HMAC-SHA256 signature (if secret configured) 2010 2011## Verifying webhook signatures 2012 2013If you configured a secret, you should verify the webhook signature to ensure requests are authentic. For example, in Go: 2014 2015```go 2016package main 2017 2018import ( 2019 "crypto/hmac" 2020 "crypto/sha256" 2021 "encoding/hex" 2022 "io" 2023 "net/http" 2024 "strings" 2025) 2026 2027func verifySignature(payload []byte, signatureHeader, secret string) bool { 2028 // Remove 'sha256=' prefix from signature header 2029 signature := strings.TrimPrefix(signatureHeader, "sha256=") 2030 2031 // Compute expected signature 2032 mac := hmac.New(sha256.New, []byte(secret)) 2033 mac.Write(payload) 2034 expected := hex.EncodeToString(mac.Sum(nil)) 2035 2036 // Use constant-time comparison to prevent timing attacks 2037 return hmac.Equal([]byte(signature), []byte(expected)) 2038} 2039 2040func webhookHandler(w http.ResponseWriter, r *http.Request) { 2041 // Read the request body 2042 payload, err := io.ReadAll(r.Body) 2043 if err != nil { 2044 http.Error(w, "Bad request", http.StatusBadRequest) 2045 return 2046 } 2047 2048 // Get signature from header 2049 signatureHeader := r.Header.Get("X-Tangled-Signature-256") 2050 2051 // Verify signature 2052 if signatureHeader != "" && verifySignature(payload, signatureHeader, yourSecret) { 2053 // Webhook is authentic, process it 2054 processWebhook(payload) 2055 w.WriteHeader(http.StatusOK) 2056 } else { 2057 http.Error(w, "Invalid signature", http.StatusUnauthorized) 2058 } 2059} 2060``` 2061 2062## Delivery retries 2063 2064Webhooks are automatically retried on failure: 2065 2066- **3 total attempts** (1 initial + 2 retries) 2067- **Exponential backoff** starting at 1 second, max 10 seconds 2068- **Retried on**: 2069 - Network errors 2070 - HTTP 5xx server errors 2071- **Not retried on**: 2072 - HTTP 4xx client errors (bad request, unauthorized, etc.) 2073 2074### Timeouts 2075 2076Webhook requests timeout after 30 seconds. If your endpoint needs more time: 2077 20781. Respond with 200 OK immediately 20792. Process the webhook asynchronously in the background 2080 2081## Example integrations 2082 2083### Discord notifications 2084 2085```javascript 2086app.post("/webhook", (req, res) => { 2087 const payload = req.body; 2088 2089 fetch("https://discord.com/api/webhooks/...", { 2090 method: "POST", 2091 headers: { "Content-Type": "application/json" }, 2092 body: JSON.stringify({ 2093 content: `New push to ${payload.repository.full_name}`, 2094 embeds: [ 2095 { 2096 title: `${payload.pusher.did} pushed to ${payload.ref}`, 2097 url: payload.repository.html_url, 2098 color: 0x00ff00, 2099 }, 2100 ], 2101 }), 2102 }); 2103 2104 res.status(200).send("OK"); 2105}); 2106``` 2107 2108# Migrating knots and spindles 2109 2110Sometimes, non-backwards compatible changes are made to the 2111knot/spindle XRPC APIs. If you host a knot or a spindle, you 2112will need to follow this guide to upgrade. Typically, this 2113only requires you to deploy the newest version. 2114 2115This document is laid out in reverse-chronological order. 2116Newer migration guides are listed first, and older guides 2117are further down the page. 2118 2119## Upgrading to v1.15.0-alpha 2120 2121With v1.15.0-alpha, a knot itself owns its members and 2122per-repo collaborators directly. Previously this data was sourced from 2123PDS records (`sh.tangled.knot.member` and `sh.tangled.repo.collaborator`) 2124that the appview and the knot both read off the firehose. 2125The knot is now the source of truth and serves them over XRPC instead: 2126 2127- `sh.tangled.knot.addMember`, `sh.tangled.knot.removeMember`, `sh.tangled.knot.listMembers` 2128- `sh.tangled.repo.addCollaborator`, `sh.tangled.repo.removeCollaborator`, `sh.tangled.repo.listCollaborators` 2129 2130Until your knot is upgraded, the appview keeps reading its 2131members and collaborators from the old firehose-sourced records. 2132Upgrade to move your knot onto knot-owned access control. 2133 2134- Upgrade to the latest tag (v1.15.0 or above) 2135- Head to the [knot dashboard](https://tangled.org/settings/knots) and 2136 hit the "retry" button to verify your knot 2137 2138## Upgrading to v1.14.0-alpha 2139 2140Starting with v1.14.0-alpha, the fully knot uses the repoDID as its 2141canonical handle for repositories. This unlocks repository 2142renames from the appview UI and changes the wire format for 2143the following lexicons (`sh.tangled.repo.pull`, `sh.tangled.repo.collaborator`, 2144`sh.tangled.repo.issue`, `sh.tangled.git.refUpdate`). 2145 2146Knots that have not been upgraded may silently drop new push 2147events, pull requests, issues, and collaborator invites for 2148repositories they host until upgraded. So upgrade please!!! 2149 2150- Upgrade to the latest tag (v1.14.0 or above) 2151- Head to the [knot dashboard](https://tangled.org/settings/knots) and 2152 hit the "retry" button to verify your knot 2153 2154## Upgrading to v1.13.0-alpha 2155 2156Starting with v1.13.0-alpha, every repository on a knot is 2157assigned a DID. This makes repositories stable across 2158renames and transfers. 2159 2160When you upgrade your knot to this version, the server will 2161automatically mint DIDs for all existing repositories on 2162startup. This is a one-time process and you may see 2163additional log output during the first boot as DIDs are 2164assigned. 2165 2166- Upgrade to the latest tag (v1.13.0 or above) 2167- Head to the [knot dashboard](https://tangled.org/settings/knots) and 2168 hit the "retry" button to verify your knot 2169 2170## Upgrading from v1.8.x 2171 2172After v1.8.2, the HTTP API for knots and spindles has been 2173deprecated and replaced with XRPC. Repositories on outdated 2174knots will not be viewable from the appview. Upgrading is 2175straightforward however. 2176 2177For knots: 2178 2179- Upgrade to the latest tag (v1.9.0 or above) 2180- Head to the [knot dashboard](https://tangled.org/settings/knots) and 2181 hit the "retry" button to verify your knot 2182 2183For spindles: 2184 2185- Upgrade to the latest tag (v1.9.0 or above) 2186- Head to the [spindle 2187 dashboard](https://tangled.org/settings/spindles) and hit the 2188 "retry" button to verify your spindle 2189 2190## Upgrading from v1.7.x 2191 2192After v1.7.0, knot secrets have been deprecated. You no 2193longer need a secret from the appview to run a knot. All 2194authorized commands to knots are managed via [Inter-Service 2195Authentication](https://atproto.com/specs/xrpc#inter-service-authentication-jwt). 2196Knots will be read-only until upgraded. 2197 2198Upgrading is quite easy, in essence: 2199 2200- `KNOT_SERVER_SECRET` is no more, you can remove this 2201 environment variable entirely 2202- `KNOT_SERVER_OWNER` is now required on boot, set this to 2203 your DID. You can find your DID in the 2204 [settings](https://tangled.org/settings) page. 2205- Restart your knot once you have replaced the environment 2206 variable 2207- Head to the [knot dashboard](https://tangled.org/settings/knots) and 2208 hit the "retry" button to verify your knot. This simply 2209 writes a `sh.tangled.knot` record to your PDS. 2210 2211If you use the nix module, simply bump the flake to the 2212latest revision, and change your config block like so: 2213 2214```diff 2215 services.tangled.knot = { 2216 enable = true; 2217 server = { 2218- secretFile = /path/to/secret; 2219+ owner = "did:plc:foo"; 2220 }; 2221 }; 2222``` 2223 2224# Bobbin 2225 2226Bobbin is an API appview for Tangled records. It serves XRPC 2227endpoints for `sh.tangled.*`, with it you can get repos, 2228issues, pulls, comments, follows, stars, labels, pipelines, 2229and profiles. It is read-only, there is no auth, since that 2230should all be handled direct-to-PDS and knot respectively. 2231 2232**Bobbin has no permanent storage**. 2233 2234It is only a glorified edge index, in the graph theory 2235sense. Additionally it has a record cache, re-filled on 2236demand. All other data that Bobbin serves comes live from 2237PDSes & knots. 2238 2239## What Bobbin needs 2240 2241The way that Bobbin is able to pull off being 2242so stateless is by moving state upstream. 2243Primarily it depends on an instance of 2244[Hydrant](https://tangled.org/did:plc:6v3ul2ptnqctyxwkz5ti4amn) 2245, which is the service that gives an event stream 2246for Bobbin to quickly backfill from on every restart. 2247Backfilling ought to take less than a couple of minutes 2248maximum. If the upstream instance of Hydrant fails 2249while Bobbin is live, its list/count endpoints stop 2250advancing and report a stale cursor. Single-lookups 2251will continue working, due to the second dependency: 2252[Slingshot](https://tangled.org/did:plc:c7mc2fn47ihdihul4vjwsuy3/tree/main/slingshot). 2253Slingshot fetches individual records & resolves identities. 2254If the upstream instance of Slingshot fails, single-lookups 2255will fail with a `502` error. There are some aggregation 2256endpoints that use Slingshot for hydrating, which will also 2257fail. 2258 2259A soft dependency that ought to exist for Bobbin to operate 2260correctly is simply the plethora of knots that are out 2261there, that Bobbin talks to directly for git data and, for 2262knots at v1.15+, members & collaborators. 2263 2264## Building Bobbin 2265 2266Bobbin is under [Tangled's core monorepo, under bobbin/](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/tree/master/bobbin). 2267Here's an easy local debug-build: 2268 2269```sh 2270cargo build -p bobbin 2271``` 2272 2273Bobbin loves being in a container. When using 2274`bobbin/containerfiles/bobbin.Containerfile`, it runs `cargo 2275build --release --bin bobbin --package bobbin` within a 2276little Debian runtime, exposing port 8090. 2277 2278## Configuration 2279 2280The best way to configure Bobbin is via a toml config file. 2281There's an `example.toml` in [Bobbin's subdir](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/blob/master/bobbin/example.toml). 2282Every value is overridable by a `BOBBIN_*` env var. 2283The load order is env, then `--config <path>`, then 2284`/etc/bobbin/config.toml`, then built-in defaults. 2285 2286Load and check a config without starting the server: 2287 2288```sh 2289bobbin --config config.toml validate 2290``` 2291 2292Minimal config is the two upstream URLs. The hydrant URL 2293takes `ws://` or `wss://`. An `http://` or `https://` 2294URL is rewritten to the matching websocket scheme at 2295connection-time. 2296 2297```toml 2298[server] 2299binds = ["127.0.0.1:8090"] 2300 2301# Loopback-only & can leave empty to disable debug introspection. 2302debug_bind = "127.0.0.1:8091" 2303 2304[hydrant] 2305url = "https://hydrant.example.com" 2306 2307[slingshot] 2308url = "https://slingshot.example.com" 2309``` 2310 2311> 🦪 Lewis 2312> 2313> At time of writing, we (Tangled) don't host public 2314> instances of Hydrant or Slingshot. You will have to 2315> find public instances or spin these up yourself! :P 2316 2317Take a gander in the project's example.toml for an 2318exhaustive list of things to configure. 2319 2320You will discover fun things such as a configurable adaptive 2321loop that watches the cgroup memory limit & throttles heavy 2322requests under pressure. It only works if it detects a 2323cgroup limit is present. The config for that is in the 2324`[backpressure]` block of the config template. 2325 2326## Running Bobbin 2327 2328Start the server using a config toml: 2329 2330```bash 2331bobbin --config config.toml 2332``` 2333Bobbin wakes up in a cold sweat and immediately gets to 2334work: 23351. It binds its listeners, connects to the Hydrant stream 2336 in the background. 23372. It serves requests from the first 2338 moment it's alive, even before the Hydrant stream connects 2339 or finishes catching up. Having a cold Hydrant itself 2340 costs only latency and approximate counts. 2341 2342## The API 2343 2344**Single lookups** take a record's AT-URI. 2345 2346- `getRepo` takes the repo URI: 2347 2348```sh 2349curl "$BOBBIN/xrpc/sh.tangled.repo.getRepo?repo=at://did:plc:boltless/sh.tangled.repo/squid" 2350``` 2351```json 2352{ 2353 "uri": "at://did:plc:boltless/sh.tangled.repo/squid", 2354 "cid": "bafyrei...", 2355 "value": { "$type": "sh.tangled.repo", "knot": "knot1.tangled.sh", "description": "...", "createdAt": "..." } 2356} 2357``` 2358 2359- `getProfile` takes the full profile record URI, so a bare 2360 handle or DID will not resolve: 2361 2362```sh 2363curl "$BOBBIN/xrpc/sh.tangled.actor.getProfile?actor=at://did:plc:boltless/sh.tangled.actor.profile/self" 2364``` 2365 2366- If Slingshot cannot serve the record, the response is `502`: 2367 2368```json 2369{ "error": "UpstreamFailed", "message": "upstream unavailable: ..." } 2370``` 2371 2372**Aggregation** endpoints come in `list*` and `count*` pairs, 2373each with a `*By` sibling, and require a `subject` query param. 2374 2375- `listRepos` and `countRepos` key on the owner DID: 2376 2377```sh 2378curl "$BOBBIN/xrpc/sh.tangled.repo.countRepos?subject=did:plc:boltless" 2379``` 2380```json 2381{ "count": 7, "distinctAuthors": 1 } 2382``` 2383 2384```sh 2385curl "$BOBBIN/xrpc/sh.tangled.repo.listRepos?subject=did:plc:boltless&limit=3" 2386``` 2387```json 2388{ "items": [ { "uri": "at://did:plc:boltless/sh.tangled.repo/squid", "cid": "bafyrei...", "value": { } } ], "cursor": null } 2389``` 2390 2391- Bobbin validates the subject per collection. Here a repo URI 2392 is passed where a bare DID is required, so the call returns a 2393 `400`: 2394 2395```sh 2396curl "$BOBBIN/xrpc/sh.tangled.graph.listFollows?subject=at://did:plc:boltless/sh.tangled.repo/squid" 2397``` 2398```json 2399{ "error": "InvalidRequest", "message": "invalid request: subject must be a bare did, got at-uri with collection sh.tangled.repo" } 2400``` 2401 2402**Search** is a single endpoint over an in-mem full-text 2403index: 2404 2405```sh 2406curl "$BOBBIN/xrpc/sh.tangled.search.query?q=tangled&limit=2" 2407``` 2408```json 2409{ "hits": [ { "uri": "at://...", "cid": "...", "nsid": "sh.tangled.repo", "score": 27.1, "value": { } } ], "cursor": null } 2410``` 2411 2412**Git data** such as blob, tree, diff, log, and archive proxies 2413straight to the repo's knot, streamed back without caching. 2414 2415## Coverage and warm-up 2416 2417- While the edge index is catching up from Hydrant, 2418 the aggregation count is a lower bound & may still climb. 2419- One endpoint reports how far along the backfill it is: 2420 2421```sh 2422curl "$BOBBIN/xrpc/sh.tangled.bobbin.getCoverage" 2423``` 2424 2425While warming up: 2426 2427```json 2428{ "ready": false, "eventsProcessed": 45588, "lastCursor": 51658 } 2429``` 2430 2431Once caught up, Bobbin flips to ready: 2432 2433```json 2434{ "ready": true, "eventsProcessed": 106085, "lastCursor": 116527 } 2435``` 2436 2437If starting up Hydrant for the first time, Hydrant itself 2438will take a decent while (a couple of hours) to backfill 2439from PDSes. Hydrant stores its backfill on disk. Bobbin 2440restart reaches `ready` in minutes by replaying event from 2441an already-populated Hydrant. If your Hydrant is new, expect 2442Bobbin to backfill in that same couple of hours that Hydrant 2443takes. 2444 2445## Loose ends and not-gonna-impl 2446 2447- **No coverage signal for per-knot rosters yet.** 2448 Coverage tracks the hydrant stream only. A v1.15 knot 2449 that is unreachable serves a stale or empty member set 2450 with nothing to flag it. 2451- **Knot eventstream fan-out isn't pooled.** 2452 Bobbin opens one websocket per v1.15 2453 knot on top of the hydrant subscription. A network with 2454 thousands of knots wants pooling or a shared subscription. 2455- **No sequential issue or PR numbers.** bobbin returns rkeys, 2456 not `#42` style ids like the web appview. A client 2457 deriving a display number does it from creation order. But 2458 why bother? rkeys are the IDs. 2459 2460# Hacking on Tangled 2461 2462We highly recommend [installing 2463Nix](https://nixos.org/download/) (the package manager) 2464before working on the codebase. The Nix flake provides a lot 2465of helpers to get started and most importantly, builds and 2466dev shells are entirely deterministic. 2467 2468To set up your dev environment: 2469 2470```bash 2471nix develop 2472``` 2473 2474Non-Nix users can look at the `devShell` attribute in the 2475`flake.nix` file to determine necessary dependencies. 2476 2477## Running the appview 2478 2479The appview requires Redis and OAuth JWKs. Start these 2480first, before launching the appview itself. 2481 2482```bash 2483# OAuth JWKs should already be set up by the Nix devshell: 2484echo $TANGLED_OAUTH_CLIENT_SECRET 2485z42ty4RT1ovnTopY8B8ekz9NuziF2CuMkZ7rbRFpAR9jBqMc 2486 2487echo $TANGLED_OAUTH_CLIENT_KID 24881761667908 2489 2490# if not, you can set it up yourself: 2491goat key generate -t P-256 2492Key Type: P-256 / secp256r1 / ES256 private key 2493Secret Key (Multibase Syntax): save this securely (eg, add to password manager) 2494 z42tuPDKRfM2mz2Kv953ARen2jmrPA8S9LX9tRq4RVcUMwwL 2495Public Key (DID Key Syntax): share or publish this (eg, in DID document) 2496 did:key:zDnaeUBxtG6Xuv3ATJE4GaWeyXM3jyamJsZw3bSPpxx4bNXDR 2497 2498# the secret key from above 2499export TANGLED_OAUTH_CLIENT_SECRET="z42tuP..." 2500 2501# Run Redis in a new shell to store OAuth sessions 2502redis-server 2503``` 2504 2505The Nix flake exposes a few `app` attributes (run `nix 2506flake show` to see a full list of what the flake provides), 2507one of the apps runs the appview with the `air` 2508live-reloader: 2509 2510```bash 2511TANGLED_DEV=true nix run .#watch-appview 2512 2513# TANGLED_DB_PATH might be of interest to point to 2514# different sqlite DBs 2515 2516# in a separate shell, you can live-reload tailwind 2517nix run .#watch-tailwind 2518``` 2519 2520## Running knots and spindles 2521 2522An end-to-end knot setup requires setting up a machine with 2523`sshd`, `AuthorizedKeysCommand`, and a Git user, which is 2524quite cumbersome. So the Nix flake provides a 2525`nixosConfiguration` to do so. 2526 2527<details> 2528 <summary><strong>macOS users will have to set up a Nix Builder first</strong></summary> 2529 2530In order to build Tangled's dev VM on macOS, you will 2531first need to set up a Linux Nix builder. The recommended 2532way to do so is to run a [`darwin.linux-builder` 2533VM](https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder) 2534and to register it in `nix.conf` as a builder for Linux 2535with the same architecture as your Mac (`linux-aarch64` if 2536you are using Apple Silicon). 2537 2538If you're on nix-darwin, you can simply add 2539 2540``` 2541nix.linux-builder.enable = true; 2542``` 2543 2544to your host's `configuration.nix`. 2545 2546Alternatively, you can use any other method to set up a 2547Linux machine with Nix installed that you can `sudo ssh` 2548into (in other words, root user on your Mac has to be able 2549to ssh into the Linux machine without entering a password) 2550and that has the same architecture as your Mac. See 2551[remote builder 2552instructions](https://nix.dev/manual/nix/2.28/advanced-topics/distributed-builds.html#requirements) 2553for how to register such a builder in `nix.conf`. 2554 2555> WARNING: If you'd like to use 2556> [`nixos-lima`](https://github.com/nixos-lima/nixos-lima) or 2557> [Orbstack](https://orbstack.dev/), note that setting them up so that `sudo 2558ssh` works can be tricky. It seems to be [possible with 2559> Orbstack](https://github.com/orgs/orbstack/discussions/1669). 2560 2561</details> 2562 2563To begin, grab your DID from http://localhost:3000/settings. 2564Then, set `TANGLED_VM_KNOT_OWNER` and 2565`TANGLED_VM_SPINDLE_OWNER` to your DID. You can now start a 2566lightweight NixOS VM like so: 2567 2568```bash 2569nix run --impure .#vm 2570 2571# type `poweroff` at the shell to exit the VM 2572``` 2573 2574This starts a knot on port 6444, a spindle on port 6555 2575with `ssh` exposed on port 2222. 2576 2577Once the services are running, head to 2578http://localhost:3000/settings/knots and hit "Verify". It should 2579verify the ownership of the services instantly if everything 2580went smoothly. 2581 2582You can push repositories to this VM with this ssh config 2583block on your main machine: 2584 2585```bash 2586Host nixos-shell 2587 Hostname localhost 2588 Port 2222 2589 User git 2590 IdentityFile ~/.ssh/my_tangled_key 2591``` 2592 2593Set up a remote called `local-dev` on a git repo: 2594 2595```bash 2596git remote add local-dev git@nixos-shell:user/repo 2597git push local-dev main 2598``` 2599 2600The above VM should already be running a spindle on 2601`localhost:6555`. Head to http://localhost:3000/settings/spindles and 2602hit "Verify". You can then configure each repository to use 2603this spindle and run CI jobs. 2604 2605Of interest when debugging spindles: 2606 2607``` 2608# Service logs from journald: 2609journalctl -xeu spindle 2610 2611# CI job logs from disk: 2612ls /var/log/spindle 2613 2614# Debugging spindle database: 2615sqlite3 /var/lib/spindle/spindle.db 2616 2617# litecli has a nicer REPL interface: 2618litecli /var/lib/spindle/spindle.db 2619``` 2620 2621If for any reason you wish to disable either one of the 2622services in the VM, modify [nix/vm.nix](/nix/vm.nix) and set 2623`services.tangled.spindle.enable` (or 2624`services.tangled.knot.enable`) to `false`. 2625 2626# Contribution guide 2627 2628## Commit guidelines 2629 2630We follow a commit style similar to the Go project. Please keep commits: 2631 2632- **atomic**: each commit should represent one logical change 2633- **descriptive**: the commit message should clearly describe what the 2634 change does and why it's needed 2635 2636### Message format 2637 2638``` 2639<service/top-level directory>/<affected package/directory>: <short summary of change> 2640 2641Optional longer description can go here, if necessary. Explain what the 2642change does and why, especially if not obvious. Reference relevant 2643issues or PRs when applicable. These can be links for now since we don't 2644auto-link issues/PRs yet. 2645``` 2646 2647Here are some examples: 2648 2649``` 2650appview/state: fix token expiry check in middleware 2651 2652The previous check did not account for clock drift, leading to premature 2653token invalidation. 2654``` 2655 2656``` 2657knotserver/git/service: improve error checking in upload-pack 2658``` 2659 2660### General notes 2661 2662- PRs get merged "as-is" (fast-forward)—like applying a patch-series 2663 using `git am`. At present, there is no squashing—so please author 2664 your commits as they would appear on `master`, following the above 2665 guidelines. 2666- If there is a lot of nesting, for example "appview: 2667 pages/templates/repo/fragments: ...", these can be truncated down to 2668 just "appview: repo/fragments: ...". If the change affects a lot of 2669 subdirectories, you may abbreviate to just the top-level names, e.g. 2670 "appview: ..." or "knotserver: ...". 2671- Keep commits lowercased with no trailing period. 2672- Use the imperative mood in the summary line (e.g., "fix bug" not 2673 "fixed bug" or "fixes bug"). 2674- Try to keep the summary line under 72 characters, but we aren't too 2675 fussed about this. 2676- Follow the same formatting for PR titles if filled manually. 2677- Don't include unrelated changes in the same commit. 2678- Avoid noisy commit messages like "wip" or "final fix"—rewrite history 2679 before submitting if necessary. 2680 2681## Code formatting 2682 2683We use a variety of tools to format our code, and multiplex them with 2684[`treefmt`](https://treefmt.com). All you need to do to format your changes 2685is run `nix run .#fmt` (or just `treefmt` if you're in the devshell). 2686 2687## Proposals for bigger changes 2688 2689Small fixes like typos, minor bugs, or trivial refactors can be 2690submitted directly as PRs. 2691 2692For larger changes—especially those introducing new features, significant 2693refactoring, or altering system behavior—please open a proposal first. This 2694helps us evaluate the scope, design, and potential impact before implementation. 2695 2696Create a new issue titled: 2697 2698``` 2699proposal: <affected scope>: <summary of change> 2700``` 2701 2702In the description, explain: 2703 2704- What the change is 2705- Why it's needed 2706- How you plan to implement it (roughly) 2707- Any open questions or tradeoffs 2708 2709We'll use the issue thread to discuss and refine the idea before moving 2710forward. 2711 2712## Developer Certificate of Origin (DCO) 2713 2714We require all contributors to certify that they have the right to 2715submit the code they're contributing. To do this, we follow the 2716[Developer Certificate of Origin 2717(DCO)](https://developercertificate.org/). 2718 2719By signing your commits, you're stating that the contribution is your 2720own work, or that you have the right to submit it under the project's 2721license. This helps us keep things clean and legally sound. 2722 2723To sign your commit, just add the `-s` flag when committing: 2724 2725```sh 2726git commit -s -m "your commit message" 2727``` 2728 2729This appends a line like: 2730 2731``` 2732Signed-off-by: Your Name <your.email@example.com> 2733``` 2734 2735We won't merge commits if they aren't signed off. If you forget, you can 2736amend the last commit like this: 2737 2738```sh 2739git commit --amend -s 2740``` 2741 2742If you're submitting a PR with multiple commits, make sure each one is 2743signed. 2744 2745For [jj](https://jj-vcs.github.io/jj/latest/) users, you can run the following command 2746to make it sign off commits in the tangled repo: 2747 2748```shell 2749# Safety check, should say "No matching config key..." 2750jj config list templates.commit_trailers 2751# The command below may need to be adjusted if the command above returned something. 2752jj config set --repo templates.commit_trailers "format_signed_off_by_trailer(self)" 2753``` 2754 2755Refer to the [jujutsu 2756documentation](https://jj-vcs.github.io/jj/latest/config/#commit-trailers) 2757for more information. 2758 2759# Troubleshooting guide 2760 2761## Login issues 2762 2763Owing to the distributed nature of OAuth on AT Protocol, you 2764may run into issues with logging in. If you run a 2765self-hosted PDS: 2766 2767- You may need to ensure that your PDS is timesynced using 2768 NTP: 2769 - Enable the `ntpd` service 2770 - Run `ntpd -qg` to synchronize your clock 2771- You may need to increase the default request timeout: 2772 `NODE_OPTIONS="--network-family-autoselection-attempt-timeout=500"` 2773 2774## Empty punchcard 2775 2776For Tangled to register commits that you make across the 2777network, you need to setup one of following: 2778 2779- The committer email should be a verified email associated 2780 to your account. You can add and verify emails on the 2781 settings page. 2782- Or, the committer email should be set to your account's 2783 DID: `git config user.email "did:plc:foobar"`. You can find 2784 your account's DID on the settings page 2785 2786## Commit is not marked as verified 2787 2788Tangled only supports SSH commit signatures. Ensure the SSH public key you use 2789for signing is uploaded to Tangled. 2790 2791To sign commits using an SSH key with git: 2792 2793``` 2794git config --global gpg.format ssh 2795git config --global user.signingkey ~/.ssh/tangled-key 2796``` 2797 2798To sign commits using an SSH key with jj, add this to your 2799config: 2800 2801``` 2802[signing] 2803behavior = "own" 2804backend = "ssh" 2805key = "~/.ssh/tangled-key" 2806``` 2807 2808## Self-hosted knot issues 2809 2810If you need help troubleshooting a self-hosted knot, check 2811out the [knot troubleshooting 2812guide](/knot-self-hosting-guide.html#troubleshooting).