alice.pds.demo.boltless.dev / core
This repository has no description
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

core / docs / src / content / docs / spindles / workflows.md
19 kB

title: Workflows description: Write CI/CD pipelines for your repositories using spindle workflows.#

Pipelines#

Spindle workflows allow you to write CI/CD pipelines in a simple format. They're located in the .tangled/workflows directory at the root of your repository, and are defined using YAML.

A workflow has a set of common fields that apply no matter which engine you pick:

  • Trigger: A required field that defines when a workflow should be triggered.
  • Engine: A required field that defines which engine a workflow should run on.
  • Clone options: An optional field that defines how the repository should be cloned.
  • Environment: An optional field that allows you to define environment variables.
  • Steps: An optional field that allows you to define what steps should run in the workflow.

On top of these, each engine has its own options for things like dependencies and images. See Engines for the per-engine fields.

Trigger#

The first thing to add to a workflow is the trigger, which defines when a workflow runs. This is defined using a when field, which takes in a list of conditions. Each condition has the following fields:

  • event: This is a required field that defines when your workflow should run. It's a list that can take one or more of the following values:
    • push: The workflow should run every time a commit is pushed to the repository.
    • pull_request: The workflow should run every time a pull request is made or updated.
    • manual: The workflow can be triggered manually.
  • branch: Defines which branches the workflow should run for. If used with the push event, commits to the branch(es) listed here will trigger the workflow. If used with the pull_request event, updates to pull requests targeting the branch(es) listed here will trigger the workflow. This field has no effect with the manual event. Supports glob patterns using * and ** (e.g., main, develop, release-*). Either branch or tag (or both) must be specified for push events.
  • tag: Defines which tags the workflow should run for. Only used with the push event - when tags matching the pattern(s) listed here are pushed, the workflow will trigger. This field has no effect with pull_request or manual events. Supports glob patterns using * and ** (e.g., v*, v1.*, release-**). Either branch or tag (or both) must be specified for push events.

For example, if you'd like to define a workflow that runs when commits are pushed to the main and develop branches, or when pull requests that target the main branch are updated, or manually, you can do so with:

when:
  - event: ["push", "manual"]
    branch: ["main", "develop"]
  - event: ["pull_request"]
    branch: ["main"]

You can also trigger workflows on tag pushes. For instance, to run a deployment workflow when tags matching v* are pushed:

when:
  - event: ["push"]
    tag: ["v*"]

You can even combine branch and tag patterns in a single constraint (the workflow triggers if either matches):

when:
  - event: ["push"]
    branch: ["main", "release-*"]
    tag: ["v*", "stable"]

To skip CI for a push, pass a Git push option:

git push -o skip-ci

ci-skip is also accepted.

Engine#

Next is the engine on which the workflow should run, defined using the required engine field. The currently supported engines are:

Example:

engine: "nixery"

Each engine also adds its own workflow fields (dependencies, images, services, and so on). These are documented under Engines.

Clone options#

When a workflow starts, the first step is to clone the repository. You can customize this behavior using the optional clone field. It has the following fields:

  • skip: Setting this to true will skip cloning the repository. This can be useful if your workflow is doing something that doesn't require anything from the repository itself. This is false by default.
  • depth: This sets the number of commits, or the "clone depth", to fetch from the repository. For example, if you set this to 2, the last 2 commits will be fetched. By default, the depth is set to 1, meaning only the most recent commit will be fetched, which is the commit that triggered the workflow.
  • submodules: If you use Git submodules (https://git-scm.com/book/en/v2/Git-Tools-Submodules) in your repository, setting this field to true will recursively fetch all submodules. This is false by default.

The default settings are:

clone:
  skip: false
  depth: 1
  submodules: false

Environment#

The environment field allows you define environment variables that will be available throughout the entire workflow. Do not put secrets here, these environment variables are visible to anyone viewing the repository. You can add secrets for pipelines in your repository's settings.

Example:

environment:
  GOOS: "linux"
  GOARCH: "arm64"
  NODE_ENV: "production"
  MY_ENV_VAR: "MY_ENV_VALUE"

By default, the following environment variables are set:

  • CI - Always set to true to indicate a CI environment
  • TANGLED_PIPELINE_ID - The AT URI of the current pipeline
  • TANGLED_PIPELINE_KIND - One of push, pull_request or manual
  • TANGLED_REPO_KNOT - The repository's knot hostname
  • TANGLED_REPO_DID - The DID of the repository owner
  • TANGLED_REPO_NAME - The name of the repository
  • TANGLED_REPO_DEFAULT_BRANCH - The default branch of the repository
  • TANGLED_REPO_URL - The full URL to the repository

These variables are only available when the pipeline is triggered by a push:

  • TANGLED_REF - The full git reference (e.g., refs/heads/main or refs/tags/v1.0.0)
  • TANGLED_REF_NAME - The short name of the reference (e.g., main or v1.0.0)
  • TANGLED_REF_TYPE - The type of reference, either branch or tag
  • TANGLED_SHA - The commit SHA that triggered the pipeline
  • TANGLED_COMMIT_SHA - Alias for TANGLED_SHA

These variables are only available when the pipeline is triggered by a pull request:

  • TANGLED_PR_SOURCE_BRANCH - The source branch of the pull request
  • TANGLED_PR_TARGET_BRANCH - The target branch of the pull request
  • TANGLED_PR_SOURCE_SHA - The commit SHA of the source branch

Steps#

The steps field allows you to define what steps should run in the workflow. It's a list of step objects, each with the following fields:

  • name: This field allows you to give your step a name. This name is visible in your workflow runs, and is used to describe what the step is doing.
  • command: This field allows you to define a command to run in that step. The step is run in a Bash shell, and the logs from the command will be visible in the pipelines page on the Tangled website. Any dependencies you added in your engine's section (see Engines) will be available to use here.
  • environment: Similar to the global environment config, this optional field is a key-value map that allows you to set environment variables for the step. Do not put secrets here, these environment variables are visible to anyone viewing the repository. You can add secrets for pipelines in your repository's settings.

Example:

steps:
  - name: "Build backend"
    command: "go build"
    environment:
      GOOS: "darwin"
      GOARCH: "arm64"
  - name: "Build frontend"
    command: "npm run build"
    environment:
      NODE_ENV: "production"

Engines#

The common fields above apply to every workflow. Each engine then adds its own fields on top. Pick an engine with the engine field and use the matching section below.

Nixery engine#

Dependencies#

When you're running a workflow you'll usually need additional dependencies. The dependencies field lets you define which dependencies to get, and from where. It's a key-value map, with the key being the registry to fetch dependencies from, and the value being the list of dependencies to fetch.

The registry URL syntax can be found on the nix manual.

Say you want to fetch Node.js and Go from nixpkgs, and a package called my_pkg you've made from your own registry at your repository at https://tangled.org/@example.com/my_pkg. You can define those dependencies like so:

dependencies:
  # nixpkgs
  nixpkgs:
    - nodejs
    - go
  # unstable
  nixpkgs/nixpkgs-unstable:
    - bun
  # custom registry
  git+https://tangled.org/@example.com/my_pkg:
    - my_pkg

Now these dependencies are available to use in your workflow!

Complete nixery workflow#

# .tangled/workflows/build.yml

when:
  - event: ["push", "manual"]
    branch: ["main", "develop"]
  - event: ["pull_request"]
    branch: ["main"]

engine: "nixery"

# using the default values
clone:
  skip: false
  depth: 1
  submodules: false

dependencies:
  # nixpkgs
  nixpkgs:
    - nodejs
    - go
  # custom registry
  git+https://tangled.org/@example.com/my_pkg:
    - my_pkg

environment:
  GOOS: "linux"
  GOARCH: "arm64"
  NODE_ENV: "production"
  MY_ENV_VAR: "MY_ENV_VALUE"

steps:
  - name: "Build backend"
    command: "go build"
    environment:
      GOOS: "darwin"
      GOARCH: "arm64"
  - name: "Build frontend"
    command: "npm run build"
    environment:
      NODE_ENV: "production"

If you want another example of a workflow, you can look at the one Tangled uses to build the project.

microVM engine#

Image#

A workflow picks the image to boot with the top-level image field:

engine: microvm
image: nixos

There are two flavours of images:

  • NixOS images (e.g. nixos): the whole guest is built with Nix, so you can configure it from the workflow file itself. The dependencies, services, virtualisation, registry and caches fields below are all understood here, and the guest builds and activates that configuration before any of your steps run.
  • Non-NixOS images (e.g. alpine): there's no NixOS to configure, so the workflow-level config fields above have no effect. You still get a full machine to run steps in.

The available image names depend on what the spindle operator has installed. nixos and alpine are examples. If image is omitted, the spindle's configured default image is used.

Dependencies#

On the microVM engine, dependencies is a flat list of packages that are made available to every step. This field only applies to NixOS images; for other images you can use the package manager included in a step.

The guest builds a nix develop-style devshell from your dependencies and uses it for each step, so you can, for example, add pkg-config and openssl and have the openssl-sys crate while compiling a Rust project just work.

A bare name like go is looked up in nixpkgs. You can also point at any flake with the flakeref#attr syntax, so github:nixos/nixpkgs#hello pulls hello straight out of that flake.

dependencies:
  - go
  - github:nixos/nixpkgs#hello

Registry#

The registry field remaps flake references, the same way nix registry does. This lets you pin or alias the flakes used by dependencies.

For example, pin nixpkgs to nixos-unstable so that the bare go above resolves from unstable, and alias your own flake so you can use myflake#tool in dependencies:

registry:
  nixpkgs: github:nixos/nixpkgs/nixos-unstable
  myflake: github:me/x

Caches#

The caches field is a map of Nix binary cache URL to its trusted public key. These are fed into the spindle's read proxy, so the guest can substitute prebuilt paths from them instead of building everything from scratch.

caches:
  https://nix-community.cachix.org: "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="

Services and virtualisation#

The services and virtualisation fields are passed straight through to NixOS. Anything you could write under services.* or virtualisation.* in a NixOS configuration, you can write here, and it's brought up before any of your steps run.

As a convenience, true works as shorthand for .enable = true anywhere an enable option exists (e.g. virtualisation.docker: true).

services:
  postgresql:
    enable: true
    ensureDatabases: ["spindle-workflow"]
    ensureUsers:
      - name: spindle-workflow
        ensureDBOwnership: true

virtualisation:
  docker: true

Recipes#

Lint, test and build a Node project#
when:
  - event: ["push", "pull_request"]
    branch: ["main"]

engine: microvm
image: nixos

dependencies:
  - pnpm

steps:
  - name: "Install dependencies"
    command: pnpm install --frozen-lockfile
  - name: "Lint and test"
    command: |
      pnpm run lint
      pnpm test
  - name: "Build"
    command: pnpm run build
Check formatting#
when:
  - event: ["push", "pull_request"]
    branch: ["main"]

engine: microvm
image: alpine # slimmer image for checking the formatting

steps:
  - name: "Install go"
    command: apk add go
  - name: "Check formatting"
    command: test -z $(gofmt -l .)

when:
  - event: ["push", "pull_request"]
    branch: ["main"]

engine: microvm
image: nixos

dependencies:
  - gcc
  - cargo
  - rustc
  - clippy
  - rustfmt
  - pkg-config # exports PKG_CONFIG_PATH for the libraries below
  - openssl # the C library + headers openssl-sys links against

steps:
  - name: "Check formatting"
    command: cargo fmt --check
  - name: "Clippy"
    command: cargo clippy --all-targets -- -D warnings
  - name: "Test"
    command: cargo test --all
  - name: "Release build"
    command: cargo build --release
Run migrations and integration tests against PostgreSQL#
when:
  - event: ["push", "pull_request"]
    branch: ["main"]

engine: microvm
image: nixos

environment:
  DATABASE_URL: "postgresql:///spindle-workflow?host=/run/postgresql"

dependencies:
  - gcc
  - cargo
  - rustc
  - pkg-config
  - openssl
  - sqlx-cli

services:
  postgresql:
    enable: true
    # has to be same name as the user for peer auth to work automatically
    ensureDatabases: ["spindle-workflow"]
    ensureUsers:
      - name: spindle-workflow
        ensureDBOwnership: true

steps:
  - name: "Run migrations"
    command: sqlx migrate run
  - name: "Integration tests"
    command: cargo test --all
Build and push a Docker image on tag#
when:
  - event: ["push"]
    tag: ["v*"]

engine: microvm
image: nixos

virtualisation:
  docker: true

steps:
  - name: "Build and push to ghcr.io"
    command: |
      set -euo pipefail

      echo "$REGISTRY_TOKEN" | docker login ghcr.io -u "$REGISTRY_USER" --password-stdin
      image="ghcr.io/$REGISTRY_USER/myapp:$TANGLED_REF_NAME"

      docker build -t "$image" -t "ghcr.io/$REGISTRY_USER/myapp:latest" .
      docker push "$image"
      docker push "ghcr.io/$REGISTRY_USER/myapp:latest"
Deploy to Cloudflare Workers on tag#
# .tangled/workflows/deploy.yml
when:
  - event: ["push"]
    tag: ["v*"]

engine: microvm
image: nixos

dependencies:
  - pnpm

steps:
  - name: "Install dependencies"
    command: pnpm install --frozen-lockfile
  - name: "Deploy worker"
    # `wrangler` picks up `CLOUDFLARE_API_TOKEN` from the env.
    # set it under **Settings → Secrets**.
    command: pnpm exec wrangler deploy
Publish a release artifact#
when:
  - event: ["push"]
    tag: ["v*"] # trigger on versions

engine: microvm
image: nixos

dependencies:
  - go

steps:
  - name: "Build release binary"
    command: |
      mkdir -p dist
      CGO_ENABLED=0 go build -trimpath -ldflags "-s -w" -o dist/myapp ./cmd/myapp

  - name: "Publish artifact record"
    command: |
      set -euo pipefail
      # change this if you're not on `tngl.sh`
      PDS="https://tngl.sh"
      # also update this to your handle or did
      ATP_IDENTIFIER="user.tngl.sh"
      ARTIFACT_PATH="dist/myapp"
      ARTIFACT_NAME="myapp"

      # set `ATP_APP_PASSWORD` under **Settings → Secrets**
      session=$(curl -fsS -X POST "$PDS/xrpc/com.atproto.server.createSession" \
        -H "Content-Type: application/json" \
        -d "{\"identifier\":\"$ATP_IDENTIFIER\",\"password\":\"$ATP_APP_PASSWORD\"}")
      jwt=$(echo "$session" | jq -r .accessJwt)
      did=$(echo "$session" | jq -r .did)

      # upload the binary as a blob
      blob=$(curl -fsS -X POST "$PDS/xrpc/com.atproto.repo.uploadBlob" \
        -H "Authorization: Bearer $jwt" \
        -H "Content-Type: application/octet-stream" \
        --data-binary @"$ARTIFACT_PATH")

      # note that this requires an annotated tag (`git tag -a v1.0.0 -m ...`)
      tag_hash=$(git rev-parse "$TANGLED_REF_NAME^{tag}")
      tag_bytes=$(printf '%s' "$tag_hash" | xxd -r -p | base64 | tr -d '=')

      # the sh.tangled.repo.artifact record for your artifact
      record=$(jq -n \
        --arg did "$did" \
        --arg tag "$tag_bytes" \
        --arg name "$ARTIFACT_NAME" \
        --arg repo "$TANGLED_REPO_URL" \
        --arg created "$(date -Iseconds)" \
        --argjson blob "$(echo "$blob" | jq .blob)" '{
          repo: $did,
          collection: "sh.tangled.repo.artifact",
          validate: false,
          record: {
            "$type": "sh.tangled.repo.artifact",
            tag: {"$bytes": $tag},
            name: $name,
            repo: $repo,
            artifact: $blob,
            createdAt: $created
          }
        }')

      # create the record on the PDS
      curl -fsS -X POST "$PDS/xrpc/com.atproto.repo.createRecord" \
        -H "Authorization: Bearer $jwt" \
        -H "Content-Type: application/json" \
        -d "$record"