alice.pds.demo.boltless.dev
This repository has no description
did:plc:3fwecdnvtcscjnrx2p4n7a…
1# Knot 2
2
3This is an alternate implementation of a Tangled knot server!
4
5In Tangled, a "knot" is simply a git server that does its auth layer over the AT Protocol.
6Essentially, it tastes like this:
7
81. Atproto users declare their public SSH key for themselves, in their PDS.
92. A knot admin (defined by atproto DID in the knot config) makes a request to the knot to allow membership to a given atproto user.
103. Said user can now create/update/delete a git repo on the knot, and do normal git things to that git repo over SSH.
114. Said user makes a request to the knot to allow more atproto users as collaborators on their specific git repo.
12
13The reader will notice this disconnect between making requests to the knot, vs. making requests to atproto (individual users' PDSes). The social side of git repos (PRs, issues, etc.) are "owned" by atproto at time of writing; in contrast, important ACL data (members, collaborators), lives on the knot as the source of truth. As time goes on we are re-assessing the idea of users owning what is "collaborative data" (issues, PRs, etc.) on their PDSes - soon may come the day that an issue also lives on the knot as a source of truth, with an accompanying pointer record on user PDS to attest that it's theirs.
14
15Back to knot 2 today:
16
17- **It has no database, or should I say, git is the only database, along with a secret-key-file.**
18- It aims to be small, fast, and modern - for example: http3/quic support, SHA256 by default, and no need for unix `git` user, to name a couple of features.
19- Config variables are in example.toml, just make a config.toml from that and go ham.
20
21Have fun!
22
23# Running a knot2
24
25The following is how I actually run `knot.oyster.cafe`. Please treat it as one possible setup.
26
27The soon-to-be knot operator will need 3 things before beginning:
281. A remote computer, preferably one that stays online
291. A domain name directed at the remote computer
301. The operator's own atproto DID, which becomes the knot's admin
31
32If the operator doesn't know their DID, resolve the handle at [pds.ls](https://pds.ls) or with any atproto tool that does identity resolution.
33
34## Configuration
35
36`example.toml` at project root is generated from the code,
37so it always ought to be up to date with what's possible on knot2.
38Copy it and fill in the required values:
39
40```sh
41cp example.toml config.toml
42```
43
44The most important values have no default and thus the knot won't start without them:
45
46- `server.hostname`: Public hostname, which will also be the knot's identity as `did:web:<hostname>`, so it oughtta be a public, good, solid, representative name.
47- `server.admins`: List of atproto DIDs, where the first one is the knot's "full owner", which is the DID the operator registers the knot under & what `sh.tangled.owner` returns. Every other admin has the same powers, just no claim to the fancy title, the way tangled is *currently* set up. Admin & minions.
48- `server.ssh_host_key_file`: Path for the SSH host key.
49- `repo.scan_path`: Path to the directory that gets the actual git repos.
50- `secrets.sealed_key_file`: Path to the sealed key store.
51- `secrets.master_key_env`: Name of the env var holding the master key.
52- `atproto.plc_directory`: PLC directory URL, which is "normally" `https://plc.directory`. There is deliberately no default here because I don't want Bluesky-defaultism. The operator chooses their own if they please.
53
54Some of those can denote files that don't exist yet, because it's more like *where* to put them, since a knot can for sure create files but won't choose paths for the operator. The host key and the sealed store both get created on demand, parent directories and all. Directories are a little different in that `repo.scan_path` and `lfs.store_path` (if the operator sets that) have to already exist and be writable, or the knot won't start. After that, `scan_path` gets filled out as repos arrive.
55
56Every key outside the `[messages]` block can also come from an environment variable, as one can see in the `example.toml`. Environment variables win over the file if both are specified btw.
57
58## Master key
59
60The master key is for unsealing the per-repo signing keys. One has to generate 32 bytes of base64 and keep it out of the config file, for opsec:
61
62```sh
63openssl rand -base64 32
64```
65
66Put it in an env file that only root can read, then `secrets.master_key_env` at the variable name:
67
68```sh
69# knot.env
70KNOT_MASTER_KEY=<base64 that was just generated>
71```
72
73**Back this up somewhere separate from the server.** The sealed key store on disk is useless without it, and every repo-DID on a knot is derived from keys inside that store. If the operator loses the master key, the repos will of course stay readable as regular git, but the knot will no longer be able to prove ownership to atproto.
74
75## Ports, Taylor's version (ha ha, 22 joke)
76
77A knot has two listeners where HTTP defaults to `[::]:5555`, SSH defaults to `[::]:2222`.
78
79Thus far in Tangled in general, SSH is heavily used because the existing Tangled knot doesn't do HTTP pushing. Tangled generally auto-generates a URL for SSH git'ing that looks like this:
80
81```
82git clone git@knot.oyster.cafe:did:plc:barnacle
83```
84
85However notice that from Tangled's client, it builds a clone URL with no port number in, because it always assumes 22. If the knot's SSH listener is listening on another port, all of its users have gotta either rewrite the URL as `ssh://git@knot.oyster.cafe:2222/did:plc:barnacle`, or add a block to their `~/.ssh/config`.
86
87So **give the knot port 22 if one can**! It doesn't need unix `git` user / shell account, so the only thing in the way is the remote computer's own sshd, which is usually already taking port 22.
88
89If the reader is nodding along instead of saying "no Lewis I won't change the remote computer's sshd because I like not bricking it" then:
90
91Moving sshd is something that can lock one out of one's own server, so do it in this order:
92
931. Open a second SSH session to the server and keep it open for this whole procedure. If step 4 goes wrong, having this session open might be the saving grace.
942. Edit `/etc/ssh/sshd_config` and set `Port 2200` or whatever free port one likes. Leave `Port 22` in place as well for now, so sshd listens on both.
953. Open the new port in the firewall if there is one. On ufw that's `ufw allow 2200/tcp` (I think!! Untested). On a cloud provider one will probably also have to deal with it / open it in their proprietary config.
964. Restart sshd, and **from the local computer, in a third terminal**, confirm `ssh -p 2200 root@the.server` works before continuing.
975. Once confirmed, only now remove `Port 22` from `sshd_config`, restart sshd once more, & give the port to the knot. When running the binary directly, that means `ssh_listen_addr = "[::]:22"`. Comparatively, in a container it entails publishing the container's 2222 as the host's 22, which is what the compose file example below does.
98
99If one would rather not move sshd at all, another cool option for having knot2 on port 22 is a second IP address on the remote computer. Bind sshd to one with `ListenAddress`, bind the knot to the other with `ssh_listen_addr = "<second-ip>:22"`, and just put the knot's DNS record on that second address. Leaving the knot on `[::]:22` would wildcard-bind every address on the box and would collide with sshd no matter which single IP that sshd listens on.
100
101HTTP can stay on 5555 behind a reverse proxy, or move to 443 if one wants the knot to terminate TLS itself.
102
103## Running with containers
104
105The `Containerfile` at project root builds a distroless image with just the `knot-server` binary in it:
106
107```sh
108podman build -t knot-oyster:latest .
109```
110
111I personally run it with a composefile. This is the file from `knot.oyster.cafe` with a few opsec adjustments:
112
113```yaml
114services:
115 knot:
116 image: localhost/knot-oyster:latest
117 container_name: knot-oyster
118 pull_policy: never
119 restart: unless-stopped
120 mem_limit: 2g
121 env_file: ./knot.env
122 ports:
123 - "0.0.0.0:22:2222"
124 - "[::]:22:2222"
125 volumes:
126 - ./config.toml:/etc/knot/config.toml:ro
127 - ./repos:/data/repos
128 - ./ssh:/data/ssh
129 - ./secrets:/data/secrets
130 - ./lfs:/data/lfs
131```
132
133The container keeps listening on 2222 internally and the host publishes that as 22, so the config file never has to change. My own instance publishes 2222 on the host because that computer already had sshd on 22 when I set it up, and my laziness has been regrettable until knot2 came with http pushing.
134
135Here's the corresponding config, with path specifying the mounted volumes:
136
137```toml
138[server]
139hostname = "knot.oyster.cafe"
140admins = ["did:plc:nel"] # well obviously this isn't a real DID but one gets the picture
141listen_addr = "[::]:5555"
142ssh_listen_addr = "[::]:2222"
143ssh_host_key_file = "/data/ssh/host_key"
144appview_endpoint = "https://tangled.org" # this is *not* Tangled defaultism, it's cosmetic for git operation messaging
145
146[repo]
147scan_path = "/data/repos"
148
149[secrets]
150sealed_key_file = "/data/secrets/sealed.bin"
151master_key_env = "KNOT_MASTER_KEY"
152
153[atproto]
154plc_directory = "https://plc.directory"
155
156[xrpc]
157trusted_proxy_header = "x-forwarded-for"
158
159[git]
160object_format = "sha256"
161
162[lfs]
163store_path = "/data/lfs"
164free_space_floor_bytes = 32212254720
165```
166
167Create the dirs, then let there be light I suppose:
168
169```sh
170mkdir -p repos ssh secrets lfs
171podman-compose up -d
172```
173
174The `mkdir` is necessary, since the knot won't start unless the repo and LFS directories are present/writable. Podman would create the bind-mount sources for the operator, but then they belong to whichever unix user podman has rather than to the operator.
175
176The knot creates the SSH host key on the first run at mode 600, aaand the sealed store on that same first run, because the knot's own signing key needs sealing before any repo exists. The knot purposefully won't load a host key that is group or other readable, so don't loosen those please.
177
178Note that this (my) config turns LFS on, since `lfs.store_path` is set. One can drop that whole `[lfs]` block if one doesn't want it. The floor of 30GiB is what I have judged for my disk (of 500GiB, doing other things at the same time), so pick something that suits one's own rather than copying mine.
179
180Speaking of LFS, I made the directory different in the first place so that we could specify a whole separate storage medium if wanted. For example, let's say I want my actual git repos to be wicked fast, so everything *else* is on an SSD, and *only* LFS is on a massive-but-relatively-cheap HDD cluster. Wouldn't want terabytes and terabytes of massive files taking up precious SSD space in this economy!
181
182## TLS
183
184My setup has Caddy in front, which gets me certificates for free and lets one machine serve several sites (which it does). The config is:
185
186```
187knot.oyster.cafe {
188 reverse_proxy knot-oyster:5555
189}
190```
191
192That talks to the knot by container name, which needs both containers on a single podman network. One creates such a network once with something like `podman network create tangled`, then add the network to the compose file above as an external one and to whatever runs the proxy. If one would rather not, publish `127.0.0.1:5555:5555` from the knot container and proxy to that instead.
193
194Set `xrpc.trusted_proxy_header = "x-forwarded-for"` when doing this, otherwise every client looks like it comes from the proxy and the ratelimiter wil treat them as one very busy mister. Only set it behind a proxy the operator controls, since a direct client can like, invent that header.
195
196The knot can also terminate TLS itself (and that's the only way to get its HTTP3 support) because a plain TCP frontend can't proxy QUIC. Using a certificate the operator already manages:
197
198```toml
199[server]
200listen_addr = "[::]:443"
201
202[tls]
203cert_path = "/data/tls/fullchain.pem"
204key_path = "/data/tls/privkey.pem"
205```
206
207Or let it fetch its own via ACME:
208
209```toml
210[server]
211listen_addr = "[::]:443"
212
213[tls]
214acme_enabled = true
215acme_cache_dir = "/data/acme"
216acme_contact = "nel@oyster.cafe"
217```
218
219ACME here uses the TLS-ALPN-01 challenge, so the knot has to be the reciever (in the phone sense) answering on 443 for the configured hostname. Set `acme_staging = true` while testing such that a typo doesn't nuke the Let's Encrypt ratelimit. Leave `trusted_proxy_header` unset in this mode, and open 443/udp in the firewall if one wants HTTP3 to be reachable.
220
221## First run
222
223```sh
224curl -s https://knot.oyster.cafe/xrpc/_health
225curl -s https://knot.oyster.cafe/xrpc/sh.tangled.owner
226curl -s https://knot.oyster.cafe/.well-known/did.json
227```
228
229`_health` reports the version, and if LFS is on it returns an error status when the LFS store isn't writable. `sh.tangled.owner` returns the first DID in `server.admins`, which is what the Tangled appview reads to confirm the operator is who they say they are when they register the knot. `did.json` is the knot's own `did:web` document, served at the hostname it was configured with.
230
231If any of the above endpoints doesn't pong, check out the stderr logs, `podman logs -f knot-oyster` in my case.
232
233## Letting people on
234
235Sharing is caring!
236
237If one wants the knot to show up nicely on Tangled the web app, register the knot on [tangled.org](https://tangled.org) with the same DID listed first in `server.admins`.
238
239Admission is closed by default, so an admin adds each member before they can create repos. Repo owners can then add their own collaborators without any admin involvement of course. If one wants a knot anyone can use:
240
241```toml
242[acl]
243admission = "open"
244```
245
246Note that open admission still respects the blocklist, naturally.
247
248Since ssh-pushing is so popular, users will usually authenticate with the SSH key they published on their PDS -> the most common support question the operator may get is that a user's ssh client agent offers five keys and gets rejected before it ever reaches the registered one. The fix on their side is:
249
250```
251Host knot.oyster.cafe
252 IdentityFile ~/.ssh/id_ed25519
253 IdentitiesOnly yes
254```
255
256## Clone URLs
257
258A git repo can be pushed/pulled by its owner + name, or by its own repo-DID. The owner can be a DID or an atproto handle.
259
260Over SSH:
261
262```sh
263git clone knot.oyster.cafe:nel.pet/squid
264git clone knot.oyster.cafe:did:plc:nel/squid
265git clone knot.oyster.cafe:did:plc:barnacle
266```
267
268Over HTTPS, the same:
269
270```sh
271git clone https://knot.oyster.cafe/nel.pet/squid
272git clone https://knot.oyster.cafe/did:plc:nel/squid
273git clone https://knot.oyster.cafe/did:plc:barnacle
274```
275
276Push works over both, of course . For HTTP pushing, it is up to the user to find a good Tangled-CLI or something that can put the right things in the git credential helper such that a service auth token is minted and used on push.
277
278A trailing `.git` on the repo name is optional, so `did:plc:nel/squid.git` goes to the same repo as `did:plc:nel/squid`. That only applies to the repo name variant though - `did:plc:barnacle.git` is read as a DID rather than as a repo-DID with a suffix, and it won't resolve. This would be made better from better DID parsing, since a `did:plc` can't have dots, only a `did:web` can.
279
280## Things worth knowing before one commits (get it?) to a config
281
282### SHA-256 is the default
283
284Yeah, sorry, let's modernize.
285
286`git.object_format` defaults to `sha256`, and it applies to repos at creation time. A SHA-256 repo cannot be pushed to or fetched from a SHA-1 repo, so if one expects users mirroring in from elsewhere, set `object_format = "sha1"` before anyone creates anything. Changing it later only affects new repos.
287
288### LFS is off until one supplies a path
289
290LFS turns on for both transports when `lfs.store_path` is set. `free_space_floor_bytes` is the disk headroom below which the knot starts refusing uploads, defaulting to 1GiB, so set it to the amount of free space one actually wants to keep.
291
292### Resources tune themselves
293
294`resources.max_threads` and `resources.max_memory_bytes` are smart-ceilings, both `0` by default meaning "use the whole computer". Under a container memory limit the knot reads the cgroup and sizes itself to that, so a `mem_limit` on the container usually suffices.
295
296### Maintenance runs on its own
297
298Commit-graphs, multi-pack indexes, bitmaps, and geometric repacks happen every 6 hours by default. Turn it all off with `maintenance.enabled = false` if one would rather do it oneself.
299
300### The homepage is replaceable, please do replace it
301
302`homepage.path` serves an HTML file of one's choice at `/`, and `homepage.enabled = false` disables the homepage entirely.
303
304## Backups
305
306Back these things up or don't come cryin' to me!
307
3081. The master key, wherever one keeps it.
3092. `secrets/sealed.bin`, the sealed key store. It's useless without the master key & the master key is useless without it, so treat them as a pair.
3103. `repos/`, which is every repo plus the knot's own ACL data. Git is the database, so that one directory is the knot's entire state.
3114. `lfs/`, if one enables it.
3125. I guess the SSH host key, though much less dire if it is lost and has to be changed.
313
314## Updating
315
316// TODO: publish to ATCR, maybe nix something something.
317
318```sh
319git pull
320podman build -t knot-oyster:latest .
321podman-compose up -d
322```
323
324The knot drains at `SIGTERM` time, so in-flight clones/pushes get up to 40s to finish before they're cut off.
325
326Happy knotting!
327