This post is the overview: what’s running and how the fleet is shaped. The parts that need real room to explain have their own posts, indexed at the bottom.

Services

Before the internals, the services the fleet runs.

Open to anyone:

Sign-in required:

Nearly all of that shares a single small container, which is a slightly absurd amount of load for an OpenVZ guest on a 3.10 kernel and works fine.

What’s not in either table is the shell access: SSH to every host in the fleet reaches the internet only through Cloudflare tunnels, so there is no listening port to publish in the first place. That asymmetry is deliberate — the interesting engineering in this setup is in tunnels, meshes and grafting rather than virtual hosts.

The fleet

By type, as the flake partitions them:

Only fourteen of them run NixOS. That turns out to be the interesting thing about the setup: the engineering effort isn’t in the NixOS hosts, it’s in everything else. A NixOS module and a home-manager module that produce the same behaviour on stock Ubuntu need to agree, so the shared logic sits in pure-data modules underneath both — one source of truth, two renderers. When the Ubuntu box needs a firewall rule that its NixOS sibling gets from networking.firewall, the same data goes through a generator that writes a nushell script into /etc/ufw/after.init instead.

On those six non-NixOS hosts there’s one decision worth stealing. /nix/var/nix/profiles/default points at root’s home-manager profile, so pkgs.nix in root’s home.packages is the machine’s Nix, and its version is a property of flake.lock. The nix-daemon units are rendered by home-manager with a profile-indirect ExecStart. Upgrading Nix on a machine that doesn’t run NixOS becomes make home-root HOST=<h> and a daemon restart.

Deep dives

Three parts of this needed more room than an overview allows:

• Nix on a Phone, and the Trick That Makes It Work — running a full Nix userland on Android, and the base-32-to-base-64 hash re-encoding that keeps store paths byte-identical in length while pointing somewhere else entirely.
• A 2009 iMac That Refuses to Die — inverted SSH ciphers on a CPU with no AES-NI, a measured 92 MiB kernel diet, an ext4 patch with an upstream submission kit, and swap over NVMe-over-TCP.
• Pins, Secrets, and Deploys That Can’t Strand You — seven nixpkgs branches each pinned for a stated reason, secrets without agenix, and two mechanisms that stop a remote switch from bricking the box.

Still to write: the cross-border connectivity layer — mesh VPN, reverse tunnels, multipath TCP — which is the other half of the repo.

What the repo is actually for

The thing I’d most want to convey isn’t any single mechanism. It’s a habit: in this repo, comments don’t say what the code does. They say what broke.

[host] demonstrated exactly this (“Failed to open …/nix-daemon.socket” three seconds before nix.mount).

1690/1693 bin-farm symlinks are 60–90 B targets.

measured: 0 kvm:kvm_userspace_exit in 10 s.

the gate boot missed by EXACTLY TWO MESSAGES.

Every one of those is a debugging session someone (me) already paid for, written down at the exact line where the next person would otherwise repeat it. Nix is what makes the fleet reproducible. The comments are what make it maintainable — they’re the difference between a configuration I can still change in two years and one I’d be afraid to touch.

That, more than the twenty-one hosts, is the infrastructure.

This post is about the unglamorous half: how versions get pinned, how secrets move, and how a deploy avoids leaving a machine unreachable. None of it is clever in isolation. All of it is what makes the rest safe to touch.

Seven nixpkgs, each for a reason

The flake tracks seven nixpkgs branches simultaneously. That sounds undisciplined until you read why each one is pinned — every input in flake.nix carries its justification inline:

• 26.05 — the default everywhere, Hydra-verified.
• unstable — opt-in per package via pkgs.unstable.*.
• 25.11 — glibc 2.40, because that’s what the phone’s runtime is built against.
• 24.11 — mesa 24.2, the last release with bundled libgbm.
• 24.05 — systemd 255, the last version whose chase_and_open still has a name_to_handle_at fallback. One host is an OpenVZ container on a 3.10 kernel that lacks the syscall.
• 22.11 — xorg-server 1.20, the newest X the NVIDIA 340.xx legacy driver can talk to.
• 22.05 — gcc-12 and a glibc 2.34 floor, for building release binaries that assume an older toolchain.

And one more that isn’t a branch at all: a specific revision hash from release-20.09, frozen in 2022, kept alive because it’s the last nixpkgs containing a cudatoolkit_6_5 expression. 21.05 dropped CUDA older than 10. I have a GPU that needs 6.5.

None of these leak into each other. The CUDA one is the sharpest case: the override swaps a modern stdenv, modern dependencies and a rewrapped older gcc into the 2020-era recipe, so it evaluates against ancient expressions but no 20.09 store path reaches the runtime closure.

The pattern worth stealing here isn’t the number of pins — it’s that each one names the specific artifact it exists for. A pin justified as “newer versions broke something” rots into a mystery within a year. A pin justified as “mesa 24.2, last release with bundled libgbm” tells you exactly what has to become true before you can delete it.

Secrets

No agenix, no sops-nix. A private Git submodule is consumed as a flake input, which means it’s pinned in flake.lock like anything else, and its contents arrive at modules as an ordinary function argument through specialArgs.

Two rules make it hold up. Secrets are passed to services as files — rendered with writeText, or staged out of band by the installer — never as literals on a command line where they’d show up in ps. And where a secret is owned by something outside Nix, it’s resolved at merge time from that system’s own env file rather than being copied into the repo, so it never enters the Nix store at all; a job whose key is missing is skipped with a warning rather than silently mis-deploying. Public keys, meanwhile, live in the open in a plain tracked file, because they’re public.

The honest caveat: anything rendered into a unit file or a config is readable by whoever can read that file, and this scheme doesn’t pretend otherwise. Where that exposure exists, the module says so in a comment rather than implying a guarantee it can’t make. I’d rather have a documented boundary than an encrypted-at-rest story that quietly decrypts into a world-readable store path anyway.

Deploying without losing the machine

Two safety mechanisms I’d recommend to anyone administering remote machines.

First, every switch runs inside a transient system-level systemd unit, so an SSH disconnect can’t kill a half-applied configuration change. The invocation looks over-engineered and each layer was earned:

systemd-run --pty --pipe --wait --collect --same-dir \
  env --ignore-signal=SIGHUP --ignore-signal=SIGINT \
  script -qefc "<cmd>" /tmp/<unit>.log

systemd-run moves the work off the SSH connection’s lifetime. The script(1) wrapper is there because when the client dies, the outer pty hangs up and every later write fails with EIO even with SIGHUP ignored — a bare --pty command dies mid-switch. script absorbs the EIO while the command’s inner pty stays healthy, which keeps live colour output and progress bars working. (nohup can’t substitute: with a tty stdout it redirects to nohup.out and you lose the live view.)

Second, reinstalling a box whose only access path is a reverse SSH tunnel. A stock kexec image would sever that tunnel and strand the machine. So the installer image inherits the target’s own SSH host key and uses it as a client identity to dial back out to a relay, re-exposing its sshd there. Two properties fall out of that: the image contains no secrets, so it can be cached and shared freely; and until disko actually partitions anything, kexec is RAM-only — a power cycle falls back to the untouched old system, which restores the original tunnel. A failed phone-home is recoverable without console access.

That second property is the one I’d underline. The interesting question about a remote reinstall isn’t whether the happy path works; it’s what state the machine is in when the process dies halfway. Making the destructive step the last step, and keeping everything before it in RAM, converts “drive four hours to attach a console” into “cut the power and try again.”

The phone runs a full Nix userland on Android, against a custom glibc 2.40 carrying Termux’s kernel-compatibility patches, with a patched fakechroot as /etc/ld.so.preload. It has the same shell, the same editors, the same package set as every other host. I wrote this post on it.

The constraint

Getting there required reimplementing nixpkgs’ replaceDependencies, and the reason is a nice constraint.

Upstream patches a dependency out of an already-built closure by binary string substitution — it walks the NAR stream and rewrites store paths in place. Rewriting bytes inside a compiled ELF file means you cannot change how many bytes there are: shifting everything after the edit would break every offset in the file. So the old and new paths must be byte-identical in length.

That constraint fails twice on Android. Standard glibc and an Android-patched glibc don’t have the same path length. And /nix/store doesn’t exist at the real path on Android at all — the store lives under the app’s data directory, which is a different length again.

The trick

The solution is a three-layer rewrite, and the middle layer is the clever part. Every store path maps to a symlink whose basename is a re-encoding of the same Nix hash:

The Nix store path already carries a 20-byte hash encoded in nix-base-32 as the first 32 chars of the basename. We re-encode it as base-64 URL-safe (27 chars for the same 160 bits — nix-base-32 is 5 bits/char, base-64 is 6) and take a prefix that makes the symlink path same total length as the original store path.

Base-32 spends five bits per character; base-64 spends six. The same 160-bit hash therefore fits in 27 characters instead of 32, and those five recovered characters are the budget you spend on a different path prefix — while keeping the total length identical, which is what the byte-level substitution demands.

It’s a pleasing kind of solution: the information you need to preserve (160 bits of hash) and the constraint you have to satisfy (exact path length) turn out to be independent, once you stop assuming the hash has to stay in the encoding Nix chose for it. For a typical 44-character basename the crop lands at exactly 27 chars, so the full hash survives. At the pathological minimum it’s 16 chars — still 96 bits, still collision-safe.

The three layers

Layer one does OLD → SYMLINK, same-length by construction, as above.

Layer two does SYMLINK → REAL structurally rather than byte-wise: ELF interpreters, RPATHs, NAR symlink targets, script literals, shebangs. These are all length-agnostic — you’re rewriting a field, not patching bytes in place — so the real path can be any length it likes. The glibc swap itself, being cross-length, rides this pass.

The result is that the whole environment gets patched at once. There are no per-package Android variants to maintain: packages come out of the ordinary binary cache and get rewritten on the way in, rather than being rebuilt from source for the platform. On a phone, that distinction is the difference between a working system and a battery-shaped space heater.

Everything above is why the Makefile in the blog you’re reading can invoke a grafted Ruby 4.0 on a phone and have it behave like Ruby on a normal Linux box — the same bundle exec jekyll build as the CI runner, on the same flake, from a device in my pocket.

Exactly one host in that flake is a desktop, and it’s a 2009 iMac: Core 2 Duo, 3.8 GiB of RAM, a GeForce 9400M. Its notes file is the longest in the repo, and it has an evidence/ directory of GPU-freeze forensics. It’s also the reason the flake carries a nixpkgs revision frozen in 2022 — the last one that still packages CUDA 6.5.

Keeping a sixteen-year-old machine genuinely useful, rather than nominally booting, turns out to be a series of specific fights. Here are four.

Inverted cipher preferences

The Penryn CPU has no AES-NI and no PCLMULQDQ, so AES is software and GCM’s GHASH is table-driven. Measured on the box: ChaCha20-Poly1305 costs 4.7 CPU-seconds per GB, AES-256-GCM costs 8.3.

This host therefore wants the opposite SSH cipher order from the rest of the fleet, whose cloud CPUs all have AES-NI and for which the ordering is reversed. So the setting is host-scoped, with a comment forbidding anyone from hoisting it into the shared module — because the shared module is exactly where a well-meaning future me would put it.

This is the general shape of the problem with old hardware in a fleet: the correct configuration isn’t a worse version of the modern one, it’s a different one, and the abstraction that serves twenty homogeneous machines actively fights the twenty-first.

A ~92 MiB kernel memory diet

On a machine with 3.8 GiB, kernel overhead is real. The diet was itemised and measured rather than guessed: printk shift, BTF off, the ftrace family off, the kexec family off, a smaller swiotlb.

Verified across a reboot — reserved memory went from 275,960K to 204,260K. Every item in that list was checked live before it was changed, which matters, because roughly half of the “obvious” savings I expected turned out to be worth nothing on this kernel.

An ext4 patch with an upstream submission kit

Kernel-side creation of inline-data symlinks was removed upstream. On this machine that’s a measurable cost, because a survey of the Nix bin farm found 1690 of 1693 symlinks have 60–90 byte targets — small enough to live in the inode, but instead each one gets a 4 KiB block and a cold seek on a spinning disk.

The patch re-adds it. Two details made it more than a local hack: the on-disk format matches e2fsck 1.47.4’s validation exactly — a mismatch would make pass 2 deallocate the symlinks, and this host boots with fsck.repair=yes, so getting it subtly wrong would eat the filesystem rather than merely fail — and the tree carries a cover letter, a documentation patch and an fstests case, ready to send.

Swap over the network

It swaps to a 16 GB NVMe-over-TCP device exported by a workstation elsewhere on the network.

The lease protocol is my favourite thing in the repo. A client hands the daemon a pidfd over a Unix socket with SCM_RIGHTS, then execvs its real payload after closing everything. The daemon polls that pidfd, which becomes readable on any exit — including SIGKILL, and immune to PID reuse, which a bare PID check is not.

The properties fall out for free rather than being enforced: nothing is inherited into the payload process, so it can’t leak the lease; concurrent users refcount naturally; and the whole thing is crash-safe by construction rather than by cleanup handler. Because it’s socket-activated, no client path needs sudo, polkit or systemctl — access control is the socket’s file mode. It tears itself down after a grace period and exits.

Why bother

The machine owes me nothing. But every one of these fights produced something reusable: the cipher lesson applies to any heterogeneous fleet, the ext4 patch is going upstream, and the swap lease is a pattern I’d use anywhere I needed a crash-safe resource handoff between unprivileged processes.

Old hardware is a good teacher because it refuses to let you paper over inefficiency with headroom.

What you’ll find here

• About — the interests behind these notes.
• The blog — practical writing about Nix, operating systems, networking, self-hosting, and debugging.
• Search — find a topic or return to a detail from an earlier post.

Why keep a notebook?

The final command rarely tells the whole story. These posts keep the constraints, failed attempts, and reasons behind a solution alongside the steps that worked.

I hope they save someone else a little time, or give them a useful place to start.