Talos boot
The boot server on spore that serves folly's signed iPXE, HookOS and SecureBoot ISO on the bootstrap VLAN, beside fleetd's enrollment name.
Talos boot is the part of spore that serves folly's signed iPXE, HookOS UKI and SecureBoot ISO to machines on the Kubernetes network, from spore's address on the bootstrap VLAN. fleetd serves the enrollment name's HTTPS at the same address, and Tinkerbell, also on spore, decides which machine boots what.
The site is the NixOS option homelab.site, declared in nix/profiles/anchor.nix and set to folly. The boot services take their state paths, metric labels, release tag and committed per-site files from it, so the paths in this page read <site> where they say folly.
Parts#
| Part | Job | Where it runs |
|---|---|---|
bootstrap VLAN interface | Gives spore its SPORE_BOOTSTRAP_IP, tagged on end0 | spore |
| dnsmasq TFTP | Serves the signed iPXE, talos/folly/ipxe-amd64-secureboot.efi, from /var/lib/tftpboot | spore |
netboot-not-allowed links | Serve the same iPXE at <mac>/netboot-not-allowed for each MAC NetBox holds, the name Smee offers while allowPXE is off, for firmware that takes Smee's offer over UniFi's (riptide's does; shale's does not). Smee is off on spore, so nothing asks for them; fleetd still makes and removes them under /var/lib/tftpboot on each reconcile pass | spore |
nginx talos-boot | Serves the ISO and HookOS on port 80 at SPORE_BOOTSTRAP_IP | spore |
| HookOS syslog | rsyslog on UDP 514 at SPORE_BOOTSTRAP_IP, into spore's journal under the identifier hookos (hookos-syslog.nix). Tinkerbell says how to read it | spore |
talos-boot-mirror (the mirror) | Fetches and checks the release each hour | spore |
| fleetd's boot listener | Serves enroll.lolwtf.ca on port 443 at SPORE_BOOTSTRAP_IP with a Let's Encrypt certificate from DNS-01: the device flow under /attest/ and a node's first config at /talos/config, or at /seed for a NixOS or Alpine class. The anchors' CoreDNS answers the name with SPORE_BOOTSTRAP_IP | spore |
talos-boot-check | Exports whether spore serves the boot files and the enrollment name, each minute | spore |
The boot check passes when dnsmasq is active and nginx serves HookOS at spore's bootstrap address. The HTTPS check passes when fleetd answers /healthz for enroll.lolwtf.ca at that address with a certificate that public roots trust.
Boot path#
The UDM relays the Kubernetes network's DHCP to Kea on spore, whose offer names SPORE_BOOTSTRAP_IP as the TFTP and boot server, so every machine on a node port netboots there, known or not. Firmware PXE loads the signed iPXE from it, and iPXE shows its menu, whose default chains HookOS from nginx at the same address. HookOS attests against fleetd before its agent waits for a Workflow. Tinkerbell has the details. HookOS is about 200 MB, too large for TFTP, so it goes over HTTP.
The bootstrap network has no DHCP: it holds spore's boot address. The Kubernetes network's DHCP names the anchors' DNS addresses, and folly's Talos nodes take time from them. CoreDNS and chrony answer on every interface of each anchor.
Netboot images#
iPXE is about 1.2 MB. Its embedded script runs DHCP, sets boot-url to http://<SPORE_BOOTSTRAP_IP>/talos/folly and asks Smee for nothing: with Smee on, Smee's per-MAC script boots the same HookOS the menu boots.
Menu#
iPXE displays a menu with a 10-second timeout.
| Item | Action |
|---|---|
hookos (default) | Load ${boot-url}/hookos-amd64-secureboot-uki.efi with the command line from nix/images/hookos-folly.nix. Hardware scripts use the same command line. |
enroll | Use sanboot to load the signed ISO. |
exit | Return to firmware. An item that cannot load also returns to firmware. |
Secure Boot enrollment#
In Setup Mode, the ISO's systemd-boot menu offers "Enroll Secure Boot Keys: auto". This loads PK, KEK and db from clusters/folly/talos/secureboot/, then reboots.
Select the enrollment item. systemd-boot enrolls automatically only in a virtual machine.
On HP firmware, clear the keys with Secure Boot off. Enroll the keys, then enable Secure Boot. HP restores factory keys if Secure Boot is enabled with no keys present.
An 800 G2 with BIOS N21 2.37 refuses the PK write. Version 2.63 accepts it.
iPXE constraints#
- sanboot requires an absolute URL, so the script sets
boot-url. - Before HookOS starts,
imgfreereleases other images. systemd-stub refuses to start while iPXE serves another image's initrd. - The script passes the command line as arguments. Without Secure Boot, systemd-stub uses those arguments instead of its signed command line.
HookOS image#
HookOS is one UKI for every node, signed with the HookOS key. Its signed command line names no worker, pins the agent image ghcr.io/jonpulsifer/tink-agent-autoid by digest, and names fleet_attest_url and fleet_ntp_server. That image sets the agent ID to the MAC of the NIC that holds the default route, and attests before the agent starts. The firmware checks iPXE, HookOS and each image iPXE chains against db, and PKI says which key signs what.
| URL path | File under /var/lib/talos-boot/folly/ |
|---|---|
/talos/folly/metal-amd64-secureboot.iso | current/metal-amd64-secureboot.iso |
/talos/folly/hookos-amd64-secureboot-uki.efi | current/hookos-amd64-secureboot-uki.efi |
The mirror#
Each image build is its own immutable GitHub release, tagged talos-folly/<talos_version>-<run id>. The mirror asks the releases API for the newest one of the version and downloads its ISO, iPXE and HookOS. It skips the download while it already serves that release's tag. .github/scripts/talos-uki-verify.sh, the image pipeline's check, requires one signature that the committed certificate's key made over the image's own hash: the UKI signing certificate for iPXE, and the HookOS certificate for HookOS. A new release moves current to prev. A failed run changes neither.
A Talos version bump fails the mirror until that version's release exists. spore serves the old release until then. A release without the ISO, iPXE or HookOS also fails it.
Alerts#
The cluster cannot reach the bootstrap VLAN, so spore's node exporter carries talos_boot_serving, talos_boot_https_serving, talos_boot_mirror_success, talos_boot_mirror_last_success_timestamp_seconds and talos_boot_uki_info, which names each slot's version and release. clusters/folly/monitoring/fleetd-rules.yaml declares the alerts.
| Alert | Meaning |
|---|---|
BootFilesNotServed | spore fails the boot check or exports no metrics |
EnrollmentNameNotServed | spore fails the HTTPS check or exports no metrics |
BootMirrorStale | The mirror has not verified a release for 6 hours |
Rules#
- spore's switch port must carry the bootstrap VLAN tagged, for the boot address and the gateway's relayed DHCP. capsule's and forge's ports do not carry it.
talos-uki-verify.shrunssbverify --certand also requires exactly one signature. sbverify alone passes a file with several signatures when any one of them verifies.- A machine on the bootstrap VLAN reaches spore directly, where the UniFi firewall cannot filter it. spore's
nixos-fw-bootstrapchain, fromtalos-boot.nix, runs before the ports the NixOS firewall opens on every interface. Over IPv4 it admits the ports declared forbootstrap(TFTP, HTTP, HTTPS, relayed DHCP, HookOS's syslog and the Tink server, and Smee's while Smee is on), DNS, NTP and ping, and drops the rest. Over IPv6 it admits only ICMPv6. Declare a new bootstrap service innetworking.firewall.interfaces.bootstrap, or the chain drops it. Theboot-planecheck fails if the chain admits ssh, NFS, Garage, an exporter or the Tinkerbell API, or if a host other than spore joins the VLAN. - Traffic from spore's bootstrap address takes the routing table numbered
BOOTSTRAP_VLAN, through a policy rule that systemd-networkd sets on thebootstrapinterface fromtalos-boot.nix. The table holds the VLAN's own route and a default via the gateway, the first host inBOOTSTRAP_CIDR. A request that the gateway routes onto the VLAN from another network therefore passes the NixOS firewall's strict reverse-path filter, and its reply leaves onbootstrap. - dnsmasq sends each TFTP transfer's data from port 69 (
tftp-single-port), so a transfer is one UDP flow. A gateway reply rule matches only the request's port, so a transfer from a server-chosen port would not cross the gateway. Theboot-planeandspore-reliabilitychecks fail without the setting. - A node on the Kubernetes network reaches spore through the gateway, whose policies
kubernetes_to_bootadmitSPORE_BOOTSTRAP_IPon TCP 80, 443 and 42113 (the Tink server), and UDP 69, 123 and 514 (HookOS's syslog). Smee's script port, 7080, returns with Smee. DHCP needs no policy: the gateway relays it onto the VLAN itself. - The
boot-planeandspore-reliabilitychecks innix/lib/checks.nixfail if dnsmasq could lease addresses or answer DHCP.
Where it lives#
nix/services/talos-boot.nix: the VLAN, firewall, dnsmasq, nginx paths, the checks and the mirrornix/services/fleetd.nix: the enrollment name's listener and certificate, and thenetboot-not-allowedlinksnix/services/hookos-syslog.nix: HookOS's syslog receivernix/services/tinkerbell.nix: Tinkerbell, with Smee behindhomelab.tinkerbell.smee.enableclusters/folly/config/anchor-topology.json:BOOTSTRAP_CIDR,BOOTSTRAP_VLAN,SPORE_BOOTSTRAP_IPterraform/network/unifi/folly/bootstrap.tf: the bootstrap network, the boot option and the switch portsterraform/network/unifi/folly/k8s.tf: the Kubernetes network's DHCPterraform/network/unifi/folly/firewall.tf: the policieskubernetes_to_bootclusters/folly/talos/talos.tf:talos_version, which names the releaseclusters/folly/talos/secureboot/uki-signing-cert.pem,hookos-signing-cert.pem: the certificates the mirror checksclusters/folly/talos/netboot.json: the HookOS release and the agent image digestnix/images/ipxe-folly.nix,nix/images/hookos-folly.nix: the unsigned iPXE, with its menu, and HookOSimages/tink-agent-autoid/: the agent image.github/scripts/talos-uki-verify.sh: the signature check, shared with the image pipeline
Related#
Linked from 10
- sporeA Raspberry Pi 5 on NVMe that serves x86 netboot, forge's HTTP boot image, the Talos boot image and Tinkerbell, lab DNS and time, and the SPIRE server.
- KubernetesThe folly and offsite Kubernetes clusters, how Flux applies clusters/ from git, and the sandbox runtimes and storage the clusters offer.
- Routing and firewallThe BGP routes and zone firewall that connect the folly and offsite clusters through the UniFi gateways, and folly's node ports and boot network.
- NixOSThe flake that builds each Linux host and OS image, the fleet baseline each host shares, and the daily upgrade from main.
- KeaThe DHCP server on spore for folly's Kubernetes network, which leases the netboot pool and names the signed iPXE, through the gateway's relay.
- NetbootThe network boot service on spore, with TFTP and HTTP for folly's boot files and a signed RAM-boot image that forge falls back to.
- NixOS node imageThe signed disk image a folly node of the nixos class boots, how CI builds and signs it, and how its first boot reads the fleet seed.
- TinkerbellThe provisioning stack on spore that power-cycles, wipes and reinstalls folly's Talos nodes, driven by fleetd from NetBox, with its own Kubernetes API.
- Node lifecycleHow a folly node goes from a NetBox device to an attested Talos node with a SPIRE identity, and back out, with the system that owns each step.
- Prepare a machineSet a new folly machine's firmware and AMT address on its node port and enroll the lab's Secure Boot keys over PXE, before NetBox provisions it.