Install Talos on a node

Use this runbook to boot a NixOS node into the Talos Linux installer without a USB drive or a site visit. kexec loads the Image Factory kernel for the cluster's schematic from the running system. Talos then waits in maintenance mode, with the disk untouched, until the clusters/<site>/talos/ root applies its machine config and installs it. Rebuild a cluster on Talos says when to run it for each node. To install a folly node with Secure Boot, use Provision a node instead.

warning

This procedure changes live state by hand. It is an exception to the GitOps rule because the node runs no Talos API until it boots the installer.

Before you start#

You need SSH access to the node as jawn, with sudo, and talosctl on your machine.
You need a path to the node's port 50000 that does not depend on the node's own cluster.
On folly, you need kubectl access, as Get cluster admin access describes, and openssl.
Get the schematic ID from tofu -chdir=clusters/<site>/talos output -raw schematic_id, and the version from talos_version in clusters/<site>/talos/talos.tf.

<node> is the host name, and <addr> is its address: on folly its address in clusters/folly/inventory/nodes.yaml, the nodes file, and on offsite its entry in NODE_ADDRESSES in clusters/offsite/config/cluster-topology.json.

Load the installer#

Download the kernel, the initramfs and the kernel command line for the schematic.

S=<schematic> V=<version>
sudo mkdir -p /var/tmp/talos && cd /var/tmp/talos
sudo curl -fsSLo kernel-amd64 "https://factory.talos.dev/image/$S/$V/kernel-amd64"
sudo curl -fsSLo initramfs-amd64.xz "https://factory.talos.dev/image/$S/$V/initramfs-amd64.xz"
sudo curl -fsSLo cmdline "https://factory.talos.dev/image/$S/$V/cmdline-metal-amd64"

Make sure that the command line names the platform.
```
grep -o 'talos.platform=metal' cmdline
```
Result: talos.platform=metal.

Load the installer, and make sure that the kernel holds it.

sudo kexec -l kernel-amd64 --initrd=initramfs-amd64.xz --command-line="$(cat cmdline)"
cat /sys/kernel/kexec_loaded

Result: 1.

caution

The next step stops every pod on the node. Maintenance mode cannot reboot. Only a machine config or a power cycle takes the node out of it.

Boot the installer.
```
sudo systemctl kexec
```

Check maintenance mode#

From your machine, make sure that Talos answers.
```
talosctl version --insecure --nodes <addr>
```
Result: A Server block with the Talos version.
Make sure that the schematic's extensions loaded.
```
talosctl get extensions --insecure --nodes <addr>
```
Result: gvisor, i915, intel-ucode, kata-containers, realtek-firmware, and a row named for the schematic ID.
Read the disks and the links.
```
talosctl get disks --insecure --nodes <addr>
talosctl get links --insecure --nodes <addr>
```
Result: The install disk named by install for the node in talos.tf, and an eno or enp link up. Cilium's devices matches those names.

Read the GPU's kernel log.

talosctl dmesg --insecure --nodes <addr> | grep -iE 'i915|wedged|GuC'

Result: i915 lines with no wedged.

note

A GPU that was never reset can wedge after kexec. Whether Talos boots cleanly with module_blacklist=i915 on the command line is not known.

Check the volumes after the install#

The clusters/<site>/talos/ apply installs Talos and reboots the node. Then do these checks before a pod lands on the node. A pod that lands first writes to EPHEMERAL, and the error is silent.

Read the volumes.
```
talosctl --context <site> -n <addr> get volumestatus
```
Result: EPHEMERAL, u-data and, on the control plane, ETCD, each in the phase ready.
Make sure that the data volume is mounted.
```
talosctl --context <site> -n <addr> get mountstatus
```
Result: /var/mnt/data with the filesystem xfs.
Make sure that the node runs the root's installer.
```
talosctl --context <site> -n <addr> version --short
```
Result: The talos_version of the root.

If the node is riptide, power-cycle it once before you use the GPU.

talosctl --context <site> -n <addr> reboot --mode powercycle

Approve the node's first certificate#

On folly, the controller manager approves no kubelet certificate that a bootstrap token asks for: an admission policy in clusters/folly/tinkerbell-access/node-client-approval.yaml refuses it the approval. fleetd approves only a worker that it installs. A node that joins with the cluster's token, such as shale or a control plane, waits for you. A cluster that Talos bootstrapped approves its first nodes itself until Flux applies that file. KubeletCSRPending fires when a request waits an hour. offsite approves its nodes itself.

Find the node's request.
```
kubectl --context folly get csr --field-selector spec.signerName=kubernetes.io/kube-apiserver-client-kubelet
```
Result: A Pending request whose requestor is system:bootstrap:<id>, the ID of the cluster's token. fleetd also journals it on the node's NetBox device.

Make sure that the request names the node you install, and nothing more.

kubectl --context folly get csr <csr> -o jsonpath='{.spec.request}' | base64 -d | openssl req -noout -subject -text | grep -E 'subject=|Alternative'

Result: subject=O=system:nodes, CN=system:node:<node>, and no Alternative line.