NetBox
The record of every machine in the lab, on capsule, reachable only over the tailnet.
NetBox is the lab's asset and lifecycle record: each machine, its place, its cabling and its status. It runs on capsule from the NixOS services.netbox module, outside both clusters. The owner edits it in the UI.
Parts#
Every part runs on capsule.
| Part | Job |
|---|---|
netbox | The UI and REST API, on gunicorn at [::1]:8001 |
netbox-rq | Background jobs |
| PostgreSQL 17 | The netbox database, on its Unix socket |
Redis netbox | Cache and job queue, on its Unix socket |
| nginx | Serves /static/ and proxies gunicorn at 127.0.0.1:8080 |
netbox-serve | Tailscale serve: TLS on capsule's tailnet address, proxied to nginx |
netbox-backup | Nightly database dump and media archive to Garage, 03:15 Halifax |
Use it#
Open https://capsule.<tailnet>, which the tailnet grants open to the owner. Sign in as admin with the password in the 1Password item netbox-folly. Scripts use the write-enabled admin token netbox/api-token in nix/secrets/capsule.sops.yaml, also api-token in that item.
Atlantis on offsite reaches the API over HTTP on port 8080 at capsule's Anchor network address, which the nest_k8s_to_capsule_netbox policy and capsule's firewall admit from offsite's node, VIP and pod ranges.
To add a node, add its entry to clusters/folly/inventory/nodes.yaml in a pull request; Atlantis applies it as a planned device. Provision a node has the steps.
Data#
terraform/netbox/folly applies the inventory to NetBox.
| Source | Declares |
|---|---|
clusters/folly/inventory/nodes.yaml | Each node's device, address, cable and custom fields |
lab-inventory.json in the root | Other physical hosts and network equipment at folly and oldschool; confirmed rack associations and stack order |
sites.tf | The racks |
networks.tf | VLANs and missing prefixes, read from UniFi catalogs and topology files |
rack-components.json | Rack elevations, including shelves, panels and power units |
The root also declares node device types, custom fields and choice sets on dcim.device.
The event rule fleet-controller-device-update sends every device update to fleetd's webhook. Its address comes from clusters/folly/config/anchor-topology.json. The webhook secret is set in NetBox by hand.
The anchors, folly site and node switch retain their seed records. Devices carry the tag managed-by: git-seed. Unverified cabling stays unset.
oldschool is the remote site whose cluster is offsite. Hardware inventory explains the nodes file and drift check.
Users and tokens#
terraform/netbox/folly declares two API-only users beside admin, the superuser. Each has a random password that nothing reads, so nobody signs in as it, and the owner mints each user's v2 token in the UI. Terraform manages no token.
| User | Token lives in | Permissions |
|---|---|---|
terraform | The 1Password item netbox-folly-terraform, which Atlantis reads | View, add, change and delete on the types terraform/netbox/folly manages, including users.user and users.objectpermission; management of sites, racks, roles, platforms, manufacturers, interfaces, MAC addresses, VLANs and prefixes for the lab inventory |
fleetd | fleetd/netbox-token in spore's SOPS file | View, change and add on only what apps/fleetd calls: view dcim.device, dcim.interface, dcim.macaddress, extras.journalentry and core.objectchange; change dcim.device and dcim.interface; add dcim.interface, dcim.macaddress and extras.journalentry. No delete, no superuser, no staff |
The provider cannot set a superuser, so terraform holds object permissions instead. It manages the permissions it holds, so it can keep managing users and permissions, and for the same reason it can grant itself anything. What the separate user gives is a name in the changelog: fleetd honours a device's approved EK binding only when the user named by homelab.fleetd.bindingAuthor (terraform) made its newest change, and a token of admin or fleetd never counts.
A token still on admin makes Terraform's writes record as admin, so no binding counts and every planned machine waits. Switch a NetBox token to its own user moves each token.
Backups#
netbox-backup streams pg_dump --format=custom and a tarball of /var/lib/netbox/media to netbox/<UTC time>/ in folly's Garage cluster, and deletes objects older than 30 days. On success it writes lab_backup_last_success_timestamp_seconds{source="netbox"}. The netbox bucket has no off-site copy.
To restore, stop netbox.service, run pg_restore --clean --if-exists -d netbox as netbox, and unpack the tarball into /var/lib/netbox.
Rules#
- Change
services.netbox.packageon purpose, after a backup.netbox.serviceruns the migrations on its first start after a version change, and NetBox has no downgrade. - Keep PostgreSQL's pinned major version. A new major version needs a dump and restore.
- Keep nginx on loopback. Tailscale serve is the only way in, and the bootstrap VLAN must never reach NetBox.
- Keep
netbox/secret-keyandnetbox/api-token-pepper. A new pepper invalidates every API token, and a new secret key ends every session. ALLOWED_HOSTSis the host's tailnet name, so another anchor serves NetBox at a new address.- Keep
CHANGELOG_RETENTIONat 0. fleetd honours a device's approved EK binding only while the changelog holds Terraform's write of it, and its token reads the changelog.
Operate#
| Alert | Meaning |
|---|---|
CapsuleNetboxDown | NetBox, its worker, PostgreSQL, Redis or the Tailscale serve unit is not active |
HostBackupStale with source="netbox" | No backup has succeeded for a day and two hours |
Where it lives#
nix/services/netbox.nix: NetBox, nginx, Tailscale serve, the superuser bootstrap and the backupterraform/netbox/folly: the nodes, the custom fields and the webhooknix/secrets/capsule.sops.yaml:netbox/andgarage/keys/netbox/nix/services/garage.nix: thenetboxbucket and key, which spore's SOPS file holds for the bootstrapclusters/folly/monitoring/capsule.yaml: the alerts
Related#
Linked from 8
- capsuleA Raspberry Pi 5 on NVMe that is one of the three lab DNS and time anchors, and runs NetBox.
- IdentitySPIRE on spore gives each folly node with a TPM 2.0 an identity rooted in its TPM, by LDevID or EK hash, gives its pods SPIFFE IDs, and lets a pod reach GCP with no key file.
- Node lifecycleHow a folly node goes from a NetBox device to an attested Talos node with a SPIRE identity, and back out, with the system that owns each step.
- Hardware inventoryThe nodes file that declares each folly Kubernetes node, the code that reads it, the OpenTofu root that applies it into NetBox, and the check that compares NetBox with the machines.
- Provision a nodeDeclare a folly machine in the nodes file, plug it into its node port, and let it attest and install as a Talos worker or a joining control plane.
- Reprovision a nodeReinstall, attest again or wipe an installed folly node by setting its NetBox device's desired_action.
- Retire a nodeDrain a folly worker, back up its volumes, and set its NetBox device to decommissioning, so that fleetd wipes it, then remove its entry from the nodes file.
- Switch a NetBox token to its own userMint a v2 token for the terraform and fleetd NetBox users, move Terraform and fleetd onto them, and revoke the admin tokens.