Topology
The two JSON files that hold the lab's network facts, the format rule they follow, and the Flux, Nix and OpenTofu code that reads them.
Two JSON files are the single source of truth for the lab's network facts: addresses, subnets, API server endpoints and BGP ASNs. Flux, Nix and OpenTofu read each fact from these files, so one change reaches every layer. Name the key and the file on a page or in code, and do not copy the value. Network says what the networks are.
Files#
| File | Holds | Keys |
|---|---|---|
clusters/<site>/config/cluster-topology.json | One cluster: its node, pod, service and load-balancer ranges, API server, gateway router and BGP ASNs. On offsite it also holds the address of each node | CLUSTER_NAME, API_SERVER_*, ROUTER_IP, *_CIDR, LB_RANGE, CLUSTER_DNS, BGP_*_ASN, on folly K8S_DHCP_CIDR, the Kubernetes network's DHCP pool inside K8S_NODE_CIDR, and on offsite NODE_ADDRESSES |
clusters/folly/config/anchor-topology.json | The folly lab: the Anchor network, future and bootstrap ranges, the bootstrap VLAN ID, and the addresses of lab hosts and Windows desktops | ANCHOR_CIDR, FUTURE_CIDR, BOOTSTRAP_CIDR, BOOTSTRAP_VLAN, <HOST>_IP, SPORE_BOOTSTRAP_IP |
Print the keys and values of a file with jq .data <file>.
Format#
Each file is a Kubernetes ConfigMap in the flux-system namespace.
It uses JSON, which is also valid YAML. Flux applies it without conversion.
The data field is a flat map. Both keys and values must be strings because Flux substitution accepts only strings.
Readers parse the values into the types they need:
| Type | Encoding | Example key |
|---|---|---|
| List | Comma-separated string | CLUSTER_DNS |
| Number | Quoted string | API_SERVER_PORT |
| Map | JSON object inside a string | offsite's NODE_ADDRESSES, which maps host names to addresses |
Readers#
| Reader | How it reads the files | Where |
|---|---|---|
| Flux | The config Flux Kustomization applies each file as a ConfigMap. Other Flux Kustomizations replace ${KEY} in their manifests with values from it, through postBuild.substituteFrom. | clusters/<site>/config/kustomization.yaml and clusters/<site>/flux-system/ |
| Nix | builtins.fromJSON at evaluation | nix/lib/networks.nix for folly, and nix/lib/anchor.nix for the Anchor network |
| OpenTofu | The cluster-topology module returns the data of one file, selected by its site and config_map inputs | terraform/modules/cluster-topology/ |
On folly, only the monitoring and storage Flux Kustomizations substitute from anchor-topology. To find the OpenTofu roots that read a file, run git grep -l modules/cluster-topology -- terraform clusters.
terraform/network/tailscale/fleet.tf.json follows the same pattern for naming facts: the tailnet name and the public DNS zone. The Tailscale root loads it as local.fleet, and nix/lib/fleet.nix reads it.
Checks#
| Check | What it enforces | Where it runs |
|---|---|---|
| Topology contract | Each cluster-topology.json is the flux-system/cluster-topology ConfigMap, has flat string data and every required key, holds valid addresses, CIDRs, port and ASNs, and does not overlap the other cluster. offsite's NODE_ADDRESSES is a JSON object of hostname to address, each address is in K8S_NODE_CIDR and belongs to one node, and API_SERVER_IP is a node's address. folly's file must not set NODE_ADDRESSES, and must set K8S_DHCP_CIDR inside K8S_NODE_CIDR, clear of ROUTER_IP and API_SERVER_IP. | .github/workflows/topology-contract.yml, with the conftest policy .github/policy/cluster-topology.rego |
| Anchor address preconditions | Each lab host address in anchor-topology.json matches its reservation in terraform/network/unifi/folly/clients.yaml | The plan of terraform/network/unifi/folly |
| Node addresses outside DHCP | On folly no hardware inventory address is in K8S_DHCP_CIDR, and a node with a static address in the Talos module is in K8S_NODE_CIDR and outside K8S_DHCP_CIDR. On offsite each node has a MAC under k8s in terraform/network/unifi/offsite/clients.yaml, reserved at its NODE_ADDRESSES value. | The plans of terraform/network/unifi/<site> and clusters/folly/talos |
| Node addresses | A folly node's address is in K8S_NODE_CIDR, is shared with no other node, and API_SERVER_IP is a node's address | The plan of terraform/netbox/folly |
| Nix evaluation | Every host still evaluates with the new values | nix-ci.yaml, on a change to any topology file |
The topology contract does not check anchor-topology.json.
Rules#
- Keep every
datavalue a string. Kubernetes rejects a ConfigMap with any other value, and Flux substitutes only strings. - A change to a topology file gets no Atlantis autoplan, because
ATLANTIS_AUTOPLAN_FILE_LISTmatches no.jsonfile. Plan and apply each root that reads the file, as OpenTofu and Atlantis describes. - Change a lab host address in
anchor-topology.jsonandclients.yamlin the same pull request, or the precondition fails the plan. - A folly node's address is in
clusters/folly/inventory/nodes.yaml; the Talos root manages the declared nodes NetBox holdsactiveorfailed. An offsite node's address is inNODE_ADDRESSESonly, and its MAC is interraform/network/unifi/offsite/clients.yaml. A folly node's DNS record and static address follow its address, and an offsite node's DNS record and reservation. The roots that read it areterraform/network/unifi/<site>,terraform/network/cloudflarefor offsite,clusters/<site>/talos, andterraform/netbox/follyfor folly. - Some files hold a literal copy of a topology value: the FRR files, whose offsite
neighborlines are theNODE_ADDRESSESvalues,policy.hujsonand the addresses pinned in app manifests. Change them in the same pull request, as the Network rules describe. - Put a setting that is not a network fact, such as a time zone or a DNS zone for Apps, in
cluster-settings:clusters/base/cluster-settings.yaml, patched byclusters/<site>/config/cluster-settings.yaml. CATHY_IPandPBX_SIP_VIPare addresses in folly'scluster-settings, for the PBX.CATHY_IPis the office phone on Management, which neither topology file covers, and the follyappsFlux Kustomization does not substitute fromanchor-topology.terraform/network/unifi/folly/voip.tfchecksCATHY_IPagainstclients.yaml.
Related#
Linked from 10
- PBXTwo Asterisk phone switches, one on folly for the four lines of the office phone and one on offsite that hands its callers to an ElevenLabs troll agent.
- KubernetesThe folly and offsite Kubernetes clusters, how Flux applies clusters/ from git, and the sandbox runtimes and storage the clusters offer.
- NetworkThe two sites, their UniFi gateways and networks, and the tunnel, routes, ingress and tailnet that connect them.
- Ingress and DNSThe Gateways, DNS records, certificates, tunnels and lab resolvers that give cluster services a name and a path from the LAN, tailnet or internet.
- Routing and firewallThe BGP routes and zone firewall that connect the folly and offsite clusters through the UniFi gateways, and folly's node ports and boot network.
- NixOSThe flake that builds each Linux host and OS image, the fleet baseline each host shares, and the daily upgrade from main.
- Talos bootThe boot server on spore that serves folly's signed iPXE, HookOS and SecureBoot ISO on the bootstrap VLAN, beside fleetd's enrollment name.
- OpenTofu and AtlantisThe OpenTofu roots that declare the network, cloud accounts, identity and cluster bootstrap, and the Atlantis server that plans and applies them from pull requests.
- Folly node lifecycle and trust designThe technical design, tradeoffs, evidence and threat boundaries for node provisioning, trusted boot and workload identity at folly.
- Hardware inventoryThe nodes file that declares each folly Kubernetes node, the code that reads it, the OpenTofu root that applies it into NetBox, and the check that compares NetBox with the machines.