infra_wiki

Network

The two sites, their UniFi gateways and networks, and the tunnel, routes, ingress and tailnet that connect them.

The network connects two sites, folly (home) and offsite (remote). Each site has a UniFi gateway, its own networks and one Kubernetes cluster. Each gateway runs a UniFi console, the web app that configures the site. folly's console calls offsite nest, and offsite's console calls folly fml.

Parts#

PartJobWhere it runs
UniFi gatewayRoutes and filters traffic, and exchanges routes with the cluster over BGPUDM Pro at folly, UCG Max at offsite
Site Magic tunnelCarries traffic and routes between the sites over WireGuard (wgsts1000)The two gateways
CiliumAssigns pod and load-balancer addresses (VIPs), and announces them over BGPEach Kubernetes node
IngressGives services DNS names, certificates and public reachEach cluster
Tailscale ConnectorRoutes Tailscale traffic into the siteEach cluster
Lab resolversServe lab DNS and NTPcapsule and spore

Networks#

SiteNetworkJob
follyManagementHome network, WLAN fml
follyAnchorThe anchors and other lab hosts, WLAN lab (open, hidden)
follyKubernetesfolly nodes. DHCP offers iPXE netboot from spore.
follyfutureIPv6 prefix delegated from the WAN. Holds the Windows desktops tallboy and atomic.
follyiotIoT devices
offsiteDefaultClient LAN
offsiteKubernetesoffsite nodes

Rules#

  • Read addresses, subnets and ASNs from cluster-topology.json and anchor-topology.json. Do not copy them.
  • Change a subnet and every copy of it in one PR, or routes and DNS break. Find the copies with git grep -nF '<old prefix>'. They include the FRR files, policy.hujson, the Tailscale Connectors and VIPs pinned in app manifests.
  • After a change to a node subnet, future or offsite's Default network, edit the Site Magic subnet list in both consoles, or the other site has no route to the subnet.
  • Change a lab host address as the Topology rules say.

Where it lives#

  • terraform/network/unifi/<site>/: networks and their VLAN IDs, WLANs, firewall and gateway BGP config. terraform/network/unifi/folly/home.tf and terraform/network/unifi/offsite/networks.tf hold the Management, iot and offsite Default subnets. terraform/network/unifi/folly/aps.tf declares folly's access points and the AP groups a WLAN is assigned to, from the aps map in clients.yaml.
  • Each console's UI: the Site Magic tunnel, its subnet lists and the console's Let's Encrypt certificate
  • clusters/<site>/config/cluster-topology.json: K8S_NODE_CIDR, LB_RANGE, ROUTER_IP and the other cluster keys. Print them with jq .data <file>.
  • clusters/folly/config/anchor-topology.json: ANCHOR_CIDR, FUTURE_CIDR and the lab host addresses

Edit on GitHub

Linked from 15
  • netbenchAn internal web tool on the folly cluster that runs iperf3 tests from a pod to Kubernetes nodes, NixOS lab hosts outside Kubernetes and the offsite cluster.
  • atomicA Windows desktop at folly on the future network that Prometheus on folly cannot scrape.
  • capsuleA Raspberry Pi 5 on NVMe that is one of the three lab DNS and time anchors, and runs NetBox.
  • cloudpi4A wired Raspberry Pi 4 at folly that is a test DNS resolver and an iperf3 server.
  • forgeA Raspberry Pi 5 on NVMe that is the lab's native arm64 build host.
  • HostsEvery host in the homelab, with its role, site, hardware, OS and status, and how to reach it.
  • radiopi0A Raspberry Pi Zero W with a pHAT BEAT that runs Raspbian and a CloudEvents LED service that is not in git.
  • sporeA Raspberry Pi 5 on NVMe that serves x86 netboot, forge's HTTP boot image, the Talos boot image and Tinkerbell, lab DNS and time, and the SPIRE server.
  • tallboyThe owner's Windows 11 desktop at folly on the future network, which sends metrics and event logs to folly.
  • HomeThe owner's homelab at two sites, folly and offsite, managed as code: its apps, platform, hosts and runbooks.
  • Remote accessThe Tailscale tailnet that reaches both sites' private addresses, and the grants that decide who reaches what.
  • Routing and firewallThe BGP routes and zone firewall that connect the folly and offsite clusters through the UniFi gateways, and folly's node ports and boot network.
  • GlossaryThe lab-specific terms that the wiki pages use, what each means, and the page that defines it.
  • TopologyThe two JSON files that hold the lab's network facts, the format rule they follow, and the Flux, Nix and OpenTofu code that reads them.
  • Inspect the UniFi networkRead the live state of a UniFi console with the read-only unifi.sh helper, and compare it with terraform/network/unifi/.

↑↓ move ↵ open esc close