Install Windows monitoring

Use this runbook to make a Windows desktop report to folly's Prometheus and VictoriaLogs, to update its agents, or to find why it stopped reporting. apps/lookout/windows/Install-Monitoring.ps1 installs three agents from downloads pinned by SHA256. Lookout is the app that keeps them installed on a desktop that runs it, such as tallboy; Install Lookout adds it to a desktop.

Before you start#

• You need an administrator account on the desktop with PowerShell 7 and git, from Install a Windows desktop.
• You need a full checkout on a Linux machine, such as the WSL distro, and SSH access to a lab host, such as spore.
• The desktop must be on a network in the Internal firewall zone, such as Management or future.

<host> is the desktop name. <HOST> is <host> in upper case, and <HOST>_IP is its key in clusters/folly/config/anchor-topology.json.

Add a desktop to Prometheus#

1. Under desktops in terraform/network/unifi/folly/clients.yaml, add <host> with its mac and its ip, the host part of its address.

2. Add <HOST>_IP to clusters/folly/config/anchor-topology.json.

3. Add <host> to local.windows_hosts in terraform/network/unifi/folly/windows-hosts.tf. If the desktop is on future, copy the tallboy entry. If it is on Management, copy the atomic entry.

4. Add an endpoint with ${<HOST>_IP} and nodeName: <host> to the EndpointSlice in clusters/folly/monitoring/windows-exporters.yaml.

5. Apply the change through a pull request. See Apply an OpenTofu change.

   Result: The Atlantis plan creates unifi_client.windows_hosts["<host>"].

Install or update the agents#

1. On the desktop, open PowerShell 7 as administrator.

2. Check out the installer, or update its checkout. The checkout holds only apps/lookout/windows.

   $infra = "$HOME\src\github.com\jonpulsifer\infra"
   if (-not (Test-Path $infra)) { git clone --filter=blob:none --sparse https://github.com/jonpulsifer/infra $infra }
   cd $infra
   git sparse-checkout set apps/lookout/windows
   git pull --ff-only

   Result: apps\lookout\windows\Install-Monitoring.ps1 exists.

3. Run the installer. To leave out the event logs, add -SkipVector.

   .\apps\lookout\windows\Install-Monitoring.ps1

   Result: The last line is running Event Log shipping to VictoriaLogs, or ==> Vector skipped with -SkipVector.

Change an agent version#

1. On a Linux machine, get the SHA256 of the new release file, <url>.

   curl -sL <url> | sha256sum

   Result: The hash, then -.

2. In Install-Monitoring.ps1, change the <Agent>Version, <Agent>Url and <Agent>Sha256 parameters together. <Agent> is Exporter, Ohm or Vector.

3. Merge the change.

4. Do Install or update the agents on each desktop.

Install Lookout#

Lookout replaces step 4 of Install or update the agents on a desktop that runs it.

1. Download Lookout.msi from the latest lookout-v<version> release of the repository, or from the lookout-msi artifact of a lookout workflow run.

2. On the desktop, run the installer. Approve the administrator prompt. If SmartScreen stops it, choose More info, then Run anyway.

   msiexec /i .\Lookout.msi

   Result: Lookout is listed in Settings under Apps, and the service runs.

   Get-Service Lookout

3. Open Lookout from the Start menu once. It starts by itself at the next logon.

   Result: The tray icon appears, and turns green once the three agents are installed.

4. Do Check a desktop. The agent's own metrics are there too.

   ssh spore.lolwtf.ca "curl -s http://<ip>:9182/metrics" | grep '^lookout_'

   Result: lookout_last_sync_success 1 and lookout_component_current{component="..."} 1 for each of windows_exporter, ohmgraphite and vector.

Check a desktop#

1. In the full checkout, read the desktop address, <ip>.

   jq -r .data.<HOST>_IP clusters/folly/config/anchor-topology.json

   Result: The desktop address.

2. Read the status of windows_exporter from a lab host.

   ssh spore.lolwtf.ca "curl -s -o /dev/null -w '%{http_code}\n' http://<ip>:9182/metrics"

   Result: 200, or 000 if the desktop is off.

3. Read the collector status.

   ssh spore.lolwtf.ca "curl -s http://<ip>:9182/metrics" | grep '^windows_exporter_collector_success'

   Result: Each collector has the value 1.

4. Count the OhmGraphite metrics.

   ssh spore.lolwtf.ca "curl -s http://<ip>:4445/metrics" | grep -c '^ohm_'

   Result: A number larger than 0.

5. Read the Prometheus targets of <host>.

   curl -s -G https://prom.lolwtf.ca/api/v1/query --data-urlencode 'query=up{job="windows-exporter",instance="<host>"}' \
     | jq -r '.data.result[] | "\(.metric.endpoint) \(.value[1])"'

   Result: http-metrics 1 and sensors 1.

6. In Grafana, query the VictoriaLogs data source with {job="windows-eventlog", host="<host>"}.

   Result: The event log lines of <host>.

Check Memory Integrity#

Memory Integrity and the vulnerable driver blocklist are Windows security settings. When either one is on, Windows blocks the LibreHardwareMonitor driver. The desktop then has no CPU temperature or fan speed, so WindowsCpuTempHigh and WindowsFanStopped cannot fire. If only those sensors are missing, do this procedure.

1. On the desktop, open PowerShell 7.

2. Read the security services that run.

   (Get-CimInstance -Namespace root\Microsoft\Windows\DeviceGuard -ClassName Win32_DeviceGuard).SecurityServicesRunning

   Result: The list includes 2 if Memory Integrity is on.

3. Read the vulnerable driver blocklist setting.

   Get-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\CI\Config -Name VulnerableDriverBlocklistEnable

   Result: VulnerableDriverBlocklistEnable is 1 if the blocklist is on.

4. If either setting is on, accept the missing sensors.

If something goes wrong#

Related#

• Observability
• Lookout
• Install a Windows desktop
• tallboy and atomic