Sign in →
CloudLab Works emblem: Waku the orca ringed by CloudLab and WorksAgenticCLOUDLAB WORKScrossed whale bones, one end a wrenchCloud City, headwaters of the agentic cloud revolution!
Waku, the orca wearing a headset Ask Me

Invisible Wires: Agentic Cloud

When infrastructure as code can't reach the router, build the drift check instead

· By Alex Alvord

Every Monday at 06:15 a job checks the router against what it should be. Here is the last result, and what a change looks like when it happens.

 MONDAY 06:15  UniFi drift check, last run

   OK no drift (11 reservations, 1 LANs)

 What a change looks like (output from the test suite, not a real event)

   DRIFT 2 difference(s)
    - reservation changed: aa:bb:cc:dd:ee:01 (cka-w1) 192.168.1.47 -> 192.168.1.250
    - LAN Default dns: ['192.168.1.75', '9.9.9.9'] -> ['9.9.9.9']

Nothing in those settings, the fixed-IP reservations and the DHCP DNS servers, can change for a week without this check noticing. That check exists because the obvious tool, OpenTofu, would not run against the router, and the failure is worth walking through.

What failed

One OpenTofu plan ended with a single line: "Unable to Connect to UniFi Controller". Every other check said the controller was fine.

The plan was meant to bring the router's fixed-IP reservations and DHCP DNS setting under code, next to the Proxmox and Kubernetes pieces already there. The provider reaches the router through the vendor's cloud connector, with an API key. The key never touches the agent that runs the job. It travels through a local proxy that re-signs TLS and injects the secret on the way out, so the secret is never in a file, a log or a prompt.

 BEFORE: the provider could not use the working path

   tofu plan
      |
      v
 +----------------+  HTTPS  +--------------------+  HTTPS  +-----------+
 | UniFi provider |-------->| local egress proxy |-------->| vendor    |
 | (a Go program) |         | adds the API key,  |         | cloud     |
 +----------------+         | re-signs TLS       |         | connector |
      ^                     +--------+-----------+         +-----+-----+
      |                              |                           |
      | "certificate is not trusted" | cert signed by            v
      +------------------------------+ the proxy's authority  +--------+
                                                              | router |
   curl, same proxy, same URL:  HTTP 404 (a normal answer)    +--------+
   Go,   same proxy, same URL:  x509 error, SSL_CERT_FILE ignored on macOS

A plain curl through that proxy worked. The Go-based provider did not. On macOS, Go programs read the system trust store and ignore the SSL_CERT_FILE variable that curl honors. We reproduced it without the provider: a ten-line Go program fetching the vendor's API host through the same proxy failed with "certificate is not trusted", with SSL_CERT_FILE set or not, while curl on the same URL returned a normal HTTP answer. The provider's insecure-skip-verify option is ignored when the cloud connector is on, so there was no flag to flip.

The three ways out

OptionWhat it costsVerdict
Add the proxy's authority to the macOS trust storeEvery program on the machine now trusts a certificate authority that can re-sign any siteNo. It weakens the control the proxy exists to provide
Run the plan from a Linux host with its own egressA second place that needs the API key, which is the thing the proxy avoidsLater, if the router config grows
Keep the router out of OpenTofuNo plan, no apply, no state for the routerYes, with one addition below

The blind spot, and the fix

Leaving the router out of OpenTofu has a real cost. Nothing declares what the router should look like, and nothing notices when someone changes it in the app. Infrastructure as code gives two things: a way to change things, and a way to see that they changed. We could do without the first for a short list of reservations (11 today) and one DNS setting. We could not do without the second.

So we built only the second, on the path that already worked.

 AFTER: a read-only check on the path that already worked

   Monday 06:15, scheduler
      |
      | 1  grant the worker one script, for this run only
      v
 +------------+  2 run  +--------------+  HTTPS  +--------------+  +--------+
 | restricted |-------->| check.py     |-------->| egress proxy |->| vendor |
 | worker     |         | curl, GET    |<--------| key + TLS    |<-| cloud  |
 +------------+         | only         |         +--------------+  +---+----+
      ^                 +------+-------+                               |
      |                        | 3  compare live vs expected           v
      | 4  revoke the grant    v                                   +--------+
      |               +-----------------+                          | router |
      |               | expected.json   |  match  -> say nothing   +--------+
      +---------------+ (committed)     |  differ -> list it, ask:
                      +-----------------+            revert or record?

The weekly job:

  1. Reads the router's fixed-IP reservations and LAN DHCP settings through curl, using GET requests only.
  2. Compares them with a committed file of the expected state.
  3. Says nothing when they match. When they differ, it lists each difference and asks whether to revert the change or record it.

The worker that runs it gets permission for that one script, for that one run, and loses it afterward. The script only reads. Changes still go through the reviewed curl jobs, and the expected file changes in the same commit as the change itself.

What we kept and what we gave up

 WHAT OPENTOFU DOES FOR YOU          THE ROUTER, WITHOUT IT

 +---------------------------+       +---------------------------+
 | 1  change things safely   | ----> | reviewed curl jobs        |  kept
 |    (plan, then apply)     |       +---------------------------+
 +---------------------------+
 | 2  notice when they change| ----> | weekly drift check        |  BUILT
 |    (drift)                |       +---------------------------+
 +---------------------------+
 | 3  record the intended    | ----> | expected.json in git,     |  kept
 |    state                  |       | updated with each change  |
 +---------------------------+       +---------------------------+
 | 4  import / diff state    | ----> | not replaced               |  GIVEN UP
 +---------------------------+       +---------------------------+

The comparison logic has four offline tests: identical state stays silent, a changed address is reported, a removed and an added reservation are both reported, and a changed DNS server is reported. The first live run read 11 reservations and 1 LAN and reported no drift against a baseline taken from the same router.

We gave up the plan and apply workflow for the router and any importable state. If the router configuration grows past a handful of settings, the Linux-host option comes back, with the key handling designed for it. Until then a read-only check is the cheaper piece of OpenTofu to keep.

The general point: when a tool can't reach a system, ask which of its jobs you actually need. Here the job was "tell me when this changes", and that does not require the tool that failed.

Comments

  1. Loading comments…

Comments are read by Alex before they appear. No email address needed; your name shows as you type it. See privacy.

← All posts

Ask Waku