A wrong DHCP gateway quietly broke a k8s cluster for 48 days

The firewall had its WAN default route pointing to the wrong address, 10.20.0.3. That address was alive, but anything that needed the internet got a “Destination Net Unreachable” response. The real working gateway was 10.20.0.1.

The routing table showed 0.0.0.0 going through 10.20.0.3. The cause was DHCP on the 10.20.0.0/24 network handing out 10.20.0.3 as the router option, likely from a stale or wrongly configured DHCP server. learned that value at boot and used it for 48 days because no workload exposed the failure clearly.

The fix was to edit WAN_DHCP in under System > Gateways > and statically override the gateway IP to 10.20.0.1.

Key points

  • was using 10.20.0.3 as the default gateway instead of the real gateway, 10.20.0.1.
  • 10.20.0.3 replied on the network but could not route traffic to the internet.
  • DHCP gave the wrong router option on boot, and kept using it for 48 days.
  • The fix was to override the WAN_DHCP gateway IP in and set it to 10.20.0.1.
  • checks should include routing table and gateway checks, not just server health.
Read original