Policy-based routing is one of the better-documented features in the Linux kernel. ip rule selects a routing table by matching on source address, incoming interface, or firewall mark; ip route table N holds the nexthops for that table. The man pages are clear, the examples work, and a five-line setup will route a subnet out a second interface. What the man pages do not say is that the routing state this mechanism depends on — the rules, the table entries, the fwmark assignments — is ephemeral, and that in production it gets silently wiped by events that have nothing to do with routing. This post is the production side of that story: a fleet of OpenWRT routers using DNS-aware policy-based routing to reach geo-restricted services, and the three outages that taught us the real engineering is not in the rules but in keeping them alive.

The mechanism

The customer infrastructure runs in a German data center. Banking, government, and tax portals in the target country reject connections from foreign source addresses — Cloudflare CAPTCHA loops, WAF 403s, hard geo-blocks. The fix is to route only the geo-restricted traffic through a WireGuard tunnel to a VPS in the target country and let everything else take the normal WAN. Six OpenWRT routers, each with its own wg0 tunnel to the same VPS endpoint in the target country.

The routing chain has four links, and each one is a standard primitive:

  1. dnsmasq resolves a geo-restricted domain and writes the answer IP into an nftables set. The directive is nftset=/.example/4#inet#fw4#geo_routing — every address resolved under the target country’s TLD lands in the geo_routing set in the fw4 table. Individual domains get their own lines: nftset=/tax-portal.example/4#inet#fw4#geo_routing. This is what makes the routing DNS-aware: the policy tracks domain names, not static IP ranges that drift.

  2. nftables marks packets bound for those addresses. A rule in the mangle chain sets fwmark 0x040000 on any packet whose destination is in @geo_routing.

  3. ip rule sends marked packets to a separate table. ip rule add priority 29993 fwmark 0x040000/0xff0000 lookup pbr_wg0.

  4. Table 220 points into the tunnel. default dev wg0 scope link.

The OpenWRT pbr package orchestrates this with resolver_set='dnsmasq.nftset', so dnsmasq populates the set in the background and the policy follows DNS automatically. The whole thing reduces to four invariants, checked in one sweep:

ip rule show | grep 29993                          # 29993: from all fwmark 0x40000/0xff0000 lookup pbr_wg0
grep pbr_wg0 /etc/iproute2/rt_tables              # 220 pbr_wg0
ip route show table 220                            # default dev wg0 scope link
ping -c1 -W2 198.51.100.130                        # 0% packet loss (target resolver)

When all four hold, geo-restricted traffic exits in the target country and everything else exits in Germany. The mechanism took an afternoon to stand up. Keeping it standing took three outages.

The first outage: forwarding died

The first sign was not a routing symptom. An entire LAN lost internet — every client, every site, not just the geo-restricted ones. DNS still resolved. The router itself could ping 8.8.8.8. Clients could ping the router. But no client could reach past it.

The diagnostic that separated the failure cleanly was one sysctl: net.ipv4.ip_forward = 0. The forwarding plane was off. This was not a WAN outage and not a routing-table problem; it was the kernel not forwarding at all.

The cause was a split-brain state in the pbr service: pbr.config.enabled='0' but pbr.config.strict_enforcement='1', with the init script still boot-enabled. The pbr init path called stop_forward() during startup and disabled ip_forward before checking whether PBR was actually in use. With strict enforcement on but the service disabled, a boot or reload could leave the router stranded with forwarding off and no path to re-enable it. The boot log said it plainly: user.notice pbr: Forwarding is disabled.

The lesson that survived: on OpenWRT with pbr, strict_enforcement='1' is a footgun when the service is left UCI-disabled but boot-enabled. The production baseline is now enabled='1' with strict_enforcement='0' on every router — PBR stays on, but it never reaches into the forwarding sysctl to enforce its idea of correctness. And for the symptom pattern “DNS works, router reachable, Internet dead,” sysctl net.ipv4.ip_forward is the first check, not the last. It separates a forwarding-plane failure from an upstream WAN failure in one command.

The second outage: the upgrade that wiped the rules

Eight days later, four of six routers lost geo-restricted routing simultaneously. General internet was fine. The WireGuard tunnels were up — recent handshakes, healthy endpoints. But ip rule show | grep 29993 returned nothing, and pbr_wg0 was gone from rt_tables. Geo-restricted traffic was still being marked in nftables, but with no rule to send it to table 220, it fell back to the main table and exited in Germany. The geo-restricted services saw a German IP and blocked it.

The cause was an unattended opkg upgrade cron that had restarted the pbr package mid-uptime. The upgrade stopped and restarted the pbr service, which cleared the kernel routing state — rule 29993 and the table 220 entry. Under normal conditions, a WireGuard interface coming up fires an ifup hotplug event, and the hotplug handler re-applies the rule. But the interfaces were already up when the upgrade ran. No interface went down and back; no ifup fired. The two safety mechanisms that existed — a hotplug handler and a boot-time repair service — were both keyed to events that did not happen. The state was wiped by a package upgrade, and nothing in the chain was listening for that.

This is the failure class the man pages do not prepare you for. The routing configuration was correct. The tunnels were healthy. The mechanism was sound. The state was gone because an unrelated package manager restarted a service, and the restoration logic was coupled to the wrong trigger.

The third: one character

Two months later, a single customer site lost DNS resolution for one service. The root cause was a one-character typo in geo-routing.conf: geo_rou instead of geo_routing on line 38. dnsmasq logged the broken nftset reference continuously, and the PBR policy for tax-portal.example silently stopped working. A conditional DNS forwarder bypassed a separately-failing upstream resolver, and the typo was corrected with a single sed. Eight minutes from ticket to verified fix.

The follow-up note that mattered: the geo-routing.conf file is managed by a centralized management template. An on-device fix can be overwritten by the control plane within two minutes. The template had to be corrected too, or the typo would have reappeared on the next sync. Configuration-as-code only protects you if the source of truth is the thing you fix.

The safety net

The three-layer safety net that re-asserts canonical PBR routing state after reboots, service restarts, and mid-uptime state loss

Three outages, three different triggers, one shared lesson: the canonical routing state has to be treated as ephemeral and re-asserted, not configured once and trusted. The production fix is three complementary layers, each keyed to a different event class:

  • Layer A — hotplug handler (99-geo-routing-rule): fires on ACTION=ifup for wg0. Covers reboots and any interface cycle.
  • Layer B — init.d service (geo-routing-fix): applies the canonical state immediately on start() and again at 70 seconds to survive the multi-WireGuard boot storm. Covers service restarts — including the pbr upgrade path that caused the second outage.
  • Layer C — cron watchdog (geo-routing-watchdog, every 5 minutes): checks whether wg0 is up and rule 29993 is missing, and re-applies the state if so. Covers any mid-uptime state loss that A and B miss. Cron survives every package upgrade; it is the one trigger that cannot be cleared by restarting pbr.

The design principle is that no single layer is trusted. A covers reboots but not upgrades. B covers upgrades but not arbitrary state loss. C covers everything but runs on a five-minute interval, so it is the backstop, not the front line. The layers are independent, idempotent, and cheap — the watchdog is a shell script that grep-checks ip rule and exits early when the state is healthy.

The general lesson

Policy-based routing is a kernel mechanism, and the kernel mechanism is reliable. What is not reliable is the userspace state around it — the service that owns the rules, the package manager that restarts that service, the config files that reference nftables sets by name. Every one of those is a separate process with its own lifecycle, and none of them coordinates with the routing plane.

The production posture that follows is to stop treating routing state as configuration and start treating it as a desired-state contract that is continuously reconciled. Write the canonical state down in one place. Check it from multiple independent triggers. Make the check idempotent and cheap so it can run every five minutes without cost. And when an outage happens, the first question is not “what changed in the routing config” but “what event cleared the state, and which layer should have caught it.”

The rules themselves took an afternoon. The safety net took three outages. That ratio is the part worth remembering — and the posture we carry into continuous-improvement engagements.