Multihoming with BGP is the point where enterprise networking stops being configuration and starts being policy. The protocol is simple; the failure modes are political, and they usually surface at 3am during a partial upstream failure rather than a clean one.
This is a working design for a dual-homed enterprise edge, with the specific guards that keep a mistake from becoming an outage.
What you need before you start
Three things, and the lead time on them is measured in weeks:
- An ASN. Your regional registry issues these. Private ASNs (64512–65534, plus 4200000000–4294967294) work if both upstreams agree to strip them, but a public ASN is the correct answer for anything permanent.
- Provider-independent address space. A /24 IPv4 minimum — anything longer is filtered across most of the internet — and a /48 for IPv6.
- A Route Origin Authorisation for each prefix, published in the RPKI. Major transit providers now drop RPKI-invalid routes. Without ROAs your announcement is not merely unprotected; increasingly it is unreachable.
Create ROAs before you announce. Announcing a prefix with no ROA is "unknown" and generally accepted. Announcing it with a ROA that specifies the wrong origin ASN or a mismatched max-length is "invalid" and dropped. A wrong ROA is worse than no ROA.
Topology
Two edge routers, one transit each, an iBGP session between them, and both feeding the core over a first-hop redundancy protocol. Nothing exotic — and deliberately so, because the complexity in this design lives in the policy, not the topology.
Session configuration
Cisco IOS-XE for the transit session:
router bgp 65001
bgp router-id 10.0.0.1
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 203.0.113.1 remote-as 64500
neighbor 203.0.113.1 description Transit-A
neighbor 203.0.113.1 password 7 <redacted>
neighbor 203.0.113.1 timers 10 30
neighbor 10.0.0.2 remote-as 65001
neighbor 10.0.0.2 description iBGP-to-edge-rtr-02
neighbor 10.0.0.2 update-source Loopback0
address-family ipv4
network 198.51.100.0 mask 255.255.255.0
neighbor 203.0.113.1 activate
neighbor 203.0.113.1 route-map TRANSIT-A-IN in
neighbor 203.0.113.1 route-map ANNOUNCE-OUT out
neighbor 203.0.113.1 maximum-prefix 1000000 90 restart 15
neighbor 203.0.113.1 send-community both
neighbor 10.0.0.2 activate
neighbor 10.0.0.2 next-hop-self
exit-address-family
Several of those lines are load-bearing.
no bgp default ipv4-unicast stops IOS activating every neighbour into the IPv4 table automatically. Without it, a neighbour you configure for IPv6 or for a VPN address family silently starts exchanging IPv4 routes.
maximum-prefix is your circuit breaker. If an upstream leaks the full table plus their customers' deaggregated noise, you want the session to drop rather than your router to exhaust memory and reload. The restart 15 reopens the session after fifteen minutes.
timers 10 30 detects failure in 30 seconds instead of the 180-second default. Pair it with BFD if the provider supports it — sub-second detection, and considerably kinder to the CPU than aggressive BGP timers alone:
interface GigabitEthernet0/0/1
bfd interval 300 min_rx 300 multiplier 3
router bgp 65001
neighbor 203.0.113.1 fall-over bfd
Outbound policy: announce only what is yours
This is the prefix filter that keeps you off the industry mailing lists.
ip prefix-list OUR-PREFIXES seq 10 permit 198.51.100.0/24
route-map ANNOUNCE-OUT permit 10
match ip address prefix-list OUR-PREFIXES
set community 64500:100
route-map ANNOUNCE-OUT deny 20
The explicit deny 20 at the end is the important part. A route-map has an implicit deny, but writing it makes the intent reviewable — and stops the next engineer from appending a permit clause without noticing what it bypasses.
Never redistribute an IGP into BGP on an edge router. Every large-scale route leak in the last decade traces back to a variation of this. If you need a prefix announced, use a network statement naming it explicitly.
Inbound policy: sanitise everything
ip prefix-list MARTIANS seq 10 permit 0.0.0.0/8 le 32
ip prefix-list MARTIANS seq 20 permit 10.0.0.0/8 le 32
ip prefix-list MARTIANS seq 30 permit 127.0.0.0/8 le 32
ip prefix-list MARTIANS seq 40 permit 169.254.0.0/16 le 32
ip prefix-list MARTIANS seq 50 permit 172.16.0.0/12 le 32
ip prefix-list MARTIANS seq 60 permit 192.168.0.0/16 le 32
ip prefix-list MARTIANS seq 70 permit 198.51.100.0/24 le 32
ip prefix-list MARTIANS seq 80 permit 224.0.0.0/4 le 32
ip prefix-list MARTIANS seq 90 permit 0.0.0.0/0 ge 25
route-map TRANSIT-A-IN deny 10
match ip address prefix-list MARTIANS
route-map TRANSIT-A-IN permit 20
set local-preference 200
set community 65001:1000
Sequence 70 rejects your own prefix coming back at you. Sequence 90 rejects anything longer than a /24, which is the de facto minimum for global routing.
Traffic engineering
Outbound is easy and entirely under your control. Local preference decides:
route-map TRANSIT-A-IN permit 20
set local-preference 200 ! preferred
route-map TRANSIT-B-IN permit 20
set local-preference 150 ! backup
Higher wins, and it is evaluated before AS path length, so it overrides everything a provider does upstream.
Inbound is where multihoming gets genuinely difficult, because you are asking the rest of the internet to prefer one of your links. You have three blunt instruments.
AS path prepending — pad the path on the link you want less used:
route-map ANNOUNCE-B-OUT permit 10
match ip address prefix-list OUR-PREFIXES
set as-path prepend 65001 65001
Prepending is coarse. One prepend often does nothing; three sometimes shifts everything. It is a suggestion, not a control.
Communities — the precise instrument, when the provider supports them. Most publish a list:
! Transit B: 64510:120 = set local-pref 120 within their network
route-map ANNOUNCE-B-OUT permit 10
set community 64510:120
More-specific announcements — announce the aggregate on both links and a /25 on the preferred one. Effective, and it costs the global table an extra entry. Use sparingly; some networks filter it anyway.
Inbound engineering is iterative. Change one thing, wait a full business day, then measure with real traffic data — not looking-glass output from three vantage points. Convergence and caching mean the effect of a change is not visible for hours.
RPKI validation
Validate what you receive as well as signing what you announce:
router bgp 65001
bgp rpki server tcp 10.0.0.50 port 3323 refresh 600
address-family ipv4
bgp bestpath prefix-validate allow-invalid
exit-address-family
route-map TRANSIT-A-IN deny 5
match rpki invalid
Run a local validator — Routinator and rpki-client are both solid — rather than trusting an upstream's validation state. Start in logging mode, review what would be dropped for a fortnight, then enforce. There are still real, reachable networks with broken ROAs, and you want to know which of them your users depend on before you drop them.
Verification
After every change:
show bgp ipv4 unicast summary
show bgp ipv4 unicast neighbors 203.0.113.1 advertised-routes
show bgp ipv4 unicast neighbors 203.0.113.1 routes
show bgp ipv4 unicast 198.51.100.0/24
show bgp rpki table
advertised-routes is the one that matters. It answers "what am I actually telling the internet?" — the question behind nearly every serious BGP incident. Check it after every policy change, without exception.
Externally, confirm what the world sees: RIPE RIS, bgp.he.net, and your providers' looking glasses. Set up alerting through BGPalerter or a monitoring service so that an unexpected origin announcing your prefix pages someone.
Failure testing
Do this in a maintenance window, before you depend on it:
- Shut the Transit A session.
neighbor 203.0.113.1 shutdown. Traffic should move to B within your BFD/timer window. Measure it. - Fail the physical link. Unplug it. This tests link-down detection, which is a different path than a clean session teardown.
- Fail edge-rtr-01 entirely. Power it off. This tests the FHRP failover and the iBGP topology together.
- Simulate a partial failure. Leave the session up but blackhole traffic beyond the provider's first hop. This is the nasty one — BGP stays up, traffic disappears. Nothing in the base configuration catches it, which is exactly why you need external reachability monitoring rather than device-level monitoring alone.
That fourth scenario is the one that produces long outages. BGP tells you the session is healthy; it says nothing about whether packets arrive. Monitor from outside your network, always.
1 comment
Sign in to join the discussion.
External reachability monitoring from three regions has been worth every penny since.