Priority: Description: Our VPLS Supplier is performing maintenance on the links to Frankfurt.
VLAN Services Frankfurt <=> Zurich
we expect to have about 30 minutes of less peerings in Frankfurt, which can take a few minutes for convergence each time.
Assuming times in Local Time:
Date of work: 23-SEP-2020
Start time: 00:00 CEST
End time: 05:00 CEST
Place of work: Frankfurt
Planned Work: EU6311
2020091562000598 (2020-09-15 08:24:19): stale route announcement causing blackholing
Priority: Description: This night, our supplier zurich basel seems to have technical issues again - our line was lost. OSPF promptly re-routed traffic.
Monitoring saw everything as green, so no further action was taken.
as it seems, one of our routers still has a static route:
show ip bgp 18.104.22.168/32
BGP routing table entry for 22.214.171.124/32
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to non peer-group peers:
126.96.36.199 from 0.0.0.0 (188.8.131.52)
Origin incomplete, metric 30, localpref 200, weight 32768, valid, sourced, best
Last update: Fri Sep 4 10:32:17 2020
this route shows towards the wrong direction though, causing a routing loop:
The router that is in this loop, is connected to swissIX. Our monitoring is on a fiber7 line and not over swissIX; therefore from the point of view of the monitoring nothing was wrong, and everything still stayed up.
But traffic via SwissIX seems to have looped.
Take-Away from this outtage: monitor the "ususal suspect" also from within the network outbound, and not just latency measurements towards them (e.g. 184.108.40.206, 220.127.116.11 as well as 18.104.22.168 were all behind swissiX)