Openfactory Status

Priority:
Description: Our VPLS Supplier is performing maintenance on the links to Frankfurt.

This Affects

VLAN Services Frankfurt <=> Zurich
- Peerings

we expect to have about 30 minutes of less peerings in Frankfurt, which can take a few minutes for convergence each time.


Assuming times in Local Time:

---
Date of work: 23-SEP-2020
Start time: 00:00 CEST
End time: 05:00 CEST
Place of work: Frankfurt
Planned Work: EU6311

---

Priority:
Description: This night, our supplier zurich basel seems to have technical issues again - our line was lost. OSPF promptly re-routed traffic.

Monitoring saw everything as green, so no further action was taken.


as it seems, one of our routers still has a static route:
show ip bgp 5.226.149.251/32
BGP routing table entry for 5.226.149.251/32
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to non peer-group peers:
5.226.148.84 5.226.150.171
Local
5.226.149.199 from 0.0.0.0 (5.226.149.233)
Origin incomplete, metric 30, localpref 200, weight 32768, valid, sourced, best
Last update: Fri Sep 4 10:32:17 2020


this route shows towards the wrong direction though, causing a routing loop:

12. (waiting for reply)
13. gw-as58299.init7.net 0.0% 4 46.3 50.2 44.0 55.6 6.0
14. (waiting for reply)
15. gw-as58299.init7.net 0.0% 4 47.7 48.5 46.2 50.7 2.0
16. eth8.r3-core-ixn.ch.as58299.net 33.3% 4 55.2 52.9 50.6 55.2 3.2
17. gw-as58299.init7.net 0.0% 4 65.6 57.6 50.9 65.6 6.3
18. (waiting for reply)
19. gw-as58299.init7.net 0.0% 4 54.6 53.4 51.9 54.6 1.3
20. (waiting for reply)
21. gw-as58299.init7.net 0.0% 4 52.8 53.5 50.1 58.4 3.5
22. (waiting for reply)
23. gw-as58299.init7.net 0.0% 4 63.4 60.4 54.7 63.4 3.9
24. (waiting for reply)
25. gw-as58299.init7.net 0.0% 4 79.6 65.3 51.9 79.6 11.8
26. eth8.r3-core-ixn.ch.as58299.net 66.7% 4 54.8 54.8 54.8 54.8 0.0
27. gw-as58299.init7.net 0.0% 3 54.6 58.8 54.6 64.7 5.3(


The router that is in this loop, is connected to swissIX. Our monitoring is on a fiber7 line and not over swissIX; therefore from the point of view of the monitoring nothing was wrong, and everything still stayed up.
But traffic via SwissIX seems to have looped.


Take-Away from this outtage: monitor the "ususal suspect" also from within the network outbound, and not just latency measurements towards them (e.g. 1.1.1.1, 9.9.9.9 as well as 8.8.8.8 were all behind swissiX)