Reports until 22:23, Thursday 04 May 2023
H1 CDS
jonathan.hanks@LIGO.ORG - posted 22:23, Thursday 04 May 2023 (69337)
WP11133 Network outage
We attempted to move to the SRX router again today.  We took an outage from about 6:20pm to 10:00pm localtime (1am-5am UTC).  We were able to have some ESnet engineers and a Juniper support engineer working with us.  Unfortunately we were unable to get bits flowing through the SRX.  Neither us, ESnet, nor the vendor understand why the packets are not flowing.

We ruled out issues with the fiber, optics.  We rewrote, removed, disabled firewall rules.  We reworked the routing setup and policies several times.

We could send packages but not receive them, a symptom of that was we were never able to get an arp entry for ESnet.  ESnet received packets from us and had a correct apr entry.  We also tried set a static arp rule on our end, but that did not help.