(Related to FRS12067)

I wanted to do another investigation of the rate of connection errors recently, and how they compare to before and after h1guardian1 was upgraded to a 40core machine [alog46464]. This is just a quick look at the connection errors in ISC_LOCK from the last tow months. A further study will follow at some other time.

In the past I had determined that the connection errors had severely decreased post machine upgrade, and that ISC_LOCK was the biggest offender for connection errors (sorry no link at the moment). This is also seen by simply trending the CONNECT channel for ISC_LOCK (1st attachment). You can see that after the machine upgrade on January 16, 2019, the rate was reduced and after the start of O3 it was brought down even further. The problem is that they are not completely gone, so I looked at the last 2 months of connection errors from ISC_LOCK to determine where/when these happen.

Out of the 20 connection error instances, 11 were true connection errors. Most of these are from restarts on maintenance days, the others are events the lightning strike, dolphin crash, or similar.

The other 9 all occur in the state CHECK_MICH_FRINGES (48). It is always at the same line:

        ezca.get_LIGOFilter('SUS-BS_M3_LOCK_L').only_on('INPUT','FM1', 'FM2','OUTPUT', 'DECIMATION')

There is nothing special about this, these only_on LIGOFilter calls are all over ISC_LOCK. Yesterday I toggled that filter bank to see if for some very odd reason, it was the bank causing trouble. Nothing abnormal. I then ran this state manually a few times yesterday and during the lock reacquisition, and I still didn't see anything (no connection errors or abnormal machine loads).

These sparse connection errors don't seem to cause any trouble, and they are very brief (see log attachment 2). I will continue to monitor these to see if their rate increases or if they become problematic.