Jim (on phone), Nutsinee
I noticed the BRSY fault message so I digged around and found BRS_ETMY_RX_INMON and BRS_ETMY_VEL to be flat -- so I called Jim. As Jim requested I attached spectum of the subtracted and the unsubtracted BRSY signal, before and after it went fault. The two signals are exact same after the BRS went to fault. I also attached timeseries of bit channels (T1600103 -- see Status Bit troubleshooting). I couldn't find DRIFT BIT so I attached DRIFTMON instead. CBIT and DRIFTMON went bad/flat around 05:27 UTC.
According to Jim BRSY is not feeding anything (flat signal?) to the ISI right now. End Y is currently using BLEND_Quiet_250_SC_BRS. As long as the wind is low this configuration should be okay.
As per the "Troubleshooting guide" mentioned above, the CBIT being down indicates that the C# code crashed. This code reads the CCD camera and calculates the BRS angle. It is the first time this code has crashed at EY. Odd...
To restart, one should remote login to BRS-Y beckhoff computer as described in the guide above, then close the current BRS2 C# code (labelled as such) and the BRSY EPICS script. Then follow the startup procedure from Step 9 onwards. If this doesn't work, there may be a hardware problem, which will need more steps to diagnose.
At 11:12 UTC the IFO lost lock. I logged into h1brsey and double clicked on the 'BRS C#' shortcut on the desktop. The program started and BRSY is no longer reporting a fault.
I've added a test to DIAG_MAIN to catch this (I think we used to have something similar when we just had one BRS?). Before DIAG_MAIN just looked for the BRS guardian nodes to go into fault, and would just report the node was in fault. I've now inluded an explicit check for the CBIT, and DIAG_MAIN will now tell you the C code has crashed.
def BRS_CHECK():
"""Check the two end station Beam Roation Sensors to make sure that
they are not in FAULT (as read from their Guardian status node state.
Also checks the status of the CBIT, to see if C code is live
"""
for end in ['X','Y']:
if ezca['ISI-GND_BRS_ETM{}_CBIT'.format(end)] == 0:
yield 'BRS {} C code has stopped'.format(end)
elif ezca['GRD-BRS{}_STAT_STATE'.format(end)] == 'FAULT':
yield 'BRS {} is in FAULT'.format(end)