10:31utc (2:31am) Begin receiving verbal alarm: "Timing system error: SYS-TIMING_Y_FO_A_ERROR_FLAG" (also "...Y_PPS_A_ERROR_FLAG")
On the CDS Overview, the Slow Controls area has been flashing RED for the Corner Station with errors coming up for C1_PLC1 & C1_PLC2 (see attached for error with both of them showing error at same time).
H1 is currently locked and OBSERVING. Not sure what if this warrants waking David up (maybe he already has texts about this alarm/issue?). Since it's the middle of the night, I'm tempted to wait since it is intermittant, and we are locked.)
I'll search through alogs to see if this issue has come up before.
Perhaps not related, but at 10:29, DIAG_MAIN has been giving the message: "ALS: Y laser oscillating"
Timing Error for Y
Now have noticed Timing Error for "Y" (screenshot shows this new red box on CDS Overview & windows associated with this).
Will give Dave a call/voicemail.
Corner station Beckhoff PLC1 and PLC2 errors show a comms error with EY (see attached). So both the Beckhoff and Timing errors suggest fiber connection issues between corner and EY.
It looks like the fiber link between the slow controls Beckhoff chassis in the corner and EY has failed. CS-PLC1 reports failure in both directions to EY, CS-PLC2 has receive error but no transmit error.
If there was a true timing system failure at EY in that the timing master in the MSR has lost connection to the timing fanout at EY then the IOP models would skew and stop driving their DAC cards. We are not seeing this, so it looks like a Beckhoff reporting problem.
Opened FRS13929
At this point in time, recommend waiting for Richard to get on site to investigate the EY fiber comms.
Fiber bus coupler EK1501-0010 replaced system working again.