david.barker@LIGO.ORG - posted 11:17, Friday 03 November 2023 - last comment - 18:23, Friday 03 November 2023(73956)
Possible timing system cause of Thu night lock-loss + IPC errors + DAQ crashes
Daniel, Erik, Dave, Fil:
Erik discovered that the DTS DAQ also crashed at the same times last night, indicating a site wide timing issue could be the cause.
Daniel just found that the MSR timing server is reporting GPS issues. We are investigating and checking the spares status.
I've opened FRS29567 for this issue
Daniel, Fil, Marc, Erik, Dave:
Daniel has taken a closer look at the timing master issues and he says it appears more likely that the timing master will need to be replaced rather than an issue with the roof antenna or its cabling.
The attached 24 hour trend shows the timing master's GPS error count (blue) GPS locked status (orange) and number of tracked satellites (green).
The error burst on the left is 19:00 - 22:00 Thu night, which caused the two crashes. The smaller middle burst is 03:00 - 04:00 this morning. The larger right hand burst is 07:00 - 08:00 which was close to causing a crash.
Daniel further points out that the number of satellites remains good through out, and is sometimes too big (>60).
This all points to an internal error in the timing master and not an antenna issue.
We are readying the spare master chassis for a swap out.
In preparation for another potential timing error this weekend, I have pushed the current pitch/yaw offsets to SDF for the following suspensions: ETMX/Y, TMSX/Y, BS, PRM, SRM, SR2/3, PR2/3, MC1/2/3, IM1/4.
Daniel, Erik, Dave:
For a sanity check we verified that the DAQ's GPS time is correct.
Currently this is a hand calculation, Erik is writing code to automate it.
I checked two times:
14:00:00 Fri 03 Nov 2023 PDT (just a few minutes ago)
21:10:00 Thu 02 Nov 2023 PDT (between the first and second crash last night)
Decoding the CNS-II GPS receivers' IRIG-B channels recorded in the DAQ as H1:CAL-PCALX_IRIGB_DQ, H1:CAL-PCALY_IRIGB_DQ I get the UTC times:
21:00:18
04:10:18
PDT/UTC diff is +7hr. The GPS/UTC leap second diff is currently 18 seconds, so the DAQ timing is correct.
Austin's having pushed the slider values to the suspensions was extremely helpful in our recovery later in the afternoon. For the future, I suggest we also (a) Offload the IMC WFS (which I had forgotten to do on Tuesday, and forgetting caused a lot of difficulty) before doing this, and then (b) including *every* suspension just in case, and (c) Jeff reminds me that we should also include the IMC PZT when doing this. But, already having had most of them pushed was already a huge time saver!