david.barker@LIGO.ORG - posted 16:09, Wednesday 03 January 2018 - last comment - 16:18, Wednesday 03 January 2018(39970)
h1lsc0 models needed to be restarted, repeat of timing error first seen 22nd Dec 2017
All models on h1lsc0 froze up at 15:21 PST due to an IRIG-B timing error on the h1ioplsc0 model. The dmesg logs suggests this is a repeat of the problem seen on Friday 22nd December 2017 which also required a full model restart.
Looking at minute trends, the IOP IRIG-B signal went from 13 uS to 202,710 uS at 15:21:42 PST, later it jumped up to 732,404 uS at 15:30:42 PST. After restarting the model, it returned to 13uS.
On Friday Dec 22 2017 at 09:44:42 PST this signal jumped to 296,598 uS until it was reset 13 minutes later (no subsequent jump into the 0.7 S region).
h1lsc0's dmesg logs for the two events are:
[6123949.214305] h1lscaux: ADC TIMEOUT 0 46035 19 46099
[6123949.214307] h1omc: ADC TIMEOUT 0 46035 19 46099
[6123949.214309] h1lsc: ADC TIMEOUT 0 46035 19 46099
[6123949.214311] h1sqz: ADC TIMEOUT 0 46035 19 46099
[6123949.214312] h1omcpi: ADC TIMEOUT 0 46035 19 46099
...
Following the restarts, I green'ed up the overview by diag-resetting all models to clear any IPC errors and cleared the DAQ accumulated CRC errors on h1lsc0. The only remaining RED item is the DAQ EDCU, which is unable to connect to two corner station Beckhoff Slow Controls channels following yesterday's code change of h1ecatc1plc3. The two channels are:
H1:TCS-ITMX_CO2_TEMP_SPARE
H1:TCS-ITMY_CO2_TEMP_SPARE
this will require restarting the DAQ to use the latest INI files to clear.