jim.warner@LIGO.ORG - posted 06:48, Tuesday 11 June 2019 - last comment - 13:59, Tuesday 11 June 2019(49799)
Something just crashed in SUS, knocked out of observing
Just got a bunch of SW watchdog tripped notifications and got kicked out of observe, but we are still locked. All of the fields for the large HAM2 suspensions on the CDS overview are white, as well as H1IOPSUSH2A. I've called Dave, he's looking into it.
The general core on h1sush2a (the one which runs the operating system) has crashed, but the five locked cores which run the models (h1iopsush2a, h1susmc1, h1susmc3, h1susprm and h1suspr3) continue to function. This means the suspensions are still working, but the EPICS reporting has failed. Because the SDF channels are no longer available for these models, Guardian will not allow H1 to go into observation mode.
We have seen this type of crash before, the only recovery action is to isolate h1sush2a from the Dolphin fabric and reset the computer using its front-panel switch. Given that maintenance starts in less than an hour and LLO has just dropped out, we will fix this problem during maintenance.
image shows that DAQ data from h1sush2a is frozen from 13:37 UTC (06:37 PDT).
opened FRS13004. Then closed it as resolved (h1sush2a was rebooted).