h1susey has crashed. Similar to h1lsc0 this morning it looks like the models are still running and the suspensions are still being damped. We have lost DAQ data from this machine and EPICS monitoring/controls.
If we need to restart this system, please call me on my cell phone and I can do this remotely.
h1susey crashed at 19:00 PDT (02:00 Sun 7th UTC).
current status of SUS-EY.
I spoke with Richard on the phone last night soon after the crash and we agreed we could leave this system overnight because both the software-watchdog (running on h1iopsusey) and the independent hardware-watchdogs were functioning.
I realized we can indirectly monitor what the h1susetmy model is doing by looking at the coil current monitors being acquired by the h1susauxey model. Attached plots show 24 hours of M0_F1 IMON against h1susetmy_cpu_meter, 7 days of M0_F1_IMON, and 24 hours of all the ETMY M0 IMONs.
It looks like the suspension has changed state around 4am this morning.
Would the first commissioner on site today please call me on my cell phone and we can schedule a restart of h1susey.
You have to be careful leaving the front end in this state, at least when it come to guardian. If a guardian node can't see EPICS channels it needs it goes into a CERROR state, in which it won't progress on it's graph until the needed channels are recovered.
We had a very similar failure at LLO last week, while the IFO was at full lock. The IFO kept operating otherwise normally, but the ISC_LOCK node went into CERROR. When there was a lockloss the node was still stuck and the DOWN state was not executed. This can be bad since certain resets don't take place in a timely manner, which can cause things like violin modes to et rung up.
Looks like that's exactly what happened here. See alog 44384 for a few notes.