J. Kissel, J. Warner, P. Thomas (R. McCarthy and D. Barker remotely)
I happened to be on site giving this Saturday's tour, for better or worse, when Jim and Patrick informed me that End X had some sort of electronics failure that took down everything (see first entry here: LHO aLOG 382517).
After a debrief from both of them while driving to the end station, we knew to start our investigation at the SUS rack power supplies: the symptom, they'd informed me after having already gone to the end station, that the coil drivers and AA/AI chassis in the SUS rack showed only one leg one via the LEDs in the back.
We don't know why this happened, and have not investigated.
As a consequence, however, this explains why the OSEM sensor report to the independent software watchdog went dead, and thus tripped that watchdog, killing the seismic system via watchdogs.
Upon arrival, we went immediately to the power supplies, and indeed found the left leg of the 18V power supply with its ON/OFF rocker switch in the OFF position. We turned it back ON, it came to life (see "after" picture IMG_2234.jpg), and this restored the SUS electronics and its OSEMs (see "after" picture IMG_2235.jpg).
However, unfortunately, we also noticed that the EX:ISC 12V power supply LED lights were flickering (see video on YouTube because .m4v format is not accepted by the aLOG). Not knowing that this was a normal thing (we found out later once we got a hold of Richard), we also power cycled both legs of those ISC 12V power supplies hoping to clear the blinking.
Sadly
(a) this did not clear the blinking,
(b) it killed the power to the neighboring IRIGB timing system, which informs (and thus killed) the end station ISC I/O chassis. That, in turn, killed the entire end station Dolphin Network.
(c) (we found out later) it killed the power supply to the relay that feeds the power to the ESD HV. Namely, once we got up to looking at the ESD HV power supply (we worked our way up the rack from the bottom) we found that with rocker switches ON, there were no displays and OFF, and they were unresponsive to flipping the ON/OFF rocker switches (as pictured in IMG_2233.jpg).
In order to fix (b) we got a hold of Dave, who is having to restart all front-ends and IO chassis remotely. He'll likely aLOG later.
In order to fix (c) we followed power cables to the HV power supplies up to the top of the power supply rack, where found what we now know as a relay in the state as pictured in IMG_2236.jpg and IMG_2237.jpg.
Eventually, we found the big red button on the "front" of the chassis (see IMG_2238.jpg), and like all good red buttons, it drove us to push it (cautiously, we had left the ESD power supplies in the OFF position before doing so, however). We heard a few clicks, and all the green lights lit up. Once that was happy, then the ESD HV power supplies came back to life with a simple flick of the power switch.
As of now, Jim is re-isolating tables, and resuming recovery with a fully functional system. "Normal" recovery from here, but we should be cautious of violin modes.
Again, we have not invested why the power supply tripped off in the first place; we'll leave that for the work week or for those more curious than us.
TCSX is back - H1 in Observe - details to follow
Initial exit of H1 from Observe:
H1 recovery / return to Observe: