Displaying report 1-1 of 1.
Reports until 04:04, Wednesday 11 September 2019
H1 CDS
david.barker@LIGO.ORG - posted 04:04, Wednesday 11 September 2019 - last comment - 04:32, Wednesday 11 September 2019(51888)
front ends back, looks like h1iscey is the only one crashed

After h1boot1 came back (took a few minutes, and FSCK was needed) the overview MEDM came back to life.

It looks like h1iscey is the only casualty, and it has taken the DAQ data down for all the other front ends which share its h1dc0 port (both ends, corner station ISC and SUSAUXB123).

I'm starting the h1iscey reboot process.

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 04:15, Wednesday 11 September 2019 (51889)

Resetting h1iscey now. Looking through the logs we had an issue with this front end in June of this year.

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=50009

david.barker@LIGO.ORG - 04:32, Wednesday 11 September 2019 (51890)

I cannot ssh into any EY Dolphin'ed front end machine (h1susey, h1seiey, h1iscey), even though SUS and SEI EPICS appears operational.

I took h1iscey out of Dolphin and remotely reset it via IPMI. This crashed the models on h1susey and h1seiey. Also, h1iscey was pingable for a short while and then froze out again.

I tried to remotely reset h1susey via IPMI, but its management port is unresponsive.

Corey is now driving to EY to capture the consoles of these three machines and then power cycle them. I'm his safety buddy while he is traveling to and from EY.

Displaying report 1-1 of 1.