Reports until 19:23, Friday 19 October 2018
H1 CDS
sheila.dwyer@LIGO.ORG - posted 19:23, Friday 19 October 2018 - last comment - 19:59, Friday 19 October 2018(44695)
SEI computer issue again

Hang, Sheila, Dave, everyone

We had a repeat of this morning's crash of HAM2,3,4,5.  44678  44671

Dave walked us through the reset over the phone:

log into h1seih23 and h1seih45 as controls (ssh controls@h1seih23)  (check secrets.ligo.org) run the command /etc/startWorld.sh to restart all the models on those machines.  After they come back you will need to reset watchdogs and dackill.  

It seems that the sensor correction gains already on this time. 

If this continues to happen overnight people can do the restart this way.  If it happens tomorrow during the day Dave might want to try shutting things down and starting them again, so we can give him a call.  

Comments related to this report
keith.thorne@LIGO.ORG - 19:46, Friday 19 October 2018 (44696)
Link 

As the proc files show, the DAC FIFOs are getting exhausted.  This typically happens if the IOP cycle time is too long (so it doesn't get the DAC FIFO filled before it gets clocked out every cycle).   We see a long cycle time on each (500 mu-sec) but unsure when it occurs.  A common factor between seih23 and seih45 is the DACKILL IPC from IOP model on sush34, or it could be a DC power supply.

I would definitely recommend I/O chassis power-cycle (or at least a front-end computer reboot). As I understand it, the SEI HAM front-ends have newer (2012 production) PCIe expansion fibers, so we hope they are not degrading.
david.barker@LIGO.ORG - 19:52, Friday 19 October 2018 (44697)
Link

Attached image show the state_word for h1iopseih23 (red) and h1iopseih45 (blue) as a second trend around the time of the problem.

The sequence is:

h1iopseih23 goes to DAC+DK for 1 second, then goes into DAC+DK+OVR for 7 seconds

h1iopseih23 returns to DAC+DK and stays there

7 seconds later, h1iopseih45 follows the same sequence.

 

Images attached to this comment
david.barker@LIGO.ORG - 19:59, Friday 19 October 2018 (44698)
Link

Attached plot shows situation during this morning's 03:29 crash, same general sequence but the IOP order is reversed.

Images attached to this comment