We have lost several corner stations front end models in what may be a Dolphin glitch. Caused the lock loss at 16:12PDT.
h1omc0 was synchronized at the wrong time, see the attached GSD TP screen. Duotone value is negative. T0 shows timing was lagging behind by 30 ms.
Output from timing card looked normal.
Full set of IOP GDS-TP screens
Jonathan, Erik, Dave, TJ:
We did not see anything useful in the front end logs.
Erik restarted the models on h1omc0, h1susb123, h1sush2a, h1sush34, h1sush56 and h1oaf0.
Note, we did not need to fence front ends from Dolphin and reboot, just model restarts were sufficient.
I cleared all the SWWD watchdogs, no SEI WDs had tripped.
We called TJ who will continue the recovery of the interferometer.
Sun30Apr2023
LOC TIME HOSTNAME MODEL/REBOOT
16:59:56 h1omc0 h1iopomc0
17:00:10 h1omc0 h1omc
17:00:24 h1omc0 h1omcpi
17:02:29 h1susb123 h1iopsusb123
17:02:34 h1sush2a h1iopsush2a
17:02:43 h1susb123 h1susitmy
17:02:48 h1sush2a h1susmc1
17:02:57 h1susb123 h1susbs
17:03:02 h1sush2a h1susmc3
17:03:11 h1susb123 h1susitmx
17:03:11 h1sush34 h1iopsush34
17:03:16 h1sush2a h1susprm
17:03:25 h1susb123 h1susitmpi
17:03:25 h1sush34 h1susmc2
17:03:30 h1sush2a h1suspr3
17:03:35 h1sush56 h1iopsush56
17:03:39 h1sush34 h1suspr2
17:03:53 h1sush34 h1sussr2
17:03:55 h1sush56 h1sussrm
17:04:01 h1oaf0 h1iopoaf0
17:04:09 h1sush56 h1sussr3
17:04:15 h1oaf0 h1pemcs
17:04:23 h1sush56 h1susifoout
17:04:29 h1oaf0 h1tcscs
17:04:37 h1sush56 h1sussqzout
17:04:43 h1oaf0 h1susprocpi
17:04:57 h1oaf0 h1seiproc
17:05:11 h1oaf0 h1oaf
17:05:25 h1oaf0 h1calcs
17:05:39 h1oaf0 h1susproc
17:05:54 h1oaf0 h1calinj
17:06:08 h1oaf0 h1bos