For the record, sush34's last DAC card (the 6th DAC card numbered 5 because counting starts at zero), in slot 2-5, is SR2's M2 and M3 coils.
Attached are some relevant fast time-series of the AS port power (as measured by H1:ASC-AS_A_DC_NSUM_OUT_DQ), the requested drive from the lowest stage of each of the SUS controlled by this chassis -- MC2, PR2, and SR2 -- (as measured by H1:SUS-$(OPTIC)_M3_MASTER_OUT_$(OSEMDOF)_DQ), and the fast current monitors from the same coils (as measured by H1:SUS-$(OPTIC)_M3_FASTIMON_$(OSEMDOF)_OUT_DQ), which are thankfully on completely different computers.
16:45:46.94 UTC -- SR2 and PR2 Coil Driver MASTER_OUT_DQ starts requesting a repetitive signal, but not MC2
16:45:47.08 UTC -- MC2 starts requesting a HUGE signal, saturating MC2's M3 stage.
16:45:47.12 UTC -- AS port registers first badness
16:45:47.15 UTC -- Power at AS port has dropped to zero
So, my supposition is that the lock loss occurred because of the MC2 blast -- not because of the SR2 DAC card failure. The MC2 blast occurred, because its DAC cards are in the same IO chassis (sush34) as SR2, and something went belly up much before the reported timing system glitch.
Also -- I attach a trend of how long it took for the IRIG-B to come back after recovery of the front-end.
Fil is in transit to the site. I'm determining which card is inaccessible.
I've opened FRS13170 to cover this fault.
Last DAC (slot 2-5) is missing.
In attachment, lower left is the card locations as shown by lspci. The last 18bit DAC (slot 2-5) is missing.
I've done with the h1sush34 card analysis, I've powered the computer down in preparation for card replacment.
Fil has replaced the 18bit DAC with a spare. The IOP model now sees all six cards, here is the autocal status:
h1sush34 ~ # dmesg|grep AUTOCAL
[ 63.863055] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds
[ 69.226743] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds
[ 75.018226] h1iopsush34: DAC AUTOCAL SUCCESS in 5342 milliseconds
[ 80.381909] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds
[ 86.173432] h1iopsush34: DAC AUTOCAL SUCCESS in 5342 milliseconds
[ 91.537123] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds
IOP has gone into a negative IRIG-B excursion, so DAQ and IPC data will be bad for the next few minutes.
Fil, Jeff K, Niko, Dave:
IRIG-B has caught up. I've cleared all IPC and DAQ-CRC errors. Looks like h1sush34 is back online.
Many thanks to Fil for coming out to the site and replacing the card.
DAC card replaced in h1sus34 IO chassis S1103886.
Old: PCIe-18AO8-8-40.32M S/N 101208-08
New: PCIe-18AO8-8-40.32M S/N 101208-06