D. Barker, F. Clara, J. Kissel, N. Lecoeuche

Recovery blow-by-blow:

Picking up from after Fil gets in and Dave gracefully shuts down the h1sus34 front-end computer:
~18:50 UTC
- Fil & Niko finish replacing DAC 2-5 (SR2 M2 & M3) 18-bit DAC card in h1sush34 IO chassis (LHO aLOG 50306). They mention they had to mess with h1susaux34 timing cable (they were zip-tied together), but thankfully this did not glitch the timing on h1susau34 -- it ran happily throughout the replacement.

~19:00 UTC 
- Jeff hits the physical restart button on the h1sush34 computer in the Mass Storage Room, beginning automatic restart of all front-end processes

~19:10 UTC 
- Input-Output Processing (IOP) model process automatically restarts, successfully calibrates the DAC cards (including the new one). All user model processes are then restarted (LHO aLOG 50304).

19:22 UTC 
- After successful start-up of all processes on h1sush34 computer, IRIG-B timing error signal reflects that the IO chassis needs to resynchronize to the world. Dave calls to confirm success of model restarts, and that it'll take ~10 minutes for the IRIG-B system to synchronize. Until then, we won't be able to reset the IOP Software Watchdog (SWWD) because inter-process communication (IPC, needed for SUS-IOP in sush34 to talk to SEI-IOPs in seih23 and seih45) error checking requires all IO chassis' IRIG-B timing to be happy.

19:27 UTC 
- Jeff and Niko begin to untrip local "user" watchdogs for MC2, PR2, and SR2, and suspension portions of SWWD that don't rely on IPC. Confirmed that all SUS's drive signals are indeed going out in to the real world, and SUS are damping.
          
- Brought IMC_LOCK guardian to "DOWN" such that the IMC doesn't starting locking (and running alignment control) while the seismic systems (namely, HEPI) are still shut down and the DC position of HAM3 / MC2 are known to be wrong. 

- Unmanaged IMC-LOCK (normally managed by ISC_LOCK) so we're in control and ISC_LOCK is not fighting to lock the IMC prematurely. 

19:30 UTC 
- h1sush34 IO chassis IRIG-B timing is now synchronized, Dave hits a reset on all front-end diagnostics to clear latched indications of IPC errors (LHO aLOG 50305)

- Untripped local "user" watchdogs for ISIs and HEPIs in HAM3 and HAM4

- Untripped SWWD for ISI&HPIHAM3 and ISI&HPIHAM4 DACs on SEI23 and SEI34 computers, allowing drive signals to reach the platforms finally.

- SEI Manager guardians for ISI&HPIHAM3 and ISI&HPIHAM4 recognized that all watchdogs have been cleared, and automatically began the process of bringing platforms to FULLY_ISOLATED, and succeeded without problem or intervention.

- Used ndscope to trend and restore MC2 alignment (used alignment sliders alone as reference; did not use any stage's OSEMs).

- Brought IMC_LOCK guardian to "LOCKED." This restored resonance and alignment of the IMC, without problem or intervention.

- Ran "INIT" and "DOWN" on ISC_LOCK guardian so as to re-manage IMC_LOCK and get the IFO prepped for locking. 

- Used ndscope to trend and restore PR2 and SR2 alignment (again -- used alignment sliders alone as reference; did not use any stage's OSEMs).

19:40 UTC
- Pressed our luck and tried to just start the lock acquisition sequence with the ISC_LOCK guardian as normal.

- Having trouble with the X ARM ALS -- after Green ASC system turned on, WFS DOF1 Pitch would drastically drive ETMX into the weeds and break the lock. Niko suggest he's heard that something has been done to the order of operations in the Green ASC system during the LOCKING_ARMS_GREEN state recently (must be Sheila's work on Friday June 14th LHO aLOG 49930). This is even though green flashes in the X ARM show normalized power above 0.95, and the spot on the green camera looks nice. 
After 4 attempts at this, we decided it prudent to just proceed with initial alignment, for starters just to see if INITIAL_ALIGNMENT guardian does any better.

20:06UTC
- Requested INITIAL_ALIGNMENT of ISC_LOCK and began to run through the laminated operator checklist. We discussed running through the new automatic system (see LHO aLOG 50190), but (a) didn't remember how to get it started (I only found that LHO aLOG 50190 now while writing this log), and (b) wanted to take things slowly given our problems with the X ARM and our knowledge of the computer crash involving SUS PR2 and SR2.

- While discussing, Niko tries to squeak a little bit more out of the the X ARM's green flashes and gets from 0.95 to 1.0 (max for this "normalized" signal is 1.1).

- X ARM INITIAL ALIGNMENT step works well. Yes, there's still a large control transient when green ASC turns on, but it doesn't go off in to the weeds. Either (a) the change in offload sequence in ISC_LOCK's version of locking the green are different and worse than INITIAL_ALIGNMENT's version, or (b) Niko's last-ditch tweaks were enough to push the DOF1 error signal on the "other side" of the WFS error signal phase curve and the signal is now valid (where before, if the supposition is correct, the alignment was "so bad" that the WFS error signal was outside the normal linear regime, where the sign on the loop flips.)

- Jeff decides to go through the rest of INITIAL_ALIGNMENT "just to see" while Niko gets lunch. Good thing -- when PRC comes up, the error / control signals for PRC1 and PRC2 are large, and it takes a while to converge. SRC_ALIGN was not so bad. Rest of initial alignment went smoothly.

20:36 UTC
- Resume normal lock acquisition with ISC_LOCK.  LOCKING_ARMS_GREEN succeeds, but we loose lock mid-way through CHECKING_IR. XARM is Glitching enough to cause lock-loss. After reacquiring to that point, there's glitching again, but this time not bad enough break green lock, and we hold it through CHECK_IR.

20:43 UTC
- After re-acquisition and slowly finding IR, we go through to PRMI_LOCKED. Locks up nicely without any need for adjustment, so we head to OFFLOAD_DRMI_ASC 
 We acquire pretty quickly, BUT PRC1 and PRC2 ASC error signals are huge, so we loose lock once there.

20:57 - 21:10 UTC
- Upon second attempt, we request DRMI_LOCKED_CHECK_ASC, and leave it there for 10 minutes as PRC1, PRC2, and some of SRC1 and SRC2 ASC loops are slowly brought in to convergence. We wait past when the state's convergence checker thinks things have converged.

21:19 - 21:50 UTC
- Here, the rest of the acquisition sequence goes well, just very slowly. We paused at each request stated state that are good stopping points for the ASC system, PREP_ASC_FOR_FULL_IFO, ENGAGE_DC_VIOLINS, INCREASE_POWER, LOWNOISE_ESD_ETMX. We waited extra long at 2W input power on DC readout (i.e. ENGAGE_DC_VIOLINS) such that the full SOFT loops and ADS system could cook for a while slowly but surely (re)finding good spot positions before the major thermalization transient of power-up.

21:50 UTC 
- We hit NOMINAL_LOW_NOISE, and it takes us a few minutes to clear out a few SDF differences (LHO aLOG 50307).

20:00 UTC
- OBSERVING.

Sub-comment about X-ARM Problems with LOCKING ARMS GREEN (at 19:40 UTC above)

I attach time-series of ALS signals during our troubles with LOCKING_ARMS_GREEN mentioned above. X ARM transmitted power is in blue in the bottom left corner, and X ARM ALS ASC signals are in the upper right corner.

The first attachment shows the first 3 attempts in LOCKING_ARMS_GREEN which show ALS WFS DOF1 yanking the ETM around, and causing lock loss.

The second attachment shows a zoom in on one of those lock losses. 

The third attachment shows success when using INITIAL_ALIGNMENT.

Sub-comment about X-ARM ALS Glitching during CHECK_IR (at 20:36 UTC above)

I attach time-series of the X-ARM ALS glitch that caused a lock loss while trying to find IR. Again, X ARM transmitted power is in blue in the bottom left corner, and X ARM ALS ASC signals are in the upper right corner.

The first attachment shows the X-ARM glitching that caused the lock loss,

The second attachment shows glitching that occurred during the next attempt, which appears to be roughly the same, but does not cause a further lock loss.

It is *not* raining, and there is only ~10 mph or less winds, so it's not obvious if this was because of exposed fiber bundles along the arms (i.e. FRS Tickets 11113 and 12209).

Activity associated with FRS Ticket 13170.

Fil is in transit to the site. I'm determining which card is inaccessible.

I've opened FRS13170 to cover this fault.

Last DAC (slot 2-5) is missing.

In attachment, lower left is the card locations as shown by lspci. The last 18bit DAC (slot 2-5) is missing.

I've done with the h1sush34 card analysis, I've powered the computer down in preparation for card replacment.

Fil has replaced the 18bit DAC with a spare. The IOP model now sees all six cards, here is the autocal status:

h1sush34 ~ # dmesg|grep AUTOCAL

[   63.863055] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds

[   69.226743] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds

[   75.018226] h1iopsush34: DAC AUTOCAL SUCCESS in 5342 milliseconds

[   80.381909] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds

[   86.173432] h1iopsush34: DAC AUTOCAL SUCCESS in 5342 milliseconds

[   91.537123] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds

IOP has gone into a negative IRIG-B excursion, so DAQ and IPC data will be bad for the next few minutes.

Fil, Jeff K, Niko, Dave:

IRIG-B has caught up. I've cleared all IPC and DAQ-CRC errors. Looks like h1sush34 is back online.

Many thanks to Fil for coming out to the site and replacing the card.

DAC card replaced in h1sus34 IO chassis S1103886.

Old:  PCIe-18AO8-8-40.32M S/N 101208-08
New: PCIe-18AO8-8-40.32M S/N 101208-06

For the record, sush34's last DAC card (the 6th DAC card numbered 5 because counting starts at zero), in slot 2-5, is SR2's M2 and M3 coils. 

Attached are some relevant fast time-series of the AS port power (as measured by H1:ASC-AS_A_DC_NSUM_OUT_DQ), the requested drive from the lowest stage of each of the SUS controlled by this chassis -- MC2, PR2, and SR2 -- (as measured by H1:SUS-$(OPTIC)_M3_MASTER_OUT_$(OSEMDOF)_DQ), and the fast current monitors from the same coils (as measured by H1:SUS-$(OPTIC)_M3_FASTIMON_$(OSEMDOF)_OUT_DQ), which are thankfully on completely different computers.

16:45:46.94 UTC -- SR2 and PR2 Coil Driver MASTER_OUT_DQ starts requesting a repetitive signal, but not MC2
16:45:47.08 UTC -- MC2 starts requesting a HUGE signal, saturating MC2's M3 stage. 
16:45:47.12 UTC -- AS port registers first badness
16:45:47.15 UTC -- Power at AS port has dropped to zero

So, my supposition is that the lock loss occurred because of the MC2 blast -- not because of the SR2 DAC card failure. The MC2 blast occurred, because its DAC cards are in the same IO chassis (sush34) as SR2, and something went belly up much before the reported timing system glitch.

Also -- I attach a trend of how long it took for the IRIG-B to come back after recovery of the front-end.

Jeff K, Nico, Dave:

h1iopsush34 is not recovering. On first startWorld try it saw all the 18bit DACs but the model did not get going. I started the IOP by hand a second time and it only autocal'ed 5 of the expected 6 18bit DACs. This was confirmed by lspci. Before calling EE out, we will try a power cycle of the IO Chassis, I powered down the front end and Jeff K is power cycling the IO Chassis.

h1sush34 ~ # lspci -v |grep 3101                                                                                                

        Subsystem: PLX Technology, Inc. Device 3101                                                                             

        Subsystem: PLX Technology, Inc. Device 3101                                                                             

h1sush34 ~ # lspci -v |grep 3357                                                                                                

        Subsystem: PLX Technology, Inc. Device 3357                                                                             

        Subsystem: PLX Technology, Inc. Device 3357                                                                             

        Subsystem: PLX Technology, Inc. Device 3357                                                                             

        Subsystem: PLX Technology, Inc. Device 3357                                                                             

        Subsystem: PLX Technology, Inc. Device 3357