From the trend data, it looks like the noise eater signal started misbehaving about a week ago.  The photodetector monitoring
the output power of the NPRO seems to be still working.  I would be inclined to check the field box that the
H1:PSL-MIS_NPRO_RRO_OUTPUT is attached too, as there maybe a large DC offset in that monitor signal's output.

NMBDs installed on ITMx are serial numbers 015 and 016.

Patrick rebooting ISC at EX: Actually h1ecatx1 EPICS IOC (alog 38970).

• IMG-2655 shows overhead view of the black glass beam dump that was added (9:00 position in photo)
• IMG-2656 shows the HR optic w/black glass which replaced the 90/10 BS that was there before
• IMG-2658 shows the vacant space where there were two HR mirrors (one of which resides where the BS used to)
• IMG-2659 is a close up of the mirror that replaced the BS

Here are a few more pics.  As TJ notes, the ROC front face is 5-6mm from the SR3 AR surface.  It is locked down in this location. 

Note, we followed a few hints from LLO's install:

https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=25831

Continuity checks at the feedthru still need to be made.  Will solicit EE for their help.

Initial continuity test failed. Found issues with in-vacuum cable, power pins not pushed in completely. Pins were pushed in until a locking click was heard.

Reading are:
Larger Outer Pins, Heater: 66.9Ω
Inner pair (left most looking at connector from air side), thermacouple: 105Ω

Found center of SR3(-X Scribe) to be 230.2mm above optical table.  By siting the top and bottom of RoC Heater, found center to be at 231.4mm. Removed available shim to put center of RoC Heater at 229.6 for 230.2-229.6=0.6mm below perfect.

Hugh's comment reminded me that to get the heater to fit, Betsy and I added a 1mm washer to raise the height of the assembly. In total we have 4mm of washers (2x1.5mm & 1x1mm).

I conducted measurement of quantity 6 of [D1600104 SR3 ROC Actuator, Ceramic Heater Assy] at CIT on 4 March 2016. Dirty state before baking.
The serial number of the heater assy installed in LHO HAM5 is S1600180 - see https://ics-redux.ligo-la.caltech.edu/JIRA/browse/ASSY-D1500258-002
S1600180 Resistance = 66.8 Ohms on 4 March 2016.
There is good agreement between the as-installed and pre-bake measurements.

We've also talked about seismic watchdogs a bit and why the ISIs trip after the isolation loops are shut off by the guardian. Both ETMs are in damped right now so we set the T240 threshold to 100 counts, and sure enough, the T240s started counting saturations, but did not trip the watchdog. Attached plot shows the T240 saturation counts, threshold and ST1 WD mon state. The dip on the top left plot is where we reduced the threshold, the spike on the bottom left is where the model started counting T240 saturations, and the flat line bottom right shows the watchdog didn't trip. This is as it should be.

However, what I think I've seen during ISI trips before, is the ST1 T240s saturate, ST1 trips and ST2 runs for a little bit then trips. This results in ST1 getting whacked pretty hard. I'll try to see if that's what happened with this earthquake.

J. Kissel, inspired by conversation from S. Cooper, S. Dwyer, J. Warner

I'll remind folks that this collective SEI/ SUS watchdog system has been built up sporadically over ~10 years in fits and spurts as reactionary and quick solutions to various problems by several generations of engineers and scientists. Also, the watchdog system is almost entirely designed only to protect the hardware from a software failure, and never designed to combat this latest suggestion -- protecting the hardware from the earth. So I apologize on behalf of that history at how clunking and confusing things are when discussing what to do in that situation. 

Also, I'll remind people that there are three "areas" of watchdogs: 
    (1) in software, inside the user model -- typically defined by the subsystem experts
    (2) in software, inside the upper level iop model -- typically defined by CDS software group, with input from subsystem experts
    (3) in hardware, either in the AA/AI chassis, or built into the analog coil drivers -- typically defined during initial aLIGO design phase

In my reply here, I'll only be referring to (1) & (2), though I still have an ECR pending approval regarding (3) -- see E1600270 and/or FRS Ticket 6100.

With all that primer done, here's what we should do with the suspension user watchdogs (1), and not necessarily just for earthquakes:
    (a) Remove all connection between SUS and the ISIs user watchdogs. The independent software watchdogs (2) should cover us in any bad scenarios that that connection was designed to protect against.
    (b) Update the RMS system to be *actually* an RMS, and especially, one that we can define a time-constant. The RMS system that is currently installed is some frankenstein brought alive before bugs in the RCG were appreciated (namely LHO aLOG 19658), and before I understood how to use the RCG's RMS function in general. The independent software's watchdog (2) is a good role model for this
    (c) We should rip out all USER MODEL usage of the DACKILL part. The way the DACKILL used across suspension types and platforms with many payloads is confusing and inconsistent. Any originally designed intent of this part is now covered by the independent software watchdog.
    (d) Once (b) is complete, we should tailor the lower the time-constants and the band-passing to better match the digital usage of the stage. For example, the worst that can happen to a PUM stage is getting sent junk ASC and Violin Mode Damping control feedback signals when the IFO has lost lock, but the guardian has not figured it out and switched off control.
    (e part 1) Upon watchdog trip, we should consider leaving the alignment offsets alone. Suddenly turning off alignment offsets often causes just as much of a kick to the system as what had originally set off the watchdog. HEPI has successfully implemented such a system.
    (e part 2) We should re-think the interaction between the remaining USER watchdog system and the Guardian. Currently, after a watchdog trip the guardian state immediately jumps to "TRIPPED" and begins to shut off all outputs and bringing the digital control system to "SAFE." 
    (f) Add a "bypass" feature to the watchdog such that a user can request the "at all costs, continue to try damping to top mass" in the case of earthquakes.

I'm attaching some more plots of what happened to the ISIs during this earthquake. The first plot is the saturation count time series for all seismometers and actuators for the test mass ISIs. All of the chambers saturated on the Stage 2 actuators first, this is the first green spike. This tripped the high gain DC-coupled isolation loops, and probably cause Stage 2 to hit it's lockers. The watchdog stopped counting all saturations for 3 seconds (by design), then immediately tripped damping loops on the saturated L4Cs or T240s. I'm not sure why the GS13s don't show up here.

The second plot I attach shows how long the ETMX was saturating different sensors. The L4Cs were saturated for about 45 seconds, the T240s and GS13s were saturated for minutes. The L4Cs never had their analog gains switched, but the chamber guardian should have switched the GS13s automatically. For this reason, if we increase the pause time in the watchdog (between shutting off the isolation loops and full shutdown), I think this shows that for this earthquake the ride-thru time needs to be more than 45 seconds.

Here are a couple photos with the feedthru protection on and the cables secured.  Please be very cautious with the CPS cables going out to the left.  The ends of the cable are barely behind the shroud plane and I certainly worry about them.

As I recall, being the operator on site for the Montana quake, one or two optics/chambers tripped, and I couldn't see any info on a website or even on the plot of our own siesmometers about an EQ, so at first I didn't know that there was an EQ, I thought it could have been some other kind of a failure, and I only knew it was an EQ when more things tripped.

I'm wondering if, given that scenario, you would recommend that in the next observing run Operators transition SEI if any chamber trips, as a precausion for big close EQs?

Would there be enough time for SEIs to transition before a big close EQ arrives?

If not, is there any down side to having an EQ arrive mid-transition?

The motivation for this code comes from the struggles commissioners had during the Mexican earthquake on Sept 8th, this year. The goal is to put the ISIs into a state where it is as easy as possible to keep the ISI damping loops on. It's a last ditch effort to protect the suspensions. So, even if the earthquake is already tripping platforms, it would be best to make the transition, anyway.

For an earthquake as close as the Montana quake, we won't get any advanced notice. I still would not advocate just switching to this configuration just because some chamber tripped. It takes about 45 seconds for a BSC to go from Fully Isolated to damped and about 2 minutes the other way, and there will still be settings to recover after (because the script switches the gains on ISI seismometers). You will also probably have to run an initial alignment. Some thought should be given to a wall FOM for the live ground seismometer signals to help identify if there is an earthquake arriving on site, unannounced. The low frequency blrms on the wall only update every minute, and are probably too slow to help, but all 3 seismometers seeing large, similar signals at the same time is a pretty good indication.