We have verified that X-Arm RFM IPC signals go to ETMX and Y-Arm to ETMY.

The h1omc model sends DARM_CTRL to ETMX with a matrix element of +1.0 and to ETMY with an element of -1.0. The DARM_CTRL signal is currently the CAL_LINE_SUM signal, a ~30Hz sine wave with a small amplitude of 0.4 counts. The attached dataviewer plot shows the sending trace (CH3: H1:LSC-CAL_LINE_SUM) the receiver on h1susetmx (CH4: H1:SUS-ETMX_L3_ISCINF_L_IN1) and the receiver on h1susetmy (Ch5: H1:SUS-ETMY_L3_ISCINF_L_IN1). The plot is 1/16th of a second on the time axis. As can be seen, ETMX is in phase with CAL_LINE_SUM and ETMY is 180deg out of phase.

The relevant part of h1omc model and the matrix settings are shown in an attachment.

To be absolutely sure of which Dolphin port the cable should be connected to, we drove out to EY and verified with h1susey.

Model processing time changes resulting from computer upgrade (see attached minute trend plot).

Ch1: h1ioplsc0. Did not noticably speed up, in fact looks like it may have slowed by 1uS. min-max range has tightened up.

Ch2: h1lsc: Sped up from 38 +/- 4uS to 25 +/- 0uS (12us (32%) faster with very little jitter)

Ch3: h1omc: Sped up from 22 +/-2uS to 15 +/-0uS (7uS (32%)  faster with very little jitter)

Ch4: h1lscaux: Sped up from 19 +/-1uS to 15 +/-0uS (4uS (21%) faster with very little jitter)

Ch5: h1omcpi: Sped up from 8 +/-1uS to 6 +/0uS (2uS (25%) faster with no jitter)

We were motivated to look at the fast shutter wires by the incident on  July 17th  where the wire from the fast shutter must have been clipping the to OM1 from HAM5. My original alog about this might not have spelled things out well enough, so here are some plots.  

The first plot is from July 17 2017 5:51:00 UTC, this was after Jim, Cheryl and I figured out that the fast shutter was malfunctioning. We locked DRMI, and opened and closed the fast shutter several times.  The top panel shows the state of the shutter, 0 is open 1 is closed.  The second panel shows AS_A_SUM, which is downstream of the shutter, and goes to zero when the shutter is blocking the beam as it should.  The third panel shows AS_C, which is upstream of the shutter but downstream of the place where the wire is close to the beam (check Jeff's annotated photo).  You can see that moving the shutter causes dips in the amount of light on AS_C, and that the wire must land in a slightly different place in the HAM5 to OM1 beam causing a different level of light to be seen on AS_C

The second attachment shows that the shutter did seem to block the beam going to the AS WFS in the July 16th lockloss before we had this malfunction.  Chandra also checked vacuum pressures for a spike in HAM6 pressures similar to what happened when we burned the OMC in August 2016, and saw nothing.  I had been wondering if the fast shutter might have failed to durring a lockloss where the OMC was unlocked, which could result in a lot of power being on the beam dump.  It seems like this didn't happen on July 16th. 

Note, we played with this shutter cable last in Aug 2016.  We struggled with getting this wire away from the beam at that time.  The pictures in the log from Aug 2016 28969 show that we left the wire in a larger arc than the pictures show now.  I suppose it's not so surprising that the wire has maybe migrated into the beam path over the numerous cycles over the last year.

Just in case, here's another labeled picture to show the beam path and clearance of the fast shutter to its high power beam dump. The above mentioned break is a result of the OMC REFL beam path, NOT the faster shutter path.

Now associated with FRS Ticket 9196.

Note, Corey and I both think (after looking at pictures) that these are the black PEEK set screws installed in the beam dumps shown.

Apologies -- in the above entry it says "when we killed the DCPDs in Aug 2016." However, we killed on of the OMC cavity mirrors, not the DCPDs (see, e.g. LHO aLOG 28820). We merely used the replacement of the entire OMC breadboard (necessary because the burned mirror is a part of the monolithic structure) as a target of opportunity to install high quantum efficiency PDs (see LHO aLOG 28807).

Sorry about the confusion!

Quick pictures from yesterday's install.

From the trend data, it looks like the noise eater signal started misbehaving about a week ago.  The photodetector monitoring
the output power of the NPRO seems to be still working.  I would be inclined to check the field box that the
H1:PSL-MIS_NPRO_RRO_OUTPUT is attached too, as there maybe a large DC offset in that monitor signal's output.

We've also talked about seismic watchdogs a bit and why the ISIs trip after the isolation loops are shut off by the guardian. Both ETMs are in damped right now so we set the T240 threshold to 100 counts, and sure enough, the T240s started counting saturations, but did not trip the watchdog. Attached plot shows the T240 saturation counts, threshold and ST1 WD mon state. The dip on the top left plot is where we reduced the threshold, the spike on the bottom left is where the model started counting T240 saturations, and the flat line bottom right shows the watchdog didn't trip. This is as it should be.

However, what I think I've seen during ISI trips before, is the ST1 T240s saturate, ST1 trips and ST2 runs for a little bit then trips. This results in ST1 getting whacked pretty hard. I'll try to see if that's what happened with this earthquake.

J. Kissel, inspired by conversation from S. Cooper, S. Dwyer, J. Warner

I'll remind folks that this collective SEI/ SUS watchdog system has been built up sporadically over ~10 years in fits and spurts as reactionary and quick solutions to various problems by several generations of engineers and scientists. Also, the watchdog system is almost entirely designed only to protect the hardware from a software failure, and never designed to combat this latest suggestion -- protecting the hardware from the earth. So I apologize on behalf of that history at how clunking and confusing things are when discussing what to do in that situation. 

Also, I'll remind people that there are three "areas" of watchdogs: 
    (1) in software, inside the user model -- typically defined by the subsystem experts
    (2) in software, inside the upper level iop model -- typically defined by CDS software group, with input from subsystem experts
    (3) in hardware, either in the AA/AI chassis, or built into the analog coil drivers -- typically defined during initial aLIGO design phase

In my reply here, I'll only be referring to (1) & (2), though I still have an ECR pending approval regarding (3) -- see E1600270 and/or FRS Ticket 6100.

With all that primer done, here's what we should do with the suspension user watchdogs (1), and not necessarily just for earthquakes:
    (a) Remove all connection between SUS and the ISIs user watchdogs. The independent software watchdogs (2) should cover us in any bad scenarios that that connection was designed to protect against.
    (b) Update the RMS system to be *actually* an RMS, and especially, one that we can define a time-constant. The RMS system that is currently installed is some frankenstein brought alive before bugs in the RCG were appreciated (namely LHO aLOG 19658), and before I understood how to use the RCG's RMS function in general. The independent software's watchdog (2) is a good role model for this
    (c) We should rip out all USER MODEL usage of the DACKILL part. The way the DACKILL used across suspension types and platforms with many payloads is confusing and inconsistent. Any originally designed intent of this part is now covered by the independent software watchdog.
    (d) Once (b) is complete, we should tailor the lower the time-constants and the band-passing to better match the digital usage of the stage. For example, the worst that can happen to a PUM stage is getting sent junk ASC and Violin Mode Damping control feedback signals when the IFO has lost lock, but the guardian has not figured it out and switched off control.
    (e part 1) Upon watchdog trip, we should consider leaving the alignment offsets alone. Suddenly turning off alignment offsets often causes just as much of a kick to the system as what had originally set off the watchdog. HEPI has successfully implemented such a system.
    (e part 2) We should re-think the interaction between the remaining USER watchdog system and the Guardian. Currently, after a watchdog trip the guardian state immediately jumps to "TRIPPED" and begins to shut off all outputs and bringing the digital control system to "SAFE." 
    (f) Add a "bypass" feature to the watchdog such that a user can request the "at all costs, continue to try damping to top mass" in the case of earthquakes.

I'm attaching some more plots of what happened to the ISIs during this earthquake. The first plot is the saturation count time series for all seismometers and actuators for the test mass ISIs. All of the chambers saturated on the Stage 2 actuators first, this is the first green spike. This tripped the high gain DC-coupled isolation loops, and probably cause Stage 2 to hit it's lockers. The watchdog stopped counting all saturations for 3 seconds (by design), then immediately tripped damping loops on the saturated L4Cs or T240s. I'm not sure why the GS13s don't show up here.

The second plot I attach shows how long the ETMX was saturating different sensors. The L4Cs were saturated for about 45 seconds, the T240s and GS13s were saturated for minutes. The L4Cs never had their analog gains switched, but the chamber guardian should have switched the GS13s automatically. For this reason, if we increase the pause time in the watchdog (between shutting off the isolation loops and full shutdown), I think this shows that for this earthquake the ride-thru time needs to be more than 45 seconds.

Do you think you should write an incident report on this?

No, I don't think this warrants an incident report, since I believe this is a laser burn which broke the glass, not some other "accident".  It does warrant an FRS however, since we will likely need a better "fix" to this.  Further details:

On Friday, Keita and I inspected this broken beam dump on the North side of the HAM6 table in chamber.  I have more pictures attached below.  When I carefully pushed the 2 pieces of broken glass back together (like a puzzle) on the mount, I could see a hole of missing glass (PIC 1 and 2 below).  As well, there is a shard of black glass sitting on the table about 8 inches in front of the beam dump, maybe "launched" from the hole site of the glass piece (PIC 3, circled in BLUE).  As well, there are 2 other burn marks on this same piece of black glass off to the left visible in the first pics.

Sheila is going to help me with another round of inspections here and help determine beam propogation.  We will also look into timing with potential shutter/toaster fussiness, and HAM6 pressure changes to hone in on when this may have occured since April 2016 when it was deemed healthy.

While inspecting HAM6 last week, I also looked a bit at the "toaster" fast shutter.  A quickish look it's wires did not reveal any burn spots.  Pictures attached.

As well, there is another black glass beam dump in the NW side of the table, close to the viewport which shows what appears to be some burn markings (last Picture).  Likely another FRS addition.

For further follow up on the broken beam dump -- identified as the OMC REFL Beam Diverter dump -- check out LHO aLOG 38998 and FRS Ticket 9196.

Regarding the dump which has a "glancing blow" burn mark, that's the dump catching the OM3 TRANS beam. Indeed, the burn mark appears on the *outside* of the functional part of the dump. So, I don't think there's a need for action there (and hence no need for an FRS ticket).