I have been investigating the seismic system watchdogs. We want to know if the watchdogs threshold amount of motion to shut down the controls of the system can be relaxed. As a first step in doing this I found the threshold values for each of the suspension watchdogs. The channel and the corresponding threshold are shown below.
H1:SUS-ITMX_M0_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-ITMX_R0_WD_OSEMAC_RMS_MAX 10000.0
H1:SUS-ITMX_L2_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-ITMX_L1_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-TMSY_M1_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-SRM_M1_WD_OSEMAC_RMS_MAX 15000.0
H1:SUS-SRM_M3_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-SRM_M2_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-SR3_M1_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-SR3_M3_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-SR3_M2_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-PR2_M1_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-PR2_M3_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-PR2_M2_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-ETMX_M0_WD_OSEMAC_RMS_MAX 25000.0
H1:SUS-ETMX_R0_WD_OSEMAC_RMS_MAX 25000.0
H1:SUS-ETMX_L2_WD_OSEMAC_RMS_MAX 25000.0
H1:SUS-ETMX_L1_WD_OSEMAC_RMS_MAX 25000.0
H1:SUS-OMC_M1_WD_OSEMAC_RMS_MAX 12000.0
H1:SUS-IM2_M1_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-OM3_M1_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-ITMY_M0_WD_OSEMAC_RMS_MAX 13000.0
H1:SUS-ITMY_R0_WD_OSEMAC_RMS_MAX 13000.0
H1:SUS-ITMY_L2_WD_OSEMAC_RMS_MAX 13000.0
H1:SUS-ITMY_L1_WD_OSEMAC_RMS_MAX 13000.0
H1:SUS-BS_M1_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-BS_M2_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-IM1_M1_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-MC3_M1_WD_OSEMAC_RMS_MAX 25000.0
H1:SUS-MC3_M3_WD_OSEMAC_RMS_MAX 15000.0
H1:SUS-MC3_M2_WD_OSEMAC_RMS_MAX 15000.0
H1:SUS-RM1_M1_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-IM4_M1_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-OM1_M1_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-MC1_M1_WD_OSEMAC_RMS_MAX 25000.0
H1:SUS-MC1_M3_WD_OSEMAC_RMS_MAX 15000.0
H1:SUS-MC1_M2_WD_OSEMAC_RMS_MAX 15000.0
H1:SUS-ETMY_M0_WD_OSEMAC_RMS_MAX 16000.0
H1:SUS-ETMY_R0_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-ETMY_L2_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-ETMY_L1_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-TMSX_M1_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-SR2_M1_WD_OSEMAC_RMS_MAX 11000.0
H1:SUS-SR2_M3_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-SR2_M2_WD_OSEMAC_RMS_MAX 8000.0
H1:SUS-OM2_M1_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-PR3_M1_WD_OSEMAC_RMS_MAX 15000.0
H1:SUS-PR3_M3_WD_OSEMAC_RMS_MAX 15000.0
H1:SUS-PR3_M2_WD_OSEMAC_RMS_MAX 15000.0
H1:SUS-MC2_M1_WD_OSEMAC_RMS_MAX 18000.0
H1:SUS-MC2_M3_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-MC2_M2_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-RM2_M1_WD_OSEMAC_RMS_MAX 25000.0
H1:SUS-PRM_M1_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-PRM_M3_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-PRM_M2_WD_OSEMAC_RMS_MAX 80000.0
H1:SUS-IM3_M1_WD_OSEMAC_RMS_MAX 8000.0

Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 1984 seconds. TC B did not register fill. LLCV set back to 18.0% open.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 3264 seconds. LLCV set back to 35.0% open.

Summary: there seems to be a correlation between computer errors (for instance timing or IPC errors) with one or two types of blip glitches in DARM.

Explanation:

With the help of Dave Barker, in the last days I have been looking at several FEC_{}_STATE_WORD channels to find times of computer errors. First I used the minute trends to go back 90 days, and after that I used the second trends to find the times of the errors with an accuracy of a second (going back two weeks). I have been looking for error codes up to 128, where 2=TIM, 4=ADC, 8=DAC, 16=DAQ, 32=IPC, 64=AWG, and 128=DK. Finally, I generated omega scans of GDS-CALIB_STRAIN for the times of errors and found that they often show glitches.

Since there were some times that did not glitch, we are trying to track down how the coupling between the errors and the glitches is happening to see if it can be fixed or if the times can be vetoed. It might not be possible to fix the issue until we go into commissioning time at the end of O2, so DetChar might want to consider creating a flag for these times.

Investigations:

With the times obtained from the second trends, I have created tables of omega scans to see how the errors propagate and where the glitches appear. All those tables can be found here as html files, and the lists of times of errors for the last ~two weeks as txt files. So far, the model with the highest rate of glitches vs non-glitches is the susetm(x/y)pi.

I have been looking at some of these glitches, and the loudest ones coincide with small range drops. Also, the three glitches that had a strong residual in the subtraction of the MASTER signal from the NOISEMON signal (see aLog 34694) appear in the minute trends as times when there were computing errors (I haven't tracked those times down to the second though) in the h1iopsusey model.

17:44 Lockloss

Had had been degrading in range, BUT we have been in the middle of a windstorm where we are currently riding through sustained 30mph winds with gusts up to 50mph. 

Have taken Observatory Mode to WIND!

We have full gradient field data from the ITMX HWS archived back to March-2016. In order to see the thermal lens, we need coincident times between low noise HWS operation and a relatively high power lock-acquisition or lock-loss of the IFO.

By mining the archived data, I can see evidence for the point source going back to 12-March-2016 [the gradient field is relative noisy]. Unfortunately, there is no archived data earlier than this time.

TITLE: 04/07 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    Wind: 21mph Gusts, 15mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.61 μm/s

Looks like there was a bit of a wind storm starting at about 5am PDT (13:00utc) with a few minutes of steady winds of 45-50mph 30min before the H1 lockloss.
QUICK SUMMARY:

• 13:30utc (6:30am) PI Mode 27 briefly rung up
• 13:31-13:44 PI Mode 19 begins ringing up and alarming
• 13:53 LOCKLOSS
  • TJ had a misaligned ALSx in yaw.
• ~14:39 Talked to William P at LLO to let them know I'm on site.
• ~14:40 Saw a misaligned X-arm, so I tweaked ETMx, Green Arms could then lock, and then I tweaked the BS & PRM during a PRMI lock, and we were on our way to....
• 15:08 NOMINAL LOW NOISE, with the CSOFT Pit gain at 0.2 (instead of nominal 0.4 noted by Jim/Sheila last night)---so, this gain was walked to 0.4 by hand.
• 15:12 Rejoining L1 in OBSERVING (so, out of OBSERVING for ~80min)

TITLE: 04/07 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Corey
SUMMARY: Locked and Observing until the last hour or so and then PI mode 19 SDF diffs and a lockloss. I could not get ALSX to relock and was unable to bring up a camera to see how it was misaligned. The normalized power was ~.5 and moving ETMX could not improve it. Without being able to see what I was doing made it basically impossible. Corey came in early and Richard went to see what he could do as well.

• 13:35 Out Observing due to a SDF diff for PI mode 19.
• 13:48 Observing
• 13:53 Lockloss

A few min after I fixed the SDF diff it dropped lock, not sure of the cause yet.

PI mode 19 started to ring up a bit to cause the SUS_PI node to change some of the PLL filters causing an SDF diff. Mode 19 is not one of the modes that we regularly damp while at 30W, but it may have been excited from the wind. I went to clear the SDF diff but just trying to click the unmonitor button would not work, there was an exclamation point next to the "MON" but I'm not quite sure if that was suppose pop up a screen for me that didn't work remotely, or if it was something else. I managed to select the unmonitor all, with only that channel having an FM3 diff, and getting it to accept it that way. I'm worried that this may have accepted the entire filter bank, and there was another screen that I couldn't see that would allow me to choose what parts on that filter bank to monitor.

We are back to Observing and there is a screenshot below of the SDF diff.

Smooth sailing so far. The wind just jumped from about 5mph to 30. It is forecasted to get very windy today, so we will see this may be the beginning.

TITLE: 04/07 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 69Mpc
OUTGOINGING OPERATOR: Jim
SUMMARY: OPERATOR IS WORKING REMOTELY OF THE DURATION OF THIS SHIFT. I still have not recovered from my pinkeye and we could not find an emergency replacement, so I am logged in and will be monitoring from home. I will do as much as I can from home, and Corey offered to drive on site in the early morning if need be. I have contacted LLO and I am on the Control Rooms TeamSpeak channel if anyone needs to get a hold of me.

TITLE: 04/07 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 69Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Commissioning changes made staying locked difficult, but sorted now. TJ attempting to do his shift remotely (otherwise he's "out sick")
LOG:
3:30 UTC Noticed range was falling off & ASC was ringing up

4:00 UTC Lockloss, started trying to detangle commissioning changes from earlier, got locked again, but range deteriorated again and ASC range up again. Sheila helped me figure out that her CSOFT changes hadn't been added to the ISC_LOCK guardian, so CSOFT currently still has to be turned up by hand. We should sort this out tomorrow.

Brett, Jeff and I have been looking a little at Brett's suspension model, and trying to asses how our damping design (both top mass and oplev) is at high power, based on some conversations at the LLO commissioning meeting.  Today I turned off oplev damping on all test masses. 

Top mass damping:

The first attached plot shows the magnitudes of the top to top transfer functions for L, P, and Y for 3 different states (no damping and no power in the arms, top mass only damping and no power in the arms, and top mass only damping and 30 Watts of input power).  You can see that while it would be possible to design better damping loops to deal with the radiation pressure changes in the plant, there isn't anything terrible happening at 30 Watts.  I also looked at 50 Watts input power, and these aren't any large undamped features.  I'm not plotting T, V, and R because the radiation pressure doesn't have any impact on them according to the model.  

Oplev damping:

The next three plots show the transfer functions from pum to test mass pitch, in the optic basis, hard and soft.  One thing that is clear is that we should not try to damp the modes above 1 Hz in the optic basis, so if we really want to use oplev damping we should do it in the radiation pressure basis and probably have a power dependent damping filter.  The next two plots show the impact on the hard and soft modes. You can see that the impact of the oplev damping on the hard modes (and therefore the hard ASC loops) is minimal, but there is an impact on the soft loops around half a Hz.  DSOFT pitch has a low bandwidth, but we use a bandwidth of something like 0.7Hz for CSOFT P to suppress the radiation pressure instability.  

Krishna had noted that the ITMY oplev seems to have been causing several problems recently, (aog 34351), and this afternoon we think that this caused us a lockloss.  I tried just turning the opelv damping offf, which was fine, except that we saw signs of the radiation pressure instability returning after a few minutes at 30 Watts.  I turned the gain up by a factor of 2 (from 0.2 to 0.4), and this instability went away.  The last attached plot is a measurement of the csoft pitch loop with 2 Watts, no oplev damping, before I increased the gain to 0.4.  

The guardian is now set to not turn on the oplev damping, and this is accepted in SDF.  Hopefully this saves us some problems, but it is probably still a good idea to fix the ITMY oplev on tuesday. 

I've been looking to see if LHO needs to pursue better L2A de-coupling in the corner station suspensions to improve our wind and earthquake robustness. The good news is I had to look for a while to find a candidate, but I know better what to look for now, so I'll see what else I can find. Looking at a couple of recent earthquakes, I noticed that we seemed to lose lock when the IM4 TRANS qpd pitch hit a threshold of -.6. After talking to Jenne about it, we looked at other QPDs close by and it was immediately obvious that MC2 trans qpd pitch was being driven by MC2 M1 length drive. The attached plot shows the story.

Both plots are time series for and earthquake on March 27 of this year, where we lost lock at around 1174648460 UTC. The top plot shows the MC2_TRANS_PIT_INMON, MC2_M1_DRIVEALIGN_L_OUTMON and MC2_TRANS_SUM_OUT16. The bottom plot is the ITMY STS in the Y direction. The first 600 seconds are before the earthquake arrives and is quiet. The spike in the STS at about 700 seconds is the arrival of the P waves. This causes MC2 sus to  move more, but the MC2 trans sum isn't affected much. At about 900 seconds the R waves arrive and MC2 starts moving more and more, moving the spot on the qpd more and driving down the qpd sum. I've looked at the other pds used for asc and only IM4 trans and MC2 trans seem to move this much during an earthquake.