• H1 has been locked for just under 13 hours, range 111Mpc.
• I put a new wind striptool on Video4, Ex, CS, and EY.

TITLE: 08/17 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 108Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 21mph Gusts, 16mph 5min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY: range is more variable tha usual, still, locked in Observe

• The OUT Gate was 1/2 way open when I arrived.
  • Ed took a picture with my headlights on the gate (will add)
  • I've noticed this before, and emailed Bubba, and this is the first time I've seen this happen again, so the frequency has gone down
  • What it means for the previous shift is that we can't guarentee that no vehicles were on site / driving down the arms

TITLE: 08/17 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 104Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:

H1 very glitchy. BNS range has trended (porpoised) downwards since it was locked 8 hours ago.
LOG:

00:07 Saw a "hump" in DARM between 100-200Hz

2:30 Exchanged the Wind Speed screen for te VE Overview screen (temporarily) as remote vieweing had become compromised.

I am unable to phone the site either via land line or Verizon Wireless.  Regardless, we need to have the Vacuum Site Overview available for remote viewing.  Someone has replaced our screen with one showing the wind speed. 

TITLE: 08/16 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
OUTGOING OPERATOR: Travis
CURRENT ENVIRONMENT:
    Wind: 18mph Gusts, 11mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.10 μm/s
QUICK SUMMARY:

Locked and Observing!

TITLE: 08/16 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 111Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:  With the BRS issues, smoking TVs, and phone system non-functionality, it has been an interesting day.  Thankfully we are back to Observing.
LOG:

15:44 Peter out

16:26 Niko, Ethan to PCal lab

18:19 Ethan to PCal lab

18:31 Ethan out

18:37 Kyle to MY

19:00 Kyle back

19:38 Fil to MY

19:58 Lockloss, PSL frontend crash

20:10 Gerardo to LVEA taking measurements

20:20 Richard to CER power cycling PSL IO chassis

20:57 Jim to EX checking out BRS

22:51 Observing

I'm putting a summary of the things we tried with BRSX today here. It might be fixed, but we need to monitor it overnight.

First thing, Patrick and I remotely restarted the BRSX beckhoff computer. This didn't resolve the glitching. 

Next, Hugh, Fil and I went to the end station. We checked nothing was hitting the box, the inside looked undisturbed. Fil and Hugh power cycled the 24V power to the beckhoff (not sure what all that involved) and I restarted the beckhoff computer again. Still glitching.

Later, I found that there were fluctuations on the reference spots on the CCD, as Jeff K put in the DCC earlier.

Hugh and I went to EX again, this time we pulled the power on the light source, then plugged it back in. While we were there, we also opened the BRS enclosure again and pulled the CCD of the top of the light pipe. After looking down into the light pipe, we noticed two pretty good sized, dead, bugs sitting on the viewport at the bottom of the light pipe.  I couldn't get my phone to focus on them through the beamsplitter. Since we couldn't reach them and didn't want to take the light pipe off, we put the CCD back on.

The BRS is more or less back in it's nominal state and I've been watching it for the last hour. It hasn't glitched since being restored after this last incursion, but it's still pretty rung up. Looking at trends from earlier, it seems possible that the glitches don't appear when the BRS is moving above some threshold. Could also be that power cycling the light source fixed it.

We should still monitor the BRS for at least overnight, before trying to run with it again.

For several days now h1sussr3 SDF was in ALL mode (denoted by purple block on overview screen, means that all channels are being scanned for sdf-diffs and not only those denoted as monitored). This means the chance of an SDF diff is much higher. I have switched it back to MASK.

At 13:03 PDT all h1psl0 models stopped running. I found that one of the 16bit DAC cards was not seen on the pci bus.

We power cycled both CPU and IO Chassis, the missing card reappeared. The IOP model did not run due to the IX Dolphin card in the CPU, but not configured on the fabric *.

Powered down CPU and removed IX611 card. IOP still wont run, its configured to use the card.

Removed pciRfm=1 directive from h1ioppsl0 and installed new IOP model. It started but we had a negative IRIG-B excursion which took about 15 minutes to clear.

Started user models, all is running and Peter verified correct DAC drives on all models.

* a description of how we got here. During the 18 Sept 2018 upgrade of Dolphin from DX to IX, h1psl0 was upgraded along with all the other IPC'ed systems. We then entered the series of Dolphin crashes, which took the PSL down each time, we quickly took the PSL out of the Dolphin fabric(%)by simply disconnecting its IX cable and taking h1psl0 out of the Dolphin manager's configuration. This kept h1psl0 stable and no reboot was necessary at that time. Fast forward to today, and h1psl0 had not been rebooted for 332 days, which is why we are seeing these startup problems today.

(%) In consultation with Keita, Daniel and Peter, the Dolphin IPC was used for ISS third loop PD readout from the end stations, and DBB Jitter transmission to LSC. Neither were needed for O3.

J. Kissel, J. Warner

In the spirit "lessons learned" -- how could we have caught the BRS EX problem (FRS Ticket 13417) sooner -- Jim reminds us that there are pre-made BLRMS EPICs channels of each BRS in the standard ground motion frequency bands, e.g. 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_30M 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_30M_100M 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_100M_300M 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_300M_1 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_1_3 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_3_10 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_10_30

The first attachment is an "All of O3" trend of these channels, and one can "clearly!" (in the "in retrospect, it's obvious!" sense) that the BLRMS in all bands gets constantly elevated above regular ambient ground motion on Aug 15 2019. 
Second attachment is a time zoomin, and an amplitude zoom out, of the past week.

The BRS rarely has extended (> 1 hour) periods in which all bands are elevated 5 nrad_rms, and during this glitchy badness, the RMS soared well above 20 in all bands.
It's not going to catch everything, but for starters, we suggest a DIAG_MAIN alarm that looks at the collection BLRMS channels, and if they're collectively seen above 20 for 1 hour, then we certainly will know to look there immediately.

Also, we plan on adding these BLRMS channels to both the BRS screens and to create a monitor screen similar to the SUS actuator saturations screen, the Violin Mode Monitor screens, and the ISI stage performance matrices, where there's a color-coded report of the BLRMS level shown to report good and/or bad levels.

Hopefully this'll help us four years down the road when this happens again!

J. Kissel, H. Radkins, J. Warner, [M. Ross remotely]
FRS Ticket 13417

As we continue to investigate the problems with BRSX glitching, we've removed it's tilt subtraction from the sensor correction path at EX (see above cited FRS Ticket and LHO aLOG 51309). Once we did, we were able to successfully get through the entire lock-acquisition sequence as normal -- hence the reported observation stretch in LHO aLOG 51324 and 51325.

As recommended by Michael we took a look at the *raw* raw BRS signal -- the fringe patterns as measured by the CCD. To do this requires killing the BRS c-code and logging in to Beckhoff ***, so we've done so and captured the video -- posted under G1901492. 
A frame-by-frame instantaneous plot of the fringes are on the left, and the corresponding long-term time-series of the calculated rotation signal from it is on the right. For the left plot, the lefter of the collection of fringes is the reference light reflection, and the righter is the reflection from the rotation sensor's suspended beam.

One can see at the start of the video, that 2 glitches have happened already in the past 500 seconds, one at ~150 seconds and another, smaller at ~250 seconds. 
Then, right at the start of the video (within 2 seconds of it), a new glitch starts, and one can see *all* the fringes -- the reference fringes a little more clearly than the that beam fringes drop in amplitude by (0.9 - 0.85)/0.9 = ~5%.
The fringe heights restore (and the glitch stops) at around 5-7 seconds. 
Best to download the movie and open in a player where you have the ability to control frame, since it's a subtle affect, best seen on repeat or flipping back and forth between YES glitch and NO glitch frames.

Where Michael's previous experience seeing this behavior was an actual insect "bug" walking across the CCD, this appears to be a more systemic problem, of either 
    - sensitivity drops in the CCD, or 
    - light level drops from the LED light source.

The investigation continues...

*** Reminder that we wouldn't *have* to kill the BRS c-code to look at diagnostic signals, if the recent attempt at a code upgrade hadn't failed for unknown reasons LHO aLOG 50178.

Another reminder -- this is the oldest BRS, installed in Aug 2014 -- see, e.g. LHO aLOG 13296.

VIDEO1 television has released its smoke and has been removed from the control room. 

OPERATORS:  This is the second CR FOM monitor that has burnt up in the past month (the last one was July 16), so something to keep an eye/nose on.

Investigating and restarting.

[The LHO H1 Team]

After a rough day and a half, we're back Observing!

Throughout this, we have reset our initial alignment setpoints, see first attachment.

Also, we have caught a tricksy situation with guardian that if we do an initial alignment and then the DRMI never has to run its down state (as happened just now, since we are locked on our first attempt!), the PRM M2 coil drivers will be left in their higher noise state.  Because we do not believe that we can switch it while locked (and we're not willing to risk a lockloss), we are leaving the PRM with the slightly higher noise coil driver state for this lock, and modifying guardian so that this cannot happen again. 

However, this means that next lock, we'll need to re-accept the correct values in SDF (the setpoint values in the second attachment are the correct ones that we'll have to re-accept).

plots of the PZT, Camera 18 (PMC Output), and Camera 29 (IMC_IN), from 15 July 2019 to 16 Aug 2019.

• CAM28 X total change: 330 to 328 = -2
• CAM28 Y total change: 157.8 to 159.7 = +2
• CAM29 X total change: 316 to to 310, then 317 to 314 = -6-3=-9
• CAM29 Y total change: 261 to 274 = +13

So Camera 28 changes with the output of the PMC.  In throey this beam should not change, however, the calculated centroid does, which is likely that the centroid of the PMC trans beam changed, however it can't be ruled out that the beam itself also changed.

Camera 29 sees the IMC_IN beam, and it can change with changes of the beam into the EOM, alignment changes of the EOM (who's mount loosesn over time, and it's been over a year since we checked these screws), and changes of the beam after the EOM, which can come from PZT power outages, and manual alignment.

Plots attached in the order of these dates of the changes seen on the PZT, camera 28 and/or camera 29:

• 15 July to 16 Aug: all changes
• 26 June 2019 - Camera 29, X and Y
• 17 July 2019 - Cameras 28 and 29, X and Y
• 22 July 2019 - PZT, Cameras 28 and 29, X and Y
• 23 to 25 July 2019 - PZT, Cameras 28 and 29 in Y
• 30 July 2019- Cameras 28 and 29 in Y

Many people (Sheila, Jenne, Jeff K, Keita, Travis, Ed, others...)

It seems like there are at least three problems right now.  The alignment problem seems nearly solved, but the difficulties we are having with ALS mean that it is hard to make progress. 

Alignment progress today:

• Earlier this morning when we managed to get past our ALS problems, we were loosing lock in the CARM offset reduction, due to the locklosses that we used to have trouble with when the recycling gain was low.  We modified the guardian to use different transmitted arm powers for the transition from TR CARM to REFL for CARM control, (see 43344 and 44075 ).  This means that the final value of the TR_CARM offset is a convient way to trend the how the recycling gain that we get from our inital alignment has changed over time, the better the recycling gain the more negative the value is (see the first attachment).  The recyclcing gain that we get from our inital alignment has been getting worse over the last few months, which has probably contributed to loosing lock durring the CARM offset reduction. 
• This morning Jenne and I were manually aligning the IFO at various points (Jenne was adjusting the PRC in DRMI and I was adjusting CHARD to maximize REFL when we were in the guardian state CARM_OFFSET_REDUCTION.  Then we were able to engage the ASC manually by stepping CHARD after some loops were on.
• Once we got the ASC engaged, we decided to try bringing the soft dofs manually towards the alingment where the beams are centered on the optics, using P2L gains of 0.85 and Y2L gains of 0, this improved all of our build ups at 2W, and gave us a recycling gain close to 48.  Jeff and Jenne used the green beams to fin the inital alignment references for these spot positions. 
• We lost lock, and Travis and Jenne did an initial alignment with the references set to bring the spots to the high recycling gain spot well centered on the optic.  This worked well for reaquiring lock, the DRMI build ups looked better than they have been, the DRMI ASC worked fine and didn't need to be manually adjusted, and we arrived at resonance with a recycling gain of ~47.  This seemed like a much more smooth acquisition, so we've left the initial alingment references there.  You can see in the attachment that the TR_CARM to REFL CARM transition is happening at a higher arm transmission as well. 
• Here is a summary of the locking attempts where we made it to engaging full IFO ASC today:
  • allowed ADS to DOF3 (ITMY) to come on with the spot position centered, and doing things manually moved ITMY to the "August" position, engaging the ETM spot servos with their August positions about half way through the process for ITMY.  This worked, but we lost lock after increasing the input power to about 15W, possibly because I had manually done something wrong with the SRC loops which didn't keep their error points 0 well durring the power up.
  • left A2L gains at their August values, acquired with the spots in the midle because that is the way the initial alignment was done, and allowed ADS to come on as it is normally written in the guardian.  This was rough, because CHARD P hit it's soft limiter, but we survived.  I think that we probably don't need to have the CHARD soft limiter on once CHARD has engaged, so it might make sense to turn that off before engaging any ADS loops.  (This is the lock where we made it as far as the ESD transition).
  • Tried the same approach, but this time when CHARD hit it's soft limiter we lost lock.
  • Set the A2L gains to be centered on the optic for the ADS engagement, this worked fine for the ITMY loop.  Then I set all the values to their "August" positions, and engaged the ETM loops at the same time.  This was too fast, and we lost lock.

ALS:

• The problem which has taken up the most time today is the ALS locking not working.  The problem seems to be with the X arm green locking alone, the arm drops lock frequently.  Jenne and Jeff noticed that the green WFS were picking up whatever glitches are happening and also misaligning the cavity, so they have disabled the WFS by setting the master gain for the X arm to 0.  This seemed to help but we still have plenty of locklosses.  We are also guessing that this is the reason why operators have been reporting that the green WFS for X aren't working well, so this problem might have been going on for a while now.
• Keita posted the observation that the X ALS laser has more glitches than Y, but this has been true for many months. 51306
• Getting tired of these locklosses, I went to the end station and tried reseting the laser current to 1.9, which was the value about a year ago, when the laser looked to be a bit more stable.  Ed and I couldn't get the wireless to work, so he remotely used the medm to help me try to set the temperature.  We didn't end up in a good position (looked like mode hopping), so we've reverted now.  Starting (and final) settings: 2.036A, 27.17C  trial settings: 1.908 A, 27.34C
• We have continued to have many ALS locklosses for the last 2-3 hours.

ESD transition lockloss:

• The one time we got to the coil driver state we made it through without a problem, unlike Corey's expirience last night 51288
• We made the transition from ETMX to ITMX ESD+ ETMY UIM+PUM without a problem, but lost lock later in this state (1249955066).  It looks like we just lost control of DARM (saturating the ETMY pum) in the fraction of a second before the PRCL gain is increased.