TITLE: 08/01 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Jenne, Jeff K., and Keita continued troubleshooting. Kara and I worked on damping ETMY mode 1 and 6. Eventually was able to damp with ETMY mode 1 set to -60 deg phase, .3 gain., and mode 6 set to 0 gain. Jenne is still troubleshooting at the end of the shift.
LOG:

00:07 UTC Timesh out of optics lab
00:15 UTC Ethan and Laurence to optics lab
01:01 UTC Ethan out of optics lab
03:28 UTC Timesh to optics lab

Jenne and Keita are troubleshooting. Kara is damping violin modes.

J. Driggers, J. Kissel, Y. Lecoeuche, K. Merfeld, P. Thomas

We continue along the path to recovery from our initial alignment reference loss. Attached are blow-by-blow notes and how-to details for perusal and future reference. I summarize the action below. I hand off the fight for alignment back to Jenne.

The afternoon, we were able to get up to and be stable at PREP_ASC_FOR_FULL_IFO (after a few more annoying tantrums from ALS X WFS DOF1 PIT, and a 12 minute "how do we fix THIS?" FSS freak-out). 
Once there, we went the ENGAGE_ASC_FOR_FULL_IFO, but immediately turned OFF ADS3 DOFs which get turned on in that state. 

Once there, Niko and I spent *several hours* slowly bringing in the ADS error signals, using a combination of the appropriate call to the "move_ARM_dev.py" script  to collectively / slowly move the arm cavity optics (while the corner station ASC followed) and adjusting PRM. 

While this was happening, Jenne and Kara were adjusting the position of the TMS in order to better center the beam spots on the red and green QPDs. Once better centered, they updated all green initial alignment offsets:
                        PIT          YAW
         Gr Camera Y =    239.3,        210.7
         Gr QPD Y A  =    -0.4 ,        0.76
         Gr QPD Y B  =    0.82,         0.01
         ITMY A2L =    -3.0          0.0
 
         Gr Camera X =    235.82,       332.9
         Gr QPD X A  =    -0.144 ,      0.571
         Gr QPD X B  =     0.709,      -0.073
         ITMX A2L =    -3.965,       0.055

We were recording "save points" along the way, and the last point before we started making explorative changes was 2019-07-31 22:07 UTC.

The arm / PRM moves were able to get all error signals but ADS3 (PRM) PIT close to zero at that save point. Any amount of moving PRM's alignment sliders had no affect on its supposed error signal

Jenne then thought that perhaps we're on "the wrong side" of the ITMY point absorber, so began moving spot positions to find this out (all the while continuing to update initial alignment references). 
We found that the ADS lines in DARM were indeed getting smaller by changing the spot position.
This confirms that we're in a collective global alignment that's hitting the other side of the point absorber.
However, in this new position, we lost the ALS lock of the green arm ... fishy.

We decided at this point there were three options, since we were essential finding out a new good spot position -- and doing it at 2W, which we don't want:
     (1) Move on to increase power and cross our fingers.
     (2) Restore initial alignment references and all sliders to "the last save point," and ... see what happens.
     (3) Restore initial alignment references and all sliders to the beginning of the day, and just cross our fingers that we get lucky again like last night.

We moved forward with option (1) first -- BUT we were too aggressive. While focused elsewhere, we hadn't noticed the ETMY MODE1 and MODE6 had rung up to all get out -- likely because in this awkward global alignment space, the phase and gain settings for this already problematic mode became wrong, and we were driving it up. As such, once we transitioned to DC readout, the DCPDs were well saturated, and we couldn't fix it in time.

So, we moved on to option (2). While down, Patrick and I reverted everything to the last save point.

After doing so, we were able to quickly and easily get back up to PREP_ASC_FOR_FULL_IFO, repeat the steps to get through ENGAGE_SOFT_LOOPS mentioned above (though thankfully now with the saved good alignment sliders, we didn't have to go through the excruciatingly slow reduction of ADS error signals), and now Kara and Patrick focused on damping ETMY MODE 1 and 6.
We spent a god 30 minutes there, damping the violins, but power buildups were wandering all over the place under the influence of the SOFT loops.

We were some of the way through getting the modes under control (probably needed another factor of 5 in peak ASD), when I think we wobbled just that much too far in power build-ups and we lost lock.

I'm handing over to Jenne.
 

TITLE: 07/31 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
    Wind: 12mph Gusts, 6mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.07 μm/s 
QUICK SUMMARY: Jenne and Jeff K. are troubleshooting at ENGAGE_SOFT_LOOPS.

TITLE: 07/31 Day Shift 15:00 – 23:00 (08:00-16:00), all times posted in UTC

STATE of H1: Corrective Maintenance

INCOMING OPERATOR: Patrick

SHIFT SUMMARY: 6.6 EQ knocked us out early in the morning. After that we began troubleshooting the issues from after the camera was bumped yesterday. This started with why we were dropping out at TR_CARM, which Jenne solved. Then we briefly had some issues during DHARD_WFS, which is also theoretically solved. We then spent some time bringing down the ADS error signals before we move past PREP_ASC_FOR_FULL_IFO. Currently we are at ENGAGE_SOFT_LOOPS.

LOG:

15:00 (08:00) Start of shift

15:06 (08:06) Ethan to Optics Lab

15:08 (08:08) Timesh to Optics Lab

15:20 (08:20) Timesh out of Optics Lab

15:26 (08:26) Lockloss from 6.6 EQ near Vanuatu

15:44 (08:44) Ethan out of Optics Lab

15:50 (08:50) Timesh to Optics Lab

16:14 (09:14) Matt to CER

17:36 (10:36) Ethan to Optics Lab

17:45 (10:45) Ethan out of Optics Lab

17:58 (10:58) Karen to H2

18:16 (11:16) Karen out of H2

18:30 (11:30) Gerardo to MX -- survey QDPAD

18:46 (11:46) Ethan to Optics Lab

18:54 (11:54) Gerardo back from MX

19:00 (12:00) Dripta, Laurence to Optics Lab

19:28 (12:28) Fil, Timesh, Matt to Optics Lab

21:30 (14:30) Gerardo to MX

21:35 (14:35) Dripta to Optics Lab

22:01 (15:01) Dripta out of Optics Lab

22:28 (15:28) Gerardo back from MX

23:00 (16:00) End of shift

While we were working on engaging ASC (see midday mini update alog 50947), we lost lock quite suddenly.  I had my locking screen open, and saw that the LSC-DARM gains had been set to zero, and the yellow ramping triangles were active.  I didn't really have time to figure out what to do, and then we lost lock suddenly (which makes sense if we stopped controlling the DARM length). 

In the attached ndscope screenshot you can see that the POP DC has good buildups, then the DARM1 and DARM2 gains are set to zero (this EPICS channel doesn't capture the fact that they were ramping down, just that the setpoint went to zero).  After that, we lost lock according to the buildups, and then ISC_LOCK noticed and started the DOWN sequence.

This is a problem.  If this is the only time it's ever happened, fine.  But, we should look to see if this is what has been causing our mysterious locklosses the past few weeks.

[Niko, Jenne, JeffK]

After a lucky 11 hour lock stretch, an earthquake kicked us out. Now, we're finding ourselves having to pick up the hunt for ITMY alignment reference that produces a robust acquisition sequence where we left off yesterday (see LHO aLOGs 50935, 50939) after the ITMY green camera got accidentally bumped (LHO aLOG 50915).

• CARM_TO_TR problem [problem (2) from LHO aLOG 50936]: The reason that we're failing at this state is that it wants to use signal from both end stations' red transmon QPDs, but the Yarm QPD doesn't have sensible signal.  We confirmed that we are able to get through this part of the sequence with just the Xarm QPD (although this made later steps wonky, and we lost lock).  Next time, we reduced the CARM offset a little more (LSC-TR_CARM_OFFSET from -0.8 to -1.2; made change permenant in ISC_LOCK guardian) so that there was more signal on the Y_B_QPD.  It's still on the edge, but now there is enough light to make it have a sensible signal, and we are able to get back to PREP_ASC_FULL_IFO.  I suspect / hope that this will be solved if we are able to reset the green initial alignment setpoints more properly today (and maybe also pico the IR QPDs).
• Engaging ASC for full IFO by hand [to address problem (3) from LHO aLOG 50936]: Performing what Keita had apparently done for Patrick last night after Jeff left -- "Keita stepped slowly through steps starting at PREP_ASC_OR_FULL_IFO;" last line in LHO aLOG 50939 -- when we're able to get to it, we have been successful engaging the non-ADS loops in the main state of ENGAGE_ASC_FOR_FULL_IFO, and were walking PRM in yaw and getting good buildups.  We will try that again after our next acquisition.

We're still having intermitten issues with problem (1) from LHO aLOG 50936 : ALS X WFS1 PIT pulls the Xarm green lock off in to the weeds [though now not during LOCKING_ARMS_GREEN, but in the middle of DRMI acquisition], so PRMI/DRMI alignment iterating is slower than we want.

Stay tuned!

Ops Shift Transition: 07/31/2019, Day Shift 15:00 – 23:00 (08:00-16:00) - UTC (PT)

State of H1: Locked

Intent Bit: Observing

Weather: 0-3mph wind

Primary 0.03 – 0.1Hz: 0.01 um/s

Secondary 0.1 – 0.3Hz: 0.1 um/s

Outgoing Operator: Jeff

Quick Summary: Been locked and Observing for 11 hours.

   The IFO remains locked and observing as it has for the past 7 hours. The current range is 114.7Mpc. 

   Given the difficulties in relocking after the Tuesday maintenance window I decided not to risk a state change and to leave the SEI_CONF in EARTHQUAKE mode. This proved to be of benefit as there have been two additional larger earthquakes and several smaller one during the first half of the shift.  

   Added 100ml water to the Crystal chiller. Diode chiller water levels are good. Filters look OK. I have new filters on order. When they arrive will setup a WP to swap out the yellowed Crystal chiller filters.  

?

Ops Shift Transition: 07/31/2019, Owl Shift 07:00 ? 15:00 (00:00 -08:00) - UTC (PT)

State of H1: Locked at NLN, with a range of 116.1Mpc

Intent Bit: Observing

Weather: ?Skies are mostly clear; winds are a Calm to Light Breeze. Temperatures are in the mid-70s.?

Primary 0.03 ? 0.1Hz: 0.06um/s

Secondary 0.1 ? 0.3Hz: 0.05um/s

Outgoing Operator: Patrick

Quick Summary: IFO is in Earthquake mode for mag5.9 EQ in El Salvador. The microseism is still a bit high to switch back to WINDY mode.??

TITLE: 07/31 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Spent the majority of the shift working with Jeff K. attempting to recover from the ITM camera move.
LOG:

23:20 UTC Observing
23:48 UTC Lock loss
23:48 UTC Rick and company to optics lab
23:51 UTC Kyle back from mid stations
00:48 UTC Rick and company done in optics lab
01:18 UTC Timesh out of optics lab
04:01 UTC Observing
04:31 UTC GRB E345536. Fermi 586240252 TRIGGER_DUR:     0.016 [sec] Stand down
Set INJ_TRANS to INJECT_KILL
05:14 UTC Set INJ_TRANS to INJECT_SUCCESS
06:03 UTC SEI_CONF to EARTH_QUAKE for earthquake from El Salvador


Relock attempts: [Jeff K., Patrick T.]

1.
Ran CHECK_MICH_FRINGES and PRMI_ASC
Skipped SHUTTER_ALS
Lock loss from ENGAGE_ASC_FOR_FULL_IFO

2.
Lockloss from LOCKING_ARMS_GREEN, WFS drove ALS X off

3.
Turned off ALS X WFS DOF 1 PIT during LOCKING_ARMS_GREEN, waited for WFS to converge, then turned it back on
Skipped SHUTTER_ALS
Lock loss from ENGAGE_SOFT_LOOPS

4.
Lockloss from LOCKING_ARMS_GREEN, WFS drove ALS X off

5.
Turned off ALS X WFS DOF 1 PIT during LOCKING_ARMS_GREEN, waited for WFS to converge, then turned it back on
Stopped at DRMI_LOCKED_CHECK_ASC and moved ITMY in attempt to increase POP80. Increased by 1 count.
Skipped SHUTTER_ALS
Lock loss from ENGAGE_SOFT_LOOPS

6.
Setting camera references to current offsets
Running initial alignment
POP80 buildup now to 66
Lockloss from CARM_TO_TR

7.
Turned off ALS X WFS DOF 1 PIT during LOCKING_ARMS_GREEN, waited for WFS to converge, then turned it back on
Waiting longer for PRC2 Yaw to converge at DRMI_LOCKED_CHECK_ASC
Skipped SHUTTER_ALS
Lockloss from ENGAGE_SOFT_LOOPS

8.
Turned off ALS X WFS DOF 1 PIT during LOCKING_ARMS_GREEN, waited for WFS to converge, then turned it back on
Lockloss from CARM_TO_TR
FSS is red and TPD is flashing on ops overview. Hit DOWN and then INIT. TPD stopped flashing but FSS is oscillating. Eventually stopped oscillating on its own.

9.
Turned off ALS X WFS DOF 1 PIT during LOCKING_ARMS_GREEN, waited for WFS to converge, then turned it back on
Lock loss from ENGAGE_ASC_FOR_FULL_IFO. I think PRC2 Yaw is running away. I tried to turn it off and clear its history, but it ran away faster.

10.
Did not need to turn off ALS X WFS DOF 1 PIT during LOCKING_ARMS_GREEN.
Back to observing. Not sure what if anything did it. Keita stepped slowly through steps starting at PREP_ASC_FOR_FULL_IFO.

Trying repeatedly and unsuccessfully to make it past ENGAGE_SOFT_LOOPS.

J. Jones, J. Kissel, K. Kawabe, N. Lecoeuche, R. McCarthy, D. Sigg, T. Shaffer, J. Warner
IIET Ticket 11931

It was identified rather early that during a tune-up of ITMY's red camera, the known-to-be-problematic (see above referenced IIET) ITM Green Camera housing got bumped (LHO aLOG 50915).

A blow-by-blow of the recovery is attached, but in summary, we
    - bypassed the ALS ITMY green camera loop during initial alignment (because we knew the ITM hadn't moved since the last nominal low noise segment), 
    - had three patterns of lock losses, but got far enough in the acquisition sequence (ENGAGE_ASC_FOR_FULL_IFO) to *roughly* measure the new reported correct position on the ITMY green camera,
    - adjusted the ALSY camera offsets to be close to the right position, 
    - re-ran initial alignment, and
    - slightly adjusted the ITM further to get past the CARM reduction.

All in all, we changed the camera offsets by ~30 cts in pitch, and ~100 cts in yaw, and we needed to move the ITM -0.4 urad in yaw to get past the COMM to TR_CARM hand-off.

We started attempting recovery at around 11:00a local (~18:00 UTC), and hit nominal low-noise at ~4:00p local (23:02 UTC).

We have now lost lock again (for an unknown reason), and are testing out the robustness of the acquisition sequence with the new ITM position. Note, that, if successful again, we're going to adjust the ITM camera offsets *again* to better match the additional move.

More details to come.

Chandra R., Kyle R.

We were able to core two more anchor holes in the floor of the LVEA in preparation for the upcoming Turbo Station installations in October.  This makes 5 of 12 completed.  This will continue during maintenance days. 

J. Kissel, N. Lecoeuche

In order to reduce the number of red herrings during any given recovery, we followed the rabbit of hole of guardian warnings on the IMC_LOCK guardian which often states "IMC WFS not centered." 

This message appears, typically, after PSL incursions (rare), site power outages (rare), or computer failures (rare) when the IMC suspensions's or PSL Periscope PT alignments get lost. Because these events are rare, instituional memory loss causes confusion for folks when they see that error message and wonder if action is needed. However, once these alignments are roughly recovered (via reseting sliders on suspensions), the WFS, again typically, eventually recover their centering all on their own as the RF loops converge [there are no "DC centering" loops on IMC WFS, as there are for many of the the WFS systems]. 

Where is the threshold set and can we modify it to better reflect this knowledge?

This message is generated by a function in ISC_Library (WFS_DC_centering_servos_OK), which is called by another decorator function (gen_check_WFS_DC) in ISC_Library which creates the class (or state as we use it in guardian) called check_WFS_DC, which is called inside the IMC_LOCK guardian state "LOCKED." The decorator function, gen_check_WFS_DC, is used any where there are WFS. 
Inside the function, WFS_DC_centering_servos_OK, the threshold is hard-coded to be abs(WFS DC Signal) > 0.5 (for PIT or YAW of any WFS DC Signal) -- and note that by convention, WFS DC signals are normalized by their SUM, such that typical signals range from +/- 1.0, and centered goodness is typically considered in regions when WFS DC signals are less than +/- 0.1-0.2.

Because WFS_DC_centering_servos_OK is a function used all over the place, we can't solely change the threshold for the IMC WFS, and thus "just increasing the threshold."

So, the short answer is "no." 
However, with a bit more thinking, I bet we can catch this fish.

I attach a short (just this past time it happened during a PSL incursion), medium (including today's incursion and the past few days of lock losses), and long (all of O3) trend of the IMC WFS signals for reference.