On Wednesday TJ and I tripped the ISI several times executing the HAM3 ISI GS13 gain switch in various ways.  I have several examples to look at but here is a comparison of one trip to a successfull switch of the gains using the SEI COMMAND perl script.

The first attachment has 20 seconds of the trip example next to the switch that does not trip.  It looks like there isn't anything different going on before the two cases: There are larger swings in X & Y in both and the other DOFs all look fairly similar too.  There is a larger glitch 5+ seconds before the trip in Z but that seems to be settled by the switch/trip time; these would be related to Reference Position ISO servoing.

In the second attachment, again side by side are the two cases zoomed in a bit closer (10 sec.)  The two stage switch by the perl script is evident with the gain switch doing more glitching and the whitening stage doing nothing.  There is a delay of the sensors wrt to the switching by the guardian which seems weird but I'm not looking at outputs from the filter bank as we have no DQ channels here...  These channels are the inputs to the blend bank after transforming to cartesian basis.   Again, nothing stands out as to why there is a problem with the guardian script.  We (TJ & I) did adjust the guardian code to separate the FMs switching like the perl script does but it still tripped the ISI.

The third attachment is a close zoom in on the ISI trip switch.  It is what it is...  Given that the Y and RZ are the dofs that ultimately go big, I'll suspect the horizontals but I could be fooled.

TITLE:    Dec11 DAY Shift 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC

STATE Of H1: Environmental

SUPPORT: Sheila, Jenne, Hugh, Evan

INCOMING OPERATOR: Nutsinee

SHIFT SUMMARY:

• Another day of record high microseism.

• Sheila and Jenne doing some workarounds for the glitchy beam splitter oplev

• no locking past PRMI

ACTIVITY LOG:

17:24 Kyle and Gerardo are going to EX and EY mechanical rooms.

17:54 Carlos to MY

18:08 Carlos back

18:10 Kyle returning to corner station towing trailer.

18:39 Kyle and Gerardo back to control room

22:18  Reset timing error on H1SUSETMY

22:27 LASER diode chiller indicator was RED on the Status screen. I added 125mL of water.

Kyle, Gerardo 

New ion pump had shut down with "excess arcing" error shortly after its initial start yesterday -> Restarted this morning and seems normal now -OK 

RichM pointed me to look at the RZ drive as a way to confirm that the Z 45mHz blend is not coupling T240 RZ into the platform.

The attached 24 hour trend at ETMX suggests the nice reduction of inertial Z motion on Stage2 GS13s does not come at the expense of increase drive to the RZ of Stage1.  Our z-RZ subtraction is doing well.  The Blend switch points are shown too.

TITLE: Dec 11 DAY Shift 08:00-16:00UTC (00:00-08:00 PDT), all times posted in UTC

STATE Of H1: Environment - µSeism extremely high

OUTGOING OPERATOR: Travis - short shift

QUICK SUMMARY: …..it’s Friday.

Title: 12/11 Owl Shift 8:00-16:00 UTC (0:00-8:00 PST).  All times in UTC.

State of H1: Unlocked due to microseism

Shift Summary: After Nutsinee's frustrating shift, I have had similar luck.  I made it as far as ENGAGE_ASC_PART3 twice, but while waiting there for the ASC loops to do their deeds, we lost lock.  Talked to Mike and he gave the OK to throw in the towel for the night.  Microseism is still above 1 um/s and winds are still 5-10 mph. 

Incoming operator: Ed

Activity log:  Attempting to lock all night.

No luck with locking so far.  Made it as far as DRMI_ON_POP once in several attempts.  Microseisem continues its upward trend above 1 um/s.  Winds have calmed from earlier but still ~10 mph.

TITLE:  12/10 EVE Shift 00:00-08:00UTC (16:00-00:00 PST), all times posted in UTC

STATE Of H1: Lock Acquisition/Environment

SUPPORT: Sheila, Jenne

Incoming Ops: Travis (failed to reach him so he's already on site)

END-OF-SHIFT SUMMARY: I've been trying to lock most of the evening. The shift started off with high wind (20-40 mph) then died down towards the end of shift (now ~10mph and below). Useism has been increasing (now at ~1.5 um/s). Locking was unsuccessful even with wind below 10mph.

Activity:

00:19 Kyle driving back from Y28

Mike suggested I should try again when the wind dies down to 0 mph. Wind speed is now ~10 mph ar EY. Without this kind of useism the ifo should have been locked without a problem.

The wind has died below 20mph but useism is getting higher and higher. ALS has become more stable that the wind is low but I still couldn't get pass DRMI locking.

Wind has already died down below 20 mph. I don't think this is wind problem anymore.......... Redoing initial alignment.

Just for fun, since the microseism is very high for LHO, I looked at the ocean wave activity near the Pacific Northwest coast. The map is a nice bright red, close to the coast, meaning 12+ m wave height.

RichM suggested we get the 45mHz Blends running on the Z axis for the ETMs.

After checking that our current version is in SVN, I saved a backup copy and then copied the blends from the X or Y dof with foton.  Attached are Trends with the transition from Quite_90s to the 45mHz Z blends fairly obvious.  Also see that there is pretty much no effect in RZ from the switch as evidenced on the optical lever.  If our Z to Rz subtraction was not too good, the lower blends might reinject tilt and show up on the oplev or some other IFO signal.  So far so good.

It probably should happen on the CS platforms as well.

I committed the ETM filter txt files to the SVN revision 12264.

I stopped the bake this morning -> The pump is still noticeably warm and will be for hours but I went ahead and removed the turbo hardware and "fired it up" this afternoon -> It is now valved-in to the BT as is the X2-8 ion pump but both of their controllers still need to be mounted in the end stations and their HV cables will need to be cut-to-length as both have a few hundred feet excess length!  

Ed, Sheila

Today we are having useism above 1um/sec, almost 2um/sec in the z direction, and wind speeds mostly between 15-35 mph with a few gusts up to 40 mph.

This morning Ed collected many examples of ALS locklosses, and together we looked at several of them.  

We found at least 3 different mechanisms-

X arm VCO runs out of range ( H1:ALS-X_REFL_CTRL_OUT_DQ, which is calibrated in um, reaches around 6 um)  An example is shown in the first attached screenshot. We found 4 example of this 19:05:22 UTC, 19:07:25, 19:44:09 and 19:48:12) 

DIFF tried to lock when the PLL is moving through its linear range rather quickly, ETMX L3 saturates before the offloading to L1 can relieve it. (2nd screenshot, 19:01:12 and 20:30)  

Lockloss while CM board gain is ramping up during COMM locking, large glitches from CM board saturate MC2 M2.  Only one example, 19:45:22

We have done a few things to try to mitigate the situation.  The most important probably was increasing the X arm PDH offloading to the suspension,  H1:LSC-X_ARM_CTRL_UGF from 0.02 to 0.08.  This matches the ugf used for the Y arm, which has not been dropping lock as often as the X arm.  

I also reduced the wait time that the COMM guardian  uses to check if the PLL is locked from 5 seconds to 2, this means less waiting for it to lock. Lastly I changed the step in the ramp of the CM board gain durring the engaging of ALS COMM from 1dB steps to 3dB steps.     

I've reverted the gain stepping back to 1dB since it didn't seem like an improvement after several locking attempts. 

TITLE:    Dec 10 DAY Shift 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC

STATE Of H1: Observing

SUPPORT: Sheila, Hugh,

INCOMING OPERATOR: Nutsinee

SHIFT SUMMARY:

• A combination of very high microseism (≥1µm/s) and high winds, for most of the day, made it less than possible for re-locking.

• Sheila did a lot of work on COMM gain stepping during less than desirable conditions

• Hugh changed a Z axis Stage 1 blend at EX on Rich Mittleman’s advice

• Looks like an IA is in order for the next shift.

ACTIVITY LOG:

16:43 Out of Observing while Livingston investigates noise. Robert is using the time to do injections.

17:09  Floor covering folks are on site. (OSB carpeting).

17:15  another carpet installer on site

17:27 Robert is going to wait for the carpet installers to unload pallets behind the corner station with the forklift before starting his injections.

17:31 Hugh out to LVEA/HAM1 area to investigate some audible HEPI noise that Robert reported.

17:35 Lockloss

18:40 Kyle out to Y-2-8

19:48 Kyle back from Y-2-8

21:47  Chris driving to MX to work in a fan room.

22:48 Kyle out of EY mech room and going to Y-2-8

23:07  kyle driving back to corner to get keys.

23:23 Kyle driving back down to Y-2-8.

The DCS C01 hoft frames are now available up to 1133178880 == Dec 03 2015 11:54:23 UTC.

See these alogs for further information:

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23966
https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=23263

MID-SHIFT SUMMARY:

After the lock loss that occurred at 17:35UTC by permitted incursions to the LVEA, the wind speeds shot up to the 35mph+ range and have been there since. µSeism is very high ( pushing on 2µm/s). I have been working with Sheila getting more acquainted qith the LockLoss tools. We aren’t missing out on coincidental time with Livingston as they are still investigating their noise /range trouble. EQ bands have even increased from ≈.09 to about .2 µm/s.