Hugh, Evan, Terra, Jeff B.  

   At 01:08 (18:10) both end station HEPI and IS WD tripped. Both Pump Controllers were down. Called Hugh. I took End-X and Evan and Terra End-Y. 

   Found OV3 error on VFD. After ramping down the servo, reset the VFD. Ramped the servo slowly up until differential pressure was over 62 and set the servo control back to Auto. System came back up OK. Fluid level were unchanged from last logged level. 

   Posted is a plot of the Diff pressure and control voltage. 

    

I have added a check to the LocklossShutterCheck guardian that will add to the guardian log a note whether the HAM6 GS13s saw a large kick or not.  Note that this does not in any way change the functionality of this guardian, in that it still requires an Operator to run and look at the plot, then manually reset this guardian. 

However, we should be able to look through the guardian log to ensure that each time we lose lock, the log notes that the GS13s saw a big kick.  When we are satisfied that this is a good measure of the shutter having closed at the last lockloss, we can consider using it as an automation metric in lieu of Operators manually resetting the guardian. 

I have set the threshold for "big kick" for the GS13 to be 30,000 nm/s (1nm/s / count, so 3e4 counts).  When the shutter closes fast, the GS13 sees a signal as large as 90,000 nm/s, but when the shutter closes slowly it only kicks the GS13s to about 10,000 nm/s.  The threshold of 30,000 nm/s should differentiate between a fast vs. slow closing of the shutter.  Normal GS13 signal is under 100 nm/s peak. 

Attached is the code, which is also checked into the svn.  The idea is (once we're ready to make this automated) that if we see the GS13 kick, the CheckShutter state will automatically go to the LowArmPower state, and allow the IFO to relock.  If there is any problem, including inability to fetch the data, the guardian will stay in the CheckShutter state, and will require human intervention.  One could also consider adding light level checks to this, but I think that's probably overkill given the other checks that we already have in place.

Sheila Evan Jenne Terra

The IMC length drive has large glitches when only the IMC is locked.  The first attachment shows the MC2 M2 length drive durring a quiet time on July 31st, and a glitchy time today.  There are glitches present in both traces, but they are much louder in the trace from today.  If you look back at data from late July, the glitches have intermittently been present even before we started having trouble locking, so this is probably not the main reason we can't lock. 

The second attachment shows 15 minutes around the time of the July 31st trace, when the glitches transition from being quiet to loud. 

Short version: The new shutter seems to be closing harder than before the vent. This is causing the ISI to trip. So far we have gotten around this by running the ISI in a low gain state, but this makes the table motion much worse above 1 hz.

After the shutter work, we started having problems with trips of the HAM6 ISI when the new shutter trips. This morning we put the ISI in the "Robust" isolated state, which uses low gain isolation loops with UGFs around 10hz. The normal loops have UGFs around 30-40 hz. For now this seems to have eliminated the tripping of the ISI, but the performance of the ISI is much reduced above 1 hz. First plot shows a comparison Y GS13s and ground STS for HAM6 for the two configurations. Red traces are with the high gain loops, blue traces are with the low gain loops. Below 1 hz they look roughly the same (low frequency diffs seem to be mostly in the ground for blue), but at 1 hz the high gain loop is doing ~100x better. Not surprising.

I've looked the GS13s in Z at shutter triggers before and after the vent and the shutter seems to be moving the table a lot more now. Attached plot shows a trigger from today with the 10hz loops (blue), yesterday with 30hz loops ( orange/ spiced cider) and Aug 1st with 30hz loops (red). I've looked over at least 3 events in each configuration and they look remarkably consistent, with respect to ISI configuration. Shutter triggers before the vent seem to have kicked the table less than they do now. Maybe this will reduce as the shutter breaks in?

We have some medium gain loops installed with slightly less gain that the high gain loops, tomorrow during maintenance I'll try testing those, but I doubt they'll be enough. Otherwise, we can come up with some loops with some better compromises between UGF and high frequency roll-off than the low gain loops. But the best option would be to figure out  what can be done to reduce the kick from the shutter.

TITLE: 08/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Commissioning work all day. HAM6 chamber manager is currently paused and the ISI is set to Robust Isolated.
LOG:

• 17:13 Evan H to LVEA looking for item.
• 17:18 Evan H out
• 17:22 Cleared ADC error on H1IOPSEIB3
• 18:18 Gerardo tp LVEA near HAM4 and HAM2
• 18:20 Sheila, Evan to LVEA to take IMC OLG measurement
• 18:34 Sheila, Evan out
• 18:39 Gerardo out
• 18:41 Fil to CER for measurement
• 19:04 Evan, Terra to EY t find item
• 19:20 Evan, Terra back
• 20:00 Jason to Optics lab
• 20:37 Jason out
• 21:00 Kyle to LVEA to poke and turn knobs at RGA
• 21:11 Kyle out
• 21:57 Sheila, Jenne to LVEA to check signals at PSL rack

Am baking Vertex RGA whilst HAM6 pumps run this week.  

1200 hrs. local -> Heating of Vertex RGA started

Comparison of Calibration Epics Values (Calibration parameters at t = t0) during O1 and ER9.

Param.        Description           2015/09/29         2016/08/03             2016/08/10 
---------------------------------------------------------------------------------------------------------

EP1           ~1/Atst0 (ftst)              2.2005e+16        2.1066e+16          4.9121e+17         

EP2          ~Delta C/(1+G)        0.9895                 0.9934                  0.9934         

EP3          1/Apu0 (fctrl)               2.1115e+16         2.1077e+16           2.1077e+16         

EP4          Atst0 (fctrl)                  5.4972e-17          5.6354e-17            5.6354e-17    
 
EP5          Apu0 (fctrl)                  4.7360e-17          4.7446e-17            4.7446e-17            

EP6          Cres(fpcal2)                1.1307e+06          8.9859e+05           8.9859e+05

EP7          D0(fpcal2)                  1.7599e+07          1.5871e+11           1.5871e+11    
 
EP8          Atst0 (fpcal2)               1.2178e-18           1.13718e-18          1.13718e-18          

EP9          Apu0 (fpcal2)               2.0898e-19           2.0235e-19            2.0235e-19

Kyle reported on Aug 12 that he thought there was a correlation between his RGA signal and locking attempts.

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=29079

On July 19 there was a programming change to the FPGAs;

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=28535

During this process there was a pressure burst in PT410 at the YEND. Dave  believes that as the system was brought back, dacs railed while connected to the ISI. This caused all actuators in the chamber to go to their maximum output. We believe this caused some heating which resulted in the pressure bump.

Perhaps Kyle saw pressure changes related to ISI activity?

Dont forget to get work permits in today for work tomorrow. Let's try to make tomorrow a short maintenance day so comissioners can get to work early.

SEI - HAM6 will trip with often the shutter now. HAM6 now runs in Damped for the beginning of locking then then switched later but is a WIP. SDF will be red.

    CS HEPI pump maintenance tomorrow.

SUS - HAM6 sus look good.

CDS - ISS outer loop work, FRS work.

    New version of GDS code (DTT updates).

PSL - PSL seems good. TCS has CO2 laser swap planned (Alastair on his way). BS OpLev investigation.

Vac - Fire up pump to bake out RGA at vertex, Yend worked well. A couple of days of pumping for HAM6

Facilites - LED lights along walkway need to be coiled up (broken?)

Ran through work permits.

Julia Kruck and I buried the high-sensitivity LEMI magnetometers this week near the vault.  This is at the approximate location for which trial spectra were taken (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=12525), and determined to be good enough to study subtraction of the effects of globally correlated fields due to, for example, Schumann resonances.

We buried them directly in the soil at a depth of about 18 inches. We used a bubble level to ensure that they were not tilted in Z, and aligned them with beam tube X and Y using lines constructed and measured on Google satellite images. We think that this alignment technique was more accurate than could be made with the simple compass that we had, though we made sure that the compass agreed with our alignment.  The connector to the X-axis magnetometer is 17m in the -X-direction from the  +Y corner of the electronics vault, and the magnetometer extends in the -X direction beyond the connector (see photos). The Y-axis magnetometer has its connector 1.22 meters +X and 1.22 m -Y of the connector for the X-axis magnetometer, so the closest parts of the magnetometers, the connectors, are 1.72 m apart. The connector  is on the +X or +Y end of the magnetometers so that the signals will have the proper phase relationship. 1.22 meters of cable is buried directly in the ground before the point where the cables come together to enter the conduit (yet to be installed) in order to damp any cable vibrations. The LEMIs lie between the lines of orange stakes that parallel them to either side (see photos). The electronics had not arrived so they have not yet been tested.

Absolute location: X-axis magnetometer  is 1020 m +X,  188 m +Y of the vertex

Robert Schofield, Julia Kruck

Evan, Sheila

This morning we had truoble with DRMI ASC.  THere are a few things that seem to have helped (in addition to intal alingment and some by hand alignment), and how the guardain is reliably getting to analog CARM. 

• The lock point of the SR3 cage servo was changed friday, changing it back to 931 (M3_WIT_P) and redoing SRC alignment seemed to help the SRC and MICH loops.
• Since the optimal yaw alingment of PRM seems to have changed quite a bit, I added an offset of 0.24 to POPA Yaw.  I also took out the lines in DOWN that were resetting this to 0 for each lock.
• Adjusting the current to the BS oplev laser reduced the glitching.
• The MICH ASC loops are now ramped on more slowly
• Evan increased the matrix element for REFL 135 Q to MICH by a factor of 2 (to 2.5) so that the MICH length ugf is around 10 Hz even when we are locked on 3F.
• This morning the mode cleaner was not relocking, and I was clearing the WFS history almost every lockloss.  This problem seems to have gone away for now.

Currently we are in a similiar situation as the last two nights, we can lock the interferometer but loose lock within a few minutes.  We have lost lock even while sitting on analog carm, we have been able to switch PRCL and MICH to POP sensors but suddenly lost lock about a minute latter. 

Our DRMI lock times have been a little long in the last 2 days, it seems like the beam splitter optical lever is glitching.  The attached screenshot shows a PRMI lock where the BS oplev damping is on for the first minute, you can glitches in both POP90 and POP18, when the oplev damping gain is ramped to 0 the glitches go away although there are some oscillations in the build up.  Turning the oplev back on brings back the glitches. 

After decoupling the pumping components used during the recent bake out of the Y-end RGA, I exposed the RGA to the Y-end vacuum volume, energized the filament and let it come into equilibrium for an hour or more.  I then let the RGA scan continuously with the multiplier (SEM) on for an additional hour or so while I gathered up my mess(es).  I periodically checked the scanning as I walked past the screen.  At one point, I noticed that the spectrum was changing rapidly towards the "dirty".  I monitored the scanning and noted that after reaching a temporary maximum, the amus which had increased then returned to near their original values.  

After consulting with the Jeff B. (the operator on shift), I feel that the observed changes in partial pressures were likely the result of IFO locking attempts as they coincide closely in time.  Perhaps something gets hot when the IFO is locked or when mirrors are steered?  

See attached scans 

The attached is just one instance and is fairly representative of the trips.  Most of the time, the ACTuator is called to blame, but as I reported in 29056, I'm not sure the Last Trip is absolutely robust.

The 2 seconds of full data show the ISI in Full Isolation (GRD STATE) when Stefan throws the shutter (FASTSHUTTER A MODE.)  You can see a fraction of a second later, the Actuator MASTER DRIVEs going very large.  All the actuators are contributing to the saturation counting as the threshold is only 32760 counts.  The watch dog trips near the end of the two seconds (WD MON STATE.)  It was going to happen anyway but I can't be sure why the WD tripped because none of the counters (iii SAT COUNT) actually reached the trip threshold.  This is a mystery to me, still.  The SAFE COUNT is 8192, 4096, and 10 for the ACT, GS13, and CPS.  None of the counters reach the SAFE COUNT level....  Regardless, the Actuators where going to reach it so that is another problem to figure out.

Unless the shutter softens its blow on the ISI, the trips are going to happen almost everytime.  For now, we are running with HAM6 in the DAMPED state.  There the HAM6 sees a few saturations on the GS13s when the Shutter triggers but literally just a few.  The HAM6 will be transitioned to ISOLATED by the ISC after the FAST SHUTTER is tested.

In an effort to eliminate the need to connect and disconnect shutter cables during vent we have moved the shutter logic controler to the rack from the table.  Also replaced the BNC cables with Yellow BNC cables so they can be easily identified.
See attachment.