Jeff B. warned me that we would likely be getting a lot of dust monitor alarms after the PEM system was reset this afternoon.  He advised that we can set arbitrary alarm values in the meantime and he will remedy them in the morning.  I have been setting the thresholds to 1000 minor and 10000 major as the alarms occur.

CP4 LTY250 pump level went into high alarm 20 mins ago.  Cell phones have received alarms.  Waiting to talk to Chandra or someone VE...

The current alarm is not the normal "glitchiness" that has been ongoing for the last few days - 20 mins ago this signal jumped from ~95 to 100 and is riding up there.  See attached.

Reloading some Guardian code knocked us out of Observe.  I took this opportunity to run a2l.  PI modes 27 and 28 started ringing up during a2l, so Terra and I tweaked some setting and beat them back down.

J. Kissel, J. Driggers, S. Dwyer, J. Bartlett, D. Barker

Today's (not that big a deal) rabbit hole: while looking for what was preventing the IFO from being OBSERVATION_READY, we found the IFO master node complaining that the IMC_LOCK gaurdian was not ready (i.e. the H1:GRD-IMC_LOCK_OK == False), even though the node was in its nominal state, managed correctly, and in EXEC. The key was that the IMC input power was just ever so slightly out-of-range of a recently added conditional statement that prevented the run portion of the ISS_DC_COUPLED state from ever completing:

        # Hack, since power_adjust() won't change the gain when it's less than 1dB different, 30Nov2016 JCD
        if self.counter == 0:
            if ezca['IMC-PWR_IN_OUTMON'] > 28 and ezca['IMC-PWR_IN_OUTMON'] < 32:
                ezca['IMC-REFL_SERVO_IN1GAIN'] = -8
                self.counter += 1

 -- and today, for the first time, we hit an a low of 27.9 [W], the self.counter never incremented, and the code never advanced because all future conditionals relied on that self.counter to have incremented. Jenne recalls that this conditional statement was added to account for the IMC-REFL_SERVO_GAIN adjustment that happens as the ISS figures out what power it wants to send into the interferometer today (see LHO aLOG 32267 for side conversation about that), and prevent SDF from complaining that this gain was set to -7 dB instead of -8 dB.

The solution: bring the incrementing of the self.counter out of the power check:

        # Hack, since power_adjust() won't change the gain when it's less than 1dB different, 30Nov2016 JCD
        if self.counter == 0:
            if ezca['IMC-PWR_IN_OUTMON'] > 28 and ezca['IMC-PWR_IN_OUTMON'] < 32:
                ezca['IMC-REFL_SERVO_IN1GAIN'] = -8
            self.counter += 1


This bug fix has been loaded, and committed to the userapps repo. 

Once the bug fix was installed, the code was able to successfully run, the IMC_LOCK status cleared, and the IFO happily went into OBSERVATION_READY.

TITLE: 12/07 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
    Wind: 13mph Gusts, 12mph 5min avg
    Primary useism: 0.28 μm/s
    Secondary useism: 0.31 μm/s
QUICK SUMMARY:  Ran through IA at the beginning of the shift.  We are back at Observing at 1:28 UTC.
 

As I was working on a different script I found that both X and Y camera can be talked to simultaneously. So I'm leaving both codes running again. If DIAG_MAIN guardian complains about HWS code stops running please do not comment out the HWS diagnostic code just to get rid of the message. I need to know if they stop running so we can try to troubleshoot it some more.

All of the dust monitor EPICS IOC's, the FMCS EPICS IOC, and the dewpoint EPICS IOC was restarted due to a reboot of the computer on which they were running.

J. Kissel (with information from S. Dwyer)

I have been blaming the rotation stage for the 5 [W] range of IFO input power from lock stretch to lock stretch, but it looks like I've been wrong this whole time. I attach a 25 day trend of the the power into the IMC, compared to 
 - the power transmitted from the PMC, 
 - the power reflected off the input of the PMC. 
 - the relative power diffracted by the AOM (servoed by 1st loop)
 - the 2nd loop Reference Voltage (servoed by 2nd Loop)

One can see 
 - the range of power into the IMC is 5 [W], with an average of 30 [W], which means a relative power range of 16%. 
 - the range of transmitted power from the PMC shows a similar range: 10 [W] or out an average of 60 [W]; a relative range of 16%.
 - the power changes are anti-correlated with the PSL diffracted power, as expected, but the channel appears to be poorly calibrated since it ranges between 2 and 6%. This is likely the cause of the power variations, and they are under reported.
 - the power reflected by the PMC shows some long-term trend over the 25 days, but no correlation with the lock-to-lock difference.
 - the second loop reference voltage seems to track the long term trend of the PMC REFL power.
 - though there may be some correlated in isolated events, the human-adjusted (and NOT every lock stretch) 1st Loop voltage reference is not correllated with the lock-to-lock stretch changes in power.

The fundamental problem -- the DC voltage reference for the ISS' 1st loop is not stable. 

Currently, we have some alarms on the diffracted power, but they're not very tight, and only designed to prevent saturation of the servo. The power fluctuations *could* be remedied with some hacky slow digital servo, or a tightening of the threshold on the diffracted power, but it seems like we should be able to get a more stable voltage reference.

the virtual machine which serves matlab licences is being rebooted. This also serves FMCS epics data, so RO alarms are being generated.

Ops Shift Log: 12/06/2016, Day Shift 16:00 – 00:00 (08:00 - 16:00) Time -  UTC (PT)
State of H1: IFO is down due to two earthquakes within about an hour
Intent Bit: Commissioning
Support: Jeff K., Keita, Jenne, Terra
Incoming Operator: Travis
Shift Summary:  After the scaled down maintenance window (due to maintenance work done on Monday), tried relocking. Found green power in low in both arms. Ran an initial alignment and relocked on the first attempt. Sat at NOMINAL_LOW_NOISE in Commissioning while Evan did some commissioning work on the ITM Reaction Chain alignment.
   Went to commissioning when Evan finished.
   Rode through a 5.8Mag EQ centered around Trinidad.
   Lost lock due to 6.4Mag EQ in Indonesia.
   Put IFO into DOWN until microseism drops below 1.0um/s.  
   
Activity Log: Time - UTC (PT)
07:00 (00:00) Take over from TJ
16:19 (08:19) Chris – Escorting pest control through LVEA
16:20 (08:20) MY_CP4_LT250_PUMP_LEVEL_PCT alarm – Cleared on own
16:34 (08:34) Take IFO to Commissioning for start of maintenance 
16:45 (08:45) Richard – Moving fiber bundles in MSR - (WP #6292)
16:45 (08:45) Chris – Finished at CS, Escorting pest control to Mid-Y and End-Y
16:46 (08:46) Christina – Going to End-X for cleaning
16:48 (08:48) Karen – Going into the LVEA to clean around Ham4, 5, 6, and high-bay
16:50 (08:50) Jim – Going to End-X to unplug the WAP
16:50 (08:50) Carlos – Going to End-Y to unplug the WAP
17:00 (09:00) Richard – Taking Scott M (LLO) into LVEA to look at vacuum racks
17:00 (09:00) Joe – Going into LVEA to check eyewash stations
17:05 (09:05) Cintas on site to service matts
17:10 (09:10) Kyle & Gerardo – Going into LVEA to work on PT180 - (WP #6377)
17:11 (09:11) Norco on site to deliver N2 to CP2
17:18 (09:18) Jim – Back from End-X
17:19 (09:19) Richard & Scott – Out of LVEA
17:31 (09:31) Lockloss – Due to maintenance activities
17:37 (09:37) Carlos – Back from End-Y – Reports the parking lot is very icy 
17:45 (09:45) Christina – Leaving End-X
17:46 (09:46) Joe – Out of the LVEA
17:46 (09:46) Chris – Pest control finished at ends and corner – Working down the arms
17:50 (09:50) Karen – Out of the LVEA
18:08 (10:08) Alfredo & Elizabeth – Going to Mid-Y to do inventory
18:30 (10:30) Dave – Going to Mid-X and Mid-Y to unplug WAP
18:44 (10:44) Joe – Going into LVEA to service first aid stations
18:47 (10:47) Norco leaving site
18:52 (10:52) Rana – Escorting tour in the LVEA (WP #6387)
18:57 (10:57) Richard & Evan – Focusing camera at SRM (WP #6371)
19:06 (11:06) Dave – Back from Mid-Stations
19:20 (11:20) Richard & Evan – Out of the LVEA
19:27 (11:27) Alfredo & Elizabeth – Back from Mid-Y
19:36 (11:36) Paradise Water on site to deliver water
19:40 (11:40) Dave – Power recycle LIGO file system server (WP #6388)
19:44 (11:44) Bubba – Inspection tour of LVEA
20:05 (12:05) Patrick – Doing sweep of the LVEA
20:10 (12:10) Reset Fiber Polarization on X Arm
20:12 (12:12) Kyle & Gerardo – Out of the LVEA
20:15 (12:15) Start to relock
21:31 (13:31) After Initial alignment – Relocked at NOMINAL_LOW_NOISE, 27.9W, 66.7MPc
21:31 (13:31) In Commissioning mode - Evan & Rana work on ITM reaction chain alignment 
22:00 (14:00) Damp PI modes 9, 26, 27, 28
22:14 (14:14) Rode through a Mag 5.8 EQ near Trinidad 
22:19 (14:19) Set mode to Observing
23:08 (15:08) Lost lock due to 6.4Mag EQ in Indonesia. Primary microseism up to 7.0um/s
23:09 (15:09) Put IFO into down until microseism settles down
00:00 (16:00) Turn over to Travis 

(see WP #6377) 

Having not asked for permission, I am now asking for forgiveness!  It was my intention to shut down the temporary pump setup @ BSC8 at the end of maintenance today.  Instead, I have left the setup running. Obviously, we will shut it down if/when directed to do so.  As is, we have just this morning entered in to the "pressure region of interest" for our long-term gauge drift data collection.  The nature of the problem doesn't lend itself to 4 hours per week of data collecting.  

Recall that ON/OFF tests of this setup while in a locked "Low Noise" IFO state produced nothing of interest to the PEM folks just prior to the start of O2.  

Not sure of the cause yet, everything seemed all good and normal. Running lockloss plots now though and I'll update if I find anything.

I used the lockloss2 script that automatically checks for sus saturations and plots them using the lockloss tool, and saw that one of the three locklosses (2016-12-05 16:02:42 UTC) in the last day or so was probably caused by a glitch on SR3.  The attached screenshot shows the timeline, there is clearly a glitch on the top mass of SR3 about 0.2 seconds before the lockloss.  

The dither outputs (which we use for the cage servo) don't show anything unual until after the lockloss, which means that this is not a cage servo problem.  Looking at top mass OSEMINF LF and RT are the two that seem to have a glitch first, at about the same time. 

I've added a lockloss template /ligo/home/ops/Templates/Locklosses/channels_to_look_at_SR3.txt  for any operators who have an unexplained lockloss and want to check if it is simlar to this one.