Per JimW's request (alog 23032), I edited the ISC_LOCK guardian such that it throws notification messages when the ISS 2nd loop fails in the engagement. I have reloaded ISC_LOCK and checked the code into the SVN.

The notification messages will be displayed (1) if the IMC_LOCK guardian falls back to the LOCKED state (which is the programmed failure sequence) and/or (2) if the engagement process takes more than 4 minutes. The below are the new version of the ENGAGE_ISS_2ND_LOOP state. The red lines are the ones I newly added.

* * * * * * in ENGAGE_ISS_2ND_LOOP * * * * * *

    def main(self):
        nodes['IMC_LOCK'] = 'ISS_ON'
        self.wait_iss_minutes = 4.0 # in [min]
        self.timer['ISSwait'] = self.wait_iss_minutes*60 # in [sec]

    @get_watchdog_IMC_check_decorator(nodes)
    @nodes.checker()
    def run(self):
        # notify when the ISS fails. 2015-Nov-3, KI
        if nodes['IMC_LOCK'] == 'LOCKED' or nodes['IMC_LOCK'] == 'OPEN_ISS':
            notify('!! 2nd loop engagement failed !!')    
        # notify when the ISS is taking too many minutes. 2015-Nov-3, KI
        if self.timer['ISSwait']:
            notify('ISS spent more than %d minutes. Check ISS'%(self.wait_iss_minutes))
        # if IMC arrives at the requested state, move on.
        return nodes['IMC_LOCK'].arrived

I reset the PSL power watchdog at 16:25 UTC (8:25 PST).

TITLE:  11/3 Owl 8-16 UTC

STATE Of H1:  Observing

SUPPORT:

SHIFT SUMMARY: Lost lock early on, switched blends around, relocking had some issues

ACTIVITY LOG:

9:30 Lockloss, cause not clear

9:30-11:00 Relocking, switched blends a couple times, ultimately left ISI's in 90mhz blends, seems to have made some low frequency issues a little better

11:00 Relocked, back to observing..

 Transition Summary:

Title:  11/03/2015, DayShift 16:00 – 00:00(08:00 – 16:00) All times in UTC (PT)              

State of H1:16:00 (08:00) The IFO is locked at NOMINAL_LOW_NOISE, in Observing mode

Outgoing Operator:Jim

Quick Summary: Preparations for the maintenance window.

On Tuesdays at 9:05 AM and/or 9:06 AM Pacific Time, we will have regular testing of the control room alerts at Hanford, WA.

Events will be type Burst CWB2G and will be labeled H1OPS, L1OPS, and DQV.
All events will also have the log comment 'This is a fake event for testing the control room alerts.'.

Lost lock ~2:30 hours ago, took 1:30 hours to get it back. I had initially switched BSC blends to 90 mhz blends because useism was down and winds were higher. I also suspected the 45mhz blends were causing the low frequency motion seem in some of the CR FOMs. After not making much progress, I tried switching back (to the 45mhz blends), but ALS-x then refused to stay locked. We are now back to observing, with 90mhz blends. The occasional bursts of low frequncy motion we were getting on IMC-F & (I think) ALS-Y Refl (or one of the Drive Aligns), and the ASC traces have not returned. Useism has been below .4 microns my entire shift, winds variable but below 20 mph.

TITLE:  Ops Eve Shift, 00:00-08:00UTC (16:00-23:59 PDT), all times posted in UTC" 

STATE Of H1: Locked, on it's way to Low Noise, in Observe

SUPPORT: Keita and Kiwamu

SHIFT SUMMARY: Locked - something rang up - lockloss - relocking issues - back to Low Noise

INCOMING OPERATOR: Jim

ACTIVITY LOG:

- IFO locked almost exactly 4 hours

- 04:13UTC - lockloss after something rang up, visible on FOMs, did not get a chance to investigate

- relocking, X arm VCO railed at -5V, and Keita and Kiwamu identified the issue, and Kiwamu fixed

- relocking, DRIM did not lock, and then PRMI did not lock

- initial alignment

- first attempt at DRMI resulted in lockloss/down

- second attempt at DRMI was successful

- transition from DRMI lock to Low Noise went well

- 07:54UTC - IFO back in Observe

Here is a latest version of our spot positions as a function of time. 

The spot colors represent the length of time that the IFO has been locked before the measurement was taken.  More blue is more cold, i.e. shorter lock duration.  The colors become more pink for longer lock durations, with anything greater than 30 min showing up as the same color pink.  

These spot positions are calculated from the P2L and Y2L gains, but now including the last coupling factor from L2 angle to L3 length, which is why the absolute values of the spot positions are somewhat changed from previous plots.  (I don't know if Marie has an LLO alog about the calculation, but she has updated the spot position calculator in the svn, and that's what I'm using.)

These plots do not have error bars, although one might hope that that helps explain why our ETM spots seem to move much more than our ITM spots.  However, the largest distance between spots for a single optic is about 2mm, which is not too bad.

[detchar whsitle team]

We see DARM whistles in H1 on October 30th. They are also seen in the REFL_SERVO channel. The plots below illustrate glitch frequency vs. PSL VCO frequency at the time of the glitch (color is log10(omicron snr)). They do not appear to couple below 2 kHz, and the whistle frequency that appears in DARM is very, very loud in REFL_SERVO.

Further investigation is ongoing for other channels and over recent locks for the last several days to try to pinpoint when this started.

Activity Log:All Times in UTC (PT)

16:00(08:00) - Take over from JIm

17:00 (09:00) Beam tube repair crew going to X-Arm 1/2 way between Mid and End stations

18:30 (10:30) Jodi, Christina, & Karen – Going to offsite warehouse

18:30 (10:30) Kyle – Going to X28 to recover equipment

19:08 (11:08) Kyle – Back from X28

20:30 (12:30) Jodi, Christina, & Karen – Back from the offsite warehouse

00:00 (16:00) Turn over to Cheryl

End of Shift Summary:

Title: 11/02/2015, DayShift 16:00 – 00:00(08:00 – 16:00) All times in UTC (PT)

Support:

Incoming Operator: Cheryl

Shift Summary:

 Good day shift. IFO has been locked in Observation mode, 22.3w, 74Mpc for the past 12.5 hours. Environmental conditions are good, with winds still a light breeze (4-7mph) or less, and microseism improving, (now between 0.4 & 0.3 um/s). The trend continues downward. There were 3 ETM-Y saturations during the shift.   

Posted below are the October temperature & humidity data from the 3IFO-Des cabinet in the LVEA and the 2 Dry Boxes in the VPW. Data for 3IFO-Des and DB1 looks relative normal. The DB4 data for October was croupt and has not been posted.  

   IFO has been locked at Low Noise in Observing mode for the past 8.5 hours. The range is currently 81Mpc.  Environmental conditions are good. Microseism is still a bit high, but is slowly coming down. All appers normal at this time.

Attached is 40 days of trends from the first Pressure Sensor on the Pump Station.  These all look fine.  There is a drop of PS1 & PS4 when I swapped some channels around for a study.  Hmmm, I don't understand why the power outage fault seen about 2/3 through the data don't show on PS2 & 3 in the corner...?  When the EndY came back after the power outage, there is a ~2 psi shift...hmm could be that the AOFF for that channel isn't in the database.  Hey though, don't ya just love those daily and weekly glitches!?

Bottomline, none of the Pump Stations have changed there output indicating the pumps are okay.

Attempting to get the IFO back into observing, but A2L tripped me up with a page or two SDF diffs, which I reverted. Now ETMY shows violin mode damping diffs. I'm accepting them but posting a screen shot of the changes, so someone who knows the whys can check.

For the second time tonight, I've lost lock because ISC_LOCK reached some state and decided it was time for a break. Each time, locking was finally going okay, no obvious issues, then Guardian reaches some state that I want it to wait at, and it then refuses to move on. No amount of, loading, pausing, execing, requesting higher states then switching to manual and going back to the current state will get it to move on. Only INITing, which then causes ISC_LOCK to go to DOWN.

TITLE:  "10/28 [DAY Shift]: 15:00-23:00UTC (08:00-16:00 PDT), all times posted in UTC" 

STATE Of H1:  Lock Acquisition

SUPPORT: Jeff K., Kiwamu, Jeff B. 

SHIFT SUMMARY: Not much happened during the day. The ifo stayed locked until toward the very end of the shift. Difficulty recovering from the lockloss due to many optics became misaligned. I restarted GraceDB script once today.

INCOMING OPERATOR: Patrick

ACTIVITY LOG:

15:25 Fil to Mid Y pick up EOM stuff

15:57 Fil back

19:04 Out of Observation to run DARM and Pcal open loop gain transfer function

20:13 Kyle to HAM3 after TF done. Cheryl to HAM12 area.

----------------Jeff Bartlett covering for me from 21:00-22:00----------------

21:33 Lockloss. 

There're few other things I couldn't make sense off his notes. Will post more later when I do!

-------------------------------------------------------------------------------------------

22:00 I came back to find the ifo have lost lock. Jeff reported ifo was locking in green but couldn't move on because the power wasn't high enough. The ALSX beam spot have moved significantly. Almost every optics I looked at moved by significant amount (including ITMs and ETMs but not TMS) and I wasn't sure where I should start touching. Everyone was out for the 3 O'clock meeting. I didn't notice that IMC was locking at the wrong mode.

23:00 Patrick took over.

Ps. I might have missed a couple of activities (outside the LVEA, low impact) during the chaos. Sorry about that.