Jeff B.  Noted white screens on EY beckhoff channels.  We logged into the computer could not find any obvious errors besides the IOC being down.  The Beckhoff system was still running but the IOC that interfaces to EPICS had crashed.  It was restarted and connections came back.

   All subsystems reported no significant problems or issues. 

   ER9 run has finished. Vern thanked all who put in so much time and effort to make this run a success. Problems brought to light during ER9 are being addressed. 

   The Tuesday Maintenance window will be given over to a site wide effort to locate missing equipment. At this time, no other maintenance items are planned.  

I wanted to look at the digital camera integrated intensities of all PRM optics during the power increase, in an attempt to find lost power.

First step: find a camera exposure setting that does not saturate at 40W:

H1:VID-CAM13_EXP 3000   (PRM)
H1:VID-CAM08_EXP 1000 (PR2)
H1:VID-CAM14_EXP 1000 (PR3)

I already recorded the corresponding sum signals during the first attempt (UTC Jul 10 23:50 - Jul 11 00:30), but had to adjust the exposure settings along the ways. The interesting sum signals are:

H1:VID-CAM13_SUM
H1:VID-CAM08_SUM
H1:VID-CAM14_SUM
 

Once at 40W, I noticed that PR3 shows quite a bit off overspilling light below PR3 (and some above, but not on the side). By moving the H1:ASC-POP_A_PIT_OFFSET from the nominal 0.36 to 0.55, I was indeed able to reduce the total light seen by the PR3 baffle camera by 32%. At the same time the recycling gain and arm transmitted power went up, but only by 2% (PR GAIN from 28.4 to 29.0 / TR_X_NORM from 1080 to 1102). There is more going on than this...

=============================

Along the way I kept loosing the FSS lock. Not sure what changed this week, so I won't debug this one.

Intent bit set at 05:30 UTC. PI modes are not well damped but they're controlled for now. There are few weird things I noticed about some of the PI modes:

• ITMY MODE 2 (14979 Hz) will start off ringing up no matter what gain I used. The nominal gain setting for this mode is +300, I flipped the sign to -300 stopped it from ringing up (and managed to damp the mode). So I updated PI guardian accordingly. However, when I went to INCREASE_POWER (after a lockloss) the mode started ringing up again. So I flipped the gain back to +300 and returned guardian setting to what it used to be.
• ETMY_PI_OMC_DAMP_MODE6_RMSMON corresponds to 18041 Hz. However, the line is not visible in DARM ASD plot. I had to put faith into whoever design the monitor and blindly flipped damp settings around. Luckily it worked.
• ETMY 15009Hz has been causing both Jim and I problems today. I was able to finally keep the mode under control with +60deg (instead of -60deg which was the nominal setting)

As I was writing this alog, the ifo went DOWN because of ETM bounce mode (not sure which ETM, possibly both). I didn't put them on my StripTool since I was assume only ITMX has problem. Huge mistake...

ITMX bounce mode was ringing up so I flipped the sign of the gain. Unfortunately I accidentally flipped the wrong gain (instead of ITMX Bounce I flipped ITMX Roll). Likely caused ITMY Roll to ring up and blew the lock.

J. Warner, R. Schofield, (J. Kissel remotely)

Jim called me this morning around 10a PT informing me that the ETMX ESD was unresponsive, preventing any lock re-acquisition. We eventually found that one leg of the HV power supply to the HV ESD driver at EX was off; unknown as to why. I reminded Jim how to restart it, then the ESD driver came back to live as normal after hitting the ON/OFF button on the HV ESD driver itself (no need to disconnect/reconnect any cables this time).

Diagnosis timeline:
I logged in an found that the HV monitor channels were all zeroes in the bottom right corner of the SUS ETMX overview, even though the digital request was the usual large bias and linearization voltages on the signal quadrants. I also noticed that because these monitor channels were reading zeros, the ISC_LOCK guardian -- which was in the DOWN state -- was hammering the remote ON/OFF button (I thought we fixed this??). After requesting that the ISC_LOCK just CHILL (i.e. pausing the guardian node), I waited a bit to let the HV ESD driver's internal mirco-processor relax then tried to remotely restarted it again. 
No dice. 
I advised Jim to head down to EX so we could debug together there (he'd gone with Robert). He went immediately to the VEA to try the ol' plug-un-plug cable trick, and found that the physical ON/OFF button did nothing, and that some indicator lights had shown that a leg of the 430 V input was red / dark. So, "we" went out to the entry foyer where the two HV power supplies live, and he found one off. We turned it back on --
 - Flip the rocker switch up to ON (wait for boot cycle to complete)
 - On the keypad, type
    - VSET > 430 ([V] for the amount of output voltage) > Enter
    - ISET > 80 ([mA] for the internal current limit) > Enter
    - Output ON/OFF
and then Jim went back out of the floor, the ESD HV driver looked much more lively, so he hit the button and the box came to live and I saw the voltage monitor channels go to their usual values. 

The only thing I can think of that trips the HV power supplies for the ESDs is the vacuum system (and when it does, it usually trips both), so I did a cursory vacuum system check on the EX overview screen, and didn't see anything crazy (i.e. I saw a bunch of green lights, and both vacuum Pirani gauges were sitting happily at ~1e-9 Torr. We'll debug more on Monday when I don't have to use the fragile remote connection to CDS.

J. Kissel (Remotely)

I've taken the lock-loss opportunity to move the frequency of super-kHz calibration line we're using on PCALX to map out the high-frequency sensing function (like was done in O1; see e.g. LHO aLOG 24843) from 3001.3 to 3501.3 Hz. Same amplitude of excitation that Evan used yesterday when restoring the line after hardware injections (39322 [ct]; see LHO aLOG 28297). The settings changes have been accepted in the h1calex SDF system under both the safe and OBSERVE.snaps.

Lost lock and went to the Down state at 1152087840 GPS, July 09, 2016 01:23:43 PDT,  08:23:43 UTC. 

Around 5:45 UTC, DARM started to glitch about every two seconds. The period is not exactly 2 seconds (I think it's just a little bit longer) but it does look pretty regular. I checked that the Pcal spectra look fine, so that's not the problem.

Attached is the change in the DARM spectrum. I wanted to use GDS-CALIB_STRAIN, but I get something that looks like it has dynamic range issues. We need to look into why that's not working. In addition, I'm attaching a Omega scan of the glitches.

Started shift in observing mode at 40 W with Michael L. and Evan G. running injections

23:25 UTC nds0 crashed. The IMC_LOCK guardian node went into error thus taking us out of observing mode. The IFO stayed locked. nds0 restarted. Keita and Sheila changed something in the IMC_LOCK guardian code that fixed the error. I set the intent bit back to observing.
01:06 UTC Injections done. Strange moving bump appeared in DARM around 110 Hz. Bump rang done.
01:07 UTC Out of observing to let Jeff K. measure DARM open loop gain.
01:48 UTC Jeff K. done with measurements
01:50 UTC Back to observing
03:56 - 04:30 UTC Stepped out of control room

The IFO has remained in observing mode without issues since the last entry. The only thing to note is approximately seven ETMY saturations. I am leaving the site now.

J. Kissel, E. Goetz

We've taken another DARMOLG TF and PCAL2 DARM transfer function in order to see if / how the SRC Detuning Optical (Anti-)Spring is changing from lock-stretch to lock-stretch. The bad (but not necessarily unexpected) news: it looks like the optical (anti-)spring frequency is moving around. One can tell by looking at the lowest frequency data points in the .pdf attachments. The model has a spring frequency of 9.381 Hz and both measurements are divided by it to form the residuals on the right column of subplots. Note also that Kiwamu and Craig's more sophisticated parametrization for the optical spring (which includes some Q between in the anti-spring poles, see LHO aLOG 28274) is not yet included. Jul 1 data points are ~ +15% discrepant in magnitude, where as Jul 09 data points are ~ +25%. 

More thoughts on this (what to do about it, how to track it, tests to try and control / reduce it) after the weekend. 

The pre-processed sensing function has been attached here, such that whomever gets to it first, can reproduce the fit using Craig's code from LHO aLOG 28274.

Data sets live here:
${CalSVN}/trunk/Runs/PreER9/H1/Measurements/DARMOLGTFs/2016-07-09_H1_DARM_OLGTF_4to1200Hz_SRCTuned.xml
${CalSVN}/trunk/Runs/PreER9/H1/Measurements/PCAL/2016-07-09_H1_PCAL2DARMTF_4to1200Hz_SRCTuned.xml