J. Warner, R. Schofield, (J. Kissel remotely)

Jim called me this morning around 10a PT informing me that the ETMX ESD was unresponsive, preventing any lock re-acquisition. We eventually found that one leg of the HV power supply to the HV ESD driver at EX was off; unknown as to why. I reminded Jim how to restart it, then the ESD driver came back to live as normal after hitting the ON/OFF button on the HV ESD driver itself (no need to disconnect/reconnect any cables this time).

Diagnosis timeline:
I logged in an found that the HV monitor channels were all zeroes in the bottom right corner of the SUS ETMX overview, even though the digital request was the usual large bias and linearization voltages on the signal quadrants. I also noticed that because these monitor channels were reading zeros, the ISC_LOCK guardian -- which was in the DOWN state -- was hammering the remote ON/OFF button (I thought we fixed this??). After requesting that the ISC_LOCK just CHILL (i.e. pausing the guardian node), I waited a bit to let the HV ESD driver's internal mirco-processor relax then tried to remotely restarted it again. 
No dice. 
I advised Jim to head down to EX so we could debug together there (he'd gone with Robert). He went immediately to the VEA to try the ol' plug-un-plug cable trick, and found that the physical ON/OFF button did nothing, and that some indicator lights had shown that a leg of the 430 V input was red / dark. So, "we" went out to the entry foyer where the two HV power supplies live, and he found one off. We turned it back on --
 - Flip the rocker switch up to ON (wait for boot cycle to complete)
 - On the keypad, type
    - VSET > 430 ([V] for the amount of output voltage) > Enter
    - ISET > 80 ([mA] for the internal current limit) > Enter
    - Output ON/OFF
and then Jim went back out of the floor, the ESD HV driver looked much more lively, so he hit the button and the box came to live and I saw the voltage monitor channels go to their usual values. 

The only thing I can think of that trips the HV power supplies for the ESDs is the vacuum system (and when it does, it usually trips both), so I did a cursory vacuum system check on the EX overview screen, and didn't see anything crazy (i.e. I saw a bunch of green lights, and both vacuum Pirani gauges were sitting happily at ~1e-9 Torr. We'll debug more on Monday when I don't have to use the fragile remote connection to CDS.

J. Kissel (Remotely)

I've taken the lock-loss opportunity to move the frequency of super-kHz calibration line we're using on PCALX to map out the high-frequency sensing function (like was done in O1; see e.g. LHO aLOG 24843) from 3001.3 to 3501.3 Hz. Same amplitude of excitation that Evan used yesterday when restoring the line after hardware injections (39322 [ct]; see LHO aLOG 28297). The settings changes have been accepted in the h1calex SDF system under both the safe and OBSERVE.snaps.

Lost lock and went to the Down state at 1152087840 GPS, July 09, 2016 01:23:43 PDT,  08:23:43 UTC. 

Around 5:45 UTC, DARM started to glitch about every two seconds. The period is not exactly 2 seconds (I think it's just a little bit longer) but it does look pretty regular. I checked that the Pcal spectra look fine, so that's not the problem.

Attached is the change in the DARM spectrum. I wanted to use GDS-CALIB_STRAIN, but I get something that looks like it has dynamic range issues. We need to look into why that's not working. In addition, I'm attaching a Omega scan of the glitches.

Started shift in observing mode at 40 W with Michael L. and Evan G. running injections

23:25 UTC nds0 crashed. The IMC_LOCK guardian node went into error thus taking us out of observing mode. The IFO stayed locked. nds0 restarted. Keita and Sheila changed something in the IMC_LOCK guardian code that fixed the error. I set the intent bit back to observing.
01:06 UTC Injections done. Strange moving bump appeared in DARM around 110 Hz. Bump rang done.
01:07 UTC Out of observing to let Jeff K. measure DARM open loop gain.
01:48 UTC Jeff K. done with measurements
01:50 UTC Back to observing
03:56 - 04:30 UTC Stepped out of control room

The IFO has remained in observing mode without issues since the last entry. The only thing to note is approximately seven ETMY saturations. I am leaving the site now.

J. Kissel, E. Goetz

We've taken another DARMOLG TF and PCAL2 DARM transfer function in order to see if / how the SRC Detuning Optical (Anti-)Spring is changing from lock-stretch to lock-stretch. The bad (but not necessarily unexpected) news: it looks like the optical (anti-)spring frequency is moving around. One can tell by looking at the lowest frequency data points in the .pdf attachments. The model has a spring frequency of 9.381 Hz and both measurements are divided by it to form the residuals on the right column of subplots. Note also that Kiwamu and Craig's more sophisticated parametrization for the optical spring (which includes some Q between in the anti-spring poles, see LHO aLOG 28274) is not yet included. Jul 1 data points are ~ +15% discrepant in magnitude, where as Jul 09 data points are ~ +25%. 

More thoughts on this (what to do about it, how to track it, tests to try and control / reduce it) after the weekend. 

The pre-processed sensing function has been attached here, such that whomever gets to it first, can reproduce the fit using Craig's code from LHO aLOG 28274.

Data sets live here:
${CalSVN}/trunk/Runs/PreER9/H1/Measurements/DARMOLGTFs/2016-07-09_H1_DARM_OLGTF_4to1200Hz_SRCTuned.xml
${CalSVN}/trunk/Runs/PreER9/H1/Measurements/PCAL/2016-07-09_H1_PCAL2DARMTF_4to1200Hz_SRCTuned.xml

For the hardware injection testing during ER9, the long duration pcal line was turned off at GPS time 1152055000. It was then turned back on at 1152067170.

It remains running at 3001.3 Hz.

Sheila, Ross

We've put together a couple of screens for monitoring PI. The dtt shows a few spectrums of regions where PI may ring up and there is also a striptool which has rms monitors of some of the more problematic of the modes. These are stored in the /ligo/home/ops/Templates/ folder and are under dtt and StripTool PI_monitoring_ER9.

If any of these modes become unstable it should be visible in the RMS monitor screen. You can then get to the damping filter for that mode by going to the from the PI medm screens which are split into each test mass. There are overview screens for either QPDs or OMC. Most of these modes are damped using OMC signal at the moment apart from the 15541Hz ETMX mode and the 15542Hz ETMY mode which are damped using QPD signals. An example damping filter for the 15541 ETMX mode is attached in the third image. The damping filter settings may need to be changed if a line is ringing up i.e. the phase can be changed from +60 degrees to -60 degrees. Also increasing the gain may help to damp a mode if it is particularly rung up (around 100 magnitude). Right now the filter settings seem to keep the modes stable.

There is also a PI wiki page which is useful for identifying which modes  relate to which test mass. 

Summary:

As reported before (28236), ISS 2nd loop MEDM screen appears as if it sometimes doesn't agree with reality. Turns out that it never truly agreed with reality. For example, in the first attachment, both SERVO ON/OFF indicator and additional 20 dB gain indicator are red while these are ON.

I was bothered enough to investigate and found that the MEDM screen has numerous nonsense logic and that the guardians are not written MEDM-friendly. I made a quick dirty fix for MEDM as well as guardians.

What was wrong:

Before going into details, you need to know that ALL of PSL 2nd loop I/O that are binary in nature is done via 16 bit DAC (don't ask me why). The output of DAC is sent to optical coupler via a resistor (see e.g. D1300439). To make anything ON, you want some large positive voltage to drive the LED in the coupler. To make anything OFF, you want to send sufficiently small (e.g. zero-ish) voltage or negative voltage.

As you can easily guess, one of the problems is the lack of convention. People sometimes send +32000 and other times +32700 to make the same thing ON. Sometimes -32000 and other times 0 to turn it OFF.

On top of that, there are many bugs in MEDM screen such that color indicators check some nonsense bits that never change, value set on pressing button but changed to a different number on releasing the button, that kind of things.

What was done:

I changed most everything in MEDM as well as IMC_LOCK and lsclibrary such that:

1. OFF is 0 so MEDM switch graphics using "if zero" and "if not zero" works correctly,
2. ON is 32000 for the sake of consistency but the color indicators check if the value is positive (instead of looking at some specific bit).

It seems as if it's working for now.

Madness example:

Just as an example. Manually pressing CHANGE PDs button could have made you believe that you selected PD1-4 when PD5-8 (3rd loop) was selected in reality.

In the second attachment, you can see that the CHANGE PDs "1-4" button sent 32000 (40V-ish differential) when pressed, but then 1 (less than 1 mV) when released.

PD1-4 is selected with large positive voltage (or ON or HIGH or whatever), otherwise PD5-8 is selected.

When you manually pressed and released "1-4",  the DAC output momentarilly went +40V differential, but immediately went down to some tiny voltage, so PD5-8 was selected in the end.

But the switch graphics was implemented assuming that non-zero output means PD1-4. Since the DAC output was 1 after pressing the "1-4" button, the graphics showed that PD1-4 was connected instead of 5-8. The PD indicator color box didn't help either as it checked the bit 2 of the DAC output, and it only turned to orange (CH5-8) only when DAC output was 32700.

The story was different for guardian as it used +32000 and -32000 for PD1-4 and PD5-8, respectively. When guardian did something, it looked as if PD1-4 was selected no matter what.

Conor, Jeff, Jim

Short summary:
- BRS results in a factor of three reduction in arm-length RMS velocity with moderate wind.
- BRS does no measurable harm at low frequencies.
- In moderate wind, all test-mass sensor corrections show the same RMS in the beam axis.
- Identical broadband sensor correction results in rejection of the primary microseism in arm-length


Moving one step further down the ISI chain, I looked at the sensor correction of the Stage 1 CPSs from the ground STS2s.

To do this, I fetched Ground and BRS data along the beamlines (ITMY_X, ITMY_Y, ETMX_X, ETMY_Y, ETMX_RY, and ETMY_RX) and
filtered the BRS using fetched zpk from Foton to tilt-correct the End-station STS2s (one could also grab the corrected channel now).

The STS2 outputs were then filtered for broadband sensor correction (using a filter very similar to the installed version). This is 
now the signal sent directly to the CPSs and at low frequency (f<0.1 Hz) is a good estimate of the relative motion of the optics.

Attachment 1 shows the CPS sensor correction inputs (in m/s/rtHz) for each test mass in the beam axis. End test masses are shown with and without 
BRS tilt correction. The RMS for each curve from 0.2Hz down is dashed. The RMS velocity is the thing I think we want to minimise at low-f.

What can be seen is that after correction and with moderate wind (~17mph), all channels except ETMX_X have identical RMS. With lower tilt and BRS correction, EX performs better.
Also note that even though there is a spectral 'bump' of BRS noise around and below 10mHz, it makes no difference at all to the RMS. BRS does no 
measurable harm after the Ground-to-CPS sensor correction filters.

Attachments 2 and 3 show the ARM DoF sensor correction, by taking the subtractions: ITMX_X - ETMX_X, and ITMY_Y - ETMY_Y. What this shows is that 
by using identical sensor correction filters, we get some (not insignificant) common-mode rejection of the primary microseism, which is *amplified*
by the sensor correction filter. This rejection will improve as the microseism increases. Additionally, in the important degree of freedom for the
interferometer, there is a factor of 3 improvement in RMS using the BRS.

Attachment 4 is the script used to produce these plots, it should be self-contained apart from an NDS fetch, Foton fetching, and these path entries:
addpath /ligo/svncommon/NbSVN/aligonoisebudget/trunk/Externals/SimulinkNb/Utils/
addpath /ligo/svncommon/SeiSVN/seismic/Common/MatlabTools/

Our first lockloss from 40 Watts today (2016-07-08 19:36:59  ISC_LOCK  NOMINAL_LOW_NOISE -> LOCKLOSS)  was caused by bounce modes.  ITMX bounce mode rang up ( I don't know why).  After this rang up I turned off all the bounce mode damping (because of confusion about which mode it was) which caused all the modes to grow and break the lock. 

This lock I have been watching and the settings we used damped the ITMX mode well when we first locked, and it has not rung up at all.  My advice to operators would be to keep open the StripTool (/ligo/home/ops/BOUNCE_DAMPING.stp).  If ITMX starts ringing up open up the damping filters (sitemap> sus> BOUCNE_ROLL>DAMPING filters)  and try a different phase for ITMX by turning on or off some of the filters like +60, -60 ect.  If it starts damping that's good, if it starts ringing up faster because of your change you can use a negative gain with that phase. 

FAMIS task 6478: PSL DIode and Crystal chilller top-off:  assigned 11PM 7/6, read by me in the morning on 7/7, on 7/8 JeffB noticed the Crystal chiller was in alarm, I trended and it went into alarm on 7/7, was previously filled on 6/30, so only lasted 8 days

• 300ml added to the Crystal chiller
• Diode chiller is not showing any alarms on it's front panel - no water added

J. Kissel, D. Macleod

Via the ODC Team's "internal status checking webpage," Duncan found that several ODC EPICs records that define good settings for the PCALX's hardware injection path were errantly all zeros. In short: I've fixed the problem and the ODC summary bit for PINJX is happy and green.

-------------
Details of how I made the summary bit green:
The subpage for PINJX shows three columns: "ini" "safe" and "current", and I *think* that the "ini" coulmn is what they want the value to be (reported in decimal), the "safe" is what the SDF system's safe.snap has stored, and "current" is the current EPICs value. 

These channels were "not monitored" in the SDF system, and the safe.snap file that the front end loads upon restart had them set to zero, so it's not surprising that no one noticed these had gone bad. 

However, given the recent rearrangement of the hardware injection path (see LHO aLOG 27383) the values listed in the "ini" column are also incorrect. 

The channels in question are
    H1:CAL-PINJX_ODC_HARDWARE_CTRL_EQ
    H1:CAL-PINJX_ODC_HARDWARE_GAIN_EQ
    H1:CAL-PINJX_ODC_CW_CTRL_EQ
    H1:CAL-PINJX_ODC_CW_GAIN_EQ
    H1:CAL-PINJX_ODC_TRANSIENT_CTRL_EQ
    H1:CAL-PINJX_ODC_TRANSIENT_GAIN_EQ
    H1:CAL-PINJX_ODC_CHANNEL_BITMASK
After digging through the h1calex front-end model to figure out what these ODC channels listed on the web actually *are*, I found the following (see first two attachments for visual aide):
- the three _GAIN_EQ records are compared against the *actual* gain of each of the three hardware injection filter banks (H1:CAL-PINJX_CW_GAIN, H1:CAL-PINJX_TRANSIENT_GAIN, H1:CAL-PINJX_HARDWARE_GAIN). These records are just standard decimal values, and since the filter bank gains are currently all +1.0, all of these records should be +1.0.

- the H1:CAL-PINJX_ODC_CHANNEL_BITMASK is a binary bit-word that masks which of the ODC bits are used to determine the summary status bit. Of course you know, in EPICs, binary bitwords must be entered in as hexidecimal. Also you know that, in ODC Bit 0 is reserved for the summary, so the mask starts functioning at bit 1. Since there are 6 comparison channels that we want to inform the summary bit, we need six bits, so the value should be
                                           Bit# 6 5 4 3 2 1 0   Decimal   Hex   What Goes In EPICs
     H1:CAL-PINJX_ODC_CHANNEL_BITMASK           1 1 1 1 1 1 0 = 126     = 7e    = 0x7e

- the three _CTRL_EQ records are compared against the output of "CtrlOut" spigot of each of the three hardware injection filter banks, respectively, which cdsCtrlFilt2 filter modules. According to the doc text inside the CDS_PARTS.mdl library, this cdsCtrlFilt2 output is a 13-bit bitword indicative of the state of all switches in the filter module: FM1's status is the least significant bit (Bit 0), FM2 through FM10's status are Bits 1 through 9, and Bits 10, 11, and 12 are the input, offset, and output switch status respectively. Because the _CTRL_EQ records are bit-wise compared to the front end's bitword, the record's value must also be a binary bit word. 

So, here's the table to translates this mess for each of the three filter banks, given the current, correct status of these filter banks (see third attached screen shot).
                                         Bit# 13 12 11 10 09 08 07 06 05 04 03 02 01     Decimal    Hex       What Goes In EPICs 
     H1:CAL-PINJX_ODC_HARDWARE_CTRL_EQ         1  0  1  0  0  0  0  0  0  0  0  0  0   = 5120     = 1400    = 0x1400
     H1:CAL-PINJX_ODC_CW_CTRL_EQ               1  0  1  1  1  1  1  1  1  1  1  1  1   = 6143     = 17FF    = 0x17ff
     H1:CAL-PINJX_ODC_TRANSIENT_CTRL_EQ        1  0  1  0  0  0  0  0  0  0  0  0  0   = 5120     = 1400    = 0x1400
                                 Switch Name  OT OF IN F1 F2 F3 F4 F5 F6 F7 F8 F9 F10
 

All of the bits are now green, and the mask is lights are on as expected, and the summary bit is green. I've switched ON the monitoring of these channels in the SDF system, and accepted these new values. I trust that Duncan will update the ODC team's page such that the "ini" column is correct.

I've also re-installed (and SDF monitored) all of the appropriate strings in each H1:CAL-PINJX_ODC_BITn channel (where n is the bit number from 0 to 13), as defined again by this ODC Team webpage.

-------------
Critiques of this instantiation of ODC:
- This ODC system is essential identically recreating the SDF system. There's really no need for this. The SDF system is required to be reconciled before going into any observational stretch, so if the hardware injection path's filter banks are monitored by the SDF system (and they are), then they by default confirmed to be correct every observation stretch.
- If someone decides that this is worth keeping, 
    - these EPICS records should be much more clearly labeled and explained on the ODC MEDM screen.
    - The actual output of the CtrlOut from each of the cdsFiltCtrl2 banks should be fed into an EPICs record as well. This way one can easily take the current value and copy it into the comparator, such that the novice doesn't have to go through the binary to EPICs hex translation game that I had to above.

State of H1: Locked at 40W and in Observe

Today's H1 Commissioners: Keita, Sheila, Kiwamu

Locking:

• locking arms green, finding IR, and locking DRMI all have gone well today
• no issues getting up to Engage Soft Loops
• in / after ENGAGE_SRC_ASC, SRM needs to be watched closely and have it's alignment adjusted to preserve the lock, however, ignore before this state
• OMC stalled about half of the time and needed me to unstall it's guardian to get to DC READOUT
• watch SRM again closely during power up

Site Activities:

• Kyle - MX and MY - vacuum work / overfill CP5
• Fil - MY - 3IFO

Current Plans for continuing ER9 (may change): continue the run the rest of today, and continue the run 8AM-midnight tomorrow

• Patrick is here for eve shift
• JimW is in tomorrow morning for day shift
• Nutsinee is in tomorrow afternoon for eve shift

Coil drivers for the quad L2 (penultimate) stage have four dewhitening filter states, of which state 2 has the largest range and state 3 the lowest noise.

State 2 is meant for lock acquisition, and is not compatible with low noise data taking (unless some form of subtraction is used). As shown in the plot below, the DAC noise in state 2 would be expected to limit the historic DARM sensitivity in the 10-50 Hz band. (For ER9 sensitivity the state 2 noise is apparently low enough to be out of the way, as noted in alog 28107.)

The IFO top node appears to be not monitoring some SEI BS and ALS nodes.  The IFO top node is reporting OK, even though multiple nodes are not reporting OK:

This is an indication of two issue:

• IFO top node is not monitoring SEI BS nodes, as well as some of the ALS nodes.
• If the SEI BS and ALS nodes are in their desired states, then their NOMINAL states are not set correctly.

The USERAPPS/h1/guardian/IFO_NODE_LIST.py that is in the SVN shows that those nodes should be managed, so my guess is that what's currently running has not been committed to the SVN.