Summary:

As reported before (28236), ISS 2nd loop MEDM screen appears as if it sometimes doesn't agree with reality. Turns out that it never truly agreed with reality. For example, in the first attachment, both SERVO ON/OFF indicator and additional 20 dB gain indicator are red while these are ON.

I was bothered enough to investigate and found that the MEDM screen has numerous nonsense logic and that the guardians are not written MEDM-friendly. I made a quick dirty fix for MEDM as well as guardians.

What was wrong:

Before going into details, you need to know that ALL of PSL 2nd loop I/O that are binary in nature is done via 16 bit DAC (don't ask me why). The output of DAC is sent to optical coupler via a resistor (see e.g. D1300439). To make anything ON, you want some large positive voltage to drive the LED in the coupler. To make anything OFF, you want to send sufficiently small (e.g. zero-ish) voltage or negative voltage.

As you can easily guess, one of the problems is the lack of convention. People sometimes send +32000 and other times +32700 to make the same thing ON. Sometimes -32000 and other times 0 to turn it OFF.

On top of that, there are many bugs in MEDM screen such that color indicators check some nonsense bits that never change, value set on pressing button but changed to a different number on releasing the button, that kind of things.

What was done:

I changed most everything in MEDM as well as IMC_LOCK and lsclibrary such that:

1. OFF is 0 so MEDM switch graphics using "if zero" and "if not zero" works correctly,
2. ON is 32000 for the sake of consistency but the color indicators check if the value is positive (instead of looking at some specific bit).

It seems as if it's working for now.

Madness example:

Just as an example. Manually pressing CHANGE PDs button could have made you believe that you selected PD1-4 when PD5-8 (3rd loop) was selected in reality.

In the second attachment, you can see that the CHANGE PDs "1-4" button sent 32000 (40V-ish differential) when pressed, but then 1 (less than 1 mV) when released.

PD1-4 is selected with large positive voltage (or ON or HIGH or whatever), otherwise PD5-8 is selected.

When you manually pressed and released "1-4",  the DAC output momentarilly went +40V differential, but immediately went down to some tiny voltage, so PD5-8 was selected in the end.

But the switch graphics was implemented assuming that non-zero output means PD1-4. Since the DAC output was 1 after pressing the "1-4" button, the graphics showed that PD1-4 was connected instead of 5-8. The PD indicator color box didn't help either as it checked the bit 2 of the DAC output, and it only turned to orange (CH5-8) only when DAC output was 32700.

The story was different for guardian as it used +32000 and -32000 for PD1-4 and PD5-8, respectively. When guardian did something, it looked as if PD1-4 was selected no matter what.

Conor, Jeff, Jim

Short summary:
- BRS results in a factor of three reduction in arm-length RMS velocity with moderate wind.
- BRS does no measurable harm at low frequencies.
- In moderate wind, all test-mass sensor corrections show the same RMS in the beam axis.
- Identical broadband sensor correction results in rejection of the primary microseism in arm-length


Moving one step further down the ISI chain, I looked at the sensor correction of the Stage 1 CPSs from the ground STS2s.

To do this, I fetched Ground and BRS data along the beamlines (ITMY_X, ITMY_Y, ETMX_X, ETMY_Y, ETMX_RY, and ETMY_RX) and
filtered the BRS using fetched zpk from Foton to tilt-correct the End-station STS2s (one could also grab the corrected channel now).

The STS2 outputs were then filtered for broadband sensor correction (using a filter very similar to the installed version). This is 
now the signal sent directly to the CPSs and at low frequency (f<0.1 Hz) is a good estimate of the relative motion of the optics.

Attachment 1 shows the CPS sensor correction inputs (in m/s/rtHz) for each test mass in the beam axis. End test masses are shown with and without 
BRS tilt correction. The RMS for each curve from 0.2Hz down is dashed. The RMS velocity is the thing I think we want to minimise at low-f.

What can be seen is that after correction and with moderate wind (~17mph), all channels except ETMX_X have identical RMS. With lower tilt and BRS correction, EX performs better.
Also note that even though there is a spectral 'bump' of BRS noise around and below 10mHz, it makes no difference at all to the RMS. BRS does no 
measurable harm after the Ground-to-CPS sensor correction filters.

Attachments 2 and 3 show the ARM DoF sensor correction, by taking the subtractions: ITMX_X - ETMX_X, and ITMY_Y - ETMY_Y. What this shows is that 
by using identical sensor correction filters, we get some (not insignificant) common-mode rejection of the primary microseism, which is *amplified*
by the sensor correction filter. This rejection will improve as the microseism increases. Additionally, in the important degree of freedom for the
interferometer, there is a factor of 3 improvement in RMS using the BRS.

Attachment 4 is the script used to produce these plots, it should be self-contained apart from an NDS fetch, Foton fetching, and these path entries:
addpath /ligo/svncommon/NbSVN/aligonoisebudget/trunk/Externals/SimulinkNb/Utils/
addpath /ligo/svncommon/SeiSVN/seismic/Common/MatlabTools/

Our first lockloss from 40 Watts today (2016-07-08 19:36:59  ISC_LOCK  NOMINAL_LOW_NOISE -> LOCKLOSS)  was caused by bounce modes.  ITMX bounce mode rang up ( I don't know why).  After this rang up I turned off all the bounce mode damping (because of confusion about which mode it was) which caused all the modes to grow and break the lock. 

This lock I have been watching and the settings we used damped the ITMX mode well when we first locked, and it has not rung up at all.  My advice to operators would be to keep open the StripTool (/ligo/home/ops/BOUNCE_DAMPING.stp).  If ITMX starts ringing up open up the damping filters (sitemap> sus> BOUCNE_ROLL>DAMPING filters)  and try a different phase for ITMX by turning on or off some of the filters like +60, -60 ect.  If it starts damping that's good, if it starts ringing up faster because of your change you can use a negative gain with that phase. 

FAMIS task 6478: PSL DIode and Crystal chilller top-off:  assigned 11PM 7/6, read by me in the morning on 7/7, on 7/8 JeffB noticed the Crystal chiller was in alarm, I trended and it went into alarm on 7/7, was previously filled on 6/30, so only lasted 8 days

• 300ml added to the Crystal chiller
• Diode chiller is not showing any alarms on it's front panel - no water added

J. Kissel, D. Macleod

Via the ODC Team's "internal status checking webpage," Duncan found that several ODC EPICs records that define good settings for the PCALX's hardware injection path were errantly all zeros. In short: I've fixed the problem and the ODC summary bit for PINJX is happy and green.

-------------
Details of how I made the summary bit green:
The subpage for PINJX shows three columns: "ini" "safe" and "current", and I *think* that the "ini" coulmn is what they want the value to be (reported in decimal), the "safe" is what the SDF system's safe.snap has stored, and "current" is the current EPICs value. 

These channels were "not monitored" in the SDF system, and the safe.snap file that the front end loads upon restart had them set to zero, so it's not surprising that no one noticed these had gone bad. 

However, given the recent rearrangement of the hardware injection path (see LHO aLOG 27383) the values listed in the "ini" column are also incorrect. 

The channels in question are
    H1:CAL-PINJX_ODC_HARDWARE_CTRL_EQ
    H1:CAL-PINJX_ODC_HARDWARE_GAIN_EQ
    H1:CAL-PINJX_ODC_CW_CTRL_EQ
    H1:CAL-PINJX_ODC_CW_GAIN_EQ
    H1:CAL-PINJX_ODC_TRANSIENT_CTRL_EQ
    H1:CAL-PINJX_ODC_TRANSIENT_GAIN_EQ
    H1:CAL-PINJX_ODC_CHANNEL_BITMASK
After digging through the h1calex front-end model to figure out what these ODC channels listed on the web actually *are*, I found the following (see first two attachments for visual aide):
- the three _GAIN_EQ records are compared against the *actual* gain of each of the three hardware injection filter banks (H1:CAL-PINJX_CW_GAIN, H1:CAL-PINJX_TRANSIENT_GAIN, H1:CAL-PINJX_HARDWARE_GAIN). These records are just standard decimal values, and since the filter bank gains are currently all +1.0, all of these records should be +1.0.

- the H1:CAL-PINJX_ODC_CHANNEL_BITMASK is a binary bit-word that masks which of the ODC bits are used to determine the summary status bit. Of course you know, in EPICs, binary bitwords must be entered in as hexidecimal. Also you know that, in ODC Bit 0 is reserved for the summary, so the mask starts functioning at bit 1. Since there are 6 comparison channels that we want to inform the summary bit, we need six bits, so the value should be
                                           Bit# 6 5 4 3 2 1 0   Decimal   Hex   What Goes In EPICs
     H1:CAL-PINJX_ODC_CHANNEL_BITMASK           1 1 1 1 1 1 0 = 126     = 7e    = 0x7e

- the three _CTRL_EQ records are compared against the output of "CtrlOut" spigot of each of the three hardware injection filter banks, respectively, which cdsCtrlFilt2 filter modules. According to the doc text inside the CDS_PARTS.mdl library, this cdsCtrlFilt2 output is a 13-bit bitword indicative of the state of all switches in the filter module: FM1's status is the least significant bit (Bit 0), FM2 through FM10's status are Bits 1 through 9, and Bits 10, 11, and 12 are the input, offset, and output switch status respectively. Because the _CTRL_EQ records are bit-wise compared to the front end's bitword, the record's value must also be a binary bit word. 

So, here's the table to translates this mess for each of the three filter banks, given the current, correct status of these filter banks (see third attached screen shot).
                                         Bit# 13 12 11 10 09 08 07 06 05 04 03 02 01     Decimal    Hex       What Goes In EPICs 
     H1:CAL-PINJX_ODC_HARDWARE_CTRL_EQ         1  0  1  0  0  0  0  0  0  0  0  0  0   = 5120     = 1400    = 0x1400
     H1:CAL-PINJX_ODC_CW_CTRL_EQ               1  0  1  1  1  1  1  1  1  1  1  1  1   = 6143     = 17FF    = 0x17ff
     H1:CAL-PINJX_ODC_TRANSIENT_CTRL_EQ        1  0  1  0  0  0  0  0  0  0  0  0  0   = 5120     = 1400    = 0x1400
                                 Switch Name  OT OF IN F1 F2 F3 F4 F5 F6 F7 F8 F9 F10
 

All of the bits are now green, and the mask is lights are on as expected, and the summary bit is green. I've switched ON the monitoring of these channels in the SDF system, and accepted these new values. I trust that Duncan will update the ODC team's page such that the "ini" column is correct.

I've also re-installed (and SDF monitored) all of the appropriate strings in each H1:CAL-PINJX_ODC_BITn channel (where n is the bit number from 0 to 13), as defined again by this ODC Team webpage.

-------------
Critiques of this instantiation of ODC:
- This ODC system is essential identically recreating the SDF system. There's really no need for this. The SDF system is required to be reconciled before going into any observational stretch, so if the hardware injection path's filter banks are monitored by the SDF system (and they are), then they by default confirmed to be correct every observation stretch.
- If someone decides that this is worth keeping, 
    - these EPICS records should be much more clearly labeled and explained on the ODC MEDM screen.
    - The actual output of the CtrlOut from each of the cdsFiltCtrl2 banks should be fed into an EPICs record as well. This way one can easily take the current value and copy it into the comparator, such that the novice doesn't have to go through the binary to EPICs hex translation game that I had to above.

State of H1: Locked at 40W and in Observe

Today's H1 Commissioners: Keita, Sheila, Kiwamu

Locking:

• locking arms green, finding IR, and locking DRMI all have gone well today
• no issues getting up to Engage Soft Loops
• in / after ENGAGE_SRC_ASC, SRM needs to be watched closely and have it's alignment adjusted to preserve the lock, however, ignore before this state
• OMC stalled about half of the time and needed me to unstall it's guardian to get to DC READOUT
• watch SRM again closely during power up

Site Activities:

• Kyle - MX and MY - vacuum work / overfill CP5
• Fil - MY - 3IFO

Current Plans for continuing ER9 (may change): continue the run the rest of today, and continue the run 8AM-midnight tomorrow

• Patrick is here for eve shift
• JimW is in tomorrow morning for day shift
• Nutsinee is in tomorrow afternoon for eve shift

Coil drivers for the quad L2 (penultimate) stage have four dewhitening filter states, of which state 2 has the largest range and state 3 the lowest noise.

State 2 is meant for lock acquisition, and is not compatible with low noise data taking (unless some form of subtraction is used). As shown in the plot below, the DAC noise in state 2 would be expected to limit the historic DARM sensitivity in the 10-50 Hz band. (For ER9 sensitivity the state 2 noise is apparently low enough to be out of the way, as noted in alog 28107.)

Here are results on a study of using iwave to control closely spaced doublets of sinusoids. This study was conducted on test data at 16384Hz sampling rate, of 100 second duration. The dataset consisted of two sine waves at 27.8 and 27.9Hz, each of amplitude 1, on a background of Gaussian white noise of RMS 0.1. Two iwave configurations were used to try and subtract the two components. In both schemes, iwave was running with a tau parameter of 5 seconds and initial guess frequencies at exactly the line frequencies. Iwave was running in phaselocked mode. The loops on the lower and upper frequency lines were set to close 5 and 6 seconds into the dataset, respectively, although these time delays are unimportant; they could both have been set to zero just as well and I mention them for completeness.

CONFIGURATION 1: the raw data is injected into iwave1 set to track the lower frequency (27.8Hz). The D phase output of iwave1 is subtracted from the raw data, and the result is injected into iwave2, set to track the upper frequency (27.9Hz).  The D phase output of iwave2 is subtracted from the input to iwave2, and the difference is the final output. In other words, the two iwave filters are cascaded.

CONFIGURATION 2: the raw data minus the D phase output of iwave2, set to track the upper frequency line (27.9Hz) is used as the input for iwave1, set to track the lower frequency line (27.8Hz). The raw data minus the D phase output of iwave1 is used as the input for iwave2. In other words, for each iwave, the input is the result of the raw data with the D phase output of the other iwave subtracted. 

The first attached pdf shows the results of studies of the the two configurations; the first two pages are for configuration 1 and the pages 3 and 4 are for configuration 2. In each case, there are three plots - the extracted line amplitudes and frequencies (together on the same page), and amplitude spectral densities of the raw data, the data after the subtraction of the first line, and the the output data after both lines are subtracted. 

The 'cross-subtraction' technique is superior, having much better line attenuation and markedly less sidebands introduced. This is because iwave uses the product of the input data and the Q phase output to sense phase shifts, but when the input data is polluted by a close by line, this causes residual beats between this line and the q phase output in the phase measurement. You can see this on page 1, where both the inferred frequency of the line and its amplitude oscillate at the difference frequency between the two lines, which is 0.1Hz, or a 10 second beat. When each iwave input has the inferred line from iwave outputs on other loud lines removed before input, this distortion goes away. 

I have attached a schematic of the cross subtraction configuration, 2, as a second attached figure, in case the written description above is not clear enough. One detail is, the output of iwave2 which is then subtracted from the input data before injection into iwave1 must be derived from the previous sample of iwave2 output, and hence has to be phase shifted forward by one sample period to accurately subtract the line. This is easy to do using both the Q and the D phase outputs of iwave2, and the cos and sin of the phase shift per sample for the frequency of the line tracked by iwave2.

1315 -1400 hrs. local -> To and from Y-mid (then X-mid, unrelated)

Opened exhaust check-valve bypass-valve.  Opened LLCV bypass-valve 1/2 turn -> LN2 @ exhaust in 55 seconds -> Restored valves to as found configuration.  

Next CP3 overfill to be Monday, July 11th.  

The IFO top node appears to be not monitoring some SEI BS and ALS nodes.  The IFO top node is reporting OK, even though multiple nodes are not reporting OK:

This is an indication of two issue:

• IFO top node is not monitoring SEI BS nodes, as well as some of the ALS nodes.
• If the SEI BS and ALS nodes are in their desired states, then their NOMINAL states are not set correctly.

The USERAPPS/h1/guardian/IFO_NODE_LIST.py that is in the SVN shows that those nodes should be managed, so my guess is that what's currently running has not been committed to the SVN.

We have operated for over 10 hours in "nominal low noise, observation, intent bit set" condition.  This was a temporary configuration to allow locking, after the laser diode box failure, for the purposes of testing ER9 pipelines. We are returning to 40W operation.  

Patrick, Sheila, Carl and Ross damping PIs

After Keita's ISS fixes we had a few hours of bad weather and earthquakes that contributed to dificulty locking.  Once things calmed down we locked and sat at several stages to see if the lock was stable, DC readout, 25 Watts, ISS 2nd loop engaged, and now we have made it to some kind of "nominal low noise" state.  Patrick hit the intent bit.  

There are several things wrong with this spectrum, some of which could be solved by going back to 40 Watts in the morning:

• We skipped the coil drivers state, since we don't expect it to help our noise and we lost lock in this state earlier. We don't see any reason not to try doing this tomorow though. 
• There is a lot of intensity noise in DARM.  (1st attached screenshot shows 2nd loop PDs before ISS was on 6:40 UTC July 8th, and after the 2nd loop is engaged at 7:30 UTC. The 2nd attachement shows that there is still a lot of coherence of intensity noise in DARM.)
• The CO2 guardians were in preheating mode for the first bit of time we had the intent bit set (this is a result of not thinking through all the consequences of changing the nominal power down to 25 Watts).  We set the CO2 guardians to NOMINAL_LOCKED_CO2 level, which means that they are now at the settings that are appropriate for 40 Watts I think, which is probably closer to the right settings for 25 Watts than preheating at least. 
• Our low noise hard ASC loops were designed for 40 Watts, so the decision to go to 25 Watts means that we would need to do some measurements and loop design work to have a low noise state for 25 Watts. 
• We probably have different spot positions on our optics now than we would at 40Watts, so we would need to run a2l for 25 Watts if we want low noise.  
• DIAG main is complaining that the POP beam diverter is not closed, this is because we are still locking on the POPX WFS, this will be open throughout ER9.
• The OMC alingment is clearly off, on the camera it is on the left half of the screen. 
• The DCPD whitening is off, which makes our high frequency noise worse.

The last attached screenshot is just to show how many locking attempts everyone made in the last 14 hours. The range displayed by the DMT viewer in the control room seems to have a problem, it should be stable at some low range but on the DMT viewer it looks like we lost lock in the last half hour.  

Started and ran the purge-air skid, Turbo, QDP80 and newly added scroll pump.  Tested the functionality of the Turbo's Safety Valve with the control cable connected to the scroll pump's relay box - demonstrating that the Safety Valve closes upon the loss of scroll pump motor AC and thus mimicking the "QDP80 Running" 
signal.

Note: The purge-air skid has developed some problems from lack of use over these past few years, namely the radiator fan never came on while running the compressors, the drying tower never cycled and the low pressure alarm never sounded.  These features worked fine when last this unit was run.  Now they don't.  Also, the Turbo spun-up normally even with the locally mounted scroll pump running (new vibration source) but tripped into EMERGENCY OPERATION upon BRAKING and when at ~1/3 NORMAL rpm.  This behavior is seen on other Turbos (XBM for example) and might be an age related issue?  

I am leaving the Turbo energized overnight to ensure that the rotor is completely stopped.  I will de-energize it tomorrow.  

With the measured sensing function (28123) in hand, we did an MCMC-based numerical fitting for the sensing function (see E. Hall, T1500553).

The fitting results are

• Optical gain = 8.980393e+05 +/- 7.965381e+02 [cnts/m]
• Cavity pole = 3.294194e+02 +/- 5.626792e-01 [Hz]
• Time delay = 1.973719e+02 +/- 3.386483e-01 [usec]
• Spring frequency = 9.825078e+00 +/- 5.453538e-02 [Hz]

[New sensing function form]

As reported by Evan (27675), it seems that the extra roll off in the DARM response at low frequencies are due to an SRC detuning (or something equivalent). While such detuning should be suppressed by some control loop ideally, we decided to include the detuning-induced functional form in addition to the ordinary single-pole response. We use the following approximated form for the DARM response

S(f) = H / (1 + i f / fc ) * exp( -2 * pi * f * tau) * f^2/(f^2 + fs^2)

where H, fc, tau and fs are the optical gain, DARM cavity pole, time delay and spring frequency. Some details of the derivation will be reported later. Apparently, we now have an additional quantity (i.e., fs) to fit.

Note that since H1 seems to be in (unintentionally) an anti-spring detuning, fs should be a real number whereas it should be an imaginary number for a pro-spring case. Obviously, Q is set to infinity for simplicity.

[MCMC-based numerical fitting]

Following the work by Evan (T1500553), we adapted his code to include the spring frequency as well. In short, it is a Bayesian analysis to obtain posteriors for the parameters that we want to estimate. We gave a simple set of priors as follows for this particular analysis.

• 0 < H < infinity
• 0 < fc < infinity
• 0 < tau < infinity
• 0 < fs  < infinity (real number)

The quad plot below shows the fitting result. The estimated parameters are obtained by taking the mean values of the resutling probability distribution. As usual, the two plots on the left hand side show a bode plot of the measured and fitted data. The two plots on the right hand side show the residual of the fit. As shown in the residuals, there are several points which are as big as 20% in magnitude and 6 deg in phase below 10 Hz. Otherwise, the residual data points seem to be within 10-ish % and 4 deg. The code is attached in pdf format.

EDIT: I am attaching the actual code as well.