Between April 21 and July 14, the HEPI Fluid system Accumulators were properly charged--19632.  The check of the charging obviously has risks though, not only does the Pump need to be spun down taking down the entire SEI isolation, the checking of the pressure via the schrader valve of the accumulator may cause it to start leaking.  This is what I expect happened at EndY on 14 July.  When I was at EndY Tuesday 21 July looking at the reference voltage of the pressure sensors (attempting to learn why the sensors are noisy,) I noted the HEPI reservoir Fluid level was dramatically down.  Yesterday I found the schrader valve was indeed leaking and needed replacement.

Fortunately, it appears the platform is not that sensitive to the loss of just a single accumulator, or it maybe more importantly depend on where that accumulator is in the system.  Attached are coherence plots between the servo pressure and the HEPI and ISI inertial sensors.  The reference traces (thick pale) are from 11 July, before my confirming measurement on July 14.  The current traces (thin dark) are from the early morning hours yesterday after I saw the low fluid level.  While there are a scattering of a few peaks of increased coherence in no instance are they in adjacent frequency bins.  There a few higher frequency (between 0.5 and 3hz) spikes that approach 0.9, too large for comfort of course but there are no broadband areas of increased coherence.  Is this meaningful?  Still, the fluid levels must be watched and accumulators checked at the first sign of lower fluid level in the reservoir.

I'll look at this again in a couple days and see if these spikes go away since the charging of this one accumulator.  Also, I need to check the affect of a lost accumulator on the fluid level in the corner station.  The actual fluid 'loss' should be the same but the reservoir is larger so the drop should be less.  Still, the loss of one accumulator would be noticeable.

There was a question yesterday about the blends we are running on the the BSC-ISIs. I'm attaching plots of the Quite90/250 and Windy90/250 blends. Pages 1-3 are the 90mhz X/Y blends, 3-6 are the 250 RX/RY blends. The Quite blends are what we are running currently on St1 in XYZ (90) and RX/RY (250), and there is some evidence that we can run them during windy times as well. The Windy blends are kind of untested, and don't currently run on the end station ISIs, I don't know why and I haven't had much of a chance to look into it. The end ISIs will switch to the Windy blends, but trip after a few seconds. Pages 3 and 6 at least show that the filters as installed on ETMY are properly complementary, ETMX is the same.

Work:

- Richard, cosmic ray work continues, not in LVEA

- no SEI

- Apollo, beam tube cleaning ongoing today

- Gerardo, vacuum, High Bay area

- SUS, no work

- PSL, yesterday's work OK, no work today

- Commissioning, intensity noise, frequency noise

- Jim, Frame Writers work continues

I turned on the 530-545 notch in FM7 of ETMY L3 LOCK, and switched to ETMY with the ESD low pass on.  As the alogs of Evan and Rana suggested, by notching the pcal lines in the drive to ETMY we can engage the low pass and still have a rms of about 3000 counts on the ESD. (screen shot of ETMY drives attached). 

The resulting DARM spectrum seems slightly better at around 50-80 Hz.  Things are worse below 20 Hz because I increased the gain of DHARD YAW.  We can see if we are at least stable at high power with this increased gain for DHARD, and then worry about the 20 Hz noise.  

We had one of the huge glitches that we have been calling beam tube glitches, at around 8:15 UTC July 24th, even though no-one is cleaning the beam tubes. I've hit the intent bit and an leaving the IFO locked.  

There is currently a descrepancy of about 25 Mpc between the range displayed in the control room and the range on the summary pages.  

Jenne, Sheila

On the theory that our slow instability at 24 Watts (alog 19855 )is caused by DHARD signal which contaminates the SOFT signals and the AS_C signal therefore gets into the SRC WFS, we decided to boost the DHARD loops.  For pitch this was easy, I just moved the pair of complex poles in the existing boost down from 0.2 Hz to 0.02 Hz. This worked the first time.  

For yaw I tried similarly  to lower the poles in the existing boost to 0 Hz, but this caused an instability that broke the lock on power up.  Jenne and I then had a look at the transfer function and remembered how not great this loop was.  In fact at 3 Watts it had about 7 ugfs, the lowest one just under 0.3 Hz.  We made a small adjustment to the plant inversion to get rid of two of the (the new plant inversion is invY2, the Q of the 1.8 Hz pole is higher), we increased the gain in the input matrix from 1 to 3, and last I modified the boost which had been a pair of 0.4Hz poles, Q of 1.2, and is now a pair of 1 Hz poles, Q of 2.  The first attached screenshot shows the transfer function before and after these changes, measured at 3 Watts.  This loop currently doesn't have any roll off, but we don't have much phase margin to spare. The second attached screenshot shows measurements at 3 different powers, reference numbers are in the legend of the phase plot.  As we increase the power, the gain below the resonance drops as expected, this means that at 24 Watts we just barely have any gain even after increasing this gain by about 20dB. 

Aside: Part of the reason that we had trouble designing this loop in the first place is that during the CARM offset reduction the gain changes by about 30dB, and the resonance move.  I thought that one way of making things easier would be to use the dynamic power normalization, which we haven't used here yet.  Scaling by POP DC would be about right, we could lower the gain by 26 dB between the point where we turn on the loop and on resonance, so we could design a loop with only one ugf above all the structure,  and not need so much gain margin. I tried this, but it didn't work.  I tried turning up the gain in the input matrix, and turning it back down.  That was fine, so I tried to use the power scaling to do exactly the same thing, and that broke the lock.  I don't know if I've been fooled by some hidden normalization, but I don't see anything in the model.

J. Kissel, D. Barker, J. Batch, J. Betzwieser

As the calibration group solidifies how to measure the slowly-time-dependent parameters of the aLIGO detector's DARM loop, we've identified that we can no longer just get away with only comparing the real-and-imaginary parts of the current DARM Open Loop Gain transfer function against those of a reference time -- at calibration line frequencies -- to produce a "gamma" coefficient (a real, scalar, multiplicative factor applied to the detector's strain output). Because of charge on the test mass evolving with time (on the actuation side of things) and the DARM coupled cavity pole frequency evolving with time (on the sensing side of things), we now must at least track, if not correct for time-varying optical gain, cavity pole frequency, and actuation strength. As such we've parametrized the interferometer response (see T1500377), and this parametrization needs the real and imaginary parts of the actuation function, sensing function, and DARM filter -- again at calibration line frequencies -- for the reference times. 

Long story short, I've added EPICs records to the CAL-CS front-end model to store these new real-and-imaginary parts to the actuation function, sensing function, and DARM filter. The new channels are
    H1:CAL-CS_TDEP_${ACTTYPE}_LINE${NUM}_REF_A_REAL
    H1:CAL-CS_TDEP_${ACTTYPE}_LINE${NUM}_REF_A_IMAG
    H1:CAL-CS_TDEP_${ACTTYPE}_LINE${NUM}_REF_D_REAL
    H1:CAL-CS_TDEP_${ACTTYPE}_LINE${NUM}_REF_D_IMAG
    H1:CAL-CS_TDEP_${ACTTYPE}_LINE${NUM}_REF_C_REAL
    H1:CAL-CS_TDEP_${ACTTYPE}_LINE${NUM}_REF_C_IMAG
    H1:CAL-CS_TDEP_${ACTTYPE}_LINE${NUM}_REF_C_NOCAVPOLE_REAL
    H1:CAL-CS_TDEP_${ACTTYPE}_LINE${NUM}_REF_C_NOCAVPOLE_IMAG
where
${ACTTYPE} = ESD, PCALX, or PCALY
${NUM} = 1, 2, (3, or 4)
and there are 4 lines for ESD and 2 lines for each PCAL.

I will update the CAL-CS MEDM screens with a look-up table for these reference values tomorrow and/or over the weekend.

Short story long again, I ran into two problems trying to compile the changes that I wanted to make:
(1) Not sure how bast to describe this bug, but it comes up when you use several FROM and GOTO tags on multiple levels of subsystems without connecting them to a CDS part on each level. There were two examples of me doing this in my desired CAL-CS model, where I'd tagged a signal, traversed down a subsystem level, tagged it again, and traversed down a few more subsystem levels before it connected to a filter bank. I show the two ways I had done this schematically below:

CASE 1:

IPC part --------+
                 | CAL_CS_MASTER
                 +----------------+
                                  | TIMEDEPENDENCE
                                  +---< TAG 1 |      | TAG 2 >---+
                                                                 | ESD_LINE1
                                                                 +---+
                                                                     | DEMOD
                                                                     +---+
                                                                         | DEMOD
                                                                         +----------- filter bank

CASE 2:
                  +---< TAG 1 |      | TAG 1 >---+
             DARM |                              | TIMEDEP
filter bank  -----+                              +--- < TAG 2 |      | TAG 2 >---+
                                                                                 | PCALX_LINE1
                                                                                 +---+
                                                                                     | DEMOD
                                                                                     +------------ filter bank

I'm not sure which of these two caused the problem, but he attached screen shots with the file tag "doesnotcompile" demonstrate what I *wanted* to compile. The correspondingly named screenshots *without* any extra file tag shows what we had to do in order to get the model to compile. Pretty darn ugly.

(2) The original "doesnotcompile" name for the PCAL and ESD line calculations were stuck in a subsystem I'd called TIMEDEPEDNENCE. Sadly, this totaly blew my channel character limit out of the water. This is a known feature that one can only have channels of 60 characters or less, I had just forgotten, and also didn't appreciate how many sub-sub-sub systems there were in this block. As such, I've reduced the sub system to the obfuscated "TDEP." Oh well.

It is my on-going attemp to damp the 501.606 Hz line as I'm slowly learning about damping filters. This mode is not touched by the Guardian and will always be off while I'm not using it. 

Added 1109 channels. Removed 524 channels.

Got notification somewhere around the CARM_ON_TR to REDUCE_CARM_OFFSET transition: NO IR in arms!!! Tried Sheila's alog suggestion of setting the ISC_LOCK guardian to manual mode and reselecting PREP_TR_CARM but it did not seem to work. Tried going back and forth between CARM_ON_TR and REDUCE_CARM_OFFSET and got brief spikes in TR_X_NORM and TR_Y_NORM but they did not last. Lost lock in middle of this. The ISC_LOCK guardian was still in manual mode when it lost lock and the guardian appeared to go haywire when it did. I think it may have been quickly flashing between all of the states. I quickly set it back to auto mode but something was still wrong. I tried different things including reloading the guardian but that ended up with a user error message. At the same time the DAQ was being rebooted and the nds servers went away. However only the SYS_DIAG guardian seemed to have an error related to this that cleared when it came back. The issue with the ISC_LOCK guardian user error message was still there. It seems to have been because when I reloaded the guardian it loaded code that Sheila was in the middle of editing. She commented something out and it was able to reload and move on. Not certain what if any of the problems leading up to this were related to nds restarts.

I got a SYS_DIAG notification that there was an issue with the NPRO noise eater. I think this may have been there since the PSL work. Sheila, Cheryl and I reset it. After that the ISS went into oscillation. Jason was able to fix it.

I have left the IFO in DC_READOUT while Sheila and Jenne investigate why an increase in power is breaking the lock.

Lessons learned:
When the ALS_COMM guardian notifies you to find IR by hand: Go to the ALS Overview, click on the lower left 80 MHZ VCO to open the H1:ALS-C_COMM_VCO medm screen and move the Set Off (Hz) slider to bring back the LSC-TR_X_NORM signal. This can usually be done by 3 clicks to the left.
When the ALS DIFF guardian notifies you to find IR by hand: Got to the LSC Overview and adjust the very small slider labeled ALS_DIFF at the bottom of this medm screen to bring back the LSC-TR_Y_NORM signal.
Dave says losing the nds servers *can* affect locking the IFO because the guardian connects to them.

Items completed today:
1. High Voltage Power Supply was installed (PEM-C1) and HV outputs were verified
2. Cosmic Ray Electronics D1500202 was installed (PEM-C1)
3. All BNC and MHV cables were terminated
4. Raymond Frey looked at outputs of cosmic detector - all within spec

Signals still need to be entered into DAQ (ongoing).

On the ISC_GUARDIANS.xml, ISC_LOCK pull down menu has two states named "Down" - why?

Picture attached.

The h1ldasgw0, h1ldasgw1, h1fw0, h1fw1, h1nds0, h1nds1, h1dc0 and h1broadcast0 computers were powered off and restarted this afternoon.  Updates were applied to the h1ldasgw0 and h1ldasgw1 computers to bring them to the same OS version.  An fsck was performed on h1fw0, which had gone 343 days without a check.  An fsck was also performed on h1nds0 which had gone 272 days without a check.

It is hoped that this maintenance will solve the instability which has caused the frame writers to restart many times per day over the last few days.

Things that probably happened, but of no impact to the IFO:

- WP5377 - Bubba, fan, LSB

Things that happened:

- EY to check SUS ESD cables, Andres, started 8:34AM, done 10AM, no cables changed

- WP5379 - JeffK, SUS and SEI IPC - done 9:00AM

- HEPI EY transition, Jim, started 9AM, done 10:49AM

- HEPI, Hugh, to EY, started 10AM, done 10:49AM

- ETMX charge measurement, Leo, started 9:50AM, done 10:54

- HEPI blend filters, Jim, EX, done 10:55

- WP5381 - Richard, Vault Seismometer, started 9AM, done 10:43AM

- High Bay door opened, Andres, 11:10AM

- WP5378 - Daniel, Beckoff, started 9AM, done 11:25

- PEM, mid-Y and EY, Vinney, started 10:46AM - done 11:34AM

- WP5350, Jason, PSL PMC, PMC and FSS electronics, Done 12:30

- TCS, Nutsinee, TCSX HWS align, started 10:50AM, Done 12:30

- WP5376 - Jonathan, CDS EPICS Gateway, started 8:24AM - assumed done

- ETMY charge measurement, Leo, started 10:50AM - assumed done

- ETMX PCAL, Sudarchan, started 9:58, Done 12:57PM

- WP5380  - JeffK, CAL-CS, started 12:54PM, done 12:57PM

- DAQ NDS0 full reboot, Dave - currently in process

Things that are waiting for a window of opportunity:

- WP5349, Filiberto, LVEA Cosmic Ray Detector, started 8:37AM, still work to do if there's a window, ~1 hour

- Low ESD at ETMX, Filiberto, still work to do if there's a window, ~30 minutes

To fix the problem of accidental PUM/ESD crossovers near the violin mode resonances, the PUM is now rolled off more aggressively above the crossover (see attachment of EY L2 LOCK FM6 vs FM7, where grey is old and purple new).

Previously, the PUM had an f² plant inversion out to about 600 Hz. It is now more like 300 Hz. There is also a broad notch around the first violin mode just to make sure that we do not have an accidental crossover there.

DARM OLTF attached (blue and red are essentially part of the same measurement). The high end of the phase bubble has flattened out a bit.

The PUM/ESD crossover was remeasured and was found to be satisfactory (attachment). Additionally, the rms drives to the three stages seem to be acceptable as well (attachment), although these were taken during low seismic activity.

We observed broadband coherence of OMC_DC_SUM with ASC_AS_C_LF_SUM and ASC_A_RF36_PIT. We made some numbers and plots, using the 64kHz version of the channels.

First the measurements we made on OCXO oscillator:
- ASC_AS_C sees a RIN of about 5e-7/rtHz above 100Hz (either from H1:ASC-AS_C_SUM_OUT_DQ or from H1:IOP-ASC0_MADC6_TP_CH11). The same is true for its segment 1.
- The calculated shot noise RIN at 20mA (quantum efficiency 0.87) detected is 4.0e-9/rtHz.
- The 4.0e-9/rtHz agrees with DCPD_NULL_OUT_DQ's prediction (8.0e-8 mA/rtHz/20mA).
- DCPD_SUM_OUT_DQ sees a slightly elevated RIN of 4.6e-9/rtHz (9.2e-8 mA/rtHz/20mA).

- The RIN in DCPDA (H1:IOP-LSC0_MADC0_TP_CH12, corrected for the whitening) is about 5.9e-8 mA/rtHz, or RIN = 5.9e-9/rtHz at 20mA/2diodes (~15pm DARM offset)...
- ...or about 3.3e-8 mA/rtHz or 1.2e-8/rtHz at 5.7mA/2diodes (~8pm DARM offset).

- ASC-AS_C_SEG1 (H1:IOP-ASC0_MADC6_TP_CH11) and OMC-DCPD_A (H1:IOP-LSC0_MADC0_TP_CH12) shows a coherence of 0.053 at 20mA, suggesting a white noise floor a factor of 0.23 below shot noise.
- At 5.7mA the same coherence is about 0.13, i.e. the white noise floor is a factor of 0.39 below shot noise.
- These two measurements are in plot 1.

- Taking the last two statements together, we predict a coherent noise of
  - 5.9e-8 mA/rtHz *0.23 = 1.4e-8 mA/rtHz at 20mA/2diodes (~15pm DARM offset)  (RIN of coherent noise = 1.4e-9/rtHz) - The pure shot noise part is thus 5.7e-8 mA/rtHz
  - 3.3e-8 mA/rtHz *0.39 = 1.3e-8 mA/rtHz at 5.7mA/2diodes (~8pm DARM offset)  (RIN of coherent noise = 4.5e-9/rtHz) - The pure shot noise part is thus 3.0e-8 mA/rtHz.

- AS_C calibration:
 - 200V/W (see alog 15431)
 - quantum efficiency 0.8 (see alog 15431)
 - 0.25% of the HAM 6 light (see alog 15431)
 - We have 39200cts in the AS_C_SUM. Thus we have
   - 39200cts / (1638.4cts/V) * 10^(-36/40) (whitening) / (200V/W) = 1.89mW and AS_C. (shot noi
   - 1.89mW/0.025 = 76mW entering HAM6. I.e. we have slightly more sideband power than carrier power (Carrier: 27mW in OMC transmission).
   - Shot noise level on AS_C_SUM is at 2.0e-8 mA/rtHz, corresponding to a RIN of 1.6e-8/rtHz. I.e. the coherent noise seen at 5e-7/rtHz is high above the shot noise. Dark noise TBD.
   - The light entering HAM 6 has a white noise of 5e-7/rtHz*76mW = 3.8e-5 mW/rtHz 
    

Bottom line:
 -We have ~1.4e-8mA/rtHz, or 1.9e-8mW/rtHz of coherent white noise on each DCPD.
 -It corresponds to 3.8e-5mW/rtHz before the OMC, i.e. the the OMC seems to attenuate this component by 2000.
 -This noise stays at the same level (in mW/rtHz) for different DCPD offsets.


Next, we switched back to the IFR for testing. plot 2 shows the same coherences (all at 5.7mA / 8pm DARM offset), but on the IFR. Interestingly now AS_C and AS_A_RF36 start seeing different noise below 2kHz. We convinced our selfs that the higher excess noise seen in AS_A_RF36 is indeed oscillator phase noise from the IFR - so that is clearly out of the picture once of the OCXO. (Evan will shortly log the oscillator phase noise predictions.)


64k Channel list:
H1:IOP-LSC0_MADC0_TP_CH12:     OMC-DCPD_A  (used in plot)
H1:IOP-LSC0_MADC0_TP_CH13:     OMC-DCPD_B
H1:IOP-LSC0_MADC1_TP_CH20:     REFLAIR_A_RF9_Q
H1:IOP-LSC0_MADC1_TP_CH21:     REFLAIR_A_RF9_I
H1:IOP-LSC0_MADC1_TP_CH22:     REFLAIR_A_RF45_Q
H1:IOP-LSC0_MADC1_TP_CH23:     REFLAIR_A_RF45_I
H1:IOP-LSC0_MADC1_TP_CH28:     REFL_A_RF9_Q
H1:IOP-LSC0_MADC1_TP_CH29:     REFL_A_RF9_I
H1:IOP-LSC0_MADC1_TP_CH30:     REFL_A_RF45_Q
H1:IOP-LSC0_MADC1_TP_CH31:     REFL_A_RF45_I


H1:IOP-ASC0_MADC4_TP_CH8:      ASC-AS_A_RF36_I1
H1:IOP-ASC0_MADC4_TP_CH9:      ASC-AS_A_RF36_Q1
H1:IOP-ASC0_MADC4_TP_CH10:     ASC-AS_A_RF36_I2
H1:IOP-ASC0_MADC4_TP_CH11:     ASC-AS_A_RF36_Q2
H1:IOP-ASC0_MADC4_TP_CH12:     ASC-AS_A_RF36_I3
H1:IOP-ASC0_MADC4_TP_CH13:     ASC-AS_A_RF36_Q3   (used in plot)
H1:IOP-ASC0_MADC4_TP_CH14:     ASC-AS_A_RF36_I4
H1:IOP-ASC0_MADC4_TP_CH15:     ASC-AS_A_RF36_Q4

H1:IOP-ASC0_MADC6_TP_CH11:     ASC-AS_C_SEG1  (used in plot)
H1:IOP-ASC0_MADC6_TP_CH10:     ASC-AS_C_SEG2
H1:IOP-ASC0_MADC6_TP_CH9:      ASC-AS_C_SEG3
H1:IOP-ASC0_MADC6_TP_CH8:      ASC-AS_C_SEG4

to help with getting the glitch rates, I'm running a script every minute which performs a DIAG clear on the models which are showing this issue. These models are: IOP-SUS[EX,EY], SUS-ETM[X,Y], IOP-SEI-E[X,Y], ALS-E[X,Y], ISC-E[X,Y].

Yesterday I started a cut-down version of this script which only cleared the ALS and ISC errors, however not every SUS glitch prodcues a remote IPC receive error so this was under counting.

We have noticed that in the past 20 hours only EY has glitched. We at still seeing two different types of IOP-SUS glitches either with or without a TIM bit setting.