11:45 Lockloss occurred without Verbal Alarm warning as it seems that VA had stopped.

12:25 FIrst re-lock attempt failed at ENGAGE_ASC_FOR_FULL_IFO - "H_T_T_S"

RF 90 started to increase, leading into the loss, causing a dip in RF 18

Second attempt is looking better. ASC loops took approx 6 minutes to converge.

13:09 increased H1:LSC-REFL_SERVO_IN1GAIN up 3dB from 9 to 12 at NLN upon RF 9/18 trace settling at 47 cts (on Craig's tutelage)

ETMY violin modes 1 and 6 elevated but ringing down. I increased the gain on Mode 6 to .6 from 0

13:10 H1 back to Observing

Today I took some frequency noise measurements.  Posted are the results and the file locations.

I increased CMB IN1 from +9dB to +12dB.  We should always do this after thermalization of the IFO.  
We can't increase the CARM digital gain too much or we'll hit the FSR with the CARM UGF.  We have to wait for the optical gain to decay and replace it with digital gain.
POP18 NORM MON is at ~47 cts now, but is at around 60 when we first lock.  We should wait until we hit around 50 cts, then up the CARM gain.


IMC Locked Alone at 35 W
- IMC REFL stitched spectra      : /ligo/home/craig.cahillane/Git/IFO/IMC/data/Spectra/StitchedSpectrum_20190423_IMC_REFL_IMON_Spectra_IMCAlone_35WInput_IN1_22dB_SecondBoostOn.txt
- IMC OLG                        : /ligo/home/craig.cahillane/Git/IFO/IMC/data/TFs/20190423_IMC_OLG_IMCAlone_35WInput_IN1_22dB_SecondBoostOn.txt
- IMC MCL Crossover              : /ligo/home/craig.cahillane/Git/IFO/FrequencyNoise/data/20190423_IMC_MCL_Crossover.xml
- MC2 to IMCF TF (Refs 16 and 17): /ligo/home/craig.cahillane/Git/IFO/FrequencyNoise/data/20190423_MC2_to_IMCF_35W.xml 

Full Lock at 35 W
- REFL B Spectra               : /ligo/home/craig.cahillane/Git/IFO/CARM/data/Spectra/StitchedSpectrum_20190423_REFL_B_35W_Input_CMBIN1Gain_9dB.txt
- REFL A Spectra               : /ligo/home/craig.cahillane/Git/IFO/CARM/data/Spectra/StitchedSpectrum_20190423_REFL_A_35W_Input_CMBIN1Gain_9dB.txt
- CARM OUT2 Spectra            : /ligo/home/craig.cahillane/Git/IFO/CARM/data/Spectra/StitchedSpectrum_20190423_CARM_OUT2_35W_Input_CMBIN1Gain_9dB.txt
- IMC REFL Spectra             : /ligo/home/craig.cahillane/Git/IFO/CARM/data/Spectra/StitchedSpectrum_20190423_IMC_REFL_35W_Input_CMBIN1Gain_9dB.txt
- IMC TEST1 Spectra            : /ligo/home/craig.cahillane/Git/IFO/CARM/data/Spectra/StitchedSpectrum_20190423_IMC_TEST1_35W_Input_CMBIN1Gain_9dB.txt

- CARM OLG w/ CMB IN1 = +9dB   : /ligo/home/craig.cahillane/Git/IFO/CARM/data/TFs/20190423_213400_20190423_CARM_OLG_FullLock_35W_Input_60mV_Exc.txt
- CARM OLG w/ CMB IN1 = +12dB  : /ligo/home/craig.cahillane/Git/IFO/CARM/data/TFs/20190423_214036_20190423_CARM_OLG_FullLock_35W_Input_60mV_Exc_12dB_CMBIN1Gain.txt
- IMC OLG w/ CMB IN1 = +9dB    : /ligo/home/craig.cahillane/Git/IFO/IMC/data/TFs/20190423_212909_IMC_OLG_FullLock_35W_Input_0dBm_Exc.txt
- IMC OLG w/ CMB IN1 = +12dB   : /ligo/home/craig.cahillane/Git/IFO/IMC/data/TFs/20190423_214142_IMC_OLG_FullLock_35W_Input_0dBm_Exc_12dB_CMBIN1Gain.txt

- LSC MCL Crossover            : /ligo/home/craig.cahillane/Git/IFO/FrequencyNoise/data/20190423_LSC_MCL_Crossover.xml
- MC2 to REFL9 Cal TF (Current): /ligo/home/craig.cahillane/Git/IFO/FrequencyNoise/data/20190423_MC2_to_IMCF_35W.xml  (This is the same as the MC2 to IMCF template above.  The current references are in full lock, Refs 16 and 17 are from IMC locked alone.)

- Frequency Noise Injs     +9dB: /ligo/home/controls/craig.cahillane/Git/IFO/FrequencyNoise/data/Injections/20190423/1240118413_GPSstart_FrequencyNoise_CMB_EXC_inj_2000_7000_Hz.pkl
                                 /ligo/home/controls/craig.cahillane/Git/IFO/FrequencyNoise/data/Injections/20190423/1240118674_GPSstart_FrequencyNoise_CMB_EXC_inj_600_2000_Hz.pkl
                                 /ligo/home/controls/craig.cahillane/Git/IFO/FrequencyNoise/data/Injections/20190423/1240118847_GPSstart_FrequencyNoise_CMB_EXC_inj_175_600_Hz.pkl
                                 /ligo/home/controls/craig.cahillane/Git/IFO/FrequencyNoise/data/Injections/20190423/1240119020_GPSstart_FrequencyNoise_CMB_EXC_inj_50_175_Hz.pkl
                                 /ligo/home/controls/craig.cahillane/Git/IFO/FrequencyNoise/data/Injections/20190423/1240119191_GPSstart_FrequencyNoise_CMB_EXC_inj_15_50_Hz.pkl
- Frequency Noise Injs    +12dB: /ligo/home/controls/craig.cahillane/Git/IFO/FrequencyNoise/data/Injections/20190423/1240119574_GPSstart_FrequencyNoise_CMB_EXC_inj_2000_7000_Hz.pkl
                                 /ligo/home/controls/craig.cahillane/Git/IFO/FrequencyNoise/data/Injections/20190423/1240119754_GPSstart_FrequencyNoise_CMB_EXC_inj_600_2000_Hz.pkl
                                 /ligo/home/controls/craig.cahillane/Git/IFO/FrequencyNoise/data/Injections/20190423/1240119927_GPSstart_FrequencyNoise_CMB_EXC_inj_175_600_Hz.pkl
                                 /ligo/home/controls/craig.cahillane/Git/IFO/FrequencyNoise/data/Injections/20190423/1240120100_GPSstart_FrequencyNoise_CMB_EXC_inj_50_175_Hz.pkl
                                 /ligo/home/controls/craig.cahillane/Git/IFO/FrequencyNoise/data/Injections/20190423/1240120272_GPSstart_FrequencyNoise_CMB_EXC_inj_15_50_Hz.pkl

- LSC MCL Noise Injection  +9dB: /ligo/home/controls/craig.cahillane/Git/IFO/MCL/data/Injections/20190423/1240123470_GPSstart_MCL_inj_5_200_Hz.pkl
- LSC MCL Noise Injection +12dB: /ligo/home/controls/craig.cahillane/Git/IFO/MCL/data/Injections/20190423/1240120857_GPSstart_MCL_inj_5_200_Hz.pkl

- PRCL Noise Injection     +9dB: /ligo/home/controls/craig.cahillane/Git/IFO/PRCL/data/Injections/20190423/1240123215_GPSstart_PRCL_inj_5_200_Hz.pkl
- PRCL Noise Injection    +12dB: /ligo/home/controls/craig.cahillane/Git/IFO/PRCL/data/Injections/20190423/1240121461_GPSstart_PRCL_inj_5_200_Hz.pkl

- Intensity Noise Inj      +9dB: /ligo/home/controls/craig.cahillane/Git/IFO/IntensityNoise/data/Injections/20190423/1240122973_GPSstart_Intensity_inj_10_7300_Hz.pkl
- Intensity Noise Inj     +12dB: /ligo/home/controls/craig.cahillane/Git/IFO/IntensityNoise/data/Injections/20190423/1240122556_GPSstart_Intensity_inj_10_7300_Hz.pkl


I upped the CMB IN1 gain from 9 to 12 dB for a couple of CARM and IMC OLGs.  From the CARM OLG (PDF 3), we seem to have about 3 dB of clearance from the FSR with CMB IN1 = +12dB.  
We know that frequency noise is starting to limit us with our squeezing levels.

TITLE: 04/24 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 106Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    Wind: 18mph Gusts, 16mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.15 μm/s
QUICK SUMMARY:

Smooth handoff of H1 in Observing following commissioning work by Georgia and Craig. A bit windy but nothing to be concerned with.

TITLE: 04/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 106Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

Managed 3min of OBSERVING for my shift!  (this was due to Maintenance recovery at the beginning and then Commissioning for the rest of the shift!)  Winds have died a little, but still blowing & useism still low!
LOG:

• 22:50 H1 kept breaking lock during ALS steps...eventually decided on doing an alignment.
• 23:07 - 23:54 Initial alignment 
  • Gave both arms over 15min for signals to converge
  • Input Align OK, PRC OK, Dark Mich was OK (took a while to go dark, but it did!), SRC OK
• 23:16 Christina back from both mid-stations
• ~23:50 SDF for PLC4 issue, noticed from Daves checker--requiring restart for its sdf
• 23:54 Locking
  • Dropping out at random spots (i.e. PRMI, DRMI, DARM_TO_DC_READOUT, etc.)
  • Could this be wind?  Alignment looks good---Locking PRMI & DRMI pretty fast.
• 0:33 Squeezer guardian work (Sheila)
• 1:30 OBSERVING
• 3:06:35:  Watched plane sweep down the DARM spectra from the bucket ~200Hz down to below 60Hz.
• 3:26 (Approved) COMMISSIONING Begins
• 3:34 - 4:48:  Craig out to the floor (PSL racks) for some set-up & measurements on the floor.
• NOTE:  Common Mode Board IN1 gain is currently 3dB higher for freq noise supression (see Craig's alog), but for next lock guardian will take it to 9 (vs 12 it is at now).
  • Additional Note:  This changed wasn't noted by SDF (so does that mean it's NOT MONITORED?)
• 6:56 Back to OBSERVING just in time for Ed!

Earlier in shift ran an alignment (took an hour, wanted to make sure ALS alignment was good).  Required several locking attempts in windy conditions, but finally made it to Observing.  Stayed in Observing for roughly 2-hrs (to let H1 heat up for Craig & Georgia).

• 3:26:  Bumped out of Observing due to Craig selecting (not running) an LSC excitation.  Decided to stay in Commissioning; planning to use ~3hrs of Approved Commissioning tonight. 
• 3:34:  Craig out to the floor (PSL racks) for some set-up.

(notified Doug L at LLO about our status via Teamspeak).

Probably happens all the time, but while we heard a plane flying by, Robert noticed it sweep on the running DARM spectra here in the Control Room (from the "bucket" down below 60Hz).

This roughly happened from 3:06:30- 3:06:45utc.

This morning while the IFO was still locked at the begining of maintence, Richard helped me cable the REFL B9I signal to the common mode summing junction.  We plugged it into REFL B IN2, which is labeled for REFLAIR on the medm screen.  We also added a label on the cable that is the same on both ends. 

We switched the CARM loop over to REFL B without any problems, using the same gain settings as are used for REFL A.  The CARM ugf stayed at around 10kHz.  We were locked with REFL B in loop and REFL A out of loop from about 15:38 UTC until 16:15 UTC.

The attachment shows the spectrum of REFL A B with no light, with A in loop, and with B in loop.  The REFL B readback is limited by ADC noise, so by switching the CARM control to REFL B we can use REFL A with it's better whitening to get a better out of loop REFL measurement.  The DARM spectrum had many large glitches during this time because the sensor correction was off and people were working on maintence in the VEAs, so it would be worth doing this again in low noise to look at coherences with DARM.  

TITLE: 04/23 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    Wind: 26mph Gusts, 23mph 5min avg
    Primary useism: 0.12 μm/s
    Secondary useism: 0.15 μm/s
QUICK SUMMARY:

• Working on getting H1 back.  Had trouble progressing past LOCKING_ALS.  Going to try for an INITIAL_ALIGNMENT.
• It's a bit breezy out with winds hovering around 20mph.  useism is just under 50% percentile.

Beckhoff Slow Controls Work:

Daniel, Patrick, Fil, Dave:

The upgrade of EY Beckhoff was stopped following problems. The corner station PLC1 was upgraded to add BUSY signels for the Beam Diverters. A DAQ restart was needed.

h1build reboot

Dave:

I've been getting errors on my daily builds due to the /tmp file system getting full on h1build. This machine had been running for 105 days, so I rebooted it.

h1edc channel stats hosted by new IOC

Dave:

I found that h1edc has problems trending its own EPICS channels (e.g. connected and non-connected counters). I am running a new IOC on h1fescript0 (under a controls screen session) which samples these channels one a minute and reposts them. The new channels have the names H1:CDS-EDCU_CHAN_[CNT, CONN, NOCON, TIMESTAMP]. I removed the h1edc channels and replaced them with the new ones in H1EPICS_DAQ.ini. A DAQ restart was needed.

Attempted scrub of /opt/rtcds ZFS file system

Dave:

The /opt/rtcds ZFS pool on h1fs0 has not been scrubed recently (a file system cleanup, similar to fsck). I started one by hand at 21:23 UTC and immediately saw problems. First it was very difficult to log back into h1fs0 to monitor the status of the scrub. After 3 minutes, the continuous wave psinject stopped running, and the DAQ-CRC error count on h1edc started ramping up from 0 to 115 in 6 steps.

While this was happening I was checking file access times on my workstation, but I could not see any slow down.

At 21:46 UTC I cancelled the zpool scrub.

I cleared the DAQ-CRC error counters. I started the psinject process on h1hwinj1 (as user hinj):

[hinj@h1hwinj1 ~]$ /home/hinj/Details/bin/x_start_psinject david.barker "restart after possible /opt/rtcds slow down"
Entered psinject_common.tcl
Sourcing /ligo/cdscfg/stdenv.tcl
Sourced stdenv.tcl
Sourcing /home/hinj/Details/psinject_config.tcl
Entering psinject_config.tcl
Sourcing /ligo/cdscfg/stdenv.tcl
Sourced stdenv.tcl
Sourced psinject_config.tcl
Note: H1 psinject WAS running as process 23420, but has died
Starting H1 injection into H1:CAL-INJ_CW_EXC using RELEASE -> preO3_H1
Injection will start at t=1240093622 (60 seconds from now)...
Injection starting now
 

Note that h1fs0 has been running since 06 June 2016 (1051 days). We have started planning its reboot/replacement.

DAQ Restart:

Dave:

DAQ was restarted at 12:41 PDT for the ecatc1plc1 and h1edc changes. Again, after the restart all DAQ data from h1sush2a was marked as bad. I restarted mx_stream on this machine and the data became good.

Test of trending h1edc channels in DAQ

Daniel restarted the ecatc1 IOCs at 15:23 PDT, which meant that about 21,000 EPICS channels were taken away from h1edc. This gave me an opportunity to check that the EDCU channel counts were trending correctly in the DAQ using the new IOC. All looks good, plot attached.

Unplugged a two unused extension cords.

Powered off the oscilloscopes next to the PSL racks after confirming with commissioners.

Noted clean room flaps on HAM5 (as in alog48549), as well on the East side of HAM3 (see picture).

Maintenance Timeline:

• 23-04-19 15:10 JaffB to End Stations
• 23-04-19 15:10 Karen to Mid Stations
• 23-04-19 15:10 Sheila to LVEA
• 23-04-19 15:10 Sheila out of LVEA
• 23-04-19 15:14 Robert to the LVEA
• 23-04-19 15:16 JeffB back from end stations
• 23-04-19 15:20 Robert to the PSL
• 23-04-19 15:23 (08:23) Chandra - Going to H2 to prep for vacuum system changes
• 23-04-19 15:34 (08:34) Niko, Sharon, - Going to End-X for PCal measurements
• 23-04-19 15:40 (08:40) Vanessa - Going to End-X
• 23-04-19 15:40 (08:40) Filiberto, Marc - Cabling in the ECR
• 23-04-19 15:42 (08:42) Niko - Taking End-X to Laser Hazard
• 23-04-19 15:45 (08:45) Chris - Going into the LVEA for battery testing
• 23-04-19 15:46 Richard and Sheila out of LVEA having swapped to a new REFL PD
• 23-04-19 15:47 (08:47) Christina - Going into the LVEA for inventory checks
• 23-04-19 15:58 (08:58) Cheryl - Focusing camera at HAM2
• 23-04-19 16:04 (09:04) Hugh - Going into the LVEA for inventory
• 23-04-19 16:10 (09:10) Sheila - Out of the LVEA
• 23-04-19 16:14 (09:14) Take IFO to Down
• 23-04-19 16:16 (09:16) Karen - Out of End-Y
• 23-04-19 16:16 (09:16) Chris - Out of LVEA - Going to End-X for battery checks
• 23-04-19 16:17 (09:17) UniFirst - On site to service mats
• 23-04-19 16:20 (09:20) Rahul - Runing charge measurements at End-Y
• 23-04-19 16:22 (09:22) Hugh - Out of the LVEA
• 23-04-19 16:32 (09:32) Vanessa - Finished at End-X
• 23-04-19 16:32 (09:32) Vanessa - Going to LVEA
• 23-04-19 16:34 (09:34) Narco - On site N2 delivery to CP2
• 23-04-19 16:35 (09:35) Kyle - Going into the LVEA
• 23-04-19 16:47 Kyle out of the LVEA
• 23-04-19 16:55 ChrisS leaving MY going to EY not in VEA
• 23-04-19 17:30 ChrisS moving the forklift
• 23-04-19 17:31 PSL down for water leak
• 23-04-19 17:51 Sheila to EY for GREEN troubleshooting
• 23-04-19 17:51 Fil to EY for cabling
• 23-04-19 17:52 Jason is restoring the PSL
• 23-04-19 17:59 DaveB to EY and EX CERs to label electronics
• 23-04-19 18:01 Paradise Water on site
• 23-04-19 18:19 Roberts crew into LVEA to remove cables-equip
• 23-04-19 18:38 ChrisS to LVEA to unplug the forklift
• 23-04-19 18:48 ChrisS out of the LVEA
• 23-04-19 18:53 DaveB back from one End Station, other deferred to
• 23-04-19 20:27 Craig to the LVEA for IMC measurements
• 23-04-19 20:27 Sheila and Fil to EY to look at and restart the green laser
• 23-04-19 21:07 Sheila and Fil back from EY - green laser is back on

There have been several times recently when operators had trouble relocking the squeezer after it unlocks and takes us out of observing, which seemed to be solved by re-requesting different states from the squeezer guardian. 

The reason the squeezer knocks us out of observing is because the SHG PZT hits it's rail, this seems to be happening ~once or twice a day. 

The reason the squeezer wasn't relocking is because the beam diverter wasn't re-opening after the squeezer relocked.  The problem was that the beam diverter won't respond to multiple requests to move that happen too soon after each other, and in some places the guardian was making the requests to rapidly.  

Daniel added a channel for the beam diverter called "busy" which can be checked before making a request of the beam diverter.  This problem should be avoided with changes to the SQZ_MANAGER for the time being, but in the future we can have a busy check for the beam diverter.

   This morning Robert noticed water on the floor of the PSL enclosure. Inspection showed the leak to be at the barb fitting on the output side of the Diode chiller filter. There was a bulge in the hose from the hose clamp to the end of the barb. The Teflon inner liner of the hose had failed at the barb, allowing water between the Teflon and the outer rubber, which created the leak.   

   Shutdown the laser and the Diode chiller. Cut out the bad section of hose and reconnected the hose to the filter. Powered up the system and noted no further leaks.     

This is a quick update on the noise budget injections that we did last week. The first attachment is the noise budget for a time with squeezing, the second attachment is for a time without. 

• To estimate the squeezing, I'm grabbing the data from the nonlinear gain monitor, which is the Q phase of the CLF diode normalized by the DC power on that diode.  This is proportional to the normalized nonlinear interaction strength x=sqrt(P/Pthreshold), and indicates that we had a nonlinear gain of 2 at the time.  The squeezing level is measured by a BLRMS normalized to the nullstream BLRMS at 4.68kHz, which reports 2dB of squeezing.  I'm assuming that the phase noise is 150 mrad, based on Nutsinee's measurements here: 47562 The noise budget then infers the losses based on the level of nonlinear gain and the measured squeezing level, in order to predict the anti squeezing level.  This is working OK for now, although the squeezing level is also limited by non-quatum noise which isn't taken into account.
• The PRCL injection does show up in the SRCL drive, but that is too small to account for the coupling to DARM, so it's added as a separate term again.  It also shows up in the out of loop frequency noise detector, but that is also small. (5th attachment)
• We are using REFL B9I to estimate the ambient level of frequency noise, similar to what was done here: 47351  There are nearly equal amounts of light on REFL A and REFL B, so the shot noise of REFL B will be causing this to be an overestimate of the frequency noise (by ~sqrt(2)) where it is refl shot noise limited.  On Tuesday I calibrated REFL 9I A into volts, so that we can compare directly REFL A+REFLB, this is shown in the 6th attachment. 
  • Upper left plot: Spectra in lock and with no light.  The whitening is different on A+B which explains the difference in the dark noise, REFL B is limited by ADC noise from 8-30Hz while we are locked, while for REFL A we must be limited by dark noise which is suppressed when we are locked.  In lock, at frequencies above ~1kHz where the loop is gain limited the two sensors agree.  According to Criag's modeling the frequency noise here is limited by the IMC detector shot noise.
  • Lower left shows the coherence between both sensors and DARM, with 1000 averages.  They are both coherent with DARM where the CARM loop is gain limited.  Where the frequency noise is limited by the REFL A shot noise, REFL B has more coherence with DARM than REFL A.  At these frequencies we could get slightly better frequency noise by summing the two detectors.
  • Craig is working on some modeling of the CARM loop that can be used to make a more accurate frequency noise projection, but for the time being the message is that our frequency noise is closer to DARM than it was before we increased the power. 
• For the ASC noise budget (third attachment), we also made injections for DC3+4 and INP1, but these don't show up in DARM for large excitations so they aren't included.
• The LSC noises are shown on their own here and the sum and residual are plotted in the last attachment.
• I'm using a logarithmic frequency vector, which makes the high frequency data look better than in past plots because of the binning.  This noise budget has never done a good job of estimating line heights, which is more noticeable now, especially if you look at the violin modes.
• I'm also now using the GDS correction calibration produced by pyDARM, instead of just compensating for the whitening of CAL-DELTAL, which helps the noise budget line up at high frequencies.
• The forest of lines around 20Hz is the ADS lines.

Some things that can still be improved in this noise budget:

• The PUM DAC noise hasn't been updated with the new measurements at LLO or the change in the coil driver state.
• A model of the CARM loop can be used to estimate frequency noise from the REFL A shot noise and the IMC shot noise separately.
• There are several other (smaller) noises that we have the data to include (DC readout upconversion, acoustic noise added by squeezer, frequency noise downconversion around the OMC dither line, PEM injection data) but aren't on here yet.
• There are some noises we haven't measured recently, like output jitter noise and any of the electrostatic noises.
• We can produce projections of the benefits of specific improvements from this.