Perhaps due to the rain that started, the h1ecatx1 Beckhoff computer has lost communications with the Wind Sensor 1,2  modules.  Both are now in the INIT NO_COMM state. I was unable to transition them to other states like PreOp, Safe-Op or Op, so it appears to be a cable issue.
- The error happened at 1:58PM PDT according the TwinCAT System Manager log

I just noticed that the h1calcs model is using its safe.snap file as its SDF reference and not OBSERVE.snap.

Looking at trends and logs, it appears that the following has happened (all times local):

Fri 05apr2019 16:01: switch_SDF_source_files.py was modified to add calcs to the exclude list, so guardian is not controlling its sdf reference

Tue 16apr2019 10:52: h1calcs model restarted, OBSERVE.snap reference set.

Wed 08may2019 00:40: h1calcs model restarted (Dolphin crash), safe.snap reference set (this is the current state).

There are differences between safe.snap and OBSERVE.snap (see attached text file).

Keita is in contact with the calibration group on this issue.

H1's back to OBSERVING after being down ~29hrs!  (~4hrs commissioning & ~25hrs due to H1 issues noted below).

Primary issues were related to:  ASC & IMC (MC1 & MC3) glitches.  Sheila will post summary about ASC & general locking for operators.  IMC glitches have been posted via Cheryl's alog+subentries and Betsy's alog this morning.

This morning I spent some time looking for more MC1 and MC3 glitches, reported by Cheryl, Jenne, Sheila yesterday https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=49257

The summary from Tuesday forward is that - 

• Bit of a struggle to recover lock from Tuesday, but then ~6ish hour lock into graveyard shift
• 5am (local) breaks lock - Cheryl find 1st glitch as culprit on MC3
• 3x more glitches:
• ~7am (MC3 only, mostly T1,2,3, little blip on other OSEMs)
• ~9am (MC3 only,mostly T1,2,3, little blip on other OSEMs )
• ~11:30am (MC1 only, mostly T2,l T3)
• NOON Fil does some power cycling (Coil drivers, AI chassis)
• ~5pm AA chassis power cycled
• There is only 1 little tiny glitch on MC3-T1 OSEM after the power cycles, at ~11pm last night...

It's not obvious that these glitches are continuing.  We scanned back a week and did not see any such glitches prior to their start at 5am local Wed. 

Also, they do not appear on PR2 (PRM and MC2 are hard to see due to all of the actuation on them which causes the signal to jump around.)

(Betsy, Fil, Marc)

I've created average spectra from days May 8, 9, 10, 11 and 12. You can see the plots in this link: https://ldas-jobs.ligo-wa.caltech.edu/~pep.covas/O3/MaySecondWeek/H1_clean-epoch_amp-wght/

One interesting new feature is around the 2nd order violin mode harmonics. Many new lines have appeared around the frequency 998.083889 Hz, which is a violin mode of ITMX which has been more excited than usual.

The first attachment shows a comparison with a previous week where this mode was not as excited (the second one is a zoom). You can see that 3 groups of lines have appeared on each side of this violin mode. Each of these groups of lines contains lines which are exactly located at frequencies which are a combination of the violin mode and one of the four calibration lines (15.6, 16.4, 17.1, 17.6), like 998.083889 - 17.6 Hz or 998.083889 + 15.6 Hz (all of these lines show values of bicoherence close to 1). In a previous alog (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=48161) this behaviour was already noticed, but now it seems that more than one group of nonlinear lines is created around the violin mode. The extra groups of lines are also located at nonlinear frequencies like 998.083889 - 2*17.6 Hz or 998.083889 + 3*15.6 Hz and also show values of bicoherence close to 1 (bicoherence between a calibration line and the previous group of nonlinear lines). I'm not sure if these extra lines are created due to a trilinearity (e.g. ASD(f1)^2 * ASD(f2) -> ASD(2*f1+f2) ) and higher order features or due to a bilinearity between the firstly created new nonlinear lines (closest group to the violin mode) and the calibration lines.

In the third image you can also see some lines around the first violin mode harmonics which are due to nonlinearities, as noticed in the past, although only one group at each side of the mode can be seen. 

In order to prevent this (which is highly contaminating the spectrum and impacting CW searches), we should try (if possible) to better damp the violin modes as soon as they are increased above "nominal" values, decrease the amplitude of the calibration lines or improve the behaviour of the DACs when driving near saturation (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=48357).

TITLE: 05/16 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    Wind: 6mph Gusts, 4mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.14 μm/s
QUICK SUMMARY:

H1 is down in the READY state due to problems  (i.e. alignment issues, IMC SUS OSEM glitches, etc.).  The OBSERVATORY MODE tool was in the "locking" state, so I transitioned it briefly to "COMMISSIONING" since [I thought] H1 is still set for higher power operations [heard this was backed out], but then second-guessed that since we really are having overall issues with H1 and will be taking us to the CORRECTIVE MAINTENANCE state (one should feel free to file an FRS....I can possibly do that).  Let's stay in the CORRECTIVE MAINTENANCE state and mark this down time until we get back to OBSERVING.  Waiting for reinforcements to arrive.

Cheryl posted an alog about possible noisy REFL signal.

I got on the LIGO Virgo Control Rooms teamspeak channel to apprise other operators of H1's status/plans.

TITLE: 05/16 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Corey
SHIFT SUMMARY:  Instability causing lock loss
LOG:

• Georgia was here here working on H1
• I've attached some plots of H1:ASC-REFL_A/B signals, and it's maybe known but I noticed REFL_B sum ringing up while REFL_A sum was steady or getting smaller.  a few other plots attached with the name describing the part of locking, signals are REFL and a couple core optics.

TITLE: 05/16 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 7mph Gusts, 4mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.14 μm/s
QUICK SUMMARY:  Georgia was here when I arrived, and worked on locking (this entry was made a bit after my arrival)

Cheryl, Georgia

We have had several ASC locklosses this evening from the INCREASE_POWER state, where we first power up from 2W to 20W. There is a 0.5 Hz instability in pitch, it looks like it shows up in the corner before the arms, but it really dominates everything so it's hard to tell.

We tried undoing the new phasing of AS_A_RF72 (SRC1 sensor), didn't help. I tried undoing the gain increase of INP1_P, still rung up. I switched the 0.5 Hz notches back on the DC yaw centering loops, also didn't help. I tried going through INCREASE_POWER by hand and we had a fast lockloss at 5W... I'm not sure why this is a problem now and I'm really stumped as to how to fix it. 

We've also had intermittent problems with the OMs saturating when the WFS are turned on in DRMI, and have had to clear their histories by hand a few times. 

I'm bypassing the RO alarms for tonight, I've contacted the control room and Bubba.

Bypass will expire:

Thu May 16 08:21:32 PDT 2019

For channel(s):

    H0:FMC-CS_WS_RO_ALARM

TITLE: 05/15 Eve Shift 23:00 – 07:00 (16:00-00:00), all times posted in UTC

STATE of H1: Locking

INCOMING OPERATOR: Cheryl

SHIFT SUMMARY: Lots of locklosses while trying to test out taking the power to 40W. It seems like there is some unrelated ASC issue that is making us lose lock within a few states of ENGAGE_SOFT_LOOPS.

LOG:

23:00 (16:00) Start of shift

23:20 (16:20) Lockloss from ENGAGE_ASC_FOR_FULL_IFO

23:43 (16:43) Lockloss from ENGAGE_SOFT_LOOPS

23:44 (16:44) Commissioners starting initial alignment/pico-ing

00:18 (17:18) Initial alignment done, relocking

00:48 (17:48) Lockloss from PREP_DC_READOUT_TRANSITION

01:17 (18:17) Lockloss from INCREASE_POWER

01:44 (18:44) Lockloss from DARM_TO_DC_READOUT

02:06 (19:06) Lockloss from ENGAGE_SOFT_LOOPS

02:20 (19:20) 5.7 earthquake near Papua New Guinea

02:36 (19:36) Lockloss from OFFLOAD_DRMI_ASC

02:49 (19:49) Lockloss from OFFLOAD_DRMI_ASC

Sheila, Georgia, Jenne

Today we took some commissioning time to try to increase the input power.  The increase in input power went fairly well, but we had many locklosses due to problems unrelated to power up, more likely related to Tuesday maintence.  In total we spent about 3-4 hours today commisioning, although the observatory mode was set to commisioning the entire time. 

Increasing power:

• We were able to increase the power to 40W requested input in the adjust power state without many problems, although we haven't had a stable low noise lock yet with the increased power.  
• We made some effort to make sure that it would be easy to revert the changes we made if necessary, and make it easy to come back to this effort another day just by changing the requested power for low noise in lscparams.
• As we increase the input power from 35W to 40W, the POP18 build up drops from ~50 cnts to eventually settle just below 40 counts.  To keep the PRCL ugf at around 50Hz, we need to increase the PRCL digital gain from 16 to 30.  We added a decorator to ISC_library that watches the value of POP18 and adjusts this gain in two steps as we thermalize, and we added this to several of the states that immediately follow the power increase.  This seems very similar to the expirence of increasing the power from 25W to 30-35W in December.  After some changes to TCS and spot positions we were able to prevent the PRCL gain and the POP18 from dropping as dramatically then.  For now it seems OK to digitially compensate for the change in optical gain until we have a chance to improve the build ups. 
• The CARM ugf stays pretty close to 7kHz in these steps (It doesn't get increased to 14kHz until the LASER_NOISE_SUPPRESSION state). 
• Georgia changed the POWER_30W state to MAXIMUM_POWER, and it is written so that all we need to do to change the requested power from 35W to 40W is to change the NLN_POWER in lscparams.
• We made some slight adjustments to the MICH and SRCL gains in LOWNOISE_LENGTH control based on ugf measurements: MICH gain is increased by 10%, SRCL gain 55%.   These are set based on the power request in lscparams.
• We had no difficulties with the noise reduction states after powering up, until we tried to reduce the 9MHz modulation depth.  This seems to have created an ASC instability, which caused enough noise that we started to saturate the ASC REFL45 sensors.  We reduced their whitening gain by 6dB and added 6dB digital filters, but we have not been able to relock since then because of other prolems.  Because we have been having unrelated ASC problems, we reverted the change to be sure that nothing we did today could be the cause of the locking issues. 
• Our rough plan for the next time that we are able to try going to 40W would be to sit and let things thermalize without reducing the 9MHz modulation depth. There we could check the phasing of the REFL WFS, measure the ASC sensing matrix, and perhaps try tuning CO2s to improve the POP18 build ups.

Other problems:

most of our locklosses today were unrelated to powering up, we lost lock twice for reasons that were related to our commissioning efforts, the rest of our many locklosses seem to be related to other problems.  We have had MC1+3 T2 osem glitches 3 times today, 49257 the first two times caused locklosses, and the third time we were able to stay locked.  

We also have had many similar locklosses in the states where we engage ASC and transition to DC readout before powering up.  We don't understand this, and the problems seem to have started yesterday.  

• We saw that the M3 picomotor which was moved yesterday (49238) wasn't set quite back to the original location according to the counter (the counter isn't a reliable way of knowing where the pico is, but it is the best thing we have).  We tried to undo this move by running the green WFS and camera, then opening the loops and moving the pico back to its original counter position (moving it by about 130 counts down).  After that we relocked the arm in green, and ran a complete initial alignment.  According to the sliders and top mass drift monitors this brought our alignment closer to what it was before maintenance day yesterday, but not all the way back. 
• We noticed that both SRC2P and INP1P were not bringing their error signals to 0 when we are engaging the ASC, so we have increased their gains, SRC2 will be reset latter in the guardian but is left with a factor of 3 higher gain for the ASC engagement and power ups.  For INP1 we have increase the gain by a factor of 10, and this will stay increased in full lock.  
• In our most recent locking attempt we phased AS72 after the ASC came on but before the soft loops were engaged.  We opened the SRC1 loops and adjusted the SRM alignment by hand, then adjusted the phases of AS72A to zero the Q signal in each quadrant.  This resulted in better POP18 and POP90 build ups, and we were able to get through the following steps.  We will see if this has fixed the problem or if we just got lucky this time.

TITLE: 05/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Niko
SHIFT SUMMARY:

Most of today has been devoted to commissioning hours with high power work (going up to 41W up from nominal 35W).  There have also been more glitches with MC1 & MC3 today (leading to power cycles of boxes in the rack).
LOG:

• 15:00 Initial Alignment
• 16:06 Commissioning for higher power work begins (Jenne, Sheila, et. al)
• 17:31 MY visit (Chandra)
• 19:08-19:15 Power cycle IMC sattelite box for MC1 & 3. (Sheila, Fil)
  • 22:50-23:00 Power cycled another box (Jenne, Fil)
• Handing off h1 to Niko for locking for the commissioners.

Ops Shift Transition: 05/15/2019, Eve Shift 23:00 – 7:00 (16:00-00:00) - UTC (PT)

State of H1: Locking

Intent Bit: Commissioning

Weather: ~10 mph wind, cloudy

Primary 0.03 – 0.1Hz: 0.02 um/s

Secondary 0.1 – 0.3Hz: 0.1 um/s

Outgoing Operator: Corey

Quick Summary: Re-locking after dropping out from REDUCE_RF9_MODULATION_DEPTH. Commissioning work going on at 40W.

The LockLoss at 12:51:54 UTC was preceded by 4 seconds, by a Verbal Alarm for MC3, at 12:51:54 UTC.

Attached is a time series of the MC3 M1 OSEMS, and the  H1:SUS-MC3_M1_OSEMINF_T2_OUT_DQ starts to glitch at 125.84 seconds into the plot, while the other MC3 M1 OSEMS, and Cal Delta and OMC DCPD A, are unaffected.

Other MC3 M1 OSEMS glitch next, then LockLoss.

Attachments:

• 1) 0 to 125 seconds into the plot, MC3 T2 has not glitched
• 2) 0 to 126 seconds, which includes LockLoss
• 3) 125 to 126 seconds into the plot, showing MC3 T2 glitches at 125.844 seconds into the plot, then the other MC3 OSEMS gltich, then LockLoss

While Relocking, DRMI lost lock and that also gave a Verbal Alarm for MC3.

Sheila, Hugh (remotely), Niko, Georgia

After an epic battle to reacquire lock (see Niko, Sheila, and Jenne's posts about alignment references and DC centering loop notches), we found a couple of strange SDF differences.

HAM3 and HAM4 GS13's

The GS13's for HAM3 and HAM4 were in a non nominal state, with filter differences for H1:ISI-HAM[3,4]_GS13INF_[dof] for each degree of freedom (H1 H2 H3 V1 V2 V3). The Gain and DWH (de-whitening) filters were on when we reached nln, where nominally they are off. Hugh told us how to fix the problem (sitemap -> ISI -> HAM3 -> Commands -> GS13 !HI). Doing this did not break the lock.

We're not sure how we ended up in this state, maybe after the large earthquake today things were not quite returned to normal?

PRM M3 output filter "differences"

The SDF reported differences with PRM M3 (see attachment), but we found the filters were correct given the state of the coil drivers. Time machining to the last lock, we don't think there was really a difference. 

M. Pirello, D. Gustafson

We installed new v3 Baluns on the following signals in the CER:

ISC-C3(37-1) 40Mhz TCS AOM Return exchanged v1-117 with v3-S002

ISC-C3(33-4) 158.8Mhz Fiber Beat Note Return exchanged v1-193 with v3-S003

ISC-C3(33-5) 79.4Mhz SQZ VCO Return exchanged v1-151 with v3-S004

ISC-C3(33-6) 203.125MHz SQZ VCXO Return exchanged v1-052 with v3-S005

ISC-C4(39-3) 79.4MHz PSL VCO Return exchanged v1-093 with v3-S010

These are all return signals, the impact should be minimal.  Work was completed per WP8150

Balun status can be seen here E1900100.