TITLE: 07/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY: Still have not recovered from the power glitch. Seems that this ~0.5Hz oscillation in MICH P is the main cultprit, but we just lost lock at CARM_ON TR. I have a list of a few things to try from Jenne if I can get to DC Readout.

TITLE: 07/25 Owl Shift 07:00 – 15:00 (00:00-08:00), all times posted in UTC

STATE of H1: Corrective Maintenance

INCOMING OPERATOR: TJ

SHIFT SUMMARY: Still working to recover from power glitch yesterday. Locklosses continue at INCREASE_POWER, despite trying a few recommended changes by commissioners. Since my mid-shift post, I tried letting the violin modes ring down before increasing power (only got down to ~4.5 and lasted the longest of any attempts), realigned the OM’s to previous lock values, and tested that the IMC stays locked when powering up to 10 W (and that PWR_SCALE matches power).

LOG:

07:00 (00:00) Start of shift

07:16 (00:16) Starting initial alignment

07:36 (00:36) Initial alignment complete, starting re-lock

07:55 (00:55) Lockloss at CARM_TO_TR, looks like we lost DRMI while ramping down LSC-REFL_SERVO_IN2GAIN.

08:43 (01:43) Lockloss from INCREASE_POWER, after waiting at ENGAGE_DC_VIOLINS for ASC channels to converge

09:38 (02:38) Lockloss from INCREASE_POWER after reducing MICH_P gain by 70%

13:04 (06:04) Lockloss from INCREASE_POWER after waiting for violins to go down a bit

14:47 (07:47) Lockloss from INCREASE_POWER after aligning OM’s to previous config (to reduce AS_A_NSUM)

15:00 (08:07) End of shift

At the beginning of the shift I did an initial alignment and (after a lockloss at CARM_TO_TR) reached ENGAGE_DC_VIOLINS. From here I waited for the ASC signals to converge a bit and went to INCREASE_POWER. At this point, several violin modes were high but not increasing (ETMY 1,6   ITMY 7   ITMX 13). We lost lock after about 15 seconds, with MICH P ringing up the loudest/maybe earliest. The last relevant command in the log was changing LSC_PRCL1_GAIN to 10. Looking at the violin modes before this lockloss, I don't see any of them ringing up significantly.

The next time I reached ENGAGE_DC_VIOLINS, I checked RF18 and RF90 against their ENGAGE_DC_VIOLINS values during the last lock (per suggestion #3 in 50796):

RF18 (before): ~98,         RF90 (before): ~33.5

RF18 (now):    ~101.5,     RF90 (now):    ~32.2

I reduced MICH P gain by 70% (per Keita's suggestion) and tried INCREASE_POWER. We lost lock again, with CSOFT and DSOFT railing right around lockloss. The last relevant command in the log was maybe setting TCSX,Y power requests to 'PRE-HEATING'?

At this point I realized that Travis' improved violin damping gains were not hard-coded yet, and that I had to change them manually. I have since done that, and am currently at ENGAGE_DC_VIOLINS waiting for ETMY modes 1 and 6 to decrease.

Additionally, (per suggestion #1 in 50796) I checked the IMC OSEM's against the last lock and they have very similar values. Optics that were off by more than 15 (somewhat arbitrarily chosen) from their values during the last lock are listed below:

IM4: 50 off in pitch

RM2: 30 off in pitch, 80 off in yaw

OM1: 30 off in pitch

OM2: 50 off in pitch

OM3: 70 off in pitch, 85 off in yaw

Ops Shift Transition: 07/25/2019, Owl Shift 07:00 – 15:00 (00:00-08:00) - UTC (PT)

State of H1: Lock acquisition

Intent Bit: Commissioning

Weather: 0-5 mph wind

Primary 0.03 – 0.1Hz: 0.01 um/s

Secondary 0.1 – 0.3Hz: 0.1 um/s

Outgoing Operator: Travis

Quick Summary: IFO is losing lock at INCREASE_POWER. Going to do an initial alignment, and then I have a list of investigation leads from Keita and Jeff to work with.

TITLE: 07/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Niko
SHIFT SUMMARY:  H1 is still not able to make it to NLN.  Worked on damping possibly problematic violin modes to no avail. 
LOG:

23:02 Kyle to MY

23:06 Laurence to optics lab

23:30 Kyle back

0:45 Laurence out

5:57 ETMy violin modes have damped to sufficient levels.  I moved on with locking, but it lost it at INCREASE_POWER, which I believe as far as it has made it today.

ETMy Modes 1 and 6 were relatively rung up.  In order to eliminate these as a problem with locking going forward, I spent several hours tonight hunting for better gain settings and damping these modes (Mode 1 was not damping with the preset gain and Mode 6 gain was 0).  The best settings I found for these two modes were:

J. Driggers, J. Kissel, T. Sadecki

We've been unable to recover further than Corey's last note in his log,LHO aLOG 50772, namely, we've been in 2W DC readout, and begin the power up sequence and lost lock 3 times.

We've since gone through an initial alignment -- and been having finnicky, intermittent and thus surpassable issues with ALS_XARM green WFS -- and are now struggling even to recover DRMI, without any knowledge of why. Assuming that some other problem hasn't cropped up, I write the rest of this aLOG assuming we'll just push through these annoying lock-losses and eventually get back to debugging the power up.

Jenne and I are running out of steam for today, but we still have a few fox holes to look into. Here're those ideas such that others can continue work.

List of current suspects:
(1) Alignment
looking through the SDF system, there's honestly too much different between the safe.snaps and current EPICs values to understand what's wrong and what's normal. However, one thing that caught our attention was the IMC optic's alignment. Because the IMC was functional after Peter, Niko, and Jason's hard work (see LHO aLOG 50779), Kissel and Corey didn't restore those optics.

(2) ISS 2nd Loop 
The diffracted power is taking quite a dip during IMC lock losses, and we lose lock right at the start of powerup with the full IFO. 
We should check health / settings of the ISS second loop. 
The theory being that if the ISS 1st loop is in a bad state / flaky, then the 2nd loop won't work.
We know at least that we've gone through initial alignment several times, so we can successfully power up with the 1st loop on, but increase power with the IFO is the first real test of the 2nd loop/ 

(3) DRMI Buildups
trend the POP18 and POP90 values at DC readout (ENGAGE_DC_VIOLINS), see if they're the same as previous successful locks before this morning issues

(4) Violin Modes
We noticed while in DC readout that ETMY MODEs 1 and 6 were particularly rung up. So, maybe some PD is saturating once we start to increase the power a bit.

(5) 9 MHz EOM driver
Just looking at the wall FOM for the 9 MHz EOM modulation index driver, it looks a little higher. would be good to compare the current ASD against prior-to-outage ASD.

TITLE: 07/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    Wind: 13mph Gusts, 10mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:  Still in recovery from last night's power glitch. 

TITLE: 07/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Travis
SHIFT SUMMARY:

H1's been down 19+hrs after last night's lightning storm.

Whew!  What a night & day!  CDS Team continued their effort from last night and finished the remaining issue with EY at around 12:20pmPDT to allow us to start locking.  Jeff K sat with me almost entire oshift to help out (& grabbed others when possible).  So far have found two issues which were the result of outtage/recovery, and it was not catching:

• ETMx ESD was OFF/stuck (needed power-cycled!) [JeffK toggled]
• OMC High Voltage PZT being OFF  [Jenne turned ON in LVEA]

Ended day with issues at INCREASE_POWER.

LOG:

• 15:12 Vibrometer work in LVEA (Robert, Matthew, others?)
  • 18:06 out and LVEA swept
• 15:13 Vacuum dustmon pumps at end stations (Jeff)
• 15:40 Work on EY communication (Richard)
• 15:54 Toggle EY ion pump (Gerardo)
• Long range reboot 
• 17:09 Reports of smoke, Jeff J checking 
• 19:20 Dolphin fixed! 
• 19:27 Initial Alignment begins
  • Y-arm was way off, touched ETMy, TMSy, and also ITMy
  • PRC was stuck in acquire
    • Jeff restored alignment for PRM (03:30utc)
  • MICH had to go DOWN once, but completed on 2nd attempt.
  • SRC was stuck in a loop (SRM was restored to 3:30utc value)...but before we could get help, it COMPLETED on its own!
    • But TJ said we could have just misaligned SRM & tweaked SR2
    • (but if guardians fight you, you can PAUSE both guardians (ALIGN_IFO & INIT_ALIGN)
• 20:00 Lock #1
  • Lockloss at LOCKING_ALS
• 20:01 Lock #2
  • Lockloss at LOCKING_ALS
  • troubleshooting work begins!
  • Keep getting stuck with early steps related to ALS_XARM & ALS_DIFF
  • ETMx ESD appears to be the issue!  (thanks for finding Jeff & Daniel/Keita for looking over our shoulders)
    • 20:42 Jeff toggled the ESD button for ETMx & immediately saw normality for ETMx & we were able to finally continue with locking!!
    • "Is there a way to get ETMx ESD issue on DIAG_MAIN?" asks JeffK.
• 20:55-21:01 LVEA anemometer check (Richard on roof)
• Lock #1 (21:00) (starting over after the ESD issue)
  • Had to FIND IR  twice so far.
  • Locked PRMI
  • Stuck at PREP_DC_READOUT_TRANSITION for a long time (this was later discovered due to the OMC PZT High Voltage being OFF).  There were complaints about the OMC not locking (due to PZT!) & since we were stuck at 2-W for so long Jenne noticed ASC ringing up (H1 doesn't like being stuck at 2W).  Temporary fix was reducing ASC-INP1 pit & yaw gains....this was to allow Jenne to run out on the floor to power on the PZT:
    • Turned ON the High Voltage for the OMC PZT (Jenne turned it "ON" by hitting "recall" + "ENTER" on the box for this)
    • This would have been another nice thing for "Diag_Main" to catch for us!
  • Lockloss at INCREASE POWER...thought it was due to us returning the INP gains (but had lockloss here again when we were NOT messing with ASC gains
• Lock #2 (22:00):  Check IR needed no action.  PRMI locked quick & with high powers.  DRMI locked fast.  OMC locked normally.  All of this looked BETTER!
  • Lockloss at INCREASE POWER for 2nd time.
• Lock #3 (22:35)
  • DRMI locked nicely (didn't need to lock PRMI)
  • Currently holding at ENGAGE_DC_VIOLINS
    • ETMY MODE 1 & 6 looked elevated (Jeff was going to increase gain
    • Lockloss at INCREASE POWER for 3rd time.
• Hand off of an unlocked H1 to Travis.

Something probably not covered in the other logs, the seismic platforms were not in their proper state when I came in this morning. ITMY_ST2_SC node was complaining and the ISI_CONFIG screen showed that HAM1 sensor correction was off. I think this happened because all of the seismic models were restarted, but the guardian machine did not get restarted. This meant that the SEI_CONF guardian and the ops overview showed that the seismic was in the nominal state, but because the way those guardians are implemented is kind of flexible, it did not force the right configuration. I fixed this by requesting the RECOVER_SC state on the SEI_CONF guardian, then going back to WINDY.

In the future, probably any seismic model restart should be followed by taking SEI_CONF to the SC_OFF state, then selecting whatever the appropriate environmental state is.

Wanted to make sure I was understanding my memory correctly.  Bottom line, I was not mistaken!

In RobertS's alog 28456, he details the PEM Anemometer directions which are not standard in discussions about wind where typically, the wind direction is from where the blows.  Robert's scheme puts zero degrees at to the North and 90^o to the East and 180^o to the South, etc.  To confirm this today, Richard graciously eased my concerns with work permit and leg work up onto the roof manually positioning the direction sensor while we watched the readout.  With the tail of the sensor flowing from the CS to the EndX and the tip pointing to the South, we read close to zero degrees and continued in this fashion moving the tail of the sensor clockwise.  The attached shows a StripTool readout while Richard did this and on the phone calling out the sensor position.

So, using the wind sensor direction info must be done with this in mind and is applicable to me as I look at historic wind direction.  Namely for a standard Matlab polar plot where the angles are plotted counterclockwise, the 90 and 270^o directions are correct for the standard from direction; but, North & South are flipped and all directions 0^o +- 90^o must be flipped to 180^o +- 90^o and vice-versa.  Anyway, I'm learning/working this but if you do it, be aware.  Come see me if you care/wish.

C. Gray, K. Kawabe, J. Kissel, D. Sigg

After initial alignment was complete (2019-07-24 ~20:00 UTC), we spent a bit of time stuck with several lock losses upon reaching LOCKING_ALS. Manually running what happens in LOCKING_ALS, we narrowed down the problem to ALS DIFF, and after 45minutes of searching, I happened to lance at the ETMX ESD driver's analog voltage monitors and noticed they looked abnormal. At 2019-07-24 20:42 UTC, I hit the H1:SUS-ETMX_BIO_L3_RESET ("HV ON/OFF") button on the QUAD overview screen once to turn OFF the driver, then once again a few seconds later to turn it ON.

It's likely that the ETMX ESD driver got locked up in a funky state after all of the power and computer issues over night. 

The bigger question is why (what we thought were several) warnings that should have been triggered didn't:
    (1) DIAG_MAIN guardian node "used to" (our memory of the O2 past) have a check of the analog voltage monitors, and would throw a warning.
    (2) The DOWN / PREP_FOR_LOCKING state should have a similar check of the analog voltage monitors, throw and error when failed, and not allow progress until fixed.

Others are investing further and will comment / fix later.

I was interested in knowing what triggered the last two LHO wind locklosses, so I ran an analysis code to try understand what dofs are the culprit.

In both case (#1 1247970127 and #2 : 1247975795), the online lockloss show EX L2 saturating first.

The drive is dominated by the length signal for both instances which I presume is from DARM. This is shown in the time series breakdown in figure1 and figure2, where we see that the blue curve (length signal) is the largest contributor to the sum of L, P and Y drives. At LLO one of our wind lockloss was rather from an angular dof saturating L2 EX (see 45237).

Looking at the spectra of the individual dofs contributing to the actuator signal, we see the length drive dominates largely up to 0.3Hz, see figure3 and figure4.

Comparing various control signals spectra for event #1 at the begining vs end of the lock (less vs more windy), we see the main difference is below 0.5Hz, with a factor of ~5 increase of the drives (for a x2 wind speed increase), see figure5.

Note that for both locklosses, L1 stage of EX is only using at most a fifth of its drive range (figure6), so it might be worth increasing the L1 EX control authority for DARM.

Richard, Fil, Rolf, Dave:

Summary: long-range Dolphin IPC data is flowing again from EY to CS  after we moved its Dolphin switch port.

Details:

Here is a summary of what we tried to get EY->CS data flowing over the past 14 hours:

rebooted h1cdsrfm

power cycled h1cdsrfm and both EX,EY Adnaco expansion chassis

power cycled EY IXS600 Dolphin switch

Checked fiber optic power to/from EY Adnaco SFP

Replaced IXH611 Dolphin card at EY with spare

Moved cdsrfm switch port from port 6 to port 4 (Fixed problem)

I'm in the process of changing the dolphin configuration to follow the port move.

J. Oberling, P. King, T. Sadeki, N. Lecoeuche

This is a catch-up alog to document my involvment with recovering the PSL from the trip caused by last night's power glitch.  I should have been entering these last night as we were recovering the PSL.  I did not think to do so, and therefore left others recovering other parts of the IFO uninformed regarding PSL status.  That's entirely on me and I apologize.

Travis called me at ~10:50pm informing me that the PSL had gone down during the thunderstorm due to an apparent power glitch, so I walked him through restarting it.  On the STAT screen we saw that the NPRO had tripped off, likely due to the glitch.  The chillers were still running nominally and the external shutter was still open, so most of the system was already ready to go and we restarted the laser without issue.  Before trying to lock the PMC, we turned off the FSS and the ISS; the FSS was turned off so it wouldn't constantly be driving the NPRO to attempt to lock the RefCav (which it couldn't do since we had no PMC), and we turned the ISS OFF as, with no PMC there was no light on the ISS PDs, so the loop was oscillating.  This done, we attempted to lock the PMC, but no luck.  As Dave was still recovering from the Dolphin crash and it was nearing the end of Travis's shift, we decided to wait until attempting to lock the PMC.  In the interim I tried to figure out why the PMC wasn't locking.  Unfortunately for me, it took my sleep-deprived brain over an hour to notice the obvious: the PMC PZT was not responding, at all.  So, first things first, I asked Niko (now on shift) to check the PZT HV power supply.  This supply lives up in the mezzanine of the CER, so Niko and I set up a quick buddy system via email for his safety while climbing up and down that steep ladder to the CER mezzanine, and he went to check on the power supply.  He got back to me in ~10 minutes with the attached picture, indicating that the power supply had indeed tripped in the power glitch.  At this point my even further sleep-deprived brain (it was after 2am) failed me completely; I could not remember the correct voltage and current settings for this power supply (they must be entered manually), so I asked that Niko, when he was ready to restart the power supply, call Peter for assistance.  Peter was able to walk Niko through recovering the PMC PZT power supply, as well as the PMC, ISS, and FSS, and this morning the PSL is up and running as normal.

For the record (and for future reference), the correct voltage and current settings for the PMC PZT power supply are posted on the power supply itself.  They are: Vset of 375V (press the Vset button, enter 375, press enter), Iset of 50mA (press the Iset button, enter 50, press enter).  Press the Output On/Off button to enable/disable the power supply output.

Corey called shortly after 14:00 PDT and informed me that the TCSy laser was off.  Looking at the MEDM screen, the RTD/IR SENS. ALARM was red and the laser was off.  I went to the LVEA and reset the laser at the front panel; I also took a picture of the front panel before I reset it, I'll attach it as a comment (it's on my phone at the moment).  This cleared the temp alarm and the laser restarted without issue.

Trending back with ndscope (see 1st attachment), the laser tripped off around 11:41am PDT.  The laser temperature was holding steady at the time of the trip, so the laser did not overheat.  While the laser was off, the chiller setpoint was stepping from 20 °C to 21 °C in 0.1 °C steps; it went through this a couple times, as seen in the attachment.  Seeing this, I went out and checked on the chiller and found it was reporting a water temperature of 22.3 °C; the setpoint for the laser after restarting was 20.9 °C.  The 2nd attachment shows the water temp reported by the chiller starting .  At the time of the trip the chiller was reporting a water temp of 22.0 °C, and is currently reporting 22.1 °C (There appears to be a descrepancy between the chiller setpoint and the water temp reported by the chiller, which I have never noticed before.  I.E. before the trip the setpoint was just over 20 °C, while the chiller was reporting a water temp of 22.0 °C.  Is this normal?  Or could this be part of the cause of the trip?).  At this point it is unclear what caused the laser to trip.  Investigation continues.

I should also note, while I was looking at the TCSy chiller, I ran into Jeff B.  He pointed out something interesting, the small display screen on the TCSx chiller has gone blank.  The chiller is still running and the TCSx laser appears to be working fine, so it appears the display has simply quit working (I've never seen them blank before (like one would expect when a display goes to sleep), everytime I go to check on the chillers they displays are active).

Edited at 15:35 PDT as I hit the POST button a little too early.