Ops Owl Shift Summary

TITLE: 07/11 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Ed
SHIFT SUMMARY: 6.6M earthaquake at the start of my shift left us waiting for a few hours, then some trouble keeping the mode cleaner locked after initial alignment, followed by ETMX violin mode 507.194hz ringing up. With +60 phase and +0.25 gain it seems to be stable(ish) at an RMS of 7. The OMC trans camera has a lot of movement, it seems to really be struggling, and the OMC_LOCK Guardian will not get to READY_FOR_HANDOFF presumably from this high violin mode.

LOG:

• 07:56 Lock loss, 6.6M earthquake south of New Zealand.
• 12:54 Restarted DIAG_MAIN
• 14:43 Peter, Jeff B to PSL to start maintenance
• 14:58 Lock loss. starting maintenance.

Head flow rates

Attached are plots of the flow rates through heads [1,4] for the last 12 hours and the last hour.  Apart
from the initial restart period, the flow rates more or less settled down and looked reasonably clean.
However in the last hour they took a downturn back to rates similar to when the chiller was restarted
last night.  The crystal chiller flow rate appears constant at this point in time, at least within the
minute trend data.

    The crystal chiller flow rate has a resolution of about -/+ 0.1 lpm.  The observed decrease in the
flow rate to the heads is ~ -0.025 lpm.  So in retrospect, perhaps the observed decrease in the head flow
rates is due to a reduced flow from the crystal chiller.

Ops Mid Shift Report

After the 6.6M New Zealand quake I ran an initial alignment with no issues, but then it would lose lock very randomly. I couldn't make it past DRMI for a long time, with the FSS oscillating often on lock losses.

I have finally made it to DC_READOUT_TRANSITION but the 507.194hz ETMX mode has been ringing up with the current settings from Guardian. Changing the phase from +60 to -60 started to help, but now it seems that to have flattened out. There are no +/-30deg filters in that bank so I'm pretty limited, but a smaller positive gain (~5) with -60 gain seems to be my best option.

Ops Owl Shift Transition

TITLE: 07/11 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
    Wind: 8mph Gusts, 6mph 5min avg
    Primary useism: 1.74 μm/s
    Secondary useism: 0.10 μm/s
QUICK SUMMARY: When I got here Patrick was battling some PI's while sitting at INCREASE_POWER. DCPD saturations were continuous and DARM looked bad. With neither of us finding the problem, seismon reported an incoming earthquake that just broke the lock.

Ops Eve Shift Report

TITLE: 07/11 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Peter and Jason brought the PSL back. I ran an initial alignment and made it to NLN. Sat at NLN trying to damp 507.194 Hz and 1463.09 Hz violin modes. I think the later was starting to come down after setting ITMX mode 3 to only FM4 and FM5 and a gain of 100. I think I might have made the former worse by playing around with EMTX mode 6 and 7. Eventually lost lock, possibly changing gain of ETMX mode 7, but I had done this many times before. Upon starting INCREASE_POWER during relocking I started getting constant OMC DCPD saturations and a whole series of peaks in an elevated DARM. Eventually the lock broke at LOWNOISE_ESD_ETMY. Relocked again and it happened again but I stopped the guardian at the completion of INCREASE_POWER. Sitting there now with constant OMC DCPD saturations and same elevated DARM. Have had to damp a whole series of PI modes and now riding through a 6.6 mag earthquake in South New Zealand.
LOG:

23:10 UTC Robert to LVEA to setup equipment for tomorrow (Tuesday)
23:42 UTC Cheryl out of optics lab
00:16 UTC Robert and Pep to end Y
00:27 UTC GRB
01:14 UTC Robert and Pep done
01:28 UTC Restarted video4
01:36 UTC Peter and Jason out of PSL enclosure. Report bad pressure regulator and no immediately available spare. Managed to get flow rate back to what it was before the trip, but may not last.
01:38 UTC Starting to relock
01:54 UTC Starting initial alignment
02:35 UTC Initial alignment done
03:28 UTC NLN
03:40 UTC Running a2l
03:45 UTC a2l done
03:48 UTC Changed filters and gain for ITMX mode 3
03:50 UTC Damped PI mode 28
03:53 UTC Changing gain for ITMX mode 3 to 10
04:18 UTC Changed gain for ITMX mode 3 to 15
04:20 UTC Changed gain for ITMX mode 3 to 50
04:22 UTC Changed gain for ITMX mode 3 to 100
04:32 UTC Changed gain for ETMX mode 7 from 15 to 20
04:32 UTC Changed gain for ETMX mode 7 to 50. Jump in DARM.
04:38 UTC Changed gain for ETMX mode 7 back to 15
04:55 UTC Changed gain for ETMX mode 7 to 0
05:08 UTC Changed gain for ETMX mode 7 to -100
05:10 UTC Changed gain for ETMX mode 7 to 0
05:15 UTC Changed gain for ETMX mode 6 from -100 to -80
05:16 UTC Changed gain for ETMX mode 6 to 0
05:17 UTC Changed gain for ETMX mode 6 to -150
05:18 UTC Changed gain for ETMX mode 7 to 15
05:22 UTC Changed gain for ETMX mode 6 to -100
05:25 UTC Changed gain for EMTX mode 6 and 7 to 0
05:26 UTC Changed gain for EMTX mode 6 to -100 and 7 to 15
05:32 UTC Reverted ETMX SUS SDF differences -> jump up in ETMX mode 5 rms mon, set back, jumped up again
05:40 UTC Changed gain for ETMX mode 7 to 150
05:45 UTC Changed gain for ETMX mode 7 to 15. Lock loss

Constant OMC DCPD saturations after INCREASE_POWER. Huge series of peaks in DARM
06:14 UTC Lock loss from LOWNOISE_ESD_ETMY with constant OMC PD saturations
06:42 UTC Going to increase power. OMC DCPD saturations again. Stopping at completion of INCREASE_POWER.
06:54 UTC Damped PI mode 27 and 19 by changing sign of gain and then had to change it back.
06:59 UTC Damped PI mode 28
07:01 UTC Damped PI mode 27 and 19 again. This time needed change of phase.
07:04 UTC Damped PI mode 28 and 20.
Damped more PI modes.

OMC DCPD saturation, OMC DCPD saturation, ...

Finished and sitting at INCREASE_POWER. Second time in a row this happened. First time lost lock trying to go further.

Tried setting ETMX violin mode 6 and mode 7 gain to 0. Did not help.

NLN

Made it to NLN at 03:28 UTC. Current range around 49 MPc. Snapshot of DARM spectrum attached.

Ran a2l and changed filters and gain for ITMX mode 3. Turned on just FM4 and FM5 and set gain to 10. (I switched the filters first during which time the gain was left large, then changed the gain to 1 and then later to 10.)

Ops Eve Mid Shift Report

Peter and Jason have brought the PSL back. I have run an initial alignment. I just got stopped in VIOLIN_MODE_DAMPING_1 by an ezca connection error to H1:SUS-ITMX_L2_DAMP_MODE7_SW2R (see attached). I tried reloading the code but the error persists. I can connect to the channel with caget from zotws3 (as myself) and from h1guardian1 (as controls).

It seems that I have managed to fix it by following the suggestion in the log and taking ISC_LOCK to STOP and then EXEC. It seemed to get stuck at PAUSE so I re-requested EXEC and it then went to EXEC. I then requested DC_READOUT and it made it to that. I noticed the attached 'invalid value encountered in divide' error go by.

Laser trip

The laser tripped this afternoon because the flow rate through the heads went to zero or fell
below its trip point.  As noted in this alog
the flow rates took a dive over the weekend and did not return to their previous values.  Upon restarting the
crystal chiller, the flow rates were stuck around 0.3 lpm.  Since all four heads were affected, the pressure
regulator in that part of the cooling circuit was exercised as the bypass valve was varied.  At some point the
flow rates returned to ~0.5 lpm, so we suspect that there's a problem with the regulator.  Beyond a certain point,
the pressure did not increase as the regulator was steadily being closed.  The same is true for the regulator in
the amplifier circuit.

    The laser has been restarted but the flow rates will have to be monitored.


  Jason / Peter

notes on H1 removal from observation mode when CW psinject process dies and is restarted

Paul, Dave:

reference Cheryl's alog from Monday evening, 3rd July and Paul's alog from today.

If the psinject (continuous wave calibration) process on h1hwinj1 dies, the monit system on that computer restarts the process. It had been found that a sudden restart of the excitation could cause issues (loss of lock?) and so additional code was added in 2015 to slowly ramp up the CW injection signal on startup (and slowly ramp it down on shutdown).

Full details of the restart procedure are in this alog. To summarize; the gain is zeroed, psinject is started, after a 2 minute wait the gain is set back to unity over a ten second ramp. The GAIN and TRAMP channels were being monitored by h1calex SDF, which meant that the DIAG_SDF guardian node took H1 out of observation when differences were reported.

Given the rarity of a psinject crash/restart, we could continue to monitor the GAIN channel as a precaution against accidental change of the CW calibration amplitude. Perhaps we should go further and change the DIAG_EXC guardian node to take H1 out of observation if the h1calex excitation is missing (currently it is being ignored). Looking at the Verbal Alarms code I'm not sure a verbal alert is raised when psinject crashes, if this is the case we should activate this.

 

 

Ops Eve Shift Transition

TITLE: 07/10 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 7mph Gusts, 4mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.06 μm/s 
QUICK SUMMARY:

Laser remains off from trip. I see Jason in the PSL enclosure investigating.

Shift Activity

TITLE: 07/10 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:
LOG:

 

15:57 Jeff B into Diode room to power cycle dust monitor. WP 7071.

15:58 begin damping exercise - ITMX mode 10 (996.252Hz)

16:14 Jeff back

16:30 Meeting in control room to discuss WPs

16:50 Richard, Jeff and Fil working in CER and Diode room with dust monitor WP7071

19:00 Vern + guest up to observation deck

19:45 Carlos up to observation deck

19:56 Carlos back

20:09 Corey to vacuum optics lab

20:55 SVN (SDF Diff) file commitments for previously modified files.

21:42 Lockloss - PSL trip due to flow error

22:01 Jason and Peter  plan to go into the enclosure to investigate that PSL trip

22:48 Thomas and Marc to MY

 

 

 

 

 

 

 

 

Violin Mode Status, A Few More Modes Killed; 1463.09 Hz Still Illusive

J. Kissel, H. Radkins, J. Warner

We're continuing to attack violin modes post Montana EQ. I attach a spectrum of "our" (all Jim and Hugh) progress, where the reference (GREEN) is this morning. We can definitely see that the violin modes begin to get hefty non-linear shoulders by the time the RMS passes ~5e2 ADC counts, which is pretty surprising for a 32e4 count ADC...

New modes that Hugh and Jim have now successfully cooled today:

EX      1475.09       MODE7     YAW      FM4(100dB), FM6(1475.09),                  G = -500
      (This mode interacts with the "already solved" 1475.25 EX mode, so that damping must also be ON and some small level while damping this mode)
IY      1472.45       MODE8     PIT      FM4(100dB), FM6(1462.45), FM7(-60d1.5k)    G = +1000
IY      1470.38       MODE4     PIT      FM4(100dB), FM6(1470.38),                  G = -500
IY      1470.82       MODE5     PIT      FM4(100dB), FM6(1470.82), FM7(-60d1.5k)    G = +1000
As before -- these are reported for the record, but the Violin Mode Table has been updated and that should be treated as cannon not this aLOG. 

I've been continuing to manipulate the damping output of the problematic pair of modes at 1463.09 Hz, including
  - Installing narrow bandpass in all test masses, exploring the unit circle with each of the 4 test masses, and all 3 degrees of freedom (L, P, Y)
  - Went back to ITMX and focused energy there (since that;s what I was driving when I originally rung it up)
  - Adding +/- 30 [deg] filters, and spinning around the unit circle there
  - Broadening the band pass, in hopes of changing the relative drive phase between the two modes.
After three hours of failure, I was *just* about to think I saw hints of progress driving the wide-band band pass in YAW, with FM4(100dB) and FM5(1463.09W) and +1 gain, but then we lost lock because the PSL tripped. Guh!

While wasting time staring at StripTool for 3 hours, I grabbed a 0.5 [mHz] resolution spectra to confirm the violin mode table's claimed exact frequency of these modes. Unfortunately, DTT (and foton's) cursor can't resolve past 2 decimal places. But at least zooming in visually, I can confirm that the two modes are:
    1463.0961 +/- 0.0005
    1463.0982 +/- 0.0005

We were focused on the 1.5k modes so far today, and made no attempts at progress on the remaining high 1.0k pair of modes at 1008.45 / 1008.49 Hz.

H1 Down PSL Tripped

10:21 21:41UTC Jason and Peter are going into the enclosure for investigation.

PSL Weekly Report - 10 Day Trends FAMIS #6156

Remnants of the last diode current adjustments are evident. The issues with the chiller plots is being dealt with. Everything else is ok.