Ops Eve Mid Shift Report

Peter and Jason have brought the PSL back. I have run an initial alignment. I just got stopped in VIOLIN_MODE_DAMPING_1 by an ezca connection error to H1:SUS-ITMX_L2_DAMP_MODE7_SW2R (see attached). I tried reloading the code but the error persists. I can connect to the channel with caget from zotws3 (as myself) and from h1guardian1 (as controls).

It seems that I have managed to fix it by following the suggestion in the log and taking ISC_LOCK to STOP and then EXEC. It seemed to get stuck at PAUSE so I re-requested EXEC and it then went to EXEC. I then requested DC_READOUT and it made it to that. I noticed the attached 'invalid value encountered in divide' error go by.

Laser trip

The laser tripped this afternoon because the flow rate through the heads went to zero or fell
below its trip point.  As noted in this alog
the flow rates took a dive over the weekend and did not return to their previous values.  Upon restarting the
crystal chiller, the flow rates were stuck around 0.3 lpm.  Since all four heads were affected, the pressure
regulator in that part of the cooling circuit was exercised as the bypass valve was varied.  At some point the
flow rates returned to ~0.5 lpm, so we suspect that there's a problem with the regulator.  Beyond a certain point,
the pressure did not increase as the regulator was steadily being closed.  The same is true for the regulator in
the amplifier circuit.

    The laser has been restarted but the flow rates will have to be monitored.


  Jason / Peter

notes on H1 removal from observation mode when CW psinject process dies and is restarted

Paul, Dave:

reference Cheryl's alog from Monday evening, 3rd July and Paul's alog from today.

If the psinject (continuous wave calibration) process on h1hwinj1 dies, the monit system on that computer restarts the process. It had been found that a sudden restart of the excitation could cause issues (loss of lock?) and so additional code was added in 2015 to slowly ramp up the CW injection signal on startup (and slowly ramp it down on shutdown).

Full details of the restart procedure are in this alog. To summarize; the gain is zeroed, psinject is started, after a 2 minute wait the gain is set back to unity over a ten second ramp. The GAIN and TRAMP channels were being monitored by h1calex SDF, which meant that the DIAG_SDF guardian node took H1 out of observation when differences were reported.

Given the rarity of a psinject crash/restart, we could continue to monitor the GAIN channel as a precaution against accidental change of the CW calibration amplitude. Perhaps we should go further and change the DIAG_EXC guardian node to take H1 out of observation if the h1calex excitation is missing (currently it is being ignored). Looking at the Verbal Alarms code I'm not sure a verbal alert is raised when psinject crashes, if this is the case we should activate this.

 

 

Ops Eve Shift Transition

TITLE: 07/10 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 7mph Gusts, 4mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.06 μm/s 
QUICK SUMMARY:

Laser remains off from trip. I see Jason in the PSL enclosure investigating.

Shift Activity

TITLE: 07/10 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:
LOG:

 

15:57 Jeff B into Diode room to power cycle dust monitor. WP 7071.

15:58 begin damping exercise - ITMX mode 10 (996.252Hz)

16:14 Jeff back

16:30 Meeting in control room to discuss WPs

16:50 Richard, Jeff and Fil working in CER and Diode room with dust monitor WP7071

19:00 Vern + guest up to observation deck

19:45 Carlos up to observation deck

19:56 Carlos back

20:09 Corey to vacuum optics lab

20:55 SVN (SDF Diff) file commitments for previously modified files.

21:42 Lockloss - PSL trip due to flow error

22:01 Jason and Peter  plan to go into the enclosure to investigate that PSL trip

22:48 Thomas and Marc to MY

 

 

 

 

 

 

 

 

Violin Mode Status, A Few More Modes Killed; 1463.09 Hz Still Illusive

J. Kissel, H. Radkins, J. Warner

We're continuing to attack violin modes post Montana EQ. I attach a spectrum of "our" (all Jim and Hugh) progress, where the reference (GREEN) is this morning. We can definitely see that the violin modes begin to get hefty non-linear shoulders by the time the RMS passes ~5e2 ADC counts, which is pretty surprising for a 32e4 count ADC...

New modes that Hugh and Jim have now successfully cooled today:

EX      1475.09       MODE7     YAW      FM4(100dB), FM6(1475.09),                  G = -500
      (This mode interacts with the "already solved" 1475.25 EX mode, so that damping must also be ON and some small level while damping this mode)
IY      1472.45       MODE8     PIT      FM4(100dB), FM6(1462.45), FM7(-60d1.5k)    G = +1000
IY      1470.38       MODE4     PIT      FM4(100dB), FM6(1470.38),                  G = -500
IY      1470.82       MODE5     PIT      FM4(100dB), FM6(1470.82), FM7(-60d1.5k)    G = +1000
As before -- these are reported for the record, but the Violin Mode Table has been updated and that should be treated as cannon not this aLOG. 

I've been continuing to manipulate the damping output of the problematic pair of modes at 1463.09 Hz, including
  - Installing narrow bandpass in all test masses, exploring the unit circle with each of the 4 test masses, and all 3 degrees of freedom (L, P, Y)
  - Went back to ITMX and focused energy there (since that;s what I was driving when I originally rung it up)
  - Adding +/- 30 [deg] filters, and spinning around the unit circle there
  - Broadening the band pass, in hopes of changing the relative drive phase between the two modes.
After three hours of failure, I was *just* about to think I saw hints of progress driving the wide-band band pass in YAW, with FM4(100dB) and FM5(1463.09W) and +1 gain, but then we lost lock because the PSL tripped. Guh!

While wasting time staring at StripTool for 3 hours, I grabbed a 0.5 [mHz] resolution spectra to confirm the violin mode table's claimed exact frequency of these modes. Unfortunately, DTT (and foton's) cursor can't resolve past 2 decimal places. But at least zooming in visually, I can confirm that the two modes are:
    1463.0961 +/- 0.0005
    1463.0982 +/- 0.0005

We were focused on the 1.5k modes so far today, and made no attempts at progress on the remaining high 1.0k pair of modes at 1008.45 / 1008.49 Hz.

H1 Down PSL Tripped

10:21 21:41UTC Jason and Peter are going into the enclosure for investigation.

ASC frontend model updated; ready for installation tomorrow

Kiwamu, Daniel,  (WP7066, ECR:E1700127-v1)

We modified the ASC front end model in order to accommodate lock-in oscillators which will be used for commissioning the new AS72 WFSs (37042). The conceptual design is summarized in T1700324. Additionally, we removed the parts that were for the AS90 WFSs because (A) we had not been using them and (B) we will take over the same electronics and ADCs for the AS72 WFSs. The below are the model files that were modified today:

• asc/common/ASC_MASTER.mdl
• isc/common/WFS.mdl (where we added new library blocks)
• sys/common/PLL_MASTER.mdl (where we added new library blocks)

They are checked into SVN. Note that Livingston is not using ASC_MASTER these days. The attached screenshots show the new components in the ASC model. As stated in the ECR, we are adding twelve new DQ channels as follows.

• ASC-AS_(A|B)_RF72_(I|Q)_(PIT|YAW)_OUT (2048 Hz)
• ASC-AS_(A|B)_RF72_(I|Q)_SUM_OUT (2048 Hz)

Also, we removed the following four DQ channels that needed to go out since we took out the AS90 system:

• ASC-AS_(A|B)_RF90_(PIT|YAW)_OUT 

I was able to compile the model without any error messages. The model is ready to install tomorrow during the maintenance period.

Mid-Shift Summary

• Damping Wars rage on. I was able to damp 2nd harmonics modes 996.252, 995.365 and 991.938.  Jeff K and Jim W are working on the higher magnitutde/higher order modes
• Jason O was able to do some chiller flow adjustments
• Richard, Fil and Jeff B getting the dust monitor in the diode room situated.
• H1 locked for 12.5hrs @ 52Mpc

SVN Diff File Cleanup

The following files were commited to SVN:

• Sheila's June 30th change  H1OAF.txt
• Sheila's June 26th change H1LSC.txt
• Sheila's June 28th change H1ASC.txt

There are files that are currently being modified by on site staff. They will respectfully be asked to commit their work.

There are some files that were modified by incoming operators. They will be asked to address their work as well.

PSL Weekly Report - 10 Day Trends FAMIS #6156

Remnants of the last diode current adjustments are evident. The issues with the chiller plots is being dealt with. Everything else is ok.

vacuum cell phone alarms reconfigured and restarted

FRS8255

Chandra, Dave:

The temporary high alarm leves for PT124 and PT144 was removed this morning (reduced from 5.0e-08 to the standard 5.0-e09 Torr) and the alarm system was restarted. In the process I discovered that the full alarm configurations for CP3 and CP4 were still commented out (we were using a subset). Now these cryo-pumps are fully functional, I have restored their full alarm configuration.

I've set FRS8255 into PENDING ready for close-out.

Diode room dust monitor code restarted

Richard, Jeff B, Dave:

the EPICS IOC which reads the diode room dust monitor was restarted, we are recording dust measurements again. The wiki page for the dustdr was found to have problems, which were corrected. Also the serial baud rate had been increased from 9600 to 19200, this change was made to the wiki page text. The startup script had an incorrect name, this was corrected.

PSL Laser Head Flow Decrease

At approximately 15:30 UTC (8:30 PDT) on Saturday, 7/8/2017, the flow through the HPO laser heads dropped by ~0.12 lpm; see the first attachment.  This is worrying as the trip point for the laser head flow is set to 0.4 lpm; the flow through heads 2 and 3 is sitting at ~0.45 lpm, just 0.05 lpm above the trip point.  Looking at the other channels in the cooling system the picture becomes a little clearer (maybe).

The 2nd attachment shows the pressures in the PSL cooling manifold; H1:PSL-OSC_PRESS1 is the pressure at the manifold input, H1:PSL-OSC_PRESS2 is the pressure at the manifold output.  As seen in the attachment, the pressure at the inlet increased slightly, and the pressure at the outlet dropped slightly.  The third attachment shows the flow through the HPO power meter circuit (H1:PSL-OSC_PWRMETERFLOW), the 35W FE laser circuit (H1:PSL-AMP_FLOW), and the overall flow out of the PSL crystal chiller (H1:PSL-OSC_XCHILFLOW); the flow out of the chiller dropped slightly (~0.1 lpm) while the flow through the HPO power meter and 35W FE circuits both increased very slightly (interestingly, the signal from the FE flow sensor cleaned up after the event, and the frequency of drops in the power meter flow sensor signal also decreased).  The most likely cause of this behavior is some kind of flow restriction, probably somewhere in the laser head cooling circuit (although it is possible there could be a blockage in the filters under the PSL table).

The 4th attachment shows the temperature of the individual laser heads.  All saw an increase of ~0.2 °C, which indicates that this is a real drop in the flow through the HPO laser heads.  Fortunately this was not enough to effect the output power of the laser (5th attachment), and the temperatures have remained steady since the loss of flow on Saturday.

In the short term, we can try increasing the overall flow at the crystal chiller to see if it clears whatever is restricting the flow; at the very least we can increase the flow so we are a bit further from the trip point.  We will also inspect the filters beneath the PSL table when we are in the enclosure during tomorrow's maintenance window.  Any further investigation will require opening up the cooling system, which is highly invasive.

PCAL Y RX PD Readout Continues to Decay With Temperature

J. Kissel

Just posting an update on this -- the H1 PCAL Y RX PD reported displacement continues to decay, likely due to some temperature dependent clipping. This had been a part of the FRS Ticket 8328, but there's little we can do during an observation run to fix this. We've since closed the issue as LONGTERMFIX. AS such, I'll raise to an integration issue, and mark as WHEN VENT (similar to the promised work in Integration Issue 4700).

For now, recall that we've switched over to using TXPD as the calibration reference (see LHO aLOG 37166).

Opened IIET Ticket 8481

I've attached my related notes on this subject, as I've been trying to draw up a history of the issue since the Fall.

"Issue History.pdf" is the information I've been able to drag out of aLOGs and long time series, "Investigations.pdf" are the questions I've had and a select few I've answered. Apologies for the rambling nature, these are my personal notes.

Short summary of my investigations: temperature correlation on the Rx side is quite clear but higher frequency coherences appear to be only in obvious places (other PCalY OFS and Tx channels, ETMY ISI channels) and I haven't figured out how to measure coherence for long term, slow frequency signals.

Further investigations and notes will now exist in DCC document G1701350 (see https://dcc.ligo.org/G1701350).

DQ Shift summary, 7/3-7/5

DQ Shifter: Alan Weinstein, Email: ajw@ligo.caltech.edu
LSC Fellow: Paul Marsh, Email: paul.mecheng@gmail.com

• High duty cycles: 93.8% on Monday, 93.2% on Tuesday, 73.5% on Wednesday (maintenance day).
• BNS range between 60-65 Mpc throughout.
• Many range drops appear coincident with ETMY ESD overflows. Dunno if it's cause or effect. Might this be fixable with more frequent ESD sign flips?
• Tuesday maintenance delayed till Wednesday due to July 4 holiday. H1 brought out of observing mode on Wednesday 7/5, shortly after 15:00 UTC, back in observing by 21:14 UTC.
• Fire 15-20 mi NW of site forces closure of rte 240, lifted on Monday evening (local). Lots of smoke & smell, but no danger to the site or interruption of lock.
• Rode through a 5.9 Mag EQ in Chile, on Tuesday morning (local). Terramon screenshot, USGS.
• Running a2l on 7/3, 19:43 UTC reduced noise curve below 20 Hz, but greatly increased glitching from 20-40 Hz. Running a2l and aligning TMSX and TMSY on 7/3, 23:56 UTC brought it back down (alog 37299 ). Similar pathologies after Wednesday maintenance, 7/5, 21:14 UTC.
• H1:CAL-PINJX_CW_GAIN change at 7/4/17, 03:52 threw H1 out of Observe ( alog 37304 ). How did that gain change??
• Lock loss on 7/4/17, 22:13-22:57 UTC - alog 37318 - looks like it was caused by a PSL FSS DAC overflow ( plot1 ,  plot2 ) .
• Hveto: H1:SUS-ETMY_L1_NOISEMON_xx_OUT_DQ was round 1 winner, all three days. (Also UPV).
• Three GRB alerts on 7/5. LLO is not getting GRB alerts, so operator has been talking to them each time ( alog 37324 )
• No significant GraceDB triggers. Most significant is a cWB trigger with FAR of 1/(0.5 months).

Full summary is here.

CW_GAIN issue resolved, see aLOG 37304.

H1 kicked out of Observe

• 03:52:44UTC - H1 out of Observe
• SDF shows that H1:CAL-PINJX_CW_GAIN should be 1, but was at 0
• channel cleared SDF by going back to 1
• 03:55:45UTC - H1 back in Observe
• not sure if this is a known thing, a new thing, or if possibly someone (not on site) might soon be claiming this change

CW GAIN issue has been resolved. The CW_GAIN and CW_TRAMP values were being monitored by Guardian, which forced the IFO out of observing when those values changed; those values were changed due to a loss of hardware injections and subsequent restart by the psinject script. Dave Barker and I have changed the gain and ramp to be unmonitored channels again.

Notes attached.

Change undone after the decision was made that losing IFO lock was the desired behavior and issue was mitigated by sparse nature of psinject reset. See aLOG 37447.