TITLE: 02/20 Owl Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Unlocked
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
Wind: 5mph Gusts, 8mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.4 μm/s
QUICK SUMMARY: There was some PSL trouble last shift, and then the ETMX RMS WDs tripped causing a lockloss at COIL_DRIVERS just before I started my shift.
TITLE: 02/20 OWL Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1:
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
Decent shift up until the latter half of the shift with a couple of PSL trips (see earlier alog).
LOG:
After 12min of OBSERVING, had another PSL trip.
Went to the PSL Chillers and there must have been an explosion of water in there. The Crystal Chiller cap did NOT pop off. The Crystal & Diode chillers both have Error Alarms (Crystal has Flow Sensor Alarm & Diode has F2-Error).
Will probably call Jason shortly.
In CORRECTIVE MAINTENANCE state.
accepted SDF Diff for ISS Ref Signal change (from -1.10 to -1.08), with Jason's approval.
Recently I looked at the highest SNR (>100) blip glitches from the low-latency monitors and noticed that most of them looked very similar in the autoscaled spectrogram (they have a lot of excess noise in the whitened spectrogram so they do not look like blip glitches there, but they are short like blip glitches).
When I generated the full omega scans, they all had in common some excess noise in the SUS-ETMY_L2_NOISEMON channels (the times I looked at -Feb 10 to Feb 14- are in https://ldas-jobs.ligo-wa.caltech.edu/~miriam.cabero/blips/O2_loud/ ). This is not true for all blip glitches, which makes me think that this might be a new sub-set whose origin could be related with the coil driver. And they are not only very loud blips, I also found other quieter ones that show something in the noisemon channels:
https://ldas-jobs.ligo-wa.caltech.edu/~tdent/wdq/H1_1165129095.9
I have looked at past aLogs and it seems like there was found a possible correlation between periods of extreme glitchiness and the ETMY L2 driver. The glitches I looked into are from last week, not from a particularly noisy period like the few days in mid December or beginning of January.
Andy was saying ( aLog 32730 ) that noisemons will show glitches whenever DARM does, and indeed there are some blips where the noisemon sees the same as DARM (e.g. https://ldas-jobs.ligo.caltech.edu/~thomas.massinger/wdq/ER10_loudest/1163443424.81/ ), but it looks somehow different for the times I mention above (and not all the blips show something in the noisemons).
So I am writing here to ask if this information is relevant for blip-glitch studies, or if the information found in the periods of extreme glitchiness is helpful for this. There seemed to be the conclusion that humidity might be the problem, as had been observed in O1 ( aLog 32885 ). That could indeed be the case for a particular time of high glitchiness, but what about the rest of the time? If the humidity is normal and we still have these kind of glitches, what could we do?
Using PyCBC Live we have created a low-latency blip glitch hunter to possibly help DetChar investigations on site.
Lists of times and SNR are generated daily in
https://ldas-jobs.ligo.caltech.edu/~miriam.cabero/blips_live/lists/
and pages in the style of the summary pages are updated every five minutes:
https://ldas-jobs.ligo.caltech.edu/~miriam.cabero/blips_live/H1/day/20170220/detchar/pycbc_live/
(note that these pages are not available from the main summary pages yet, but they might be added in the near future).
P.S: I am not cleaning up these lists like I did in O1, so there can be some times that do not show a blip. However, I expect these times to be a small percentage.
13:01utc: H1 & the H1 PSL Front End went down.
Will make a call to Jason.
On the PSL SYSSTAT medm, we had the Oscillator/Head 1-4 flow error in a Fault/red state. Jason said this meant that we had a "glitch in laser head flow sensors". Jason brought back the PSL remotely. I topped off the PSL Chiller (the cap did NOT pop off) with 375mL of water.
13:58 Back to LOCK ACQUISITION
FRS #7460 Filed For ~1hr of CORRECTIVE_MAINTENANCE DOWN time.
Cause appears to be a trip of the Laser Head 1-4 Flow interlock, specifically Head 2. The first attachment shows the 3 laser head flow rates (the 4th is currently being forced to a value in the Beckhoff software, so cannot contribute to the trip). It is clear that head 2 became erratic and began to drop in flow, eventually passing below the trip point and tripping the laser. The 2nd attachment shows a zoomed in view of only the Head 2 flow. Possible debris passing through the flow sensor?
Pressure 1 values for the EY & EX look more glitchy than the flat-lined PS1-PS4 channels.
CONTROL_VOUT values don't show anything odd over the last 45 days.
Trends are attached. This closes out FAMIS 4529.
Out of Observing for 32min for a lockloss around 9:37utc (1:37amPST).
For this current lock, the range has drifted down a few Mpc over the first 2hrs. A2L looks fine. I did not had to do anything for PI modes for this lock so far.
Other than the lockloss bit of excitement, it's been a quiet night on the H1 front.
H1 drops lock 24hrs after lockloss during Travis OWL shift last night. Nothing obvious on our wall monitors (Striptools normal, SEIS BLRMS quiet).
HAM6 ISI tripped, and I forgot to untrip the WD, so I paused the ISC_LOCK at DRMI ON POP to get HAM6 ISI back to ISOLATED.
Yesterday we had a lockloss at just before 1am PST local time & tonight we had a locklos just after 1:37amPST local time. Hmmmm...
Starting my shift, one check we do is the Alarm Handlers. The Vacuum one has an alarm for the EX PT524 vacuum gauge. Chandra noted this went down on Tues (Fire Dept Glitch?) & looks like it's remained down ever since. Since other gauges look good with pressures of 1e-9Torr at EX, I'll assume we do not need to pay any mind to this alarm for the moment.
These cold cathode gauges sometimes take a while to turn back on at low pressures.
TITLE: 02/20 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 68Mpc
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
Wind: 5mph Gusts, 4mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.29 μm/s
Nice and warm out and no fog to worry about. useism looks like it's been increasing a little over the last 6hrs.
QUICK SUMMARY:
As Jeff was handing off, saw no issues, except a 5.4 Peruvian EQ popped up (followed by a 4.9 New Zealand EQ) on USGS webpage---Terramon webpage apears to be down. Will monitor these.
Jeff mentioned an A2L would be useful to run if the opportunity arises.
Fundamental violin modes are down near 3e-19.
Have noticed that the H1 & L1 Summary Pages have been down for the last ~12hrs.
Shift Summary: A2L DTT script shows pitch elevated between 20 and 25 Hz. Will run the A2L script if LLO drops out of lock.
Mostly a smooth shift. Have been Observing for the past 22.5 hours. With the exception of A2L pitch, the interferometer has been humming along all shift. There was a GRB alert, which LLO received but LHO did not. As both sites were in observing, observed the one hour stand down. Environmental and seismic conditions remain favorable.
IFO in Observing for 18.5 hours. The A2L Pitch is well above the reference and Yaw is starting to climb as well. Will run the A2L script at the first appropriate opportunity. No other problem or concerns apparent at this time.
TITLE: 02/19 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 66Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY:
LOG:
Quiet shift. Robert connected remotely, that's been pretty much the only excitement.
TITLE: 02/19 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 69Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: Observing for 6.5 hours. No issues since relocking early in the shift.
LOG: See previous aLogs. I left a message with Bubba regarding the fire panel alarm.
At 11:34 UTC, I heard a sustained alarm tone near the door of the control room. I tracked it down to the fire system panel in the Computer User's Room. Other than the tone, I noticed that the "System Trouble" light was blinking. I'll call Bubba when it get closer to normal waking hours.
Seems to have fixed itself as it is no longer alarming.
The alarm stopped for an hour or so, but has since started again. I'll call Bubba now.
As with the first trip documented here, this one also appears to be caused by an erratic flow reading in the Laser Head 2 flow sensor. As before, the flow reading from the head 2 sensor becomes erratic before the trip and begins to drop, eventually hitting the trip point and tripping the interlock. The first attachment shows the 3 flow rates from the Laser Head circuit (once again, the 4th flow sensor is forced to a specific value in Beckhoff so cannot contribute to the trip). The 2nd attachment is a zoomed in view of the Laser Head 2 flow sensor signal.
As for the F2 error on the diode chiller, this error did not cause the chiller to turn off. I remotely turned it off and asked Corey to power cycle the chiller. This cleared the error.