Observing at ~58Mpc. Winds are gusting into the 30s and the Handford commute is going. Locked for 4hours so far
TITLE: 03/06 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 58Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
Wind: 15mph Gusts, 9mph 5min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.30 μm/s
QUICK SUMMARY: PSL tripped as my shift started. I called Jason and he got things going again. We are now back to low noise and ~60Mpc.
TITLE: 03/06 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Bad news PSL
LOG:
Jeff was recovering from an EQ when I arrived. I managed to get the IFO back up for a few hours, but then the PSL went on the fritz. I called Jason, after a few rounds of filling the chiller and resetting the noise eater I was able to get locked again. As soon as TJ arrived, though, the PSL went down again. He's working with Jason (again) on getting back up. Re-locking has been easy though.
Shift Summary: Ran A2L script – Pitch is close to 0.6 and Yaw is at 0.5. PSL AC Temp alerts. Per Jason, went into LVEA to check Makeup Air fan and AC controls. See aLOG #34586.
Lost lock due to a Mag 6.5 earthquake near Kandrian, Papua New Guinea. Primary microseism shot up to 2.0 um/s. Put IFO into DOWN until tremors subside.
Mag 6.5 Earthquake near Kandrian, Papua New Guinea
Seen on USGS, Seismon, and Terramon
USGS - EQ at 22:47 UTC
Terramon - EQ at 22:47 UTC
Rise in the BLRMS at around 23:00 UTC, to 3.0um/s
H1 lost lock, LLO was already down
In Observing for 7.5 hours. IFO seems to be running well at this time. Wind is a steady Moderate Breeze (up to 18 mph) with gusts up in the low 30s. Primary microseism in X and Y is a bit elevated, secondary microseism is flat at about 0.35um/s. A2L Pitch is up to 0.7, Yaw is below the reference. Everything else is OK.
TITLE: 03/05 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 63Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
H1 overnight issues now understood / corrected:
Added -30deg phase to ITMX bounce (FM7) - can't find it in guardian, so SDF should be accepted when relocking
LOG:
No analysis, but wanted to post a more comprehensive plot that shows temperature change -> power change and pointing change.
h1broadcast0 went down last night, called DaveB this morning, he had me power cycle it, and the lights on the front say it came back.
I was notified that the h(t) pipeline were hung up (thanks to Patrick Brockill) at about 18:20UTC. I found that the primary (h1dmt0) pipeline was producing data, but with a latency ~5 hours. The redundant pipeline (h1dmt2) was not producing output frames. I restarted both at about 18:25 (h1dmt0) and 10:29 (h1dmt2). Both pipelines seem to be producing data now.
HofT data gap. Path is /gds-h1/dmt/frames/hoft/H1/{gps-time}/. PST times.
Mar 4 22:59 H-H1_DMT_C00-1172732480-4.gwf
Mar 5 10:32 H-H1_DMT_C00-1172774052-4.gwf
I'm working with Cheryl on current CDS issues. Looks like h1broadcast0 machine locked up, Cheryl is rebooting this machine.
The timing system has no errors currently, looks like this was a transient problem.
DMT monitors are back after Cheryl rebooted h1broadcast0. I verified that no full frames were lost from h1fw0 or h1fw1 overnight. We did lose HofT frames when the DMT broadcaster was down. Keith is testing redundant broadcasters at LLO to prevent a problem like this.
Redundant GDS broadcasting now working at LLO to one DMT machine. Need to install network switch(es), additional Ethernet cards for redundancy on the redundant DMT machines..
That's interesting. When I checked the make-up air on Wednesday (after Travis first witnessed this temp drop) it was definitely running at 20%. Did you check the make-up air fan in the acoustic closet (East of the PSL enclosure) to confirm it was running at 100% (it's very loud and drowns out the power supply in the closet when at 100%)? There have been times when the PSL environmental controls indicate the fan is 100%, but when the make-up air fan itself is checked it is running at 20%. There seems to be some odd link in the evironmental controls software where it shows 100% on all fans if one set of fans (i.e. the anteroom or laser room HEPA fans) was set to 100%. This also works the other way, where the HEPA fans will sometimes show 20% (they are either 100% or off) after the make-up air has been set to 20%. Essentially, the environmental controls appear to hold the last user set fan speed that then follows from fan screen to fan screen, but appears to not be applied until the user does so. I don't know if this is how the system is supposed to behave, but I have seen this behavior on more than one occasion.
Received alerts on PSL AC/Temp at 21:29 & 21:59 UTC. Spoke with Jason. Dropped out of Observing (LLO down) to go into the LVEA and check Makeup Air fans and AC controls. (1). Makeup Air fan is running at LOW SPEED. It is NOT running at 100%. (2). The AC Controls - Makeup Air fan speed at 100%. I reset it to 20%. Verified that the Makeup Air fan was still at low speed. Trended the five temperature sensors located in the PSL Laser enclosure for the past 30 days. Results are posted below. There were 8 low temperature event starting on 03/01 at around 20:02 UTC. These events are consistent across the five sensors. The Dust Monitor looks a bit odd but this is due to the 1 minute sample taken every 10 minutes. There is some time lag but reasonably good coherence none the less. There is less coherence between the temperatures in the PSL and in the LVEA.
(John W, Gerardo M)
There appears to be something that is turning on, then off, maybe something triggered by a thermostat. Noise can be observed on the accelerometers on the PSL table. See attachments, all plots are 5 days long. On first plot the signal for the accelerometer (color blue) was modified to match the temperature signal. Other plot no mods done.
The picture is more likely to be the pre-modecleaner reflection, not the reference cavity transmission.
If it really is the reference cavity transmission, then we have a big alignment and mode matching
problem.
The picture seems to indicate that the horizontal alignment has changed.
Tom Dent, Miriam Cabero
We have identified a sub-set of blip glitches that might originate from PSL glitches. A glitch with the same morphology as a blip glitch shows up in the PSL-ISS_PDA_REL_OUT_DQ channel at the same time as a blip glitch is seen in the GDS-CALIB_STRAIN channel.
We have started identifying times of these glitches using omicron triggers from the PSL-ISS_PDA_REL_OUT_DQ channel with 30 < SNR < 150 and central frequencies between ~90 Hz and a few hundreds of Hz. A preliminary list of these times (on-going, only period Nov 30 - Dec 6 so far) can be found in the file
https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt
or, with omega scans of both channels (and with a few quieter glitches), in the wiki page
Only two of those times have full omega scans for now:
The whitened time-series of the PSL channel looks like a typical loud blip glitch, which could be helpful to identify/find times of this sub-set of blip glitches by other methods more efficient than the omicron triggers:
The CBC wiki page has been moved to https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLblips
I ran PCAT on H1:GDS-CALIB_STRAIN and H1:PSL-ISS_PDA_REL_OUT_DQ from November 30, 2016 to December 31, 2016 with a relatively high threshold (results here: https://ldas-jobs.ligo-wa.caltech.edu/~cavaglia/pcat-multi/PSL_2016-11-30_2016-12-31.html). Then I looked at the coincidence between the two channels. The list of coincident triggers is: ----------------------------------------------------- List of triggers common to PSL Type 1 and GDS Type 1: #1: 1164908667.377000 List of triggers common to PSL Type 1 and GDS Type 10: #1: 1164895965.198000 #2: 1164908666.479000 List of triggers common to PSL Type 1 and GDS Type 2: #1: 1164882018.545000 List of triggers common to PSL Type 1 and GDS Type 4: #1: 1164895924.827000 #2: 1164895925.031000 #3: 1164895925.133000 #4: 1164895931.640000 #5: 1164895931.718000 #6: 1164895958.491000 #7: 1164895958.593000 #8: 1164895965.097000 #9: 1164908667.193000 #10: 1164908667.295000 #11: 1164908673.289000 #12: 1164908721.587000 #13: 1164908722.198000 #14: 1164908722.300000 #15: 1164908722.435000 List of triggers common to PSL Type 1 and GDS Type 7: #1: 1166374569.625000 #2: 1166374569.993000 List of triggers common to PSL Type 1 and GDS Type 8: #1: 1166483271.312000 ----------------------------------------------------- I followed-up with omega scans and among the triggers above, only 1164882018.545000 is a blip glitch. The others are ~ 1 sec broadband glitches with frequency between 512 and 1024. A few scans are attached to the report.
Hi Marco,
your 'List of triggers common to PSL Type 1 and GDS Type 4' (15 times in two groups) are all during the known times of telephone audio disturbance on Dec 4 - see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=32503 and https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLGlitches
I think these don't require looking into any further, the other classes may tell us more.
The GDS glitches that look like blips in the time series seem to be type 2, 7, and 8. You did indeed find that the group of common glitches PSL - GDS type 2 is a blip glitch. However, the PSL glitches in the groups with GDS type 7 and 8 do not look like blips in the omega scan. The subset we identified clearly shows blip glitch morphology in the omega scan for the PSL channel, so it is not surprising that those two groups turned out not to be blips in GDS.
It is though surprising that you only found one time with a coincident blip in both channels, when we identified several more times in just one week of data from the omicron triggers. What was the "relatively high threshold" you used?
Hi. Sorry for taking so long with this. I rerun PCAT on the PSL and GDS channels between 2016-11-30 and 2016-12-31 with a lower threshold for glitch identification (glitches with amplitude > 4 sigma the noise floor) and with a larger coincidence window (coincident glitches within 0.1 seconds). The list of found coincident glitches is attached to the report. Four glitches in Miriam's list [https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt] show up in the list: 1164532915.0 (type 1 PSL/type 3 GDS), 1164741925.6 (type 1 PSL/type 1 GDS), 1164876857.0 (type 8 PSL/type 1 GDS), 1164882018.5 (type 1 PSL/type 8 GDS). I looked at other glitches in these types and found only one additional blip at 1166374567.1 (type 1 PSL/type 1 GDS) out of 9 additional coincident glitches. The typical waveforms of the GDS glitches show that the blip type(s) in GDS are type 1 and/or type 8. There are 1998 (type 1) and 830 (type 8) glitches in these classes. I looked at a few examples in cat 8 and indeed found several blip glitches which are not coincident with any glitch in the PSL channel. I would conclude that PCAT does not produce much evidence for a strong correlation of blip glitches in GDS and PSL. If there is, PSL-coincident glitches must be a small subset of blip glitches in h(t). However, some blips *are* coincident with glitches in the PSL, so looking more into this may be a good idea.
Hi,
thanks Marco for looking into this. We already expected that it was a small sub-set of blip glitches, because we only found very few of them and we knew the total number of blip glitches was much higher. However, I believe that not all blip glitches have the same origin and that it is important to identify sub-sets, even if small, to possibly fix whatever could be fixed.
I have extended the wiki page https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLblips and the list of times https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt up to yesterday. It is interesting to see that I did not identify any PSL blips in, e.g., Jan 20 to Jan 30, but that they come back more often after Feb 9. Unfortunately, it is not easy to automatically identify the PSL blips: the criteria I used for the omicron triggers (SNR > 30, central frequency ~few hundred Hz) do not always yield to blips but also to things like https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=156436, which also affects CALIB_STRAIN but not in the form of blip glitches.
None of the times I added up to December appear in your list of coincident glitches, but that could be because their SNR in PSL is not very high and they only leave a very small imprint in CALIB_STRAIN compared with the ones from November. In January and February there are several louder ones with bigger effect on CALIB_STRAIN though.
The most recent iteration of PSL-ISS flag generation showed three relatively loud glitch times:
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/scans/1170732596.35/
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/scans/1170745979.41/
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170212/latest/scans/1170950466.83/
The first 2 are both on Feb 10, in fact a PSL-ISS channel was picked by Hveto on that day (https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/#hveto-round-8) though not very high significance.
PSL not yet glitch-free?
Indeed PSL is not yet glitch free, as I already pointed out in my comment from last week.
Imene Belahcene, Florent Robinet
At LHO, a simple command line works well at printing PSL blip glitches:
source ~detchar/opt/virgosoft/environment.shomicron-print channel=H1:PSL-ISS_PDA_REL_OUT_DQ gps-start=1164500000 gps-end=1167500000 snr-min=30 freq-max=500 print-q=1 print-duration=1 print-bandwidth=1 | awk '$5==5.08&&$2<2{print}'
GPS times must be adjusted to your needs.
This command line returns a few GPS times not contained in Miriam's blip list: must check that they are actual blips.
The PSL has different types of glitches that match those requirements. When I look at the Omicron triggers, I do indeed check that they are blip glitches before adding the times to my list. Therefore it is perfectly consistent that you find GPS times with those characteristics that are not in my list. However, feel free to check again if you want/have time. Of course I am not error-free :)
I believe the command I posted above is an almost-perfect way to retrieve a pure sample of PSL blip glitches. The key is to only print low-Q Omicron triggers.
For example, GPS=1165434378.2129 is a PSL blip glitch and it is not in Miriam's list.
There is nothing special about what you call a blip glitch: any broadband and short-duration (hence low-Q) glitch will produce the rain-drop shape in a time-frequency map. This is due to the intrinsic tiling structure of Omicron/Omega.
Next time I update the list (probably some time this week) I will check the GPS times given by the command line you suggest (it would be nice if it does indeed work perfectly at finding only these glitches, then we'd have an automated PSL blips finder!)
