Not much going on. Been Locked and Observing for nearly 9 hours. Fire alarm panel beeped when I arrived. Acknowledged the alarm.
TITLE: 02/21 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 68Mpc for 7hrs
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY: Wind is starting to reach the 30s. Relocked at the beginning of the shift and stayed locked till then. ETMX violin modes were a bit rung up, but slowly damped over time. They did not really show up on the DARM spectrum, so I only noticed from the Summary page.
LOG:
Locked for 3.5hrs at 68Mpc. Violin modes 505.707 and 505.710 are slightly high and are showing up in the gravitational wave strain on the summary pages, but it is not high enough to cause me alarm for lock loss. I will damp these if LLO drops out again.
Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 17 seconds. LLCV set back to 16.0% open. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 369 seconds. LLCV set back to 36.0% open.
Laser Status: SysStat is good Front End Power is 34.01W (should be around 30 W) HPO Output Power is 159.0W Front End Watch is GREEN HPO Watch is GREEN PMC: It has been locked 0 days, 1 hr 10 minutes (should be days/weeks) Reflected power = 15.35Watts Transmitted power = 57.38Watts PowerSum = 72.73Watts. FSS: It has been locked for 0 days 0 hr and 52 min (should be days/weeks) TPD[V] = 2.149V (min 0.9V) ISS: The diffracted power is around 4.4% (should be 3-5%) Last saturation event was 0 days 0 hours and 52 minutes ago (should be days/weeks) Possible Issues:
I only had to wait a short time to damp bounce and roll a bit, and violin went down very quickly as well. Now we are cruising at 70Mpc,
TITLE: 02/20 Owl Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Unlocked
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
Wind: 5mph Gusts, 8mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.4 μm/s
QUICK SUMMARY: There was some PSL trouble last shift, and then the ETMX RMS WDs tripped causing a lockloss at COIL_DRIVERS just before I started my shift.
TITLE: 02/20 OWL Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1:
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
Decent shift up until the latter half of the shift with a couple of PSL trips (see earlier alog).
LOG:
After 12min of OBSERVING, had another PSL trip.
Went to the PSL Chillers and there must have been an explosion of water in there. The Crystal Chiller cap did NOT pop off. The Crystal & Diode chillers both have Error Alarms (Crystal has Flow Sensor Alarm & Diode has F2-Error).
Will probably call Jason shortly.
In CORRECTIVE MAINTENANCE state.
As with the first trip documented here, this one also appears to be caused by an erratic flow reading in the Laser Head 2 flow sensor. As before, the flow reading from the head 2 sensor becomes erratic before the trip and begins to drop, eventually hitting the trip point and tripping the interlock. The first attachment shows the 3 flow rates from the Laser Head circuit (once again, the 4th flow sensor is forced to a specific value in Beckhoff so cannot contribute to the trip). The 2nd attachment is a zoomed in view of the Laser Head 2 flow sensor signal.
As for the F2 error on the diode chiller, this error did not cause the chiller to turn off. I remotely turned it off and asked Corey to power cycle the chiller. This cleared the error.
The plug not popping off can be a problem. When the pump stops there is a back surge of water into the reservoir, which is what blows off the plug and sends a shower of water onto the floor. If the plug is too tight to allow the pressure from the back surge to relieve, there is a chance of cracking the reservoir. I will call Technotrans to see if there is a better way to relieve this back pressure. Until I hear from them PLEASE DO NOT reef down tight on the plug. Leave it lose. Cleaning up a bit of water on the floor is a lot better than swapping out a damaged chiller.
accepted SDF Diff for ISS Ref Signal change (from -1.10 to -1.08), with Jason's approval.
Recently I looked at the highest SNR (>100) blip glitches from the low-latency monitors and noticed that most of them looked very similar in the autoscaled spectrogram (they have a lot of excess noise in the whitened spectrogram so they do not look like blip glitches there, but they are short like blip glitches).
When I generated the full omega scans, they all had in common some excess noise in the SUS-ETMY_L2_NOISEMON channels (the times I looked at -Feb 10 to Feb 14- are in https://ldas-jobs.ligo-wa.caltech.edu/~miriam.cabero/blips/O2_loud/ ). This is not true for all blip glitches, which makes me think that this might be a new sub-set whose origin could be related with the coil driver. And they are not only very loud blips, I also found other quieter ones that show something in the noisemon channels:
https://ldas-jobs.ligo-wa.caltech.edu/~tdent/wdq/H1_1165129095.9
I have looked at past aLogs and it seems like there was found a possible correlation between periods of extreme glitchiness and the ETMY L2 driver. The glitches I looked into are from last week, not from a particularly noisy period like the few days in mid December or beginning of January.
Andy was saying ( aLog 32730 ) that noisemons will show glitches whenever DARM does, and indeed there are some blips where the noisemon sees the same as DARM (e.g. https://ldas-jobs.ligo.caltech.edu/~thomas.massinger/wdq/ER10_loudest/1163443424.81/ ), but it looks somehow different for the times I mention above (and not all the blips show something in the noisemons).
So I am writing here to ask if this information is relevant for blip-glitch studies, or if the information found in the periods of extreme glitchiness is helpful for this. There seemed to be the conclusion that humidity might be the problem, as had been observed in O1 ( aLog 32885 ). That could indeed be the case for a particular time of high glitchiness, but what about the rest of the time? If the humidity is normal and we still have these kind of glitches, what could we do?
Using PyCBC Live we have created a low-latency blip glitch hunter to possibly help DetChar investigations on site.
Lists of times and SNR are generated daily in
https://ldas-jobs.ligo.caltech.edu/~miriam.cabero/blips_live/lists/
and pages in the style of the summary pages are updated every five minutes:
https://ldas-jobs.ligo.caltech.edu/~miriam.cabero/blips_live/H1/day/20170220/detchar/pycbc_live/
(note that these pages are not available from the main summary pages yet, but they might be added in the near future).
P.S: I am not cleaning up these lists like I did in O1, so there can be some times that do not show a blip. However, I expect these times to be a small percentage.
13:01utc: H1 & the H1 PSL Front End went down.
Will make a call to Jason.
On the PSL SYSSTAT medm, we had the Oscillator/Head 1-4 flow error in a Fault/red state. Jason said this meant that we had a "glitch in laser head flow sensors". Jason brought back the PSL remotely. I topped off the PSL Chiller (the cap did NOT pop off) with 375mL of water.
13:58 Back to LOCK ACQUISITION
FRS #7460 Filed For ~1hr of CORRECTIVE_MAINTENANCE DOWN time.
Cause appears to be a trip of the Laser Head 1-4 Flow interlock, specifically Head 2. The first attachment shows the 3 laser head flow rates (the 4th is currently being forced to a value in the Beckhoff software, so cannot contribute to the trip). It is clear that head 2 became erratic and began to drop in flow, eventually passing below the trip point and tripping the laser. The 2nd attachment shows a zoomed in view of only the Head 2 flow. Possible debris passing through the flow sensor?
Pressure 1 values for the EY & EX look more glitchy than the flat-lined PS1-PS4 channels.
CONTROL_VOUT values don't show anything odd over the last 45 days.
Trends are attached. This closes out FAMIS 4529.
Out of Observing for 32min for a lockloss around 9:37utc (1:37amPST).
For this current lock, the range has drifted down a few Mpc over the first 2hrs. A2L looks fine. I did not had to do anything for PI modes for this lock so far.
Other than the lockloss bit of excitement, it's been a quiet night on the H1 front.
Starting my shift, one check we do is the Alarm Handlers. The Vacuum one has an alarm for the EX PT524 vacuum gauge. Chandra noted this went down on Tues (Fire Dept Glitch?) & looks like it's remained down ever since. Since other gauges look good with pressures of 1e-9Torr at EX, I'll assume we do not need to pay any mind to this alarm for the moment.
These cold cathode gauges sometimes take a while to turn back on at low pressures.
Tom Dent, Miriam Cabero
We have identified a sub-set of blip glitches that might originate from PSL glitches. A glitch with the same morphology as a blip glitch shows up in the PSL-ISS_PDA_REL_OUT_DQ channel at the same time as a blip glitch is seen in the GDS-CALIB_STRAIN channel.
We have started identifying times of these glitches using omicron triggers from the PSL-ISS_PDA_REL_OUT_DQ channel with 30 < SNR < 150 and central frequencies between ~90 Hz and a few hundreds of Hz. A preliminary list of these times (on-going, only period Nov 30 - Dec 6 so far) can be found in the file
https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt
or, with omega scans of both channels (and with a few quieter glitches), in the wiki page
Only two of those times have full omega scans for now:
The whitened time-series of the PSL channel looks like a typical loud blip glitch, which could be helpful to identify/find times of this sub-set of blip glitches by other methods more efficient than the omicron triggers:
The CBC wiki page has been moved to https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLblips
I ran PCAT on H1:GDS-CALIB_STRAIN and H1:PSL-ISS_PDA_REL_OUT_DQ from November 30, 2016 to December 31, 2016 with a relatively high threshold (results here: https://ldas-jobs.ligo-wa.caltech.edu/~cavaglia/pcat-multi/PSL_2016-11-30_2016-12-31.html). Then I looked at the coincidence between the two channels. The list of coincident triggers is: ----------------------------------------------------- List of triggers common to PSL Type 1 and GDS Type 1: #1: 1164908667.377000 List of triggers common to PSL Type 1 and GDS Type 10: #1: 1164895965.198000 #2: 1164908666.479000 List of triggers common to PSL Type 1 and GDS Type 2: #1: 1164882018.545000 List of triggers common to PSL Type 1 and GDS Type 4: #1: 1164895924.827000 #2: 1164895925.031000 #3: 1164895925.133000 #4: 1164895931.640000 #5: 1164895931.718000 #6: 1164895958.491000 #7: 1164895958.593000 #8: 1164895965.097000 #9: 1164908667.193000 #10: 1164908667.295000 #11: 1164908673.289000 #12: 1164908721.587000 #13: 1164908722.198000 #14: 1164908722.300000 #15: 1164908722.435000 List of triggers common to PSL Type 1 and GDS Type 7: #1: 1166374569.625000 #2: 1166374569.993000 List of triggers common to PSL Type 1 and GDS Type 8: #1: 1166483271.312000 ----------------------------------------------------- I followed-up with omega scans and among the triggers above, only 1164882018.545000 is a blip glitch. The others are ~ 1 sec broadband glitches with frequency between 512 and 1024. A few scans are attached to the report.
Hi Marco,
your 'List of triggers common to PSL Type 1 and GDS Type 4' (15 times in two groups) are all during the known times of telephone audio disturbance on Dec 4 - see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=32503 and https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLGlitches
I think these don't require looking into any further, the other classes may tell us more.
The GDS glitches that look like blips in the time series seem to be type 2, 7, and 8. You did indeed find that the group of common glitches PSL - GDS type 2 is a blip glitch. However, the PSL glitches in the groups with GDS type 7 and 8 do not look like blips in the omega scan. The subset we identified clearly shows blip glitch morphology in the omega scan for the PSL channel, so it is not surprising that those two groups turned out not to be blips in GDS.
It is though surprising that you only found one time with a coincident blip in both channels, when we identified several more times in just one week of data from the omicron triggers. What was the "relatively high threshold" you used?
Hi. Sorry for taking so long with this. I rerun PCAT on the PSL and GDS channels between 2016-11-30 and 2016-12-31 with a lower threshold for glitch identification (glitches with amplitude > 4 sigma the noise floor) and with a larger coincidence window (coincident glitches within 0.1 seconds). The list of found coincident glitches is attached to the report. Four glitches in Miriam's list [https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt] show up in the list: 1164532915.0 (type 1 PSL/type 3 GDS), 1164741925.6 (type 1 PSL/type 1 GDS), 1164876857.0 (type 8 PSL/type 1 GDS), 1164882018.5 (type 1 PSL/type 8 GDS). I looked at other glitches in these types and found only one additional blip at 1166374567.1 (type 1 PSL/type 1 GDS) out of 9 additional coincident glitches. The typical waveforms of the GDS glitches show that the blip type(s) in GDS are type 1 and/or type 8. There are 1998 (type 1) and 830 (type 8) glitches in these classes. I looked at a few examples in cat 8 and indeed found several blip glitches which are not coincident with any glitch in the PSL channel. I would conclude that PCAT does not produce much evidence for a strong correlation of blip glitches in GDS and PSL. If there is, PSL-coincident glitches must be a small subset of blip glitches in h(t). However, some blips *are* coincident with glitches in the PSL, so looking more into this may be a good idea.
Hi,
thanks Marco for looking into this. We already expected that it was a small sub-set of blip glitches, because we only found very few of them and we knew the total number of blip glitches was much higher. However, I believe that not all blip glitches have the same origin and that it is important to identify sub-sets, even if small, to possibly fix whatever could be fixed.
I have extended the wiki page https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLblips and the list of times https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt up to yesterday. It is interesting to see that I did not identify any PSL blips in, e.g., Jan 20 to Jan 30, but that they come back more often after Feb 9. Unfortunately, it is not easy to automatically identify the PSL blips: the criteria I used for the omicron triggers (SNR > 30, central frequency ~few hundred Hz) do not always yield to blips but also to things like https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=156436, which also affects CALIB_STRAIN but not in the form of blip glitches.
None of the times I added up to December appear in your list of coincident glitches, but that could be because their SNR in PSL is not very high and they only leave a very small imprint in CALIB_STRAIN compared with the ones from November. In January and February there are several louder ones with bigger effect on CALIB_STRAIN though.
The most recent iteration of PSL-ISS flag generation showed three relatively loud glitch times:
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/scans/1170732596.35/
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/scans/1170745979.41/
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170212/latest/scans/1170950466.83/
The first 2 are both on Feb 10, in fact a PSL-ISS channel was picked by Hveto on that day (https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/#hveto-round-8) though not very high significance.
PSL not yet glitch-free?
Indeed PSL is not yet glitch free, as I already pointed out in my comment from last week.
Imene Belahcene, Florent Robinet
At LHO, a simple command line works well at printing PSL blip glitches:
source ~detchar/opt/virgosoft/environment.shomicron-print channel=H1:PSL-ISS_PDA_REL_OUT_DQ gps-start=1164500000 gps-end=1167500000 snr-min=30 freq-max=500 print-q=1 print-duration=1 print-bandwidth=1 | awk '$5==5.08&&$2<2{print}'
GPS times must be adjusted to your needs.
This command line returns a few GPS times not contained in Miriam's blip list: must check that they are actual blips.
The PSL has different types of glitches that match those requirements. When I look at the Omicron triggers, I do indeed check that they are blip glitches before adding the times to my list. Therefore it is perfectly consistent that you find GPS times with those characteristics that are not in my list. However, feel free to check again if you want/have time. Of course I am not error-free :)
I believe the command I posted above is an almost-perfect way to retrieve a pure sample of PSL blip glitches. The key is to only print low-Q Omicron triggers.
For example, GPS=1165434378.2129 is a PSL blip glitch and it is not in Miriam's list.
There is nothing special about what you call a blip glitch: any broadband and short-duration (hence low-Q) glitch will produce the rain-drop shape in a time-frequency map. This is due to the intrinsic tiling structure of Omicron/Omega.
Next time I update the list (probably some time this week) I will check the GPS times given by the command line you suggest (it would be nice if it does indeed work perfectly at finding only these glitches, then we'd have an automated PSL blips finder!)
Lockloss 03:30 UTC. Wind is low, no earthquake. The range looked stable.
Back to Observe 4:10 UTC. Accepted IMC-REFL_SERVO gain SDF diff (screenshot attached -- why would there be a diff though).