TITLE: 01/23 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 66Mpc
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
Wind: 5mph Gusts, 3mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.73 μm/s
QUICK SUMMARY:
BRSY is in fault. Leaving alone unless we lose lock.
TITLE: 01/23 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 67Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: A bit of commissioning earlier in the evening. BRSY is faulty (alog33522). Patrick suggested I should leave the Beckhoff alone until we lose lock so I did.
LOG:
02:44 Out of observing, Sheila and Robert doing some injection
03:44 Back to Observe.
05:27 BRSY went fault.
Jim (on phone), Nutsinee
I noticed the BRSY fault message so I digged around and found BRS_ETMY_RX_INMON and BRS_ETMY_VEL to be flat -- so I called Jim. As Jim requested I attached spectum of the subtracted and the unsubtracted BRSY signal, before and after it went fault. The two signals are exact same after the BRS went to fault. I also attached timeseries of bit channels (T1600103 -- see Status Bit troubleshooting). I couldn't find DRIFT BIT so I attached DRIFTMON instead. CBIT and DRIFTMON went bad/flat around 05:27 UTC.
According to Jim BRSY is not feeding anything (flat signal?) to the ISI right now. End Y is currently using BLEND_Quiet_250_SC_BRS. As long as the wind is low this configuration should be okay.
As per the "Troubleshooting guide" mentioned above, the CBIT being down indicates that the C# code crashed. This code reads the CCD camera and calculates the BRS angle. It is the first time this code has crashed at EY. Odd...
To restart, one should remote login to BRS-Y beckhoff computer as described in the guide above, then close the current BRS2 C# code (labelled as such) and the BRSY EPICS script. Then follow the startup procedure from Step 9 onwards. If this doesn't work, there may be a hardware problem, which will need more steps to diagnose.
At 11:12 UTC the IFO lost lock. I logged into h1brsey and double clicked on the 'BRS C#' shortcut on the desktop. The program started and BRSY is no longer reporting a fault.
I've added a test to DIAG_MAIN to catch this (I think we used to have something similar when we just had one BRS?). Before DIAG_MAIN just looked for the BRS guardian nodes to go into fault, and would just report the node was in fault. I've now inluded an explicit check for the CBIT, and DIAG_MAIN will now tell you the C code has crashed.
def BRS_CHECK():
"""Check the two end station Beam Roation Sensors to make sure that
they are not in FAULT (as read from their Guardian status node state.
Also checks the status of the CBIT, to see if C code is live
"""
for end in ['X','Y']:
if ezca['ISI-GND_BRS_ETM{}_CBIT'.format(end)] == 0:
yield 'BRS {} C code has stopped'.format(end)
elif ezca['GRD-BRS{}_STAT_STATE'.format(end)] == 'FAULT':
yield 'BRS {} is in FAULT'.format(end)
Robert and I made some slow excitations on both compensation plate, with results similar to what we saw in October (30979) The times of 600 count L injection in CPY DAMP L were 2:51:12 Jan 23rd UTC, 100 counts in CPX L was 3:05 UTC, and 600 counts on CPX was 3:08:38 UTC.
We also tried moving CPY, and we saw that we could reduce the two bright spots on the PRM camera by moving from its nominal position of -150 P 0Y to +100 P 0Y. We tried an on off test and saw that it made no difference for DARM so we left it at its nominal position.
I also made a series of OMC length measurements earlier, plots coming soon.
I made a series of mains injections, each phase at each building. We stayed in observing mode since I stayed outside the LVEA/VEA and turned a heater on/off. But there is a record of my activities below.
UTC Jan 22
17:12 Started down Y-arm
17:21 arrived EY
Circuit 20 (phase A) 17:36:05 - 17:36:55
Circuit 4 (phase B) 17:38:05 - 17:38:55
Circuit 6 (phase C) 17:39:35 - 17:40:25
17:45 left EY
17: 54 CS
Circuit 26 (phase A) 18:06:05-18:06:55
Circuit 28 (phase B) 18:07:35-18:08:25
Circuit 30 (phase C) 18:09:05-18:09:55
18:15 left for EX
18:23 arrived EX
Circuit 20 (phase A) 18:34:05 - 18:34:55
Circuit 9 (phase B) 18:35:05-18:35:55
Circuit 11 (phase C) 18:37:05-18:37:55
18:40 left EX
18:48 arrived CS
Quick Summary: Turned off the 4735Hz notch in LSC Darm2 filter and back to Observe.
16:30 Robert on site looking to do injections. See ensuing aLogs
16:45 spoke to Livingston. They don't anticipate being able to lock anytime soon due to deteriorating weather conditions and high microseism.
18:41 PI Mode 28 damped. Switched polarity on phase.
20:15 Sheila on site. Troubles doing INPUT_ALIGN. She's adjusting the gain configuration.
22:30 NLN
23:05 Robert back out to LVEA
23:20 Robert back and finished
23:23 Intention bit Undisturbed
23:29 Intention bit Preventative Maintenance for Violin damping?
23:23UTC
23:29UTC coming out of observing to damp 4735 which is currently aroun 1e-17
21:20UTC Switched config on Sheila's recommendation.
Since we're finally back locked at NLN I've accepted the diffs caused by the changed configuration so that I can set the Intent bit when Sheila and Robert are done with their work.
This configuration uses 45mhz blends on the corner station BSCs, making us much more vulnerable to wind. The forecast for the the next week looks okay as far as wind goes, but if it winds pick up, we should switch back.
19:28UTC Here are the lockloss plots. Re-locking attempts aern't looking to good in the present state of alignment. It looks as if IA is in order.
Robert is going into the LVEA to do some vibration injections.
17:08 Robert heading out to do vetting injections regarding mains monitor spikes. We will remain in Observing unless he should call and inform me that he has to do something invasive. He is beginning with EY
I spoke to Radar at Livingston and he told me that high microseism and wind has them down for at least the day. Weather there is deteriorating. Outlook for L1 locking unknown at this time.
Robert back from EY and into CER.
Robert now heading to EX.
Robert back.
Tom Dent, Miriam Cabero
We have identified a sub-set of blip glitches that might originate from PSL glitches. A glitch with the same morphology as a blip glitch shows up in the PSL-ISS_PDA_REL_OUT_DQ channel at the same time as a blip glitch is seen in the GDS-CALIB_STRAIN channel.
We have started identifying times of these glitches using omicron triggers from the PSL-ISS_PDA_REL_OUT_DQ channel with 30 < SNR < 150 and central frequencies between ~90 Hz and a few hundreds of Hz. A preliminary list of these times (on-going, only period Nov 30 - Dec 6 so far) can be found in the file
https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt
or, with omega scans of both channels (and with a few quieter glitches), in the wiki page
Only two of those times have full omega scans for now:
The whitened time-series of the PSL channel looks like a typical loud blip glitch, which could be helpful to identify/find times of this sub-set of blip glitches by other methods more efficient than the omicron triggers:
The CBC wiki page has been moved to https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLblips
I ran PCAT on H1:GDS-CALIB_STRAIN and H1:PSL-ISS_PDA_REL_OUT_DQ from November 30, 2016 to December 31, 2016 with a relatively high threshold (results here: https://ldas-jobs.ligo-wa.caltech.edu/~cavaglia/pcat-multi/PSL_2016-11-30_2016-12-31.html). Then I looked at the coincidence between the two channels. The list of coincident triggers is: ----------------------------------------------------- List of triggers common to PSL Type 1 and GDS Type 1: #1: 1164908667.377000 List of triggers common to PSL Type 1 and GDS Type 10: #1: 1164895965.198000 #2: 1164908666.479000 List of triggers common to PSL Type 1 and GDS Type 2: #1: 1164882018.545000 List of triggers common to PSL Type 1 and GDS Type 4: #1: 1164895924.827000 #2: 1164895925.031000 #3: 1164895925.133000 #4: 1164895931.640000 #5: 1164895931.718000 #6: 1164895958.491000 #7: 1164895958.593000 #8: 1164895965.097000 #9: 1164908667.193000 #10: 1164908667.295000 #11: 1164908673.289000 #12: 1164908721.587000 #13: 1164908722.198000 #14: 1164908722.300000 #15: 1164908722.435000 List of triggers common to PSL Type 1 and GDS Type 7: #1: 1166374569.625000 #2: 1166374569.993000 List of triggers common to PSL Type 1 and GDS Type 8: #1: 1166483271.312000 ----------------------------------------------------- I followed-up with omega scans and among the triggers above, only 1164882018.545000 is a blip glitch. The others are ~ 1 sec broadband glitches with frequency between 512 and 1024. A few scans are attached to the report.
Hi Marco,
your 'List of triggers common to PSL Type 1 and GDS Type 4' (15 times in two groups) are all during the known times of telephone audio disturbance on Dec 4 - see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=32503 and https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLGlitches
I think these don't require looking into any further, the other classes may tell us more.
The GDS glitches that look like blips in the time series seem to be type 2, 7, and 8. You did indeed find that the group of common glitches PSL - GDS type 2 is a blip glitch. However, the PSL glitches in the groups with GDS type 7 and 8 do not look like blips in the omega scan. The subset we identified clearly shows blip glitch morphology in the omega scan for the PSL channel, so it is not surprising that those two groups turned out not to be blips in GDS.
It is though surprising that you only found one time with a coincident blip in both channels, when we identified several more times in just one week of data from the omicron triggers. What was the "relatively high threshold" you used?
Hi. Sorry for taking so long with this. I rerun PCAT on the PSL and GDS channels between 2016-11-30 and 2016-12-31 with a lower threshold for glitch identification (glitches with amplitude > 4 sigma the noise floor) and with a larger coincidence window (coincident glitches within 0.1 seconds). The list of found coincident glitches is attached to the report. Four glitches in Miriam's list [https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt] show up in the list: 1164532915.0 (type 1 PSL/type 3 GDS), 1164741925.6 (type 1 PSL/type 1 GDS), 1164876857.0 (type 8 PSL/type 1 GDS), 1164882018.5 (type 1 PSL/type 8 GDS). I looked at other glitches in these types and found only one additional blip at 1166374567.1 (type 1 PSL/type 1 GDS) out of 9 additional coincident glitches. The typical waveforms of the GDS glitches show that the blip type(s) in GDS are type 1 and/or type 8. There are 1998 (type 1) and 830 (type 8) glitches in these classes. I looked at a few examples in cat 8 and indeed found several blip glitches which are not coincident with any glitch in the PSL channel. I would conclude that PCAT does not produce much evidence for a strong correlation of blip glitches in GDS and PSL. If there is, PSL-coincident glitches must be a small subset of blip glitches in h(t). However, some blips *are* coincident with glitches in the PSL, so looking more into this may be a good idea.
Hi,
thanks Marco for looking into this. We already expected that it was a small sub-set of blip glitches, because we only found very few of them and we knew the total number of blip glitches was much higher. However, I believe that not all blip glitches have the same origin and that it is important to identify sub-sets, even if small, to possibly fix whatever could be fixed.
I have extended the wiki page https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLblips and the list of times https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt up to yesterday. It is interesting to see that I did not identify any PSL blips in, e.g., Jan 20 to Jan 30, but that they come back more often after Feb 9. Unfortunately, it is not easy to automatically identify the PSL blips: the criteria I used for the omicron triggers (SNR > 30, central frequency ~few hundred Hz) do not always yield to blips but also to things like https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=156436, which also affects CALIB_STRAIN but not in the form of blip glitches.
None of the times I added up to December appear in your list of coincident glitches, but that could be because their SNR in PSL is not very high and they only leave a very small imprint in CALIB_STRAIN compared with the ones from November. In January and February there are several louder ones with bigger effect on CALIB_STRAIN though.
The most recent iteration of PSL-ISS flag generation showed three relatively loud glitch times:
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/scans/1170732596.35/
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/scans/1170745979.41/
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170212/latest/scans/1170950466.83/
The first 2 are both on Feb 10, in fact a PSL-ISS channel was picked by Hveto on that day (https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/#hveto-round-8) though not very high significance.
PSL not yet glitch-free?
Indeed PSL is not yet glitch free, as I already pointed out in my comment from last week.
Imene Belahcene, Florent Robinet
At LHO, a simple command line works well at printing PSL blip glitches:
source ~detchar/opt/virgosoft/environment.shomicron-print channel=H1:PSL-ISS_PDA_REL_OUT_DQ gps-start=1164500000 gps-end=1167500000 snr-min=30 freq-max=500 print-q=1 print-duration=1 print-bandwidth=1 | awk '$5==5.08&&$2<2{print}'
GPS times must be adjusted to your needs.
This command line returns a few GPS times not contained in Miriam's blip list: must check that they are actual blips.
The PSL has different types of glitches that match those requirements. When I look at the Omicron triggers, I do indeed check that they are blip glitches before adding the times to my list. Therefore it is perfectly consistent that you find GPS times with those characteristics that are not in my list. However, feel free to check again if you want/have time. Of course I am not error-free :)
I believe the command I posted above is an almost-perfect way to retrieve a pure sample of PSL blip glitches. The key is to only print low-Q Omicron triggers.
For example, GPS=1165434378.2129 is a PSL blip glitch and it is not in Miriam's list.
There is nothing special about what you call a blip glitch: any broadband and short-duration (hence low-Q) glitch will produce the rain-drop shape in a time-frequency map. This is due to the intrinsic tiling structure of Omicron/Omega.
Next time I update the list (probably some time this week) I will check the GPS times given by the command line you suggest (it would be nice if it does indeed work perfectly at finding only these glitches, then we'd have an automated PSL blips finder!)