So it seems there WAS a cause for my previous locklosses, not that I could tell from StripTool. It seems that mode 26, although on a steady rise, was the culprit. It never broke the visual threshold of what is normally considered 'ringing up' but after speaking to Terra on the phone from LLO control room she had me wait at DC readout until it rang down below the noise floor. She called back at that point and I began taking H1 all the way up. Upon continuing the it began to shoot up linearly with the power increase so she began damping remotely. Long Story short, this slow moving PI mode kept the PLL for damping at bay. When it finally locked it took 10s of thousands of counts in gain to finally keep the mode at bay and begin to turn it. Previously, on Patrick's shift she reported that it took an almost equal dramatic DECREASE to achieve damping.
New Modes 22 and 23, I'm told, ring up after ≈ 4 hours into the lock. I may not see them tonight. Those gains and phases are currently set to zero. Initially the gains can be adjusted 1000 counts at a time according o what Terra told me.
Not much else to tell about the night so far that I haven't already logged.
Note: This and last lockloss, the Observatory Mode wsn't switched from Observing.
04:01:51UTC again, reason unknown.
04:20 DC readout. Re-locking going well so far.
04:28 NLN
04:29 Checking a2l alignment again.
04:30 Intention Bit - Undisturbed
02:08:24 Lockloss. Reason uinknown. Below is a default lockloss plot(s)
after a bit of Y-Arm, BS, PRMI re-aligning and OMC threatening not to lock...
03:17 NLN
03:27 Ran passive a2l measurement. Result was pretty ugly in YAW (see figure below)
03:28 running a2l script
03:30 damping PI modes 27 and 28
03:38 a2l script done. Checking SDF.
03:40 Accepted a TCS CO2 SDF diff with Nutsinee's assist
03:42:03 Intention Bit - Observe
Here is a prediction of the OMC length noise with the configuration we have been using for the last few weeks (dither amplitude750 counts, length servo digital gain 24, resulting in 6Hz ugf, before the addition of the boost described in 33989)
The length noise is larger than it was the first time I checked this, because of the reduced dither amplitude, but not really a problem as is.
I described how I made these measurements in detail in alog 30510, which includes links to other meaurements. This time I relied on 12 Hz excitations to increase the OMC length noise to be visible in DARM. To briefly summarize, at high frequencies I estiamted the length noise by locking the OMC on the sideband with the ISS second loop gain increased, at low and mid frequencies using the in loop error signal, and the excitation amplitide was estimated using the height of the second harmonic. Then the resulting RIN in transmission of the OMC is estimated, and the DARM loop supression corrected for.
The three attachments show:
We recently found that the TCSX FLIR camera has been turned on for a while. I just switched it off after a lockloss.
WP #6472 Today I modified the electric actuators (LLCVs) controlling CP4, CP3, CP6 and CP5 (the mid-stations). I will do the four remaining ones (end-stations and corner-station) during this upcoming Tuesday maintenance day. This process required pulling the vacuum rack fuses for the 24VDC supplying the actuators and the 4-20 mA read back signals. For some unknown reason, it appears that the PT245A pirani interlock was disabled as a result of pulling these fuses. This, in turn, caused PT245B to be disabled. We will investigate (some new Beckhoff interrelationship? Voltage spike via common DC pwr supply?). The "as found" problem needing fixing is that water has been entering the actuator housings due to a lack of sealing where the 1/2" conduit elbow threads into the housings. The issue is that the tapped hole is too tight/shallow effectively preventing the "Seal Tight" plastic elbow from penetrating deep enough so as to compress its sealing O-ring (see attached image). I remedied this by running a tap deeper into the hole (see attached image). Now the sealing O-ring is compressed and should be water tight (see attached image). Embracing my eternal pessimism, I went ahead and drilled and tapped a drain hole (see attached images) which should prevent the level of water from rising high enough within the housing to cause any problems should water some how still get into the housing. The hose was added to allow for a later insertion of steel wool to discourage insect entry into the housing (see attached image).
Ops Shift Log: 02/10/2017, Day Shift 16:00 – 00:00 (08:00 - 16:00) Time - UTC (PT)
Shift Summary: Encountered relocking problems with the OMC LSC_I filter bank, which Sheila resolved. During and after locking worked with Terra to setup Ops StripTools to monitor the two new PI_Modes (Mode19 & Mode20). The PI2_monitoring.stp has been reloaded on Nuc6. Had to damp PI_Modes 26, 27, & 28 several times during the shift. Also took advantage of LLO not having fully recovered from this morning’s lockloss to run the A2L script. By the end of the shift, the A2L Pitch was elevated again.
Sheila & Terra implemented new filters to damp an additional PI_Mode that was implicated in the last two lockloss, (see aLOG #34046 and WP #6474). Dropped out of Observing for 2 minutes to clear SDF differences.
The wind is starting to come up again, now at Moderate to Strong Breeze (13 to 30 mph); End-Y microseism is up sharply.
After seeing terrible glitches right before both locklosses today (see figure 1) I started digging around to see what the cause was. The noise looked similar to times of other PIs. Looking at the summary page which monitors the PI behaviour you can see the 28-32 kHz region is elevated right before both locklosses (see figure 2). After plotting (see figure 3) the timeseries of H1:OMC-BLRMS_32_BAND{1-7}_
Mode is ETMY 32761.5 Hz.
First attachment shows the mode as seen in H1:OMC-PI_DCPD_64KHZ_AHF_DQ while rung up before the first lockloss ~7 UTC today. Second attachment shows that this mode (in black) is from ETMY by following other known ETMY modes (in blue) during the 19 hr lock that ended in that lockloss. We see that it is indeed at 32761.5 Hz - as opposed to aliased down from 32775.5 Hz - since it follows the slope of the ETMY drumhead and 15kHz mode, contrary to the 18kHz mode which is aliased down from 47kHz and thus is seen to move opposite.
I will set up damping asap - it will drop us out of Observe and since we are currently in coincident time, I'll wait until one site drops or the mode starts to ring up. Currently monitoring.
Sheila, Terra
There's 4 modes in this 32.76kHz region, two on each end test mass: 32761.5 Hz ETMY, 32764.5 Hz ETMX, 32766 Hz ETMX, 32767.5 (aliased down from 32768.5) Hz. ETMY, as shown here.
The first and last have sat the highest over the past several locks (the first breaking a few locks), so we've set up damping loops for them:
MODE 23: 32761.5 Hz ETMY and MODE 22: 32767.5 Hz ETMY
Note that the Nyquist is 32768 Hz, so MODE 22 sits right there on it and none of this region can be seen in DTT (something something it cuts off before the Nyquist?). They can be seen in the OMC-PI channel and they've been added to the PI2 StripTool. Have added all 4 to PI Wiki.
Mode 22 and 23 are set up to damp on ETMY. both using channel 5 on ETMY. Both 22 and 23 are on the PI2 stip tool, but niether is added to verbal alarms yet so operators will want to keep an eye on the StripTool this weekend.
The damping gains and phases for these modes are unmonitored in SDF. If you see one of the modes ringing up, try damping it. Since we don't know what shape these modes have, Terra suggested that we try damping it with the same sign on UR and LL. If this isn't working, you can try flipping the matrix element for LL to -1, see the attached screenshot for how to do that.
A rough few hours of relocking this morning. After Sheila fixing an OMC/Guarding problem, damping many PI modes, and running the A2L script the IFO is locked at NOMINAL_LOW_NOISE with 30.3 W of power and 65.0 Mpc or range. The wind is rising up to a Fresh Breeze (up to 24 mph); seismic activity is coming up as well. A large tour of 6th graders is expected shortly in the control room.
Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 706 seconds. TC B did not register fill. LLCV set back to 14.0% open. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill not completed after 3600 seconds. LLCV set back to 39.0% open.
Raised CP3 LLCV to 15%.
Kyle fixed CP4 actuator by melting the ice and draining the water (pulled the fuse which killed autofill 19 min. into operation), and tightened the elbow seal. Manual fill took 41 min. with 1/2 turn on bypass valve, in addition to 19 min. of autofilling at 70% open on LLCV. Raised LLCV from 39% to 42% open.
We saw a pressure spike in HAM1 yesterday at 16:27 UTC (8:30 am local). Nothing unusual in the ion pump voltage during that time.
On tuesday evening I added a boost to the OMC length loop, to prevent 1083 Hz glitches due to more motion of the interferomter with BRSY being off (33989).
This has been on since then, and seems fine. This morning Jeff B had trouble relocking the OMC because this boost was on, so I moved the boost into the OMC-LSC_SERVO filter bank and added engaging it to the guardian.
Jim W., Jeff B., Terra
We've used the earthquake down time to add the following modes to PI damping:
MODE 19: 18041 Hz ETMX and MODE 21: 18061 Hz ETMX
Both are now fully set up in the PI MEDM screen, receiving error signal from OMC and driving ETMX. Both do NOT have a gain auto set to turn on (so will send zero drive); these modes haven't gone unstable in the past month or so but should be watched and if ops/commissioners find it best to set a gain, we can add that to the SUS_PI guardian. Jeff has added these modes to PI2 StripTool though can someone make sure they show up on the front wall StripTool?
I added the new modes 19 and 20 to the PI2_monitoring.stp StripTool. The nuc6 screen has been reloaded with the new modes.
Tom Dent, Miriam Cabero
We have identified a sub-set of blip glitches that might originate from PSL glitches. A glitch with the same morphology as a blip glitch shows up in the PSL-ISS_PDA_REL_OUT_DQ channel at the same time as a blip glitch is seen in the GDS-CALIB_STRAIN channel.
We have started identifying times of these glitches using omicron triggers from the PSL-ISS_PDA_REL_OUT_DQ channel with 30 < SNR < 150 and central frequencies between ~90 Hz and a few hundreds of Hz. A preliminary list of these times (on-going, only period Nov 30 - Dec 6 so far) can be found in the file
https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt
or, with omega scans of both channels (and with a few quieter glitches), in the wiki page
Only two of those times have full omega scans for now:
The whitened time-series of the PSL channel looks like a typical loud blip glitch, which could be helpful to identify/find times of this sub-set of blip glitches by other methods more efficient than the omicron triggers:
The CBC wiki page has been moved to https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLblips
I ran PCAT on H1:GDS-CALIB_STRAIN and H1:PSL-ISS_PDA_REL_OUT_DQ from November 30, 2016 to December 31, 2016 with a relatively high threshold (results here: https://ldas-jobs.ligo-wa.caltech.edu/~cavaglia/pcat-multi/PSL_2016-11-30_2016-12-31.html). Then I looked at the coincidence between the two channels. The list of coincident triggers is: ----------------------------------------------------- List of triggers common to PSL Type 1 and GDS Type 1: #1: 1164908667.377000 List of triggers common to PSL Type 1 and GDS Type 10: #1: 1164895965.198000 #2: 1164908666.479000 List of triggers common to PSL Type 1 and GDS Type 2: #1: 1164882018.545000 List of triggers common to PSL Type 1 and GDS Type 4: #1: 1164895924.827000 #2: 1164895925.031000 #3: 1164895925.133000 #4: 1164895931.640000 #5: 1164895931.718000 #6: 1164895958.491000 #7: 1164895958.593000 #8: 1164895965.097000 #9: 1164908667.193000 #10: 1164908667.295000 #11: 1164908673.289000 #12: 1164908721.587000 #13: 1164908722.198000 #14: 1164908722.300000 #15: 1164908722.435000 List of triggers common to PSL Type 1 and GDS Type 7: #1: 1166374569.625000 #2: 1166374569.993000 List of triggers common to PSL Type 1 and GDS Type 8: #1: 1166483271.312000 ----------------------------------------------------- I followed-up with omega scans and among the triggers above, only 1164882018.545000 is a blip glitch. The others are ~ 1 sec broadband glitches with frequency between 512 and 1024. A few scans are attached to the report.
Hi Marco,
your 'List of triggers common to PSL Type 1 and GDS Type 4' (15 times in two groups) are all during the known times of telephone audio disturbance on Dec 4 - see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=32503 and https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLGlitches
I think these don't require looking into any further, the other classes may tell us more.
The GDS glitches that look like blips in the time series seem to be type 2, 7, and 8. You did indeed find that the group of common glitches PSL - GDS type 2 is a blip glitch. However, the PSL glitches in the groups with GDS type 7 and 8 do not look like blips in the omega scan. The subset we identified clearly shows blip glitch morphology in the omega scan for the PSL channel, so it is not surprising that those two groups turned out not to be blips in GDS.
It is though surprising that you only found one time with a coincident blip in both channels, when we identified several more times in just one week of data from the omicron triggers. What was the "relatively high threshold" you used?
Hi. Sorry for taking so long with this. I rerun PCAT on the PSL and GDS channels between 2016-11-30 and 2016-12-31 with a lower threshold for glitch identification (glitches with amplitude > 4 sigma the noise floor) and with a larger coincidence window (coincident glitches within 0.1 seconds). The list of found coincident glitches is attached to the report. Four glitches in Miriam's list [https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt] show up in the list: 1164532915.0 (type 1 PSL/type 3 GDS), 1164741925.6 (type 1 PSL/type 1 GDS), 1164876857.0 (type 8 PSL/type 1 GDS), 1164882018.5 (type 1 PSL/type 8 GDS). I looked at other glitches in these types and found only one additional blip at 1166374567.1 (type 1 PSL/type 1 GDS) out of 9 additional coincident glitches. The typical waveforms of the GDS glitches show that the blip type(s) in GDS are type 1 and/or type 8. There are 1998 (type 1) and 830 (type 8) glitches in these classes. I looked at a few examples in cat 8 and indeed found several blip glitches which are not coincident with any glitch in the PSL channel. I would conclude that PCAT does not produce much evidence for a strong correlation of blip glitches in GDS and PSL. If there is, PSL-coincident glitches must be a small subset of blip glitches in h(t). However, some blips *are* coincident with glitches in the PSL, so looking more into this may be a good idea.
Hi,
thanks Marco for looking into this. We already expected that it was a small sub-set of blip glitches, because we only found very few of them and we knew the total number of blip glitches was much higher. However, I believe that not all blip glitches have the same origin and that it is important to identify sub-sets, even if small, to possibly fix whatever could be fixed.
I have extended the wiki page https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLblips and the list of times https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt up to yesterday. It is interesting to see that I did not identify any PSL blips in, e.g., Jan 20 to Jan 30, but that they come back more often after Feb 9. Unfortunately, it is not easy to automatically identify the PSL blips: the criteria I used for the omicron triggers (SNR > 30, central frequency ~few hundred Hz) do not always yield to blips but also to things like https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=156436, which also affects CALIB_STRAIN but not in the form of blip glitches.
None of the times I added up to December appear in your list of coincident glitches, but that could be because their SNR in PSL is not very high and they only leave a very small imprint in CALIB_STRAIN compared with the ones from November. In January and February there are several louder ones with bigger effect on CALIB_STRAIN though.
The most recent iteration of PSL-ISS flag generation showed three relatively loud glitch times:
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/scans/1170732596.35/
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/scans/1170745979.41/
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170212/latest/scans/1170950466.83/
The first 2 are both on Feb 10, in fact a PSL-ISS channel was picked by Hveto on that day (https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20170210/latest/#hveto-round-8) though not very high significance.
PSL not yet glitch-free?
Indeed PSL is not yet glitch free, as I already pointed out in my comment from last week.
Imene Belahcene, Florent Robinet
At LHO, a simple command line works well at printing PSL blip glitches:
source ~detchar/opt/virgosoft/environment.sh
omicron-print channel=H1:PSL-ISS_PDA_REL_OUT_DQ gps-start=1164500000 gps-end=1167500000 snr-min=30 freq-max=500 print-q=1 print-duration=1 print-bandwidth=1 | awk '$5==5.08&&$2<2{print}'
GPS times must be adjusted to your needs.
This command line returns a few GPS times not contained in Miriam's blip list: must check that they are actual blips.
The PSL has different types of glitches that match those requirements. When I look at the Omicron triggers, I do indeed check that they are blip glitches before adding the times to my list. Therefore it is perfectly consistent that you find GPS times with those characteristics that are not in my list. However, feel free to check again if you want/have time. Of course I am not error-free :)
I believe the command I posted above is an almost-perfect way to retrieve a pure sample of PSL blip glitches. The key is to only print low-Q Omicron triggers.
For example, GPS=1165434378.2129 is a PSL blip glitch and it is not in Miriam's list.
There is nothing special about what you call a blip glitch: any broadband and short-duration (hence low-Q) glitch will produce the rain-drop shape in a time-frequency map. This is due to the intrinsic tiling structure of Omicron/Omega.
Next time I update the list (probably some time this week) I will check the GPS times given by the command line you suggest (it would be nice if it does indeed work perfectly at finding only these glitches, then we'd have an automated PSL blips finder!)