Unfortunate turn of events considering the trouble I'm having.
Appears to be a general problem with the authentication system across the Lab/LSC this morning, not just the Ops wiki server.
LIGO-SAML-DS, which finds authentication servers, is unhappy with some very recent metadata changes (example from an affected GC server):
2017-01-14 07:51:56,396 ERROR Metadata signature cannot be verified: xmlsec1 returned non-zero code 1
If you (or CDS admins) need a workaround in the mean time while the central system is worked on, give me a call.
here are the recent ssl_error logs from cdswiki when I try to open the web service
[Sat Jan 14 09:59:00.588103 2017] [wsgi:error] [pid 23984:tid 140688044332800] [remote 71.84.186.186:38281] mod_wsgi (pid=23984): Exception occurred processing WSGI script '/usr/lib/cgi-bin/wsgi/ligo-saml-discovery-service/LIGO_SAML_DiscoveryService.wsgi'.
[Sat Jan 14 09:59:00.588145 2017] [wsgi:error] [pid 23984:tid 140688044332800] [remote 71.84.186.186:38281] Traceback (most recent call last):
[Sat Jan 14 09:59:00.588170 2017] [wsgi:error] [pid 23984:tid 140688044332800] [remote 71.84.186.186:38281] File "/usr/lib/cgi-bin/wsgi/ligo-saml-discovery-service/LIGO_SAML_DiscoveryService.wsgi", line 212, in application
[Sat Jan 14 09:59:00.588206 2017] [wsgi:error] [pid 23984:tid 140688044332800] [remote 71.84.186.186:38281] if returnURL.split('?')[0] in spDict[entityIDclaimed]:
[Sat Jan 14 09:59:00.588234 2017] [wsgi:error] [pid 23984:tid 140688044332800] [remote 71.84.186.186:38281] KeyError: 'https://lhocds.ligo-wa.caltech.edu/shibboleth-sp'
Bubba turned on heater 3A yesterday to help deal with the cold conditions. You can see in the plot that the heater is cycling, probably on a protective temperature switch.
Nutsinee reported some misbehavior in the LVEA 10-30 Hz seismic activity.
I'm not sure there is any correlation between these two events but someone might look at this. The heater is likely ~25kw so there is some significant EMI from the contactor opening and closing.
Re-locking has been a daunting task to say the least. PRC gave me a really hard time trying to get it locked. Upon getting it locked and the control signals converged, DRMI alignment still looked pretty terrible. BS alignment was checked repeatedly but to no avail to the alignment. I then decided to do a complete initial alignment. The IMC became unstable while trying to re-align the arms and I had to clear he WFS and re-align the optics. After letting the loops have the alignment for tuning during a brief break I continued with the initial alignment. Locking attempts following have been more sucesssful but not completely successful. So far, the farthest I've gotten is DC Readout.
4.8M Indonesia
Was it reported by Terramon, USGS, SEISMON? Yes, Yes, No
Magnitude (according to Terramon, USGS, SEISMON): 4.8, 4.8, NA
Location: 56km ENE of Taniwel, Indonesia
Starting time of event (ie. when BLRMS started to increase on DMT on the wall): ~06:30 UTC
Lock status? Caused lockloss.
EQ reported by Terramon BEFORE it actually arrived? Yes
Note: There were multuple earthquakes happened around the same time. It's not neccessary that this one in particular caused the lockloss. But the timing reported by Terramon matches.
TITLE: 01/14 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Ed
SHIFT SUMMARY: The range was awfully low but the IFO stayed locked until an earthquake knocked us out. Otherwise nothing exciting.
LOG:
07:19 Lockloss. There were several earthquakes happened right around the same time but the arrival time of the 4.8M Indonesia earthquake is closest to the peak time observed in the control room. The peak reached 0.4 um/s on the 0.03-0.1Hz FOM.
About 2 hours into this lock stretch the range started to drop very fast. About the same time 10-30 Hz seismic activity (LVEA vertex) is behaving funny. I don't believe that it's real (no one is out there). I don't see funny things in magnetometers. Still looking for more clues. Electronics related?
The temperature is dipping a couple of degrees in the LVEA again. This morning, Bubba had to turn on some extra heat in the LVEA, which was close to the time of the drop in range. Are some new fans getting turned on somehwhere producing extra seismic noise??
If that's the case shouldn't we see this behavior more often in the past 24 hours?
The weird glitches have stopped but the range doesn't recover. They seem to be 15 minutes spacing.
It's been set to KILL since Jan 4th. Are we leaving it at KILL on purpose?
This applies to the transient injections (not continuous pulsars). It is likely that we are trying to accumulate a good stretch of data without such injections since we came back up in January to better measure sensitivity backgrounds.
Patrick handed me a locked interferometer. The range still low. DMT Omega shows low frequency up to 20Hz is more glitchy compared to the past 12 hours. at 3:17 UTC (about 50 minutes into the lock) I watned to hit the button to run a2l dtt but instead I hit run a2l button. Back to Observe 3:34 UTC.
02:35 UTC Set to observing. I had to accept an SDF difference (see attached). I'm not sure what caused it. Would someone mind taking a look?
The OMC length locking ugf was increased at 18:30 today (while the interferometer was unlocked). It will stay this way permanently now, and we will have fewer glitches. (see alog 33104 for the work detch has done checking out that this configuration change is OK)
We intend to do this during the days when we are here as an experiment. If the clog is due to ice, reverse sublimation would be a convenient fix but we haven't looked at this closely.
TITLE: 01/14 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC STATE of H1: Observing at 54.8268Mpc INCOMING OPERATOR: Nutsinee SHIFT SUMMARY: Ran initial alignment after losing lock. Stopped at DC readout transition while Jim damped 4735.09 Hz violin mode. Sheila increased the ISS diffracted power and I accepted the SDF differences (see attached). Sheila and Jenne took time at NLN to investigate the loss in range. LOG: 16:42 UTC Chris to end Y to remove snow with John Deere tractor 16:49 UTC Bubba to end X to remove snow with front end loader 16:52 UTC Turned off sensor correction at end X 17:23 UTC Jim and Carlos to end X to check power supply for network switch 17:34 UTC Jim turning off another degree of sensor correction at end X, dropped to commissioning 17:36 UTC Jim accepted SDF difference, set back to observing 17:40 UTC Krishna changed filter that kicked us out of observing, set back 17:41 UTC Set back to observing 18:00 UTC Jim and Carlos back 18:11 UTC Lockloss. HAM6 ISI trip. Spike to ~ .1 um/s in end Y 0.03 - 0.1 Hz seismic band 18:15 UTC Jason to LVEA to increase SR3 laser power 18:19 UTC Karen driving to warehouse 18:24 UTC Starting initial alignment. ALS X is ~ .7 18:26 UTC Jason done 18:59 UTC Kyle to mid Y to turn on diaphragm pump for CP4 sensing line 19:08 UTC Turned off sensor correction at end Y 19:09 UTC Jim and Carlos to end Y 19:18 UTC Gerardo to mid Y 19:29 UTC Chris back from end Y 19:35 UTC Kyle back 19:40 UTC Jim and Carlos back 19:43 UTC Initial alignment done. Had some trouble locked SRC. 19:48 UTC Bubba done snow removal 19:49 UTC Sensor correction turned back on at both end X and end Y 20:01 UTC Stopped at CHECK_IR. Sheila adjusted ISS diffracted power from 1 to 4 percent following procedure in alog 31262 20:10 UTC Gerardo back 20:27 UTC Stopping at DC_READOUT_TRANSITION per Sheila's request. Jim damping 4735.09 Hz violin mode using procedure in alog 32081. Set H1SUS-ETMY_L2_DAMP_MODE10 gain to .1 20:52 UTC NLN. Set 4735.09 Hz damping gain back to .1, guardian must have reset it? 20:57 UTC Sheila starting investigation in low range 20:59 UTC Accepted SDF differences for ISS diffraction change 21:05 UTC Setting observation mode to corrective maintenance for Sheila's investigation 21:57 UTC Batteries in UPS for portable atomic clock died 22:30 UTC Filiberto and interns to mid Y to look for equipment 23:36 UTC Damped PI mode 28 23:41 UTC Gerardo to mid Y to retrieve power supply 23:43 UTC Filiberto back 23:50 UTC Sheila and Jenne done investigation. Set to observing. 00:28 UTC Lock loss after steadily falling range.
No flashes for DMRI or PRMI (later Sheila told me this is because she moved optics). 00:59 UTC Starting initial alignment. MC not locking. FSS is oscillating. Set to down a couple of times. Fixed. Struggled locking ALS X arm. Can not get the COMM beatnote above 6 by adjusting PR3. SRC locked, was converging then unlocked. Set ALIGN_IFO to down. Misaligned SRM. Moved SR2 to center pointing on ASC_AS_C. Realigned SRM. SRC locked. 02:00 UTC Initial alignment done. ** Note that this is the second time PR3 has needed to be moved a large amount to bring back the COMM beat note. ** On the way to NLN.
Summary of locking issues: 4735.09 Hz violin mode needed to be damped. Jim W. did this. Ran initial alignment twice. Each time I had to move PR3 a large amount to bring back the COMM beat note. The second time I could not bring it above 6. During both initial alignments I has trouble locking SRY. Sheila showed me a trick: Bring IFO_ALIGN to down. Misalign SRM. Open Sitemap -> LSC -> Photo Detectors Overview. Click on AS_C (box to the furthest right in red circle). Move SR2 to center beam spot in the X,Y plot on the right. Realign SRM. Take IFO_ALIGN back to SRC_ALIGN. The ISS diffracted power was low. Sheila fixed this (see summary).
J. Kissel, S. Dwyer, E. Goetz, K. Venkateswara We're lamenting last night's severe decay in range, and the gross long term trend toward zero since the restart after the break. Our most said theory (which doesn't make it the best or the right theory) is temperature adversely affecting the global alignment system's operating point, which has steered the IFO into a place with much worse scattering coupling. As such the anthropogenic noise (3-10, 10-30 Hz BLRMS of ground motion) is now adversely affecting the IFO much more than in the past. We'd like DetChar's help to confirm: - We've lost our fellow whom has run BruCo for us. Does detchar in general know how to run BruCo, or does Gabriele need to teach new people again? We suggest that no coherence with any particular channel would confirm the non-linear process of scattering. - We see summary page spectrograms showing excess noise, but they're too long of a time scale to tell if the features are scattering arches. Note that Bubba had started plowing at the X-End this morning, so data from ~17:00 UTC will show elevated 3-10 and 10-30 Hz BLRMS seismic noise there. So we understand that ground motion is much higher then, and because of the greater sensitivity to it (again, claiming the alignment operating point increases scattered light coupling theory) -- the problem is overnight when there appears to be no elevation in seismic noise, but the range is still terrible. Any other ideas are welcome. -------- Also because Bubba was going down to the X-end, we turned off sensor correction at 16:52 UTC (LHO aLOG 33221). This looks to coincide with the 1080 Hz glitching getting worse on the summary pages. This is consistent with the OMC length noise increase investigations we've done over the past few days (LHO aLOG 33104 and LHO aLOG 33037).
Andy ran Bruco on this already: https://ldas-jobs.ligo.caltech.edu/~lundgren/aDQ/bruco/H1_1168346418_OMC_DCPD/
Josh, TJ, Jess, Alex
This looks like PI ringing up. In the spectrogram posted above there is a line at 4734.75Hz that grows around the time of the noise. Here are some plots:
Fig 1: the line at 4734.75
Fig2: a BLRMS of this line vs time it grows alot.
Fig3: 8.8 hours of the line growing from onset of noise to really ratty time.
We will work harder to tie this to the noise. We've started looking back at earlier pages and seeing this line rising around times that noise rises but have work to do to tie them together.
There are three things that we can try to improve or investigate the drop in range as Patrick relocks.
Josh Smith, Jess, TJ and I have investigated a bit and found that there is a line around 4700 Hz (Fig. 1) that rings up around the same time as a scratchy forest of lines in mid-range frequencies, right in the bucket. At Josh's suggestion I took a comparative look at the beginnings and ends of three lock stretches on Jan 6 where this pattern rings up (the circled regions in Fig. 1). The last three figures show 10-second averaged spectra of 100 seconds of data gathered at the start and end of these lock stretches (GPS times given in the plots). "Good" periods, where there is no 4.7 kHz line, are shown in blue; "bad" periods where the line shows up are plotted in red. You can clearly see the ASD of H1:GDS-CALIB_STRAIN is systematically worse between 70-500 Hz when the high-frequency line is rung up, compared to when it wasn't. (We also note this behavior on Friday, 13 Jan, and on a couple of other days back in december, all coincident with a wandering downward trend in BNS range.)