Moved IM3P alignment offset from 29542 to 29864 to move the WIT P from 1945 to 1961.

Very directly related.

https://login.ligo.org/idp/shibboleth needs to return a signed XML document, not a human-readable error about missing files. LIGO-SAML-DS service on every SP running it trips over this. The very short-term fix is to modify your ligods.ini 's "master" property to point to the site IdP (login5. for LHO) rather than the master (login.) and restart ligods service. This is not something you should do outside of emergencies, since you really do want the latest metadata. Just happens to be a failure mode that hadn't been thought of at the time it was written -- it does handle "cannot talk to main IdP at all" case fine (e.g. Internet outage or similar).

I was just able to log in.

Ryan, thanks for the description of the problem. Should we go ahead with the short-term fix you mention or do we know if service restoration is imminent?

I would go ahead and make the change, with an eye towards reverting once folks are back at work next week. You can check if the central service has been fixed by hitting the first URL in my other comment (if you get a large blob of XML, you're good. If you get "missing file", not so good). No ETA since I have no management involvment with the central IdP server.

... and in bad form replying to myself, permanently changing sleeptime to something more like 300 seconds would not be amiss either, since the default 15 seconds induces a large load on all of the IdPs from purely "hello, are you there?" checks.

If the interferometer is up I will spend some time damping them tonight. 

It is important to notice that this 2kHz glitch line has been appearing and dissapearing quite irregularly in the past, but when it is present the associated Omicron glitches are of high SNR. In fact the last time this line showed up was all the way back to 29-30th November:

* 2kHz glitch line started to show on first 29th Nov lock

* 2kHz glitch line disappears on the 30th Nov

Originally I thought that the 2kHz glitch line could have been related to PCALX roaming calibration lines, based on Evan's alog on PCALX roaming calibration line frequency changes. The 2kHz glitch line seem to start as soon as the detector locked after the PCALX calibration line at 1001.3Hz was activated on UTC 2016-11-30 17:16:00, and then the glitch line disappeared around the time the cal line at 2001.3Hz was moved to 2501Hz at 2016-11-30 22:07:00. The fact that the time coincidence was not precise made me believe that the time coincidence may have been casual. It can now be confirmed that it must be unrelated because not such PCALX roaming line was not set at 2001.3Hz during the time of the current appearence of the 2kHz glitch line.

The obvious question is if the November 2kHz high SNR glitch line shows a similar 21 second spacing between bursts of glitches. The answer is yes ,as seen next during the dissapearance of the 2kHz glitch line on the 30th Nov (attached are the original images from which this image was made):

A zoom around the beginning of the above spectrum shows the ~21secs periodicity of the features:

A closer look to the 2nd harmonic violin modes for 30 mins during the time of the 2kHz glitch line (in blue) and 30 mins after the glitch line dissapears (in red) shows that only few violin modes were higher during the time of the 2kHz glitch line:

There are two cases when the blue lines are higher than the red:  

* At about 1003.7Hz:

* At about 1009_4Hz:

It is clear that only the pair at around 1009.45Hz would beat with a periodicity of about 20 seconds. And in this case while the lower frequency violin mode of the pair does not change much in amplitude however it is the higher frequency violin mode of the pair which increases by 30.

I have also compared the 1009.45Hz pair peaks amplitude for 4 different cases, two of them correspond to times when the 2kHz glitch line was present, and 2 other (dashed lines) to cases when the glitch line was not present. It shows how the higher frequency line has to be high enough to cause the nonlinearity for the 2kHz glitch to be present:

Nutsinee has just now created a damping filter for this pair of violin modes, so hopefully this will be enough to avoid growth of this peaks to the point of causing appearance of the 2kHz glitch line but only time will tell.

A small note for the operation purpose: Borja's 2kHz glitches thresholds on 1009.44Hz and 1009.49Hz correspond to

8e-13 and 6.5e-13 m/sqrt(Hz) in the (dtt calibrated) CAL-DELTAL_EXTERNAL_DQ channel in November

and

7e-13 and 4.4e-13 m/sqrt(Hz) in January.

Now that the guardian is turning the damping on, these two modes should be well controlled. But if something bad happened and the modes ring up I would suggest operators to take some time to damp the mode when they get close to 1e-12 on DARM FOM.

ITMY Roll Mode seems to be an ongoing obstacle to locking.

17 minutes for for damping. Added 30degrees to phase (60 total) and increased gain from 10 to 20.

Unsuccessful lock attempt @ LOWNOISE_ASC. decrease in ASAIR_B_RF90

11:08UTC Begin Initial Alignment

08:21:45UTC H1 back to Observing

For the meanwhile, it is probably worth trying to keep the AOM drive to be ~0.5 V.  One does this by
adjusting the first loop ISS reference signal (bottom left hand corner slider of the first loop ISS
MEDM screen).  The adjustment should be done with the second loop ISS off.

Plots of the ISS reference signal and its monitor are attached.  The reference signal comes from a DAC.  The
monitor is read off the ISS board from an OP-27.  One thing to note is that the monitor output is somewhat
noisier than the reference signal.  This might be related to the grounding issue (if there is indeed one).

Problem was tracked down to the loss of -18V in  PSL ISS AA chassis. Power cycling seems to have fixed it.

I was called and came to the site at 6PM to work with Peter who was also called earlier.

We have found that some of the ISS 1st loop signals were much smaller than they should be, starting 23:20 local time yesterday. PDA readback was only 0.7V or so when it should be 3.2, PDB was also too small, and ISS reference voltage readback was only -0.1 when it was set to -0.48.

We were first suspicious about analog failure of the ISS chassis, but when we went to the floor, everything looked normal in analog land. We measured the OLTF of ISS 1st loop and the UGF was about 38kHz.

Voltages on one of the DB9 that goes to the AA looked correct.

Then we went to the PSL ISS AA and found that LEDs for negative voltage was off, and the positive voltage LED was dim (will attach picture later). Peter power cycled it and everything seems to be back to normal.

Things to do for Tuesday:

Check PSL ISS AA chassis power board.

Things unrelated to this but are troubling:

We also went to the power supply area in mezzanine, and things look OK except that MANY power supplies have faulty power indicator lamps, some of them were flickering on and off.

Also, oplev power supply has a label saying +18V, but the actual voltage is more like +10V.

The reason why the readback failure matters is because we are now using the 1st loop PDA and PDB values for the power normalization of 2nd loop digital AC coupling.

With the AA chassis failing, H1:PSL-ISS_SECONDLOOP_INPUT_NORM hit the limitter bottom and no power normalization was done. Withrout a proper power normalization this AC coupling is known to cause loss of lock.

Fisrt picture: LEDs of the front panel with negative power failue (top chassis).

Second picture: LEDs of the back panel with negative power failue.

Third picture: UGF of ISS 1st loop.