I had to set the following environment variables:

EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_ADDR_LIST=10.101.0.255 10.106.0.255 10.105.0.255 10.20.0.255 10.22.15.255

I did this in the systemd config file at /lib/systemd/system/conlog.service:

[Unit]
Description=Conlog daemon
After=mysql.service

[Service]
Type=simple
Environment="EPICS_CA_AUTO_ADDR_LIST=NO"
Environment="EPICS_CA_ADDR_LIST=10.101.0.255 10.106.0.255 10.105.0.255 10.20.0.255 10.22.15.255"
ExecStart=/usr/sbin/conlogd

[Install]
WantedBy=multi-user.target

There are now 1992 unmonitored channels (attached), but it appears that these are channels that no longer exist.

J. Oberling, F. Clara

Fil removed the comparator box and ran some quick tests in the EE lab; he found nothing obviously wrong with it.  The box was reinstalled without issue.  Using a multimeter, Fil measured the input signal from the IR sensor.  The interesting thing here is that the input signal changed depending on how the box was positioned (trip point is at ~96mV).  Hanging free in mid-air the input signal measured ~56mV; with Fil touching the box, still hanging free, the signal dropped to ~36mV; holding the box on the side of the TCS enclosure the signal changed yet again to ~25mV; and finally, placing the box on top of the TCS enclosure (its usual home), the signal dropped yet again to ~15mV.  There is definitely something fishy going on with this comparator box; grounding issue or cable/connection problem maybe?

As a final check I opened the side panel to the TCS enclosure to check the viewport to ensure there were no obvious signs of damage.  Using a green flashlight I found no obvious signs of damage on either optic in the TCS viewport; in addition, nothing obviously looked amiss with the viewport itself, so for know now this seems to be an issue with either the comparator box or the IR sensor.  Seeing as how it appears to be working again (famous last words, I know...) we restarted the TCSy CO2 laser.  Everything came up without issue, we will continue to monitor this.

Wondering if this is related to the glitchy sigals from the TCSY laser controller.  They all run through that same controller box (though of course we did try swapping that out).  The lifting it up / putting it down, sounds like it could be a weird grounding issue.

Give me a call if you need any help on this tonight - I'll email my number to you.

Here are a couple pictures for informational purposes.  The first is the TCS laser controller chassis, and shows which light is lit when the IR sensor is in alarm.  The second shows the comparator box in alarm.  This box sits on top of the TCSy enclosure, on the north-east corner.

FRS has been filed (#7293), and this is ongoing.

Time out thus far:

• 21:18-22:12
• 22:38-ongoing

Tagging DetChar and OpsInfo.

This may mean that suspensions will start to wander around and sag. Maybe. Keep your eyes peeled!

Tagging DetChar, OpsInfo, and ISC.

There is suspicion from DetChar and Schofield that humidity, or the lack there of, at EY is causing electronics to glitch which propagate through the interferometer. So this *may* start to improve the IFO glitch rate. Maybe.

Also note the "cycling several times a minute." The humidifier's water heater is power from the YVEA, and it's controlled with a bang-bang servo, so look for changes in electronic / electromagnetic signals that show some characteristic ON / OFF signatures.

Again -- keep your eyes peeled!

We have been working on a low latency blip glitch hunter. It uses PyCBC Live but with a smaller template bank focused on finding possibly only blip glitches. We also generate summary pages, updated every 5 minutes:

https://ldas-jobs.ligo.caltech.edu/~miriam.cabero/blips_live/H1/day/20170209/detchar/pycbc_live/

Unfortunately it has only been running stably since Feb 3, so we cannot really compare directly if the humidifier added on Feb 1 made a significant change in the rate of blips. However, I started using this code on Jan 30, so there are some results for previous days but the code was still a little bit unstable. It can be looked at anway:

https://ldas-jobs.ligo-wa.caltech.edu/~miriam.cabero/blips_live/H1/day/20170131/detchar/pycbc_live/

By glancing at this plot from Jan 31

https://ldas-jobs.ligo-wa.caltech.edu/~miriam.cabero/blips_live/H1/day/20170131/plots/H1-OBSERVING_2FC680_PYCBC_LIVE_END_TIME_TEMPLATE_DURATION_SNR_TRIGGERS-1169856018-86400.png

and from e.g. yesterday

https://ldas-jobs.ligo.caltech.edu/~miriam.cabero/blips_live/H1/day/20170208/plots/H1-OBSERVING_2FC680_PYCBC_LIVE_END_TIME_TEMPLATE_DURATION_SNR_TRIGGERS-1170547218-86400.png

it doesn't look like there has been an improvement...

Pressure is looking good!

Based on drawing and measurements, the blockage in CP4 sensing line is just past the bibraze joint. For CP3 it is just before the bibraze joint.

A set of omega scans of similar events on 26th Jan (identified by thresholding on ISI-GND_STS_HAM2_Y) can be found at https://ldas-jobs.ligo-wa.caltech.edu/~tdent/wdq/isi_ham2/

Wow that is pretty loud, seems like it is even seen (though just barely) on seismometers clear out at EY with about the right propagation delay for air or ground propagation in this band (about 300 m/s). Like a small quake near the corner station or something really heavy, like the front loader, going over a big bump or setting its shovel down hard. Are other similar events during working hours and also seen at EY or EX?

It's hard to spot any pattern in the GPS times.  As far as I have checked the disturbances are always much stronger in CS/LVEA than in end station (if seen at all in EX/EY ..).

More times can be found at https://ldas-jobs.ligo-wa.caltech.edu/~tdent/wdq/isi_ham2/jan23/ https://ldas-jobs.ligo-wa.caltech.edu/~tdent/wdq/isi_ham2/jan24/

Hveto investigations have uncovered a bunch more times - some are definitely not in working hours, eg  https://ldas-jobs.ligo-wa.caltech.edu/~tjmassin/hveto/O2Ac-HPI-HAM2/scans/1169549195.98/ (02:46 local)   https://ldas-jobs.ligo-wa.caltech.edu/~tjmassin/hveto/O2Ab-HPI-HAM2/scans/1168330222.84/  (00:10 local)
 

Here's a plot which may be helpful as to the times of disturbances in CS showing the great majority of occurrences on the 23rd, 26th-27th and early on 28th Jan (all times UTC).  This ought to be correlated with local happenings.

The ISI-GND HAM2 channel also has loud triggers at times where there are no strain triggers as the ifo was not observing.  The main times I see are approximately (UTC time)

Jan 22 : hours 13, 18 21-22

Jan 23 : hours 0-1, 20

Jan 24 : hours 0, 1, 3-6, 10, 18-23

Jan 25 : hours 21-22

Jan 26 : hours 17-19, 21-22

Jan 27 : hours 1-3, 5-6, 10, 15-17, 19, 21, 23

Jan 28 : hours 9-10

Jan 29 : hours 19-20

Jan 30 : hours 17, 19-20 

Hmm.  Maybe this shows a predominance of times around hour 19-20-21 UTC i.e. 11-12-13 PST.  Lunchtime??  And what was special about the 24th and 27th ..

Is this maybe snow falling off the buildings? The temps started going above the teens on the 18th or so and started staying near freezing by the 24th. Fil reported seeing a chunk he thought could be ~200 lbs fall.

Ice Cracking On Roofs?

In addition to ice/snow falls mentioned by Jim, thought I'd mention audible bumps I heard from the Control Room during some snowy evenings a few weeks ago (alog33199)....Beverly Berger emailed me suggesting this could be ice cracking on the roof.  We currently do not have tons of snow on the roofs, but there are some drifts which might be on the order of a 1' tall.

MSR Door Slams?

After hearing the audio files from Thomas' alog, I was sensitive to the noise this morning.  Because of this, thought I'd note some times this morning when I heard a noise similar to Thomas' audio, and this noise was the door slamming when people were entering the MSR (Mass Storage Room adjacent to the Control Room & there were a pile of boxes which the door would hit when opened...I have since slid them out of the way).  Realize this isn't as big of a force as what Robert mentions or the snow falls, but just thought I'd note some times when they were in/out of the room this morning:

• 19:00:55, 19:05:22, 19:10:16, 19:43:40-19:44:00 Mass Storage Room door slam (not seen on DARM spectra).

I took a brief look at the times in Corey's previous 'bumps in the night' report, I think I managed to deduce correctly that it refers to UTC times on Jan 13.  Out of these I could only find glitches corresponding to the times 5:32:50 and 6:09:14.  There were also some loud triggers in the ISI-GND HAM2 channel on Jan 13, but only one corresponded in time with Corey's bumps: 1168320724 (05:31:46).

The 6:09 glitch seems to be a false alarm, a very loud blip glitch at 06:09:10 (see https://ldas-jobs.ligo-wa.caltech.edu/~tdent/wdq/H1_1168322968/) with very little visible in aux channels.  The glitch would be visible on the control room glitchgram and/or range plot but is not associated with PEM-CS_SEIS or ISI-GND HAM2 disturbances.

The 5:32:50 glitch was identified as a 'PSL glitch' some time ago - however, it also appears to be a spinosaurus!  So, a loud enough spinosaurus will also appear in the PSL. 
Evidence : Very loud in PEM-CS_SEIS_LVEA_VERTEX channels (https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=155306) and characteristic sail shape in IMC-IM4 (https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=155301).

The DetChar SEI/Ground BLRMS Y summary page tab has a good witness channel, see the 'HAM2' trace in this plot for the 13th - ie if you want to know 'was it a spinosaurus' check for a spike in HAM2. 

Here is another weird-audio-band-disturbance-in-CS event (or series of events!) from Jan 24th ~17:00 UTC :
https://ldas-jobs.ligo-wa.caltech.edu/~tdent/detchar/o2/PEM-CS_ACC_LVEAFLOOR_HAM1_Z-1169312457.wav

Could be someone walking up to a piece of the instrument, dropping or shifting some heavy object then going away .. ??

Omega scan: https://ldas-jobs.ligo-wa.caltech.edu/~tdent/wdq/psl_iss/1169312457.3/

The time mentioned in the last entry turns out to have been a scheduled Tuesday maintenance where people were indeed in the LVEA doing work (and the ifo was not observing, though locked).

For reference:

I have learned that the H1:PEM-CS_ACC_HAM6 accelerometers while being not in vacuum but just beside the HAM6 chamber, they are sensitive to the shutter closing down in order to protect the output PDs after lockloss. Therefore the spikes in this channels will always be present during lockloss (see attached picture where I plot this channel together with the power built up on PR cavity so a good indicator of lockloss).

I also looked at the possibility that the overflow of H1:ASC-AS_C_SEG4 and H1:ASC-AS_C_SEG2 may actually have caused the lockloss (more to come from this analysis worthy of an aLog in itself), but notice that these signals are used, together with SEG1 and SEG3, to generate the aligment signals for SR2 and SRM mirrors.

Finally, in order to look at the possible effect that the A2L script caused on the excess low frequency noise I will run BruCo before and after the running the A2L script, which is run ocassionaly to recenter the beam into the mirrors. Sheila has mentioned that it should not affect frequencies above 25Hz though.

The Bruco reports before and after the A2L script was run. In each case I look at 600 seconds from GPS time 1169564417 for the case 'before' and 1169569217 for the case after and look at the frequency band between 40 and 100Hz, as an example this is the command used for the 'Before' case:

./bruco.py --ifo H1 --channel=OMC-DCPD_SUM_OUT_DQ --gpsb=1169564417 --length=600 --outfs=4096 --naver=100 --dir=~/public_html/detchar/O2/bruco/Before_at_1169564417_600_OMC_DCPD --top=100 --webtop=20 --xlim=40:100 --ylim=1e-10:1 --excluded=share/lho_excluded_channels.txt

Another look at accelerometers/seismometers.

In addition to the external (i.e. in-air) accelerometers, we also have in-vacuum seismometers (GS13s mounted within the Seismic Isolation [ISI]tables) which we can also look at.  Attached is a look at one of the HAM6 ISI GS13s (H1:ISI-HAM6_GS13INF_H1_IN1_DQ) & a HAM6 (H1:PEM-CS_ACC_HAM6_OMC_X_DQ)  in-air/external (probably mounted on the HAMdoor) accelerometer.  So, when the lockloss occurs, the HAM6 Fast Shutter (referred to as "the Toaster") pops up & this shakes the HAM6 table (somes trips the HAM6 ISI).  This motion is seen by the GS13 inside and accelerometer outside (which was a surprise to me).