TITLE: 12/14 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC STATE of H1: Observing at 69.2873Mpc OUTGOING OPERATOR: Jeff CURRENT ENVIRONMENT: Wind: 6mph Gusts, 4mph 5min avg Primary useism: 0.02 μm/s Secondary useism: 0.19 μm/s QUICK SUMMARY: Starting shift in observing with IFO at NLN.
Ops Shift Log:12/13/2016, Day Shift 16:00 – 00:00 (08:00 - 16:00) Time - UTC (PT)
Activity Log: Time - UTC (PT)
I inspected the laser eyewear protection in the area outside the OSB Optics Lab. I pulled 5 pairs for a closer inspection, all others looked okay. Of the 5 pairs pulled, only one pair exhibits damage beyond what I'd term cosmetic damage (see attached images). This pair I am removing from service. I checked with the vendor and what I term a cosmetic scratch is not enough damage to void the protection offered. The same is not true for polycarbonate eyewear but we do not have any of those for work in the LVEA.
Damped PI Mod-27 at 22:36 (14:36) and 22:47 (14:47) - DID NOT drop out of Observing when damping.
Attached are the long trends with today's ETM chargement data appended. Will confer with J. Kissel regarding the turn over in trend.
Because the file system for trend files has become corrupt again, I have stopped the daqd process on h1tw0 since it can't write it's files. This will remain down until we can repair the file system.
(see WP #6386) Used the North Crane to "fly" a leak detector over the XBM. Leak tested the CF joints on the newly assembled annulus ion pump piping associated with the 2K IMC A spool - B spool flange. Upon attempting to start the ion pump, I discovered that the orientation of the HV connector was 180 degrees off from what was needed. This resulted in an interference by the aluminum mounting plate such that the HV cable couldn't be connected. Doh! Had to vent the ion pump and reinstall it in the correct orientation. Afterwards, I dumped the vented ion pump volume into the rest of the annulus volume and pumped it back down but ran out of time (at noon, anyone still in the LVEA is turned into stone!) before being able to get the ion pump to run unassisted. So this task will be left to finish up on the next maintenance day.
Completed maintenance window. Realigned and relocked the IFO at NLN. Set intent bit to Observing at 22:03 (14:03).
The H2 Enclosure phone was confirmed as off. Does not even turn on. Coincidentally I believe it was turned off when we moved from H2 to H1.
Continued with installation of the Beckhoff based safety system at EY. Network and power cables from the control unit to EP1908 terminal (ISCTEY) were terminated. As of now, we are in a monitor state only. Same configuration was done in the corner station early in November.
Whilst the interferometer was down, I took the opportunity to touch up the alignment into the reference cavity. Again mostly vertical adjustments on the periscope in front of the vacuum chamber.
Jeff B, Jim B, Carlos, Fil The EPICS IOC for the LVEA and PSL dust monitors was restarted this morning because it stopped working. Upon a lengthy investigation it was determined that someone unplugged and removed the power cord for the Comtrol unit which was under the beam tube behind the vacuum rack on the X arm. The cord appears to have been removed this morning based on trends of dust channels.
The network switch port reports that the Comtrol unit went down at 9:58 PST.
HWSX Code has been resumed. HWSY is not running.
I also want to add an interesting finding while doing this test. So I unplugged one fiber at a time to see which camera stopped streaming. Instead of a complete black out both camera seemed to be streaming off the other while being unplugged. This is exactly what happened as reported in alog32019.
I've restarted the primary and redundant h(t) pipelines at Hanford at GPS second 1165664237. This restart picks up a software update to gstlal-calibration-1.1.2-v1, which contains a couple of bug fixes:
(1) The h(t) data product should now be identical between the primary and redundant pipelines when time-dependent corrections are applied (i.e., the previously observed numerical roundoff error has been addressed);
(2) The pipeline will now run properly in offline mode, which had become a serious problem after the first bug fix referred to above.
The filters file has also been changed; see this aLOG: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=32389
Note, time-dependent corrections to h(t) (the kappa factors) are still being applied in low latency at Hanford, with the exception of changes to the coupled cavity pole frequency.
The most recent bug fix that prevents the pipeline from hanging does so by adding a 20 second latency to the low-pass-filter used in demodulating the time-dependent calibration factors. This was necessary so the the timestamps on this stream are aligned with the timestamps of the coherence uncertainty channels. Otherwise, the gate element that receives these misaligned streams will lock up, causing the pipeline to hang. However, I have now realized that this introduces minor subtlety -- the coherence gating does not take this additional 20 seconds into account. This introduces the possibility that there could be undesirable jumps in the kappas in C01 frames produced by this version of the code that do not show up in C00 (Similar to what is discussed in https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=31911 , which has been addressed in a recent bug fix). I have a different fix to this, for which I will request another release for next Tuesday. Instead of adding a latency to the low-pass filter, I have added queues before the gate, which allow it to run properly. (Note that we never encountered this problem online; the reason is because there are queues that have this effect when running online that were not previously in all the right places for the offline version.)
The PSL Enclosure phone has been switched off. A reminder that it needs to be re-enabled when work is being done in the Laser Area Enclosure. The power supply for it is in Rm 163.
Please note: The DetChar-noticed ringtone (see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=32503) forwarded to a second nearby phone, with an audible speakerphone auto response, that must also be disabled. More generally, the VEAs must be re-checked for active telephones.
The "2nd phone" heard in the audio recording from DetChar is the phone in the PSL enclosure anteroom. It is important to note that there is no call forwarding involved, both phones always ring if you call the PSL enclosure extension. The anteroom phone always lags behind the laser room phone though, which is why it appears to be forwarding the call. It sounds so faint because of the acoustic damping of the laser room (these phones are quite loud). Switching off the PSL enclosure phone turns off both the laser room and anteroom phones. This was tested this morning, as we can hear the anteroom phone from outside of the PSL enclosure; when the enclosure extension was called, no ringing was heard. To my knowledge neither of the PSL enclosure phones have the capability to auto-answer, they will continue to ring until physically answered or the caller hangs up.
Tagging a few more subsystems so they're made digitally aware of this fix. This also closes FRS Ticket 6923.
Andy, Tom Dent, Josh Nutsinee (alog 32159) and Patrick (alog 32163) both reported a string of glitches during their shifts on Dec 4th. These showed up again in connection with an investigation into some PyCBC loudest events. At Tom's suggestion, we listened to the PSL microphone, which is one of many PEM channels seeing a huge disturbance at these times. I've attached the wav file of the PSL center mic. It seems like the PSL intercom (or some other phone) rings twice and afterwards forwards the call somewhere else. You can hear a faint third ring, and Josh has another file where talking can be heard after the faint ring. When this happened at LLO during an ER, (L1 alog 8451 the intercom would answer and then make a busy signal. It's not clear why we haven't seen this more often, or when the intercom could have gotten connected. This is in the middle of a long stretch of science mode. Edit: To be clear, the loud rings make something very strong in DARM. We're investigating to what extent the quieter stuff shows up.
Here's a spectrogram of the PSL Mic annotated. The second set of more distant rings followed by auto answer starts around second 19. Also attached is the STRAIN spectrogram. The coupling to STRAIN is clear for the PSL rings and it looks like there is faintly also the second set of rings seen. Also attached is the longer Wav file with the speakerphone and talking for the first call and the rings of the second call a minute later.
Opend FRS Ticket 6923
Both PSL Phones have been disabled, see LHO aLOG 32508. FRS Ticket 6923 closed. Tagging other relevant subsystems to envoke L-Mail.