TITLE: 11/30 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 7mph Gusts, 6mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY: locked in Observe, winds less than 10mph
12:11 Attempt #1
12:23 Attempt #2
12:57 NLN
12:59 H1 Observing
FSS began to Oscillate during PRC Aligning
Cycling the Autolocker on the FSS OFF and then ON again locked the cavity but the PZT is going wild.
Re-engaged FSS Auto-Locker and ran ISC_LOCK down script.
11:56 Re-Starting Initial Alignment
Between 4:10UTC and 10:50UTC I dont have any records on that computer of my locking attempts.
My current locking attempts have been erased due to aLog ditching all my data while trying to re-lock and get something to eat while I now have no history to turn to on Verbal.
Consistently failing at CARM_OFFSET_REDUCTION.
Doing IA again.
08:30 Attempt #1
08:50 Lockloss @ Acquire DRMI
08:51 attempt #2
09:05 Attempt #3
09:19 Attempt #4
09:37 initial alignment
09:54 IA COMPLETE
Initial Alignment ran automatically
Ed, Dave:
I restarted CAM05 (ALS-Y) and CAM16 (AS-AIR) because they were not giving images. This is presumably a result of the CER POE switch going offline at 21:10 this evening. Because ALS_Y centroid is being sent to the h1alsy front end model, starting the camera image caused a lock loss.
TITLE: 11/30 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 114Mpc
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
SEI_CONF state: EARTH_QUAKE
Wind: 8mph Gusts, 5mph 5min avg
Primary useism: 0.31 μm/s
Secondary useism: 0.20 μm/s
QUICK SUMMARY:
Guatemalan EQ ringing down rather steeply. H1 still in EQ mode. I'll keep it that way for a bit expecting aftershocks. L1 is down.
....and lockloss. I'm blaming the EQ.
TITLE: 11/29 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC STATE of H1: Observing at 117Mpc INCOMING OPERATOR: Ed SHIFT SUMMARY: Lost lock early on due to earthquake in Canada. Had some trouble relocking. Dropped out of observing for about 7 minutes likely due to network switch error. Currently in earthquake mode due to an earthquake in Guatemala. LOG: 01:14 UTC Dick in electronics lab 01:48 UTC Lock loss, 4.1 magnitude earthquake in Canada 01:50 UTC Robert to end Y to replace illuminator on viewport 01:54 UTC Both arms need green alignment. No flashes at all for Y arm. 01:56 UTC Trying INCREASE_FLASHES state on X arm. 02:04 UTC Going to start aligning the Y arm by hand while INCREASE_FLASHES is running on the X arm. 02:07 UTC Got Y arm above .8. Had to move ETMY largely in pitch. Took to ETM_TMS_WFS_OFFLOADED. 02:24 UTC INCREASE_FLASHES has dropped the flashes to 0. Stopping it and aligning the X arm by hand. 02:32 UTC Got X arm above .8. Had to move both ETMX and TMSX. Took to ETM_TMS_WFS_OFFLOADED. 02:33 UTC Lock loss from LOCKING_ALS. 02:44 UTC Automatically went to ACQUIRE_PRMI. 02:45 UTC PRMI locked 02:47 UTC DRMI locked 03:12 UTC Lockloss from MAXIMUM_POWER 03:24 UTC Lockloss from CARM_OFFSET_REDUCTION 04:04 UTC Nominal low noise. There is a SDF difference H1:SUS-ETMX_M0_TEST_Y_OFFSET. This is likely left over from the INCREASE_FLASHES script that I ran and ended up canceling. I think I need to accept it for now, since changing it will probably break the lock. 04:08 UTC Observing 05:11 UTC Dropped out of observing by connection error on LASER_PWR guardian node. 05:18 UTC Back to observing 06:05 UTC "GRB-Long" verbal alarm E356051 Fermi Trigger ID 596786686 TRIGGER_DUR: 4.096 [sec] Alert may be ignored 07:50 UTC Changed SEI_CONF to EARTH_QUAKE for 5.8 magnitude earthquake in Guatemala.
At 05:11 UTC the IFO dropped out of observing when the LASER_PWR guardian node went into error. At the same time the centroid calculations for at least two of the digital video cameras froze (see attached) and the EDCU lost connection to ~357 PSL channels. The following is from the log of the LASER_PWR guardian. 2019-11-30_05:11:19.568166Z CA.Client.Exception............................................... 2019-11-30_05:11:19.568166Z Warning: "Virtual circuit unresponsive" 2019-11-30_05:11:19.568166Z Context: "h1pslctrl0.cds.ligo-wa.caltech.edu:5064" 2019-11-30_05:11:19.568166Z Source File: ../tcpiiu.cpp line 947 2019-11-30_05:11:19.568166Z Current Time: Fri Nov 29 2019 21:11:19.567809882 2019-11-30_05:11:19.568166Z .................................................................. 2019-11-30_05:11:19.580550Z LASER_PWR [POWER_38W.run] USERMSG 0: CONNECTION ERRORS. see SPM DIFFS for dead channels 2019-11-30_05:11:19.648744Z LASER_PWR EZCA CONNECTION ERROR. attempting to reestablish... 2019-11-30_05:11:19.649386Z LASER_PWR CERROR: State method raised an EzcaConnectionError exception. 2019-11-30_05:11:19.649386Z LASER_PWR CERROR: Current state method will be rerun until the connection error clears. 2019-11-30_05:11:19.649386Z LASER_PWR CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume. 2019-11-30_05:16:48.637175Z Unexpected problem with CA circuit to server "h1pslctrl0.cds.ligo-wa.caltech.edu:5064" was "Connection reset by peer" - disconnecting 2019-11-30_05:16:48.637662Z CA.Client.Exception............................................... 2019-11-30_05:16:48.637662Z Warning: "Virtual circuit disconnect" 2019-11-30_05:16:48.637662Z Context: "h1pslctrl0.cds.ligo-wa.caltech.edu:5064" 2019-11-30_05:16:48.637662Z Source File: ../cac.cpp line 1223 2019-11-30_05:16:48.637662Z Current Time: Fri Nov 29 2019 21:16:48.637156550 2019-11-30_05:16:48.637662Z .................................................................. 2019-11-30_05:16:53.643095Z LASER_PWR connections reestablished
Opened FRS13893
Outage from 05:10:42 UTC to 05:16:38 UTC (21:10 - 21:16 PST).
Evidence is now very strong that this was a freeze up of the Cisco network switch in the CER. This switch serves the PSL Diode Room (Beckhoff computer), the PSL LVEA enclosure Axis cameras and all the digital video cameras. All three systems exhibited a network error between 21:10 and 21:16. The digital video centroids are not directly trended by the DAQ, but some are copied to the end station and the received data shows the freeze (see plot).
Keita, Patrick, Dave:
What to do if this happens again and does not come back after 6 minutes.
The big question is if this were to happen again, with the PSL network down, H1 locked out of OBSERVE and the network not quickly coming back. Keita has agreed that we can make a guardian execption to remove the PSL Diode IOC from the OBSERVE veto and clear any associated SDFs. Patrick is looking at the code to see how that can be achieved. This will only be used it the network is down for at least 30 mins.
In the case of another network outage lasting more than 10 minutes I would like the operator to:
1) try to ping the switch (ping sw-lvea-aux) to see if it is running.
2) go into the CER and take a photograph of the switch, seeing if its error LEDs are lit. The switch is the Cisco with the WAP ethernet port. Only do this if it can be done safely (e.g. using LLO ops for budy).
I think we would need to add "LASER_PWR" to the EXCLUDE_NODES list in /opt/rtcds/userapps/release/sys/h1/guardian/IFO_NODE_LIST.py and reload the IFO node. The IFO node can be accessed from the 'GRD IFO' button at the very top left of the GUARD_OVERVIEW medm screen.
If the LASER_PWR node is in error, then ISC_LOCK will not have its OKAY channel as True since ISC_LOCK manages LASER_PWR. I don't think that then placing ISC_LOCK on the exclude list is a good idea.
04:08 UTC Observing. The attached SDF difference is from when I cancelled the running of the INCREASE_FLASHES state for the green alignment of the X arm. I accepted the SDF difference, since I believe reverting it would break the lock. It should be undone the next time we lose lock.
In the process of relocking after a 4.1 magnitude earthquake in Canada at 01:48 UTC.
TITLE: 11/29 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 11mph Gusts, 10mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY: Robert is back from end Y. No issues.
TITLE: 11/29 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:
LOG:
17:10 Back to Observing
23:00 Robert doing ISI injections at EY
23:50 Robert done, back to Observing