From /opt/rtcds/userapps/release/sys/h1/scripts/VerbalAlarms/Lock_Summary.py. End of Day Summary Current Status: Observing, Locked since Dec 3 14:49:59 UTC (1164811816.0) Total Day Locked: 33hr 10min 4sec [138.2%] (119404/86400) Total Day Science: error [error%] (error/86400)
TITLE: 12/04 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC STATE of H1: Observing at 73.3703Mpc INCOMING OPERATOR: Ed SHIFT SUMMARY: IFO locked and in observing during entire shift. Lock stretch now past 33 hours. CP4 pump level is still glitching in and out of alarm. Evan G. is here as fellow. LOG: 16:22 UTC Kyle left. He was working in the vacuum prep lab at the end of the hallway when I arrived. 17:44 UTC Multiple EY saturations. A comb appeared in DARM from 500 Hz to 1 kHz. Lasted ~ 1 minute. Looks similar to what Nutsinee posted in her owl shift summary. 23:10 UTC TCSY chiller flow is low verbal alarm. Trend showed it recovered. 23:44 UTC Single EY saturation. Everything in DARM below ~300 Hz jumped. 00:00 UTC Thought I briefly saw red squares on the CDS overview under end Y, but they disappeared.
Verbal alarms announced 'TCSY chiller flow is low' at 23:10:09 UTC. Plot attached (1 hour of full data). Seems to have recovered.
Lock is now over 29 hours. Evan G. is here. ~17:44 UTC Saw a reappearance of the glitches that Nutsinee reported in her Owl Summary alog (EY saturations and a comb in DARM from 500 Hz to 1 kHz). Also lasted about a minute.
The problem is reported in alog: 32148
The LHO DCS cluster seems fine, but the load on the DCS DetChar box is very high. I've sent an email to the DetChar group, Duncan Macleod, and LDAS about this, seeing if Duncan can get the summary page jobs working again, or if the box needs to be rebooted.
The detchar box has been rebooted, the detchar jobs are running again, and detchar has been informed. It may take a while for the summary pages to catch up.
TITLE: 12/04 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 75.9413Mpc
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
Wind: 6mph Gusts, 3mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.58 μm/s
QUICK SUMMARY: Locked for over 25 hours. Kyle is working in the vacuum prep lab down the hall.
Kyle just left.
TITLE: 12/04 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 72.7098Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Been locked since last night. 25 hours and counting! LLO is still down.
LOG:
10:46 Got a TCSY low flow alarm. Time series showed a quick drop of flow rate and then back to normal. I went to check the chiller on the mezzanine anyway. The flow rate read 4 pgm (normal).
14:12 Suddenly there were nasty group of lines appeared in DARM between 500-1000 Hz. Verbal alarm complained EY saturation multiple times (well, that doesn't tell us much). Lasted less than a minute (14:11:47-14:12:29).
TITLE: 12/04 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 67.6212Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT: Wind: 36mph Gusts, 31mph 5min avg Primary useism: 0.15 μm/s Secondary useism: 0.62 μm/s
QUICK SUMMARY: Still locked since last night. 17 hours and counting. I went through Ops sticky note and was thinking about running a2l since I didn't run it the first two hours into the lock last night. The coherence plot (from Keita's template) looks fine. Then I saw Ed's comment about the plan changed. So no a2l.
H1 Locked for 13hrs45min. Current range is 71.9Mpc. Winds are less than 20mph. useism is riding at about .6um/s. Keita and Mike have called to check in. Any activity that has occurred has previously been logged.
Lock# 10 Times are in: GPS (UTC) Start time: 1164768731.0 (Dec 3 02:51:54 UTC) Itbit Engage: [0, 1164769704.0, 1164770146.0] Itbit Disengage: [0, 1164770020.0, 1164793996.0] End time: 1164794028.0 (Dec 3 09:53:28 UTC) Total length: 25297.0 , 7hr 1min 37sec Total Science: 24166.0 , 6hr 42min 46sec End of Day Summary Current Status: Observing, Locked since Dec 3 14:49:59 UTC (1164811816.0) Total Day Locked: 16hr 11min 41sec [67.5%] (58301/86400) Total Day Science: 15hr 50min 32sec [66.0%] (57032/86400)
Posted from TJ's script: /opt/rtcds/userapps/release/sys/h1/scripts/VerbalAlarms/Lock_Summary.py
02:26UTC Verbal Alarm said Intention bit set to Commissioning. Checking the SDF i found that the EX ECAT PLC2 was blinking. Below is a screen cap of the channel that was blinking. DIAG_MAIN was also showing a message in the log seen in the other screenshot. The channel listed in the SDF and the Message in the DIAG_MAIN correlate in terms of time of occurence and subsystem (ALS). Some troublesome ALS channels were unmonitored last night by Keita. Looking back to his aLog, this appears to be a straggler. I was able to Un-Monitor this channel and reset the Intention bit.
02:27UTC Intention bit set to Observe
Good Catch, Ed.
I am going to paste the channel name in here so it can come up during a search (because we had the same issue with the Y-arm a week later). The channel in question here (which is in Ed's snapshot) is:
H1:ALS-X_FIBR_LOCK_TEMPERATURECONTROLS_ON
TITLE: 12/04 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC STATE of H1: Observing at 73.8494Mpc INCOMING OPERATOR: Ed SHIFT SUMMARY: Remained in observing for duration of shift. LLO is down to work on their laser. A couple of EY saturations but otherwise quiet. LOG: 17:38 - 17:43 UTC Evan G. into MSR to look for USB cable for external drive 17:46 UTC EY saturation 18:47 UTC EY saturation 21:03 UTC LLO starting diode replacement 22:52 UTC Restarted video2 (MEDM screens not updating) 23:11 - 23:28 UTC Rick leading small tour through control room
We have remained in observing. LLO has lost lock and is starting their laser diode replacement. They will likely be down for the rest of the day. Rick will be leading a tour on site.
Looks like both the live 12hr page & the running 24hr UTC Day pages both froze around 5:00utc. I've sent an email to the Detchar group.
In the meantime, using the LHO Control Room Screenshots (https://lhocds.ligo-wa.caltech.edu/cr_screens/).
Now that we have a few days of O2 under H1's belt. I wanted to give a shout out to the absolute latest Operator Sticky Notes we have so far. And a reminder is that these Sticky Notes are a wiki page here: https://lhocds.ligo-wa.caltech.edu/wiki/OperatorStickyNotes. Some of the older ones have been moved to an "old sticky notes" section at the bottom of the page. Anyone should feel free to update this list to make it pertinent and useful for operations.
Operators please note these latest Sticky Notes:
12/2-12/9: High freq Calib Line Roaming changes (WP#6368)
12/1: TCS Power Stabilization Guardian knocking out of Observing (Kiwamu alog#32090)
12/1: Surviving rung up resonant modes via "RemoveStageWhitening" (Jenne alog#32089)
12/1: ISI_CONFIG during earthquakes (Sheila's comment for alog#32086)
12/1: Resonant Mode Scratch Pad! (Jeff's alog #32077)
11/30: LVEA SWEPT on 11/29&11/30 by Betsy, Dave, & Fil (alogs 31975, 32024).
11/30: Run A2L ( once a day until ~12/8. Always run A2L on Tues Maintenance Day. (Jenne alog#32022))
Curious...After reading the aLog about Operator Sticky notes regarding running a2l once a day until 12/8 and running Kiwamu's DTT coherence templates to determine if the script would even be necessaryat this point and asking Patrick if he had run a2l script today (he had not), Keita called 03:15UTC to check on the IFO status. I told him that I had run the DTT measurement and didn't see any loss of coherence in the 20Hz area and asked If I still needed to follow the once-a-day prescription that was aformentioned. He said to me that that plan had changed. If I understood him correctly, only during the first two hours of the lock would it be necessary to run the script if the coherence showed to be out. If this is the case (i'm still not 100% certain) then the Sticky Note needs to be updated and the new plan needs to be disseminated amongst the operators?
