Lock is now over 29 hours. Evan G. is here. ~17:44 UTC Saw a reappearance of the glitches that Nutsinee reported in her Owl Summary alog (EY saturations and a comb in DARM from 500 Hz to 1 kHz). Also lasted about a minute.
The problem is reported in alog: 32148
The LHO DCS cluster seems fine, but the load on the DCS DetChar box is very high. I've sent an email to the DetChar group, Duncan Macleod, and LDAS about this, seeing if Duncan can get the summary page jobs working again, or if the box needs to be rebooted.
TITLE: 12/04 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 75.9413Mpc
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
Wind: 6mph Gusts, 3mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.58 μm/s
QUICK SUMMARY: Locked for over 25 hours. Kyle is working in the vacuum prep lab down the hall.
Kyle just left.
TITLE: 12/04 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 72.7098Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Been locked since last night. 25 hours and counting! LLO is still down.
LOG:
10:46 Got a TCSY low flow alarm. Time series showed a quick drop of flow rate and then back to normal. I went to check the chiller on the mezzanine anyway. The flow rate read 4 pgm (normal).
14:12 Suddenly there were nasty group of lines appeared in DARM between 500-1000 Hz. Verbal alarm complained EY saturation multiple times (well, that doesn't tell us much). Lasted less than a minute (14:11:47-14:12:29).
TITLE: 12/04 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 67.6212Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT: Wind: 36mph Gusts, 31mph 5min avg Primary useism: 0.15 μm/s Secondary useism: 0.62 μm/s
QUICK SUMMARY: Still locked since last night. 17 hours and counting. I went through Ops sticky note and was thinking about running a2l since I didn't run it the first two hours into the lock last night. The coherence plot (from Keita's template) looks fine. Then I saw Ed's comment about the plan changed. So no a2l.
H1 Locked for 13hrs45min. Current range is 71.9Mpc. Winds are less than 20mph. useism is riding at about .6um/s. Keita and Mike have called to check in. Any activity that has occurred has previously been logged.
Lock# 10 Times are in: GPS (UTC) Start time: 1164768731.0 (Dec 3 02:51:54 UTC) Itbit Engage: [0, 1164769704.0, 1164770146.0] Itbit Disengage: [0, 1164770020.0, 1164793996.0] End time: 1164794028.0 (Dec 3 09:53:28 UTC) Total length: 25297.0 , 7hr 1min 37sec Total Science: 24166.0 , 6hr 42min 46sec End of Day Summary Current Status: Observing, Locked since Dec 3 14:49:59 UTC (1164811816.0) Total Day Locked: 16hr 11min 41sec [67.5%] (58301/86400) Total Day Science: 15hr 50min 32sec [66.0%] (57032/86400)
Posted from TJ's script: /opt/rtcds/userapps/release/sys/h1/scripts/VerbalAlarms/Lock_Summary.py
02:26UTC Verbal Alarm said Intention bit set to Commissioning. Checking the SDF i found that the EX ECAT PLC2 was blinking. Below is a screen cap of the channel that was blinking. DIAG_MAIN was also showing a message in the log seen in the other screenshot. The channel listed in the SDF and the Message in the DIAG_MAIN correlate in terms of time of occurence and subsystem (ALS). Some troublesome ALS channels were unmonitored last night by Keita. Looking back to his aLog, this appears to be a straggler. I was able to Un-Monitor this channel and reset the Intention bit.
02:27UTC Intention bit set to Observe
Good Catch, Ed.
I am going to paste the channel name in here so it can come up during a search (because we had the same issue with the Y-arm a week later). The channel in question here (which is in Ed's snapshot) is:
H1:ALS-X_FIBR_LOCK_TEMPERATURECONTROLS_ON
TITLE: 12/04 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC STATE of H1: Observing at 73.8494Mpc INCOMING OPERATOR: Ed SHIFT SUMMARY: Remained in observing for duration of shift. LLO is down to work on their laser. A couple of EY saturations but otherwise quiet. LOG: 17:38 - 17:43 UTC Evan G. into MSR to look for USB cable for external drive 17:46 UTC EY saturation 18:47 UTC EY saturation 21:03 UTC LLO starting diode replacement 22:52 UTC Restarted video2 (MEDM screens not updating) 23:11 - 23:28 UTC Rick leading small tour through control room
We have remained in observing. LLO has lost lock and is starting their laser diode replacement. They will likely be down for the rest of the day. Rick will be leading a tour on site.
Looks like both the live 12hr page & the running 24hr UTC Day pages both froze around 5:00utc. I've sent an email to the Detchar group.
In the meantime, using the LHO Control Room Screenshots (https://lhocds.ligo-wa.caltech.edu/cr_screens/).
I have attached trends over the past 12 hours of the ALS X fiber channels that were taking us out of observing mode yesterday. I believe the drop from observing mode in the plot is from a lock loss.
TITLE: 12/03 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 66.5159Mpc
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
Wind: 6mph Gusts, 5mph 5min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.85 μm/s
QUICK SUMMARY: CP4 LN2 pump level is still noisy.
TITLE: 12/03 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 69.8653Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Lockloss as I tried to switch the ISI config to prepare for a "big" earthquake as reported by Terramon (6.0M in Alaska). Turned out Terramon predicted R-wave amplitude wrong by a factor of 7 and this ISI config change didn't have to be done. Most of the struggles were due to sudden drop of AS90/POP90 after DRMI ASC was engaged (it wasn't obvious to me at the time since the drop happened further down the path, mostly at DRMI_LOCKED). As I started an initial alignment I went through Jenne's troublehoot document that reminds me I should have looked at the error signals. After the initial alignment was done I went back to locking DRMI and zeroed the SRC1 and SRC2 error signals by moving SRM and SR2. I had to walk both SRM and SR2 to zero the error signal. The rest of the locking sequence went fine. 500-510Hz violin modes came back high and saturated the OMC DC PD so I sat at DC_READOUT and wait until the modes are somewhat damped. All the ISI ST2 configs are still BLEND_250_SC_A as Ed had them.
Operators should follow Sheila's directions for earthquakes, which are to not change the SEI_CONF state until after lockloss. Then, only change to the LARGE_EQ state if the earthquake is in the several micron RMS range.Terramon gives peak velocity, which explains some of the difference between Terramon and the wall FOM. We don't have good numbers on what the current WINDY configuration can ride out and we don't have a good configuration for the current high microseism.
Now that we have a few days of O2 under H1's belt. I wanted to give a shout out to the absolute latest Operator Sticky Notes we have so far. And a reminder is that these Sticky Notes are a wiki page here: https://lhocds.ligo-wa.caltech.edu/wiki/OperatorStickyNotes. Some of the older ones have been moved to an "old sticky notes" section at the bottom of the page. Anyone should feel free to update this list to make it pertinent and useful for operations.
Operators please note these latest Sticky Notes:
12/2-12/9: High freq Calib Line Roaming changes (WP#6368)
12/1: TCS Power Stabilization Guardian knocking out of Observing (Kiwamu alog#32090)
12/1: Surviving rung up resonant modes via "RemoveStageWhitening" (Jenne alog#32089)
12/1: ISI_CONFIG during earthquakes (Sheila's comment for alog#32086)
12/1: Resonant Mode Scratch Pad! (Jeff's alog #32077)
11/30: LVEA SWEPT on 11/29&11/30 by Betsy, Dave, & Fil (alogs 31975, 32024).
11/30: Run A2L ( once a day until ~12/8. Always run A2L on Tues Maintenance Day. (Jenne alog#32022))
Curious...After reading the aLog about Operator Sticky notes regarding running a2l once a day until 12/8 and running Kiwamu's DTT coherence templates to determine if the script would even be necessaryat this point and asking Patrick if he had run a2l script today (he had not), Keita called 03:15UTC to check on the IFO status. I told him that I had run the DTT measurement and didn't see any loss of coherence in the 20Hz area and asked If I still needed to follow the once-a-day prescription that was aformentioned. He said to me that that plan had changed. If I understood him correctly, only during the first two hours of the lock would it be necessary to run the script if the coherence showed to be out. If this is the case (i'm still not 100% certain) then the Sticky Note needs to be updated and the new plan needs to be disseminated amongst the operators?
The detchar box has been rebooted, the detchar jobs are running again, and detchar has been informed. It may take a while for the summary pages to catch up.