09:10 UTC Kyle is driving to Y2-8 to add fuel to the generator. I'm not sure if we have been taking it out of observing for this, so I am playing it safe.
Nutsinee was explaining the violin mode damping filter banks to me. We turned off the output to one of the banks that had a gain of 0 and were dropped out of observing. We quickly turned it back on and went back to observing. According to the verbal alarms log we went out of observing at 08:24:59 UTC and back to observing at 08:25:11 UTC.
TITLE: 12/09 EVE Shift 00:00-08:00UTC (16:00-00:00 PST), all times posted in UTC
STATE Of H1: Observing
SUPPORT: Sheila, Jenne, Evan
LOCK DURATION: 3 hrs
END-OF-SHIFT SUMMARY: Two locklosses. One caused by wind the other caused by the 45mHz rung up. FIND IR was difficult for both relocking. Low wind towards the end of shift. High useism. Using 45 mHz blend everywhere.
Activity:
00:08 Cleared ETMX timing error. Didn't go out of Observing.
00:18 Kyle driving back from Y28
00:53 Evan changed ETMX UIM offset. Out of Observation from 00:53:12-00:59:17
01:00 GRB alert received
01:52 Lockloss. Wind gust reached 36 mph. See alog24094 for details.
03:15:13 GRB alert received
03:15:34 Observing
03:55 Tidal signal started to ring up. Tried turning off ALS Tidal Servo. Didn't help. Tried switching ETMX blend to 90 mHz since we were going to lose lock anyway. Lockloss as a result.
04:40 Switched EX blend back to 45 mHz after two unsuccessful relocking attempts.
05:11 Back Observing.
TITLE: 12/10 [OWL Shift]: 08:00-16:00 UTC (00:00-08:00 PDT), all times posted in UTC STATE Of H1: Observing @ ~ 78 MPc OUTGOING OPERATOR: Nutsinee QUICK SUMMARY: From the cameras: The lights are off in the LVEA. The lights are off in the PSL enclosure. The lights are off at end X. The lights are off at end Y. I can not tell if the lights are on or off at mid X and mid Y. See attached screenshot for seismic. The winds have trended down to between ~ 0 and 5 mph. The ISI blends are on 45mHz everywhere. From pinging: CDS WAP is off at the LVEA. CDS WAP is off at end X. CDS WAP is off at end Y. CDS WAP is on at mid X. CDS WAP is on at mid Y.
Wind speed below 20 mph. Useism still high (at 90th percentile). I accetped the changes Evan made to ETMX L1 LOCK Tramp and gain during the last relock. Guardian still turn the violin mode damping back on despite the fact the Evan has already added a code to turn it off for his ringdown measurement.
Evan, Sheila
We lost lock a few minutes ago (24093) due to a 36 mph wind gust at EY. We are running 45mHz blends on all BSC ISIs now, and microseism is around 0.8um/sec. The first plot shows the wind gust that we think knocked us out.
We then ran both of our saturation lockloss scripts. The second attached plot shows that a saturation of ETMY L1 by a 50mHz signal was the proximal cause of lockloss, which must be from the length drive since this is our only dirve signal sent to ETMY.
Since we can't offload to HEPI above the blend frequency, it seems that we would need length feedback offloaded to M0 to avoid this kind of lockloss in the future.
Many ASC signals also see a large excursion around the time of the lockloss.
Following up yesterday's alog 24062.
We think that the ~650 Hz peak was caused by the ALS COMM VCO beating against the DIFF VCO. At times when the COMM VCO is parked at the lower value (as discussed in the earlier alog) the two VCOs are < 700 Hz apart and this beat shows up in darm. To get rid of this peak, we added a line in guardian so that H1:ALS-C_COMM_VCO_TUNEOFS is set to 2 in full lock. This way, the COMM VCO gets parked at a high value far from the DIFF VCO. See figure 1 if you want to see the actual values of the two VCOs; the top plot is from yesterday (peak present; the two VCOs are within 700 Hz of each other) and the bottom plot is from today (the two VCOs are >10 kHz apart, peak is not present).
Also attached (figure 2) is a darm spectrum comparing yesterday, when this noise was present, to the current lock. It is clear that the peak between 600 Hz and 700 Hz (and the higher harmonic) is now gone. No other peaks have been observed to appear due to this change. We think this has solved the issue, but we should keep an eye out to make sure it does not return. Also, it will be good to keep beating VCOs in mind if similar wandering peaks show up again.
Jordan, Robert, Sheila
Possibly due to a wind gust that reached almost 40 mph.
When we transition DARM control from EX to EY, we ramp off the DARM input into the EX test/PUM/UIM filter banks.
Since the UIM filter bank has an integrator, this leaves a static offset on the output of this filter bank which is cancelled by the offloading of IMC-F to the UIMs.
However, depending on the size of the offset, this can use up a significant amount of the control range of the offloading (which is set by a digital limiter). Today, the EX offloading was hitting this limiter, causing IMC-F offloading to stop and start (see attachment).
Since the offloading signals were trending even further toward saturation (i.e., greater likelihood of IMC-F saturating and causing lockloss), I dropped the interferometer out of observing and zeroed this offset using two one-minute ramps.
We should modify the ISC_LOCK guardian to do this automatically during lock acquisition.
The offloading completely railed at the end of the most recent lock acquisiton (ca. 03:00:00 Z). I have edited the guardian so that this EX UIM offset is ramped down in the CLOSE_BEAM_DIVERTERS state.
This will result in two SDF diffs on EX.
The alert was received at 01:00:08 UTC. GraceDB said the event actually happened at 00:59:16.64 UTC. We were out of Observing mode for a few minutes for a non-invasive change to ETMX UIM offset from 00:53:12 - 00:59:17 UTC. We think it's reasonable to not veto the alert.
Another GRB alert received at 03:15 UTC (LLO received it first and contacted us). We were at NLN clearing SDF at the time. The data should be good for analysis even though it wasn't in Observing intent (we hit "Undisturbed" button right after).
Attached are 45 day trends of the pressure out from the pumps and the control drive to the motor. After seeing the dips very evident in the End control drives, I replaced the CS pump output for 1 2 & 3, which are as flat as PS4, with the outside air temp. Very clear, the change in temp drives the pumps' need for speed.
Still much noisier than the others but the EX pressure cleaned itself up for some unknown reason a week ago...
And like clock work, the EndY daily and weekly glitches are still present. Anybody?
Jenne, Sheila, EvanH, Hugh
We had a fortuitous rapid drop in the wind speeds around 1400pdt and while not attempting to lock, it was a quick study looking at the GS13 signal while changing the blends to see that the 45 mHz were much quieter. Of course there still is the worry about the ETMX X 45 blend ringing up MCF but we decided that if it does,maybe we can study it or try something.
The blend switch, and why we though it was a good idea given the reduced winds ad the very elevated microseism is clearly seen on the attached plot.
It still took some time to lock, maybe still a little much wind or something else. Still we decided to not switch the ETMX X blend back fearing that it woould likely break the lock. So for over an hour we've only had minor flare ups of MCF but it really hasn't grown and faded after just a couple cycles.
TITLE: Dec 9 DAY Shift 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC
STATE Of H1: Observing
SUPPORT: Sheila, Hugh, TJ, Jenne, Robert and Jordan
LOCK DURATION: 36min
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY:
A combination of high microseism and high winds, for most of the day, made it less than possible for locking.
Twice today the call for doing maintenance was made.
Hugh and Sheila were able to determine the best blends to use at the End Stations for locking today.
Sheila, Jordan and Robert tweaked the COMM VCO to get rid of some unwanted lines in DARM
DRMI Lock was finally achieved at 23:02UTC
Intention Bit set to Undisturbed at 23:50UTC
There is a timing error showing in the ETMX CDS overview that got forgotten. It can be reset in the case of lockloss.
ACTIVITY LOG:
16:15 Fil out to LVEA to do some more cable pulling
16:25 Jeff Bartlett and Mitch out to LVEA to do some more work on dust monitor plumbing
16:30 Peter into LDR
16:40 Peter out of LDR
17:01 Jeff and Mitch out of LVEA
17:03 Jeff and Mitch to EX
17:21 Richard called to let me and Mike know that he has started his work in the CER.
17:40 Gerardo out to Y-2-8 to refuel the generator
17:54 Richard, et al, are finished in the CER and are heading down to EY.
18:19 Bubba heading to MX to work on a fan.
18:39 Gerardo back from Y-2-8
21:22 Kyle to Y-2-8 to take some temp readings
21:55 Kyle back to corner station
23:21 H1 is locked at Nominal Low Noise!
23:50 Intention Bit set to UNDISTURBED
23:50UTC
TJ & Hugh
With IFO down from elevated winds, we attempted to have the Guardian switch the gains more like how the old SEI COMMANDS script switch the gains--WP5647.
Original svn update 24 Nov alog 23695--Change tested on HAM5 and worked.
7 Dec, ISI trips with EQ reveal problem of this switching but only for HAMs 2 & 3, aLOG 24028.
In alog 24054 I report how the GS13 gain switch when done by the guardian well after the platform was isolated and quiet would still trip the HAM3 ISI. This led us to believe it was not the timing of the guardian GS13 switching--at the end of Isolation process after/while finishing the driving to the reference CPS location; but, that it was more related to how the guardian was doing the switching. The guardian switches both FM4 & 5 at the same time whereas the COMMAND script switches all the FM4 and then all the FM5s with a 1 sec sleep between.--See SEI Log 894 comment for code comparison.
Today,TJ modified the guardian code, see comment below, to try simulating the way the COMMAND script switches the Gain. Bottom line it still tripped.
We even tried reversing the order of the FM4 & 5, that is first FM5 then FM4 to turn off the analog whitening before we increased the analog gain, still tripped... So we still scratch our heads, it's winter, my skin is dry and it itches anyway.
Below is the code that we tried. I just added in a sleep for 1 second between the switching of the FM4 and FM5 banks in all dofs. When executed, it looked to have worked the same as the old perl script, but there must be some deeper magic that I need to look into.
def switch_gs13_gain(command, doflist):
"""Swtich GS13 analog gain between HIGH and LOW"""
lf_gs13 = LIGOFilterManager(['GS13INF_'+dof for dof in doflist], ezca)
if command == 'HIGH':
log('Switching GS13s to high gain')
#lf_gs13.all_do('switch', 'FM4','OFF','FM5','OFF')
lf_gs13.all_do('switch', 'FM5', 'OFF')
time.sleep(1)
lf_gs13.all_do('switch', 'FM4', 'OFF')
elif command == 'LOW':
log('Switching GS13s to low gain')
#lf_gs13.all_do('switch', 'FM4', 'ON', 'FM5', 'ON')
lf_gs13.all_do('switch', 'FM5', 'ON')
time.sleep(1)
lf_gs13.all_do('switch', 'FM4', 'ON')
# To account for the 2s zero-crossing timeout
time.sleep(3)
return
(I switched this back to what it was before and committed it back in the svn).
Note to me: Had to restart the node in order for it to take any bits of new code. I've seen this before but I'm not sure what warrents a restart vs just a reload. I'm writing this here so I can look back and find the trend.
Mike Landry has given the final "OK" to proceed with maintenance tasks this morning due to the IFO being down due to high winds. Livingston has been contacted regarding our plan. They are currently back "UP".
This period ended at ≈ 19:00UTC
Mike L. called and notified me that there was a SNEWS test alert entered into GraceDB (E206974, see attached screenshot). I did not get a verbal alarm.
There was no alarm for this in the control room because the script that queries GraceDB (ext_alert.py) will only look back 36000 seconds and the delay for this alarm was 73681.000000sec.
We have been using the default 'lookback' time, but can set to be any value we choose from the command line at the start of ext_alert.py. This delay was a special case, but it might be worth looking back a bit more than 10 hours...
Lookback time code is below if anyone is curious.
actions['run'].add_argument('-l', '--lookback-time', type=float, default=36000,
dest='lookback',
help='how far back in time to query, '
'default: %(default)s')
...(later, in the loop)
# query gracedb
now = gps_time_now(ifo=args.ifo)
start = now - args.lookback
client, events = list(
query_gracedb(start, now, ifo=args.ifo, connection=client,
far=args.far, test=args.test))