travis.sadecki@LIGO.ORG - posted 13:05, Monday 17 June 2019 (49999)
Out of Observing 19:39-19:41
SEI SDF transients still happening knocking us out of Observe.
SEI SDF transients still happening knocking us out of Observe.
DQ Shifter: Evan Goetz DQ Shift Mentor: Beverly Berger LSC Fellow(s): Laurence Datrier DQ shift wiki page Summary (highlights) of the shift: * LHO cluster problems crashed summary pages Friday and recovery is ongoing. This was due to the large number of files from summary pages/plots and other DetChar tools (~500 million combined). There are some missing or incomplete plots starting Thursday, but many more Friday/Saturday. Omicron and Hveto are incomplete or non-existant from Friday - Sunday. * Observing summary: (71.06 + 59.53 + 62.41+ 98.06 + 35.29 + 90.35 + 67.60) / 7 = 69.19%; more or less in line the mean observing time for H1 during O3 thus far. * Several earthquakes in the latter part of the week took away from observing time. Also, when H1 was locked or re-locked near earthquake times, the whistle glitches were much more problematic because of the increased ground motion. * Whistle glitches remain problematic for the early part of the week [Omicron problems prevented late week Hveto updates] as evidenced by round 1 or 2 channel winner LSC-REFL_A_LF_OUT_DQ. The other ongoing common round 1 or 2 winner remained LSC-POP_A_LF_OUT_DQ. * Low frequency narrow spectral artifacts in H1 remain problematic. They are quite dynamic (changing day by day) and very prevalent below 100 Hz compared with L1. * The 48 Hz feature is still present with varying amplitude, as reported by previous DQ shifts. I looked through some accelerometer spectra and I am unsure if I see correlation by eye. I will continue to look in more detail. * Several other features seem to have cropped up this weekend at the start of lock stretches, but then seem to reduce in amplutide during the course of a lock stretch (~550 Hz, ~850 Hz, possible violin harmonic near 1010 Hz) - see below. * PCALX showed some differences in the spectrum [likely not immediately problematic for h(t)], perhaps indicative of an unstable optical follower servo as resetting the offset has helped the stability. * Later in the week violin resonances seem more rung up at the start of locks than early in the week. Note that I'll update the shift summary once some of the backfilled summary pages are updated.
All look to be within the acceptable range.
BS ST2 has a pair of peaks between 1-2 Hz. Note that we are currently in the EARTHQUAKE SEI configuration with an EQ ringing down, so not sure if this effects these spectra.
This spectra is taken some number of hours before the 'run' time, eight or something hours I'd guess. This puts this auto spectra right in the middle of a nearly four hour out of full IFO period where in the SEI BS will be in ISOLATED_DAMPED mode for potentially much of the time.
Transitioning to EARTHQUAKE mode for SEI. This kicked us out of Observing due to H!ISIETMX SDF diffs (cleared themselves faster than I could get a screenshot). Also, Sheila suggested also taking ISC_LOCK to EARTHQUAKE state, which also kicked us out of Observe since Guardian wants it in NLN to be in Observing. I'll transition back soon as this was a smallish EQ, but we'll be out of Observing for a bit.
Back to Observing at 17:24 UTC. ISI_CONFIG taken back to WINDY (I managed to get a screenshot of the transient SDF diffs this time, see attachement). Also, SQZ was causing ZM2 to saturate. Sheila took SQZ_MANAGER to SQZ_READY_IFO which cleared the saturations. Upon returning to SQUEEZING, I got an SDF diff that the COMMBOOST was now 2.0 (which is the correct value according to Sheila) rather than 0. Our range has returned to ~115 MPc, so we're thinking that SQZ got stuck in some strange state during the last relock.
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
TITLE: 06/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 107Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: After relocking from the earthquake and a end X Beckhoff crash, we are a bit down on range.
LOG:
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
Out to find a new lock point on the TCSY CO2.
As we started this work, we got a notification for an earthquake from Chile (6.4mag)
0032 UTC - Switched to the SEI_CONF EARTHQUAKE state, as this was the predicted state from the Seiplot.
0041 UTC - Changed ISC_LOCK state to EARHQUAKE to increase ADS line gains.
The wind has also picked up to mid 20s, not helpful.
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
Noticed an EQ on SEISMON. Only a 4.3 but it was off southern CA coast and R-wave was within a minute of arrival when I noticed.
LOG Of Activity:
10:52 Notice EQ fastly approaching on SEISMON (no verbal given):
Another 4.3 Southern CA EQ at 14.36 (screenshot attached)
5.0 Japan EQ at 11:19 (Seismon lists this as barely on the line of EARTHQUAKE-worthy).
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
TITLE: 05/31 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 111Mpc
INCOMING OPERATOR: Niko
SHIFT SUMMARY: No issues to report after relocking. Rode through several EQs without issue.
LOG:
7:29 GRB alert but we were unlocked at the time
7:50 Started initial alignment. Relocking seemed to be going well after the lockloss, getting past DRMI a couple of times, then things started to walk off.
9:41 Back to Observing
10:23 SEI_CONF taken to EARTHQUAKE in response to 6.1 Mag EQ in Palau
11:57 5.8 Mag EQ in Mexico. We were still in EARTHQUAKE mode for SEI, so we survived. LLO did not.
12:02 GRB alert
13:01 GRB alert
13:38 GRB alert
14:00 SEI_CONF returned to WINDY
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
Range is actually closer to 110 Mpc
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
TITLE: 05/26 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
LOG:
9:35 Begin resetting Watchdogs -
9:57 BLRMS below 1um/s - AS spot is low and right
10:07 Everything un-tripped. Everything re-aligned - AS Spot still low and right
11:40 Just got off the phone with Sheila (she called in to check on the quads). I didn't push the big green RECOVER EQ button!!!!!! -_-
11:45 Begin Initial Alignment
12:10 Raine from LLO called to let me know that ALSY tripped and requires a trip to the end station - phone buddy system in effect
12:33 Raine reported back at the corner station.
12:52 begin re-locking H1
14:00 Lock failed at INCREASE POWER
14:23 Lock failed at INCREASE POWER (again)
15:00 Handing off to Cheryl
Cheryl, Sheila
Here are links for the two locklosses from increase power that Ed logged: 1242914440 and 1242915786
In both cases the DSOFT P and CSOFT P control signals became large right after we entered the INCREASE POWER (504) state. About a week ago we were having trouble suppressing the radiation pressure instability in the X arm because the beams were falling off the QPD. As a temporary solution we introduced an offset on TMSX TEST P before we increase the power, which is turned off again for lock acquisition so that we do not loose our green alignment references. We set the ramp time on the offset to 30 seconds, but this must have been reset to 2 seconds this morning either by the suspension guardian when recovering from a watchdog trip or by the dither alignment script.
We set the ramp time to 30 seconds again, and I added a line to the guardian that sets the ramp time to 30 seconds before turning on the offset in INCREASE_POWER. Cheryl loaded this and was able to get through the first few steps of the power increase.
Both the lockloss tool and the new remote access made it much easier than it might have been to diagnose and solve this issue remotely than it would have been in O2.
So we don't think this was simply a symptom of ETMY Modes 1 and 6 being rung up after a very large seismic event? I can say that I hadn't noticed them until the third attempt.
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
Good first half of the shift in Observing mode.
01:39 (18:39) Put SEI_CONFG to EARTHQUAKE for a mag 6.8 EQ in New Caledonia. Rode through without losing lock. Will be transiting back to WINDY shortly as the ground motion rings down.
The Mid-Columbia Ballet and the associated tours are concluding.
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
TITLE: 05/16 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ed
SHIFT SUMMARY:
Troubleshooting was the focus in the morning (Betsy on IMC & Sheila on ASC). Since thinking about any hardware changes since Maintenance Day, Sheila & Marc went to check baluns (no issue there). Ultimately, Sheila (& Keita later joining her) was able to get h1 back to NLN by focusing on ASC. We've been locked for all of the second part of the shift.
NOTE: See Sheila's alog about how to handle H1 if you have trouble after a lockloss.
LOG:
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
Noticed a notification of an earthquake from Canada after seeing the DARM spectrum get noisy. Quickly transitioned SEI_CONF to EARTH_QUAKE per the plot of responses to take. Managed to hold look through the transition, but lost it a couple of minutes later.
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
Here are notes for transition to EARTHQUAKE state tonight. Took an overview screenshot of various windows when I transitioned (attachement #1). Here are the notes:
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
We have noticed during the transition to the earthquake state for the seismic systems that a lot of IFO signals get worse at microseismic frequencies. I've done a little modelling of this and I think that this can be improved by changing the filter used for the common mode subtration. The earthquake mode uses the average of the seismometers for all the VEAs to as a common ground signal, this average is then low passed and subtracted from the local ground sensor to get a local differential sensor correction signal.
I've been modeling the effects of different low passes by reconstructing the local differential sensor correction signals and seeing how that differential control signal subtracts the local ground. I forgot to attach the plot where I compare my calculated performance with the original filter, but it does a very good job up to about 1 hz. The original low pass filter was just a .3hz pole, so it's pretty simple, but using this model, I've come up with an elliptical filter that I think is a little better. The first attached plot compares the calculated performance of my new filter to the actual performance of the original .3hz pole during an earthquake on March 28. During this earthquake, SEI_CONF was in EARTHQUAKE. The dotted blue line is the ITMY Y ground, the gold and dashed darker blue (purple?) lines are St1 motions seen by the ITMY and ETMY St1 T240s, the red line is the calculated performance given by my new eq mode low pass. You can see that both ITMY and ETMY are getting no isolation relative to the local ground motion up to about .3 hz. The new low pass gives some isolation starting at .1hz, so this should be better, and the motion isn't really worse below .1hz (the region we are winning by using the common ground subtraction). The thin green (?) line is the performance given by using the normal sensor correction path and filter. The suppression of the microseism is good, but the amplification of the low frequency ( < .1hz ) motion is pretty bad.
The second plot is a repeat of the first, but for an earthquake on March 30th, where SEI_CONF was left in the nominal, WINDY configuration. Comparing the red line to the dashed dark blue and gold lines shows that the new eq mode low pass should give similar microseismic performance to the nominal seismic configuration. Note also that the .030 - .070 hz motion should be better than the motion seen with the nominal configuration. The largely good agreement between the thin green line and gold and dashed blue line shows this is a relatively good model to predict st1 motion. The gap from .1-.3 hz is because I'm not including St1 RX tilt.
The third plot shows bode plot of some of the low passes I've looked at. The red line is the original .3hz plot low pass. The blue line is the new ellptical filter that I think we should try. The green line is just as .1hz pole, one of the first improvements I looked at, but I think the ellipse is better. I've installed the ellipse and turned it on, we'll see if it works any better for the next couple earthquakes.
I'm attaching a plot comparing the expected performance of the new low pass versus the expected and measured performance of the old low pass during the March 28th earthquake that we rode out. For this earthquake SEI_CONF was in the EARTHQUAKE state. The dotted blue line is the ground, the red is the calculated subtraction of the old low pass, the dashed darker blue (purple?) line is the measure St1 T240, the gold is the calculated new low pass and the thin line is the calculated subtraction of the nominal configuration.
Second plot is the coherence between the ETMY BRS/STS and ITMY STS during this earthquake.
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.
One comment about the lines below 100 Hz which are not present at LLO.
Marie made a plot comparing HAM6 seismometers at LLO and LHO (see figure 2) 46603 Are the extra lines in the HAM6 GS13s at the same frequencies as the extra lines in DARM?