When the Operator found the ISIs in the wrong SDF state, 21609, they reported digging through the logs but not finding anything. After reporting what the filters do, they accepted the changes and went to observing. This is just the opposite of what we should do. If no supporting evidence is found for the change, reverting to the .snap values is the correct path. Of course there is some risk to the IFO lock in filter switching, not very appealing after being out for so long.
In the Evening shift operator's summary, The TM ISIs seem to be reported back in high gain:
02:05 Attempt to bring back End Station ISIs seems succesful. GS 13s back to high gain. Seismic graph is showing 3µ/S.
02:09 Ditto for ITMs
02:10 Begin bringing up Input and Output HAM SEI.
02:15 Kissel brought ESDs back and restored ETM alignment.
02:31 All SEI back to nominal isolated states. GS13 gains switched back to HIGH.
This sort of implies the ITMs were put back into High gain but that is not the case. They were switched to low gain (H1:ISI-ITMY_ST2_GS13INF_H1_SW1R: FM4 & 5 On) 0005 utc and remain there still. At this point the SDF should have been consulted for proper configuration.
The SEI system is untouch by IFO locking guardians. SEI SDFs should be green well before looking at going into Observation mode.
One may argue that this difference is not critical: these filters compensate for changes in the analog gain and whitening. The output level of the filter bank is unchanged. On the other hand in nominal seismic environment, having greater SNR when digitizing the GS13 signal gives better ISI Damping and Isolation.
Four burst signals have been injected into H1 since the new inverse actuation filter was installed Sunday evening (Timeline of CAL-CS Calibration Filter Updates), with different waveforms: GPS 1126270500 - White noise burst, frequency 125 Hz, bandwidth 150 Hz GPS 1126280500 - White noise burst, frequency 300 Hz, bandwidth 350 Hz GPS 1126300500 - Sine-gaussian, frequency 20.82 Hz, Q=7.32 GPS 1126310500 - Sine-gaussian, frequency 413.0 Hz, Q=85.1 The low-frequency, low-Q sine-gaussian offers the clearest check of the calibration amplitude and phase among these signals. (It would be better to use detchar injections, i.e. a set of sine-gaussians at a range of frequencies, as was done during ER7, but we have not performed any detchar injections during ER8 yet.) I used a matlab script (attached) to plot the intended strain waveform (the file produced by Chris P. and read by tinj to inject the signal) and the H1:GDS-CALIB_STRAIN channel from the hoft frames in the archive at Caltech. Both have been bandpass filtered bidirectionally with 'filtfilt' around the frequency of interest. I also did a crude matched filter, stepping the intended strain waveform by integer samples and considering both upright and inverted overlaps with the data. As you can see from the plot, the signal that came through in the H1:GDS-CALIB_STRAIN channel is consistent with being *inverted* relative to the intended strain signal. That was also the case in ER7, but we thought the recent calibration work and a careful look at sign conventions would have resolved that. Is there any possibility that the GDS calibration process running on Monday, producting H1:GDS-CALIB_STRAIN, had not yet been updated with the final resolution of sign convention?
TITLE: Sep 17 OWL Shift 07:00-15:00UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Observing at 77 Mpc
SUPPORT: Sheila
SHIFT SUMMARY: After 12+ hours of not locking, we are back in business. I redid the initial alignment followed the printed Initial Alignment Checklist on the desk after the seismic nosie came down to almost nominal because PRMI looked hopeless. LLO is still trying to acquire lock.
INCOMING OPERATOR: Patrick
Activity log:
8:00 Begin initial aligment. Adjusted PR3 to maximize COMM beatnote. Spent 10 minutes at INPUT_ALIGN without luck. Trended IM4 and PR2 back to the last time the ifo was locked. I brought IM4 back to where it was. Touched PR2 YAW to get 00 mode on As Air camera.
8:04 Darkhan left
9:11 Craig left
9:12 BS ISI WD tripped during MICH_DARK_LOCK twice. Requested down and waited until the optic became settled and moved on.
ENGAGE_ASC_PART3 took ~5 minutes.
Guardian stalled at DC_READOUT_TRANSITION (alog21608). OMC was not ready for handoff and was locked at the wrong mode. I called Sheila for help but the ifo lost locked shortly.
10:52 Locked at NOMINAL_LOW_NOISE
11:13 Back to Observing after I cleared the SDF diff.
13:15 Lockloss. More earthquake....
13:46 Locked at NOMINAL_LOW_NOISE
13:48 Undisturbed.
15:00 Hand off to Patrick.
The ifo is locking and observing again. I did an initial alignment as soon as the seismic noise settled. Wind speed below 10mph. Minimal seismic activity. LLO is down.
Lockloss 13:15 UTC. I'm not sure if we lost lock due to a single 5.3M earthquake in Chile or two consecutive 5.3M earthquakes that happened just 5 minutes apart from each other. Requested ISC_LOCK to DOWN and waiting for the earthquake to pass....
Observing again at 74Mpc.
After almost 12 hours of not locking, the ifo is back at 70Mpc. OMC locked at the right mode this time so I still have no idea what's wrong earlier. Terramon has just reported another 5.5M earthquake in Chile. We might lose lock again...
SDF is reporting 6 differences in ISI ITMX and ISI ITMY. Unable to go to Observe mode. I'm digging through the alog.
I couldn't find anything about the change in ITM ISI. FM4 simply adds gain of 20dB everywhere with 0 phase. FM5 labeled DWH reduces gain at high frequency with some phase change (maximum phase change of -40dB at ~20 Hz). I'm just gonna go ahead and accept the changes since the interferometer seems happy (so far). Now Observing.
The ifo hardly noticed the 5.5M earthquake. The violin fundamentals seem higher than normal but none seems to be ringing up. LLO recently joined us in locking. We are BACK in the game!
C. Cahillane
Per Jeff's request, here are the Actuation stage residual weighted means in time based on our three measurement dates, August 26, 28, and 29.
Essentially, I took the residuals already computed in , separated them by measurement date (26, 28, 29), separated them by acutation stage (L1, L2, L3), and computed each weighted uncertainty and weighted mean.
For plot values run the compare_actcoeffs_ER8_craig_extrapolation.m file and use the plotmagwm, plotmagunc, plotphasewm, and plotphaseunc matrices.
The matrix elements are as expected:  First number corresponds to data, second to actuation stage:
            |  L1  |  L2  |  L3  |
Aug 26-
Aug 28-
Aug 29-
			
			
		
		TITLE: Sep 17 Owl Shift 07:00-15:00UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Lock Acquisition
OUTGOING OPERATOR: Ed M.
QUICK SUMMARY: Attempting to lock the interferometer. I couldn't get any flashes on POP 18 and 90 while trying to lock PRMI. Since Ed did the initial alignment when the ground motion was still high I think I might try to do the initial alignment again when the ground motion become less active.
I finished the initial alignment. After having trouble with INPUT_ALIGN I looked at IM4 and PR2 alignment. IM4 alignment was way off from the last time the ifo was locked. I paused the Guardian and brought IM4 back to where it was. After that it locked in the wrong mode so I adjusted PR2 YAW. I had a little trouble with MICH_DARK_LOCK. BS ISI Watchdog tripped twice. I requested DOWN and waited until the optic settled then tried again. I also had to touch PR3 to maximize COMM beatnote. Everything went well. Now begin lock aquisition. The ground motion has come back to its nominal after series of earthquakes.
Guardian stalled at DC_READOUT_TRANSITION. OMC locked at a wrong mode. Lost lock at DC_READOUT shortly. T2, T3, LF and RT osems had huge number of counts. I called Sheila and she suggested I monitor where the offsets came from. Now relocking.
TITLE: Sep 16 EVE Shift 23:00-07:00UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Down
LOCK DURATION: N/A
SUPPORT: Hugh, Jeff K, Sheila
INCOMING OPERATOR: Nutsinee
End-of-Shift Summary: Recovering from Chilean Earthquake(s) barrage is an understatement as there have been close to 20 after-shocks (and a smattering of New Guinea and Alaska to round out the competition), most of significant magnitude. I was able to complete an initial alignment at, around, 03:10UTC and began the locking sequence but never made it past CHECK_IR. At that point the seismograph started to climb again. Fortunately there was no further tripping of systems. Mother Nature is really letting her hair down! ( and having a few). Handing off to Nutsinee.
Activity log:
23:00 IFO locked
23:23 LOCKLOSS Chilean Earthquake 8.4MAG 23:45UTC
23:30 Began damping Seismic systems with Hugh’s tuteledge
00:32 Jeff Kissel taking advantage of the earthquake break to do some charge measurements
00:45 Darkhan also doing some open loop measurements in this down time.
02:05 Attempt to bring back End Station ISIs seems succesful. GS 13s back to high gain. Seismic graph is showing 3µ/S.
02:09 Ditto for ITMs
02:10 Begin bringing up Input and Output HAM SEI.
02:15 Kissel brought ESDs back and restored ETM alignment.
02:31 All SEI back to nominal isolated states. GS13 gains switched back to HIGH.
02:34 IMC re-locked. Looks like the ground motion is still too great to begin aligning arms.
03:10 started initial alignment. Waited at DarkMich to watch excessive ground motion
03:34 GRB alarm. https://gracedb.ligo.org/events/view/E184908 .
03:58 began locking sequence. ot as far as CHECK_IR...too much ground motion and then more earthquakes.
04:00 switched Observation mode to Environmental
05:38 Ground motion starting to drop below 1µm/s. Attempt at locking DRMI shows way too much motion.
A little late with this entry due to Mother Nature:
TITLE: Sep 16 EVE Shift 23:00-07:00UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Observing
OUTGOING OPERATOR: Patrick
QUICK SUMMARY:Full control room.. Wind is blowing just under the 20mph mark. Seismic activity quieting down from earlier earthquakes. All lights at Mid, End and LVEA stations are off. Then...Chilean earthquake 8.3Mag caused lockloss. Working with Hugh to get tripped seismic systems to happy places to ride this out.
Terramon hadn't reported the arrival of this event until it had already "knocked us for a loop".
C. Cahillane The five sensing function measurements and associated uncertainties at LHO have been analysed. Plot 1 shows the five measurements and models plotted together, alongside the residuals. The measurements are from August 26, 28, 29, and September 8 and 10. Plot 2 shows the weighted mean of the five measurement residuals in black, a naive systematic fit in red, and the weighted mean / systematic fit in blue. To be consistent with what I've done for actuation, right now I'm going to quadratically sum the systematic and statistical uncertainty, and zeroth order extrapolate it. For the sensing function though this is very clearly the incorrect approach, since the systematics on low freq sensing mag or high freq sensing phase are not simply going to stop blowing up at the end of the measurement. I will certainly have to go back and fix this. Also, since the systematics are so large here, I am thinking about applying my own "correction" to the sensing function the model gives me and then plotting only statistical uncertainty. I think this plot will display the directionality of our error alongside the statistical uncertainty. Things to do: 1) Calculate weighted mean of actuation residuals for each measurement time for Sudarshan 2) Zeroth Order -> First Order Extrapolation 3) Find systematics of actuation stages 4) Apply correct A_pu uncertainty calculations. 5) Apply systematics "corrections" to my uncertainty models to get directionality of error combined with statistical uncertainty 6) Go to LLO and do it all again!
Attached is agallery of 5 "dust" glitches. Still clueless of what they are, but - ETMY saturation is a symptom, not a cause - it is not possible to produce such a white glitch from saturating a drive. - The DCPD spectrum shows a roll-off for all of them - But the roll-off frequency (i.e. glitch duration) varies significantly = from about 300Hz to 3kHz. Example 2: GPS: 1126294545 UTC: Sep 14 2015 19:35:28 UTC ETMY saturation: yes Example 3 GPS: 1126437892 UTC: Sep 16 2015 11:24:35 UTC ETMY saturation: yes Example 4 GPS: 1126434798 UTC: Sep 16 2015 10:33:01 UTC ETMY saturation: yes Example 5 GPS: 1126441165 UTC: Sep 16 2015 12:19:08 UTC ETMY saturation: yes Example 6 GPS: 1126442379 UTC: Sep 16 2015 12:39:22 UTC ETMY saturation: yes
WIth Hang's help, I managed to investigate these glitches with the new lockloss tool using SUS-ETMY_L3_MASTER_OUT_LL_DQ as a reference channel. The script couldn't find any other optics that glitch prior to the ETMY. And sometimes the glitches are seen by ETMX 30-40 miliseconds after.
I've attached the plot of the glitches at the time you've given. I've also attached the list of channel I told the script to look. Basically all the SUS MASTER OUT DQ channels. Please let me know if you have any suggestions on whereelse I should look at.
Attached are time traces of the DCPD_SUM for the 5 examples.
Nairwita Mazumder, Rich Abbott A few days back Jim noticed (alog ) that the "Bumbling line" which varies over a large frequency range is again back on ETMX seismic channels . This was first noticed on March and disappeared before ER7 and again was seen from 4th August. One can see the lines at all the horizontal and vertical sensors on ETMX. I have attached a pdf containing some follow up work done during Rich's recent visit to LHO. The first plot in the pdf is the spectrogram of ETMX GS13 on 26th August. It can be seen that there are multiple wandering lines having a fixed offset. We were suspecting that some magnetometers at the End X might be the culprit (as we could not find any correlation between temperature fluctuation with the line ). The second and third plots are the spectrum of H1:PEM-EX_MAG_EBAY_SEIRACK_Z_DQ and H1:ISI-ETMX_ST2_BLND_Z_GS13_CUR_IN1_DQ for 2nd August and 26th August respectively. The red one is for 2nd August when the bumbling line could not be found and the blue one is the recent data (26th August). It is clear that the peaks appearing on ISI-ETMX_ST2_BLND_Z_GS13 after 3rd August are correlated with the peaks of the spectrum (which also appeared around the same time) of SEIRACK magnetometer . The plots on the second page shows the coherence between GS13 and the magnetometers in the VEA and SEIRACK. It looks like the magnetometer on the SEI rack has stronger coherence with GS13 sensors than the magnetometer located at VEA . I have marked two points (blue and red cross) in the coherence plots to highlight two of the many peaks.
Adding to Nairwita's comments, the signal seen in the GS13 spectra is also present in the magnetometer data. This being the case, it's most likely that the harmonic series points to an electromagnetic artifact associated with the HEPI pump variable frequency drive. The fact that the same signature does not exist at the other end station (I assume this to be true, but have not verified) may point to an enhanced susceptibility in the X-end electronics for some reason. No reason to panic much yet, but duly noted.
I have attached the coherence plots computed between PEM-EX_MAG_SEIRACK and GS13 , ST1 CPS and ST2 CPS over the frequency range 0.4Hz-900Hz to check the following two points: (1) If there exists any coherence between CPS and the Magnetometer at frequency above 256 Hz (2) What the low frequency behavior is I can be seen that the coherence between CPS and the magnetometer above ~25Hz is pretty low compared to GS13, but them have relatively high coherence with PEM-EX_MAG_SEIRACK near 20Hz .
Operators, please always feel free to contact me anytime reqarding such matters, even at 4am--Hugh