IFO has not returned to locking after the big earthquakes 7 hours ago.
I've called Jenne Driggers and we did as much as was possible over the phone. She'll be in this morning to help and then do measurements.
I've been working to relock the IFO since 4 earthquakes in Mexico.
- restore HEPI and ISI, and HAM5 and ETMX are not yet back to full nominal state.
- green is locking in the arms, red is not
- I aligned IM1, IM2, IM3, and IM4 to eliminate their alignment as an issue for locking red, and those diffs will probably show up and can be accepted
- the AS_AIR camera was fixed when Jim fixed HAM5 ISI
- ETMX HEPI and ISI Guardian was unhappy, but the isolation was OK, so this was not an issue for locking red
- with AS_AIR back on the camera, the alignment of the red for x arm looks not too bad, and is popping to 0.6+
Jim and Jenne are here for day shift
Since I set up a schedule of burst hardware injections last night, three of the scheduled injection times have passed. The first one, at GPS 1126160499 (injection start at 1126160499, actual burst signal centered at 1126160500), should have been executed at H1 (not at L1 since it was down at the time), but didn't work. The tinjburst.log file says: "1126160499.000000 hwinj_1126160499_2_ 1.00e+00 Compromised". (It says that five times, because tinj retries a few times if it initially fails.) The more verbose tinj.log says: Injection imminent in ~297s... 1126160499 2 1 hwinj_1126160499_2_ GRB alert at 1126151534: injection allowed. Detector locked. Intent bit is on. calling awgstream... awgstream H1:CAL-INJ_TRANSIENT_EXC 16384.000000 ../config/Burst/Waveform/hwinj_1126160499_2_H1.txt 1.000000 1126160499.000000 awgstream failure: injection compromised! That's strange... Maybe it has to do with environment variables. I seem to be able to run awgstream fine from the command line in an interactive session, with this zero-amplitude test while H1 was down: [hinj@h1hwinj1 tinj]$ awgstream H1:CAL-INJ_TRANSIENT_EXC 16384.000000 ../config/Burst/Waveform/hwinj_1126160499_2_H1.txt 0.0 1126187950.000000 Channel = H1:CAL-INJ_TRANSIENT_EXC File = ../config/Burst/Waveform/hwinj_1126160499_2_H1.txt Scale = 0.000000 Start = 1126187950.000000 The fact that a zero-amplitude injection succeeded rules out a problem with awg. I tried to do another test by adding a zero-amplitude injection to the schedule at GPS 1126191200, but since the detector was not locked, tinj did not even call awgstream, so I didn't learn anything from that. The second and third scheduled injections, at GPS 1126180499 (injection start at 1126180499, actual burst signal centered at 1126180500) and 1126190499, did not occur because the detector was not in observing mode at either time; that's the correct behavior.
State of the IFO:
- recovery from the earthquakes has gone ok
- nothing broken, but SEI at EX not in the correct state, though it's good enough to lock, so will work with JIm to understand the issue and fix it when he comes in at 15:00UTC
Timeline since the earthquake:
9:32UTC - I start relocking and IMC comes back good
10:30UTC - no AS_AIR image on the camera concerns me, but I move on to locking ALS and both arms are good
11:37UTC - I return to ETMX SEI, I wasn't able to get it into the correct state after the earthquake, and now it's a problem, so I get it to Robust_Damped and leave it there.
11:53UTC - ALS is done, and I start on locking red in the X arm - no signal on the qpd at the end station, but I can see it spike when the arm power does, so I know there's light.
I tried to fix the AS_AIR camera, nothing worked, it's a red hering, the loose optic in HAM2 must have shifted again, or soe=mething else not an in-vacuum optic.
Current: 13:34UTC - writing an update and then returning to alignment
We've seen that CBC hardware injections in ER8 have caused ETMY saturations - see this alog for the injection, and my comment about the overflows. These are often called 'ETMY saturations' but what is happening is that the DAC is being driven beyond its maximum range of 2^17 counts. This is occurring much more often now that the driver has much more analog lowpassing. Peter Fritschel wrote this DCC document tabulating the available actuation amplitude before overflowing the DAC, which allows us to check whether a hardware injection of a binary neutron star coalescence would overflow the DAC. The summary is that a BNS hardware injection at a typical SNR would saturate the digital actuation at H1 as soon as it gets into the 500 to 800 Hz range. And a BNS typically goes up to 1.5 kHz. This has nothing to do with the inverse actuation or notches, it's just the limits of how much the digital system can push on the mirrors using the ESD, which has a lot of low-passing to reduce noise. At these frequencies, the DARM loop has almost no feedback so I'm assuming that the injection just moves the mirrors freely. I made a simple fit of Peter's tables, and plotted it along with the amplitude of a BNS signal as a function of frequency. My Python code is attached below and should be well commented. There's no interface, but you can edit the distance or masses in the file. The output plot is also attached. The waveform is from Duncan Brown's thesis, because that's where it's given most clearly in the time domain with the amplitude fully specified. I chose a system with masses 1.4,1.4 Msolar and a distance 120 Mpc and optimal orientation. The waveform, over a short period of time, looks like h(t) = A f^(2/3) cos (2 pi f t) where f is the gravitational wave frequency. Since this is strain, I plot A f^(2/3) times the arm length (4000 meters). I tried to avoid the limit by changing the distance to 180 Mpc, and multiplying Peter's limit by 3.3. He assumed we've got 30,000 counts to work with, but we can manage 100,000 counts if we're lucky. It's still not enough and hits the DAC limit at 800 Hz. As soon as the DAC overflows (hits 2^17 counts), there's a huge glitch produced so the injection becomes pretty much useless. The likely path forward, in the short term, is to push the masses up into the NSBH range, like to make one object 10 solar masses. We also might need to roll off the high frequencies artificially.
As a side note, I think the situation must get worse if we include merger and ringdown, as these go to even higher frequency (the ISCO is really an artificial cut-off, and BNS merger will have signal up to ~3kHz). Do the current CBC injections stop at f_ISCO? If so, why not set the cut-off frequency to avoid saturation instead?
Magnitude 4.9, 55km SSW of Topolobampo, Mexico 2015-09-13 07:40:40 UTC
The seismometer traces are going up again after the first spike from the initial earthquake.
Not yet able to even hold arms in lock in green.
Seismic traces are now off the top of the plot and a number of watchdogs tripped. All optics are now either tripped or in safe mode.
There are now 3 big earthquakes:
Screenshot of ops overview and list of HEPIs/ISIs/Optics that are tripped.
From the time optics started to trip, until the last optic tripped, was a total of 18 seconds.
Five optics tripped at 8:26:26UTC - eight optics tripped at 8:26:44, which could have gone to safe mode, if I could have clicked faster.
I am proposing we add the capability of putting all optics in safe mode with one button, once it's clear an earthquake is going shake enough to trip optics.
TITLE: 9/12 EVE Shift: 23:00-7:00UTC (16:00-0:00PDT), all times posted in UTC
STATE OF H1: Has been fairly steady at 70Mpc with a few glitches & locked for all but 40min of the shift.
SUPPORT: Evan & Jenne were here fairlyl late
DAY'S ACTIVITIES:
Restart CW postponed till tomorrow due to GRB. Will restart Burst Injection "schedule", but it will not be active during GRB stand down time.
I still don't know where the glitches that are correlated with ETMY saturations come from, but I do know why ETMY saturates.
We have many very deep notches in the L3 LSC LOCK filter bank in ETMY. Some of these are as deep as -180 dB! (There are probably others in the LSC-DARM filter bank, but I didn't look there to plot filters). The top half of the first attachment shows all of these notches. (Since I zoomed in and captured references of each notch with many points to see the actual depth, I had to split the filter over 2 plots - sorry).
Later, we have 2 digital filters (for each quadrant) in the ESDOUTF filter banks. These are compensating for the analog "Acquisition" filter, and the analog low pass. Together, they have magnitudes of more than 70 dB above 1 kHz. The bottom half of the first attachment shows the magnitude of these filters together with the nearly-negligible-at-high-frequency filters in the DriveAlign filter bank.
So, we are multiplying our signals by very small numbers, then multiplying them up by very large numbers. This seems like a recipe for badness. I don't actually know if numerical precision is the cause of these problems, but perhaps it is?
The effect of these filters is shown in the second attachment. Note here that brown and green traces on the top and bottom plots are the same data, but with different y-axes for clarity. The pink trace is the MASTER_OUT signal for one quadrant of the ESD, right before it goes to the DAC. The glitch is much larger than the 132k count DAC range, so we are certainly saturating the DAC. The blue trace is the output of the L3 LOCK L filter bank, after the filters shown in the top half of the first attachment. The green trace is that signal, filtered by FMs 3 and 4 of the DRIVEALIGN_L2L filter bank. There is still no obvious glitch. The brown trace is again the output of the Lock L filter bank, filtered by the same DriveAlign filters, and then also filtered by the AntiLP in the ESD output filter bank. Notice that due to the high gain at high frequency of that filter, we clearly see the glitch. DTT is somehow unable to filter a signal by the AntiAcq filter (zpk([152],[3250],1,"n")), so I don't have that included in the brown trace. But, the pink MasterOut trace is the actual signal, after all of the filters.
The third attachment shows the time just before the lockloss around 21:00 12Sept2015, and we can clearly see that this filtering effect is causing the signal out to the DAC to be quite weird. However, since the Lowpass and Acq filters are engaged, the signal actually going to the ESD may be mostly okay. In this plot, the blue trace is the length signal as soon as it gets to the suspension, the red trace is after all of the notch filters, black is the MasterOut after all of the digital filtering, and green is a readback of the analog signal going to the ESD. The glitches are basically impossible to see in any trace other than the MasterOut.
To conclude, maybe we don't really care if the ETMY ESD saturates occasionally, but the reason it does is the huge high gain at high frequency of the digital filters that compensate for the analog acquisition and low pass filters. Note that in ER7 we used the acquisition filter, and so some amoung of high gain at high frequency, but now in ER8 (and O1) we are also using the lowpass, so we need much more high freq high gain in our compensation filters. As Evan pointed out in alog 21315 this is why we're seeing more of these saturations now than we used to.
(Now that I've written all of this out, it all seems very obvious. But, when I was looking at the 3rd attachment (which started this all....), it was not at all obvious to me why the black trace looked so different from the others.)
At 5:26 UTC (10:26pmPST), received a LOW/MAJOR alarm for H0:FMC-EX_CY_H2O_SUP_DEGF with a value of 34.99degF. Looking at trends, it appears this signal approaches this lower limit value every 24hrs (so perhaps it should be lowered more, or something needs to be addressed at EX). It stayed in alarm for 7minutes and then the signal started turning up.
Attached is a 3-day trend of the channel. Will send an email to Bubba about this, so he can check it out Monday.
(Peter Shawhan, Corey Gray) After the evening's GRB alert stand-down period expired, I asked Corey to turn off the Intent bit briefly so that I could start the CW hardware injections with the new ER8B configuration. At about 21:55 PDT I did "bin/x_start_psinject Peter Start with new ER8B config", at the injections were supposed to start at GPS 1126155410. However, it didn't work. The file log/psinject_H1.15_09_12_21:55:33 says: "../../bin/psinject: error while loading shared libraries: libCore.so: cannot open shared object file: No such file or directory". I won't try to debug that tonight.
I added 17 burst hardware injections to the future schedule, at the following GPS times: 1126160500, 1126180500, 1126190500, 1126210500, 1126220500, 1126240500, 1126270500, 1126280500, 1126300500, 1126310500, 1126330500, 1126340500, 1126360500, 1126390500, 1126400500, 1126420500, 1126450500 As described the last time we started a sequence of burst injections (alog), an injection will automatically be skipped if the instrument is not in observing mode, if there is a GRB alert one-hour stand-down, if the operator disables or pauses injections, etc.
Stefan, Evan
We spent some time trying to see if some of the unexplained noise in DARM is caused by jitter (or other nonlinear pointing noise) in HAM6.
First, some things that didn't work:
We were able to make jitter noise by driving OM3 (I guess it would be surprising if we weren't able to do this). The first attachment shows some example bilinear coupling made by driving OM3 in yaw at 94.3 Hz.
Then we opened the OMC REFL beam diverter and tried some bandlimited injection between 35 and 45 Hz. The result is attached. We saw no coherence between OMCR and DARM (consistent with the notion that the coupling is all bilinear). Also, pitch and yaw of OMC REFL are highly cross-coupled. [Sorry for choosing a polluted band—I wanted a place where the DARM noise was low but where the OMC REFL QPDs are still plausibly seeing displacement noise.]
Finally, we made a broadband injection from 10 Hz to 300 Hz, in pitch and yaw. The times were 01:44:40–01:51:40 for yaw and 01:54:30–01:59:30 for pitch, both 2015-09-13 Z. The result for yaw is attached.
At 3:52 (8:52pmPST), Verbal Alarm notified us of a GRB. Event info is here. Verbal Alarm was quick with the notification (TJ and others mentioned it had been getting notices minutes/hours after an event).
Following GRB Checklist (p.2 of L1500117), we have the following:
9/12 EVE Shift: 23:00-7:00UTC (16:00-0:00PDT), all times posted in UTC
At 3:08UTC H1 was taken to Observation Mode since Evan was done with his injections.
Prior to this we saw some oscillations on IMC-F on the Tidal StripTool of nuc1, but we saw no other oscillations or seismic noise.
Support: Jenne & Evan are also in the Control Room, but sounding like they are finishing up for the night.
Winds have just quickly jumped up to over 20mph at around 3:10utc.
TITLE: 9/12 EVE Shift: 23:00-7:00UTC (16:00-0:00PDT), all times posted in UTC
STATE OF H1: Relocking, but also riding out remnants of New Zealand EQ from 2hrs earlier.
OUTGOING OPERATOR: Jim
QUICK SUMMARY: Walked in to H1 in Lock Acquisition, and saw several attempts during high seismic noise.
Once in NOMINAL_LOW_NOISE, Evan took over for commissioning (I talked with Joe at LLO and they were having Bounce Woes. When they get it locked, Anamaria/Robert would probably take L1. So they will be out of Observation for a while. In the meantime, Evan will conduct Noise Budget injections.
Seismic coming down & winds inching up.
EARTHQUAKE COMING!! EARTHQUAKE COMING!!
1:55 UTC: 5.8 earthquake from Indonesia on the way--> ETA of Rayleigh wave: 2:10UTC
Evan's trying to finish a measurement. Jenne might try to do some offloading if there's time.....
False Alarm: Did not see anything in the control room for the above-mentioned incoming EQ. Note: It's velocity was 0.59um/s (compared to 1.1um/s of similar magnitude New Zealand quake which kept us down for 2+hrs.
Josh, Andy, David, Jess
Conclusion: We suspect that the 50Hz glitches in DARM are caused by EX Mains glitches that happen 400+/-70ms earlier (reliably matched) and may(?) couple through EX Seismic.
Throughout September there has been a persitent line of glitches at 50Hz in DARM with a pretty high rate. See the attached glitch-gram from today's summary pages. Based on the one hour plots on the summary pages, the rate of these 50Hz glitches in DARM is about once per minute.
We found (see e.g. the first round here) that these DARM glitches are correlated in time with glitches in EX channels, particularly PEM EX SEIS/ACC*, ISI ETMX ST1 BLND*, and ASC-X_TR*.
It looks like the cause of this may be a big glitch in EX Mains. Wew see the EX and DARM channels are all glitching with a 400+/-70ms delay (averaged five examples followed up by hand) after EX mainsmon. Attached are two examples where we lined up the glitch in EX mainsmon and showed that the glitches in EX seismic and DARM are 400ms later.
Notes:
We would be happy to follow up further leads, but wanted to report what we found so far.
PS. Should you wish to repeat this yourself, attached is a list of these glitches in DARM today (with nice GPS times) and a screenshot of the ldvw settings I used.
Note that the signal in the seismometer is a bit later than the mainsmon glitch, and that the seismometer glitch is narrowband around 50 Hz in contrast to the mainsmon. A guess is that there is a mechanical coupling -- maybe a motor running slow due to it hitting something, or a transformer core pulsing -- which induces the 50 Hz characteristic. Seems like it might be easy to hear. The delay suggests that it could be something a bit away -- maybe a chiller?
The chiller compressor is probably on the order of 100 hp (>100 amps at 480v) and is cycling with a period of ~1hour during most of the day. The period depends on the heat load into the building. During the hot time of day one chiller compressor will continue to run while a second compressor will cycle as needed. In the attached plot the peaks represent the start of the chiller compressor. The valleys represent the shutdown. Times are PDT.
After emails with Robert S and John W we did an extensive search in the "HVE-EX:*" and "H0:FMC-EX*" channels for anything that is switching on or off at the same time as the glitches reported above. John Worden suggested that the instrument air compressor (a ~5hp motor at EX) would switch on and off with about the right period. I attach a plot that shows that the glitches discussed above for EX seismic (left) and DARM (right) are correlated with the compressor turning ON at the bottom of the triangle waveforms.
Terramon told us that we had an incoming earthquake large enough to likely break lock, so I asked Jim to take us out of Observing mode so that I could quickly try to increase the SRM offloading, to see if that would save the lock. In the end, we still lost the lock.
I changed the SRM M1 Lock L gain from the nominal -0.02 to -0.05 with a 10 second ramp.
I need to look at the data to know more, but the verbal alarm did not complain about SRM saturations. It did however complain about ETMY saturations.
If that really did help prevent SRM saturations from causing our lockloss, I wonder if we should think about increasing every optics' offloading gain by a factor of 2 or 3 when we have an earthquake coming. Anyhow, I need to look at the data to see what really happened, and to see if I can figure out what really helped.
SRM offloading by just increasing the gain was no good.
I think that I must have made the crossover unstable, which made something not good at the AS port. The HARD ASC loops (particularly DHARD Pitch) start to oscillate, which is probably why all 4 test masses are saturating their L2 drives at the time of lockloss (and ETMY is also saturating the L3 drive).
So, oopsies. While trying to save the lock, I seem to have killed the lock.
To actually fix the SRM offloading, I'll need to measure the crossover to check its stability, and maybe tune the filters a bit.