Unable to maximize the ALS transmitted power just by touching ETMs and TMS alone. ITMs didn't move much compared to where it was after the lockloss. COMM beatnote is low (3ish) and touching PR3 didn't improve it.
TITLE: 11/17 [OWL Shift]: 08:00-16:00UTC (00:00-08:00 PST), all times posted in UTC
STATE Of H1: "Environment"
OUTGOING OPERATOR: Ed
QUICK SUMMARY: Sitting out a 6.5M eathquake. Wind ~30mph.
TITLE: Nov 17 DAY Shift 00:00-08:00UTC (04:00-00:00 PDT), all times posted in UTC
STATE Of H1: Down
SUPPORT: Jenne
LOCK DURATION: Lock clock had us at over 12 hours. Lockloss reset. Don’t know actual time.
INCOMING OPERATOR: Nutsinee
END-OF-SHIFT SUMMARY: IFO Down. Earthquake
ACTIVITY LOG:
02:56 I believe I heard a plane pass overhead.
05:28 Livingston lost lock about 20 minutes ago. I called Will in their control room and he reported that winds had picked up to 15mph. sheesh.
05:45 I noticed to back-back 45MhzRFAM glitches. There may have been a couple more since 35 minutes ago. The Omega graph doesn’t show glitchy activity prior to that. As of now it doesn’t seem to be chronic.
06:42 Winds have calmed down to ≤20mph. Mich Live is fidgety between 10Hz and 11Hz. I don’t see any RF45 correlations to this. Range chart shows Livingston dropping lock almost exactly that we have a very large glitch and after that, we’re a glitchy mess. The RF45 business from my previous log is part of that and there was one really large glitch that wan’t 45Mhz or a Saturation. Jordan is checking it out.
06:32 45RFAM glitching for 5 minutes straight.
06:38 With Livingston down I came out of Observing to try and reduce the RF Set on the EOM to try and settle the glitching down.
06:44 Glitches still seem to be present only not as prominent to us. ASAIR_B_LF_90 is becoming unstable looking. No saturations being reported.
06:55 Concerned about saving the lock, I called Jenne. There doesn’t seem to be any action to take regarding this issue other than to aLog it. While we’re talking RF45 is settling down. I’ll wait until our range is back to where it was previously before setting the intent bit back to observing.
07:34 Lockloss. 6.5mag earthquake off the coast of Greece. I have to laugh because after the really tough time locking y-arm last night, it was the first to lock right up despite all the ground motion while x-is still hopping crazy. Handing of to Nutsinee
| ifo | P-phase Arrival Time | S-phase Arrival Time | R-wave Arrival Time | R-Wave Velocity (micro m/s) | EQ Distance (km) | GPS P-phase Arrival Time | GPS S-phase Arrival Time | GPS R-wave Arrival Time |
|---|---|---|---|---|---|---|---|---|
| H1 | 23:22:56 PST | 23:31:33 PST | 23:56:33 PST | 6.87893 | 9750.374 | 1131780193.4 | 1131780710.7 | 1131782210.6 |
| L1 | 01:22:48 CST | 01:31:30 CST | 01:55:38 CST | 7.04519 | 9557.989 | 1131780185.0 | 1131780707.8 | 1131782155.6 |
| G1 | 08:13:45 CET | 08:16:37 CET | 08:18:17 CET | 47.7759 | 1714.404 | 1131779642.1 | 1131779814.0 | 1131779914.6 |
| V1 | 08:12:17 CET | 08:14:00 CET | 08:14:53 CET | 84.3645 | 1000.73 | 1131779554.6 | 1131779657.1 |
1131779710.7 |
Looks like 72Mpc is the range du jour for now. It's not great but it seems stable...for the moment.
Excerpt from my activity log;
06:32 45RFAM glitching for 5 minutes straight.
06:38 With Livingston down I came out of Observing to try and reduce the RF Set on the EOM to try and settle the glitching down.
06:44 Glitches still seem to be present only not as prominent to us. ASAIR_B_RF90 is becoming unstable looking. No saturations being reported.
06:55 Concerned about saving the lock, I called Jenne. There doesn’t seem to be any action to take regarding this issue other than to aLog it. While we’re talking RF45 is settling down. I’ll wait until our range is back to where it was previously before setting the intent bit back to observing.
Starting at approx 5:30 UTC we are seeing an (small) increase in glitches on the control room glitchgram FOM (Figure 1). We think this is a symptom of the RF45 related issues again as we noticed a jump in the RF45 coherence FOM during one of these glitches.Right now, it seems much less damaging then the last episode.
See spectrograms (specifically feature at ~350 seconds in):
Strain: https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=108900
RF45: https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=108901
This time, the trend of LSC-MOD_RF45_AM_CTRL_OUT_DQ shows nothing strange (Figure 2).
The only reason I said the increase in glitches was 'small' is because I was comparing it to this.
It seemed to go away for ~40 mins but now it has returned again at ~6:32 UTC, looking pretty bad. I think Ed is going to try something.
MID-SHIFT SUMMARY: IFO is still locked and Observing @ 72Mpc. Tidal and ASC have been a little “active” from time to time probably due to increasing winds peaking to ≈35mph at times. We’ll see.
Correction...peaking to 40mph since that last entry.
Most of the 8:00-16:00 PST shift was fairly smooth. There was one lock loss caused by an earthquake (I didn't have time to localize the source of this seismic event). Several ETMY DAC saturations.
One strange glitch affected the range of H1 simultaneously as a glitch in L1's range. It didn't look like a usual ETMY DAC saturation, so Nutsinee and I ran an omega scan here. Nothing obvious in H1:CAL-DELTAL_EXTERNAL_DQ but several other channels show glitches. I'm not sure which ones to focus on for this particular event. Obviously, I need to stare at a few more omega scans to see what is typical.
h1build hung last Friday (13 Nov). This locked up the the slow controls SDF monitors which are being tested. As the slow controls SDF is not used by guardian yet to determine observation state we rebooted h1build and restarted the slow controls SDF monitors.
At this time we do not know what caused it to lock up, but it was noticed when Betsy was examining difference counts and noticed a discrepancy between values in the SDF display and on other MEDM screens.
TITLE: Nov 16 EVE Shift 00:00-08:00UTC (08:00-04:00 PDT), all times posted in UTC
STATE Of H1: Observing
OUTGOING OPERATOR: Jim
QUICK SUMMARY: IFO is locked and observing @ 73.8Mpc. µSei ≈ .6µm/s. EQ bands are ≈.22µm/s. Winds are ≤20mph.
TITLE: 11/16 [Day Shift]: 16:00-24:00UTC
STATE Of H1: Observing at ~80 Mpc
SUPPORT: Usual CR crowd
SHIFT SUMMARY: Locked most of the shift. Winds quietish <20 mph, useism is medium low
ACTIVITY LOG:
17:00 JohnW Bubba driving around site looking at tumbleweed situation
17:00 Keita & Me running excitations on ETMX & ETMY ISIs, earthquake arrives at the same time, lockloss
I rotated the HAM2 STS2 to see if elevated power in the Y channel was from tilting or a bad channel. The first attachment shows the 5 site SEI STS2s when the wind was fairly quiet. The top row of panels is from 19 October before the HAM2 STS2 rotation; the bottom row is from 12 Nov after the sensor was rotated. The HAM2 Y channel still shows elevated signal even though it is now rotated to align with the X axis. So this isn't related to orientation wrt SW facing wall. Additionally, these are quiet wind times (<10mph) so hard to explain this away with wind tilt noise.
The second attachment is similarly arranged with the same 19 October panels in the top row but now the lower panels are from 13 November when the wind was really honking (20 to 60mph.) Notice that all the X & Y signals are much larger at lower frequencies (<~0.3hz,) except the HAM2 Y channel which in the lower panels is actually pointing in X. This channel has a larger magnitude until you get to about 10mHZ but it does not respond anywhere near to amount that the other sensors do. This indicates to me that the HAM2 Y channel is definitely not healthy.
O1 day 59
model restarts logged for Sun 15/Nov/2015 No restarts reported
Jess, Laura, Paul
We have been following up on the periodic 60Hz EY glitches which have been seen at LHO since at least June (alog 18936). These glitches are witnessed by various magnetometers and couple into DARM.
It seems that the glitches indicate the switch-on of some electronic device. Spectrograms of H1:PEM-EY_MAG_EBAY_SUSRACK_QUAD_SUM_DQ show lines at several frequencies (30, 42, 60, 80, 120 Hz) that start when the glitch occurs, then stop with a rather softer glitch about 27 minutes later. Attached are spectrograms from yesterday and last Thursday. There doesn't appear to be anything correlated in the HVE or H0:FMC channels.
We recently investigated something similar at LLO (alog 22549), which we managed to trace to an unmonitored air-conditioning unit outside the EY station. The behavior at LLO is somewhat different in that the glitch is much louder and the subsequent noise (in both PEM channels and DARM) is quieter, as well as the fact that it stays on longer each time. However, because the turn-on and turn-off are so predictable it should be possible for someone to go down to EY and listen for anything turning on and off at the expected time.
Trying to understand better the 60Hz problem. I looked at 16 hours of H1:PEM_EY_MAG_EBAY_SUSRACK_QUAD data on friday (from 2015-11-13 00:00:00 to 16:00:00) when the detector was mostly unlocked to see if the 60Hz bursts were there while the detector is unlock. As seen on the attached time plot ('60Hz_Burst_locked_and_unlocked.png'), yes they are there. This is good because that means we can look for the source while the detector is unlocked during maintenance tomorrow. In this plot I also show that the first 6 hours the detector was locked and the rest had the detector unlocked.
Each burst begins with a spike that last about 0.22 seconds (see attached figure 'Zoom-spikes_Mag_VEA_and_EBAY.png'). Notice that the spike looks different on the magnetometer in VEA and the one in EBAY_SEIRACK. While the former has a frequency of about 60.24 Hz, the later has a frequency of 121.65Hz.
Each spike is spaced between 88.5 and 90.5 minutes from each other, however in the past this was reported to be spaced by 75 minutes. After each spike there is an excess signal that last about 23 minutes, I looked at these segments of 23 minutes during the time when the detector was locked (blue arrow on the time figure '60Hz_Burst_locked_and_unlocked.png'), when the detector was unlock (green arrow on the time figure '60Hz_Burst_locked_and_unlocked.png') and to compare also when there was no burst (red arrow on the time figure '60Hz_Burst_locked_and_unlocked.png'). And then using the same color coding I plotted the spectrum of each segments around the 60Hz and its 2nd and 3rd harmonic (as attached in figures labelled as 'Spectrum_left_sideband_at_...png'). Interestingly these 60Hz harmonics show a one sided sideband at about 1Hz from the 60Hz carrier (this is particularly intense on the 2nd harmonic).
Then I looked at a Microphone (MIC_VEA_MINUSY) and an accelerometer (ACC_VEA_FLOOR) and also plotted spectrum of the same 3 time segments around the 60Hz and 2nd harmonic (figures 'Spectrum_MIC_VEA_...png' and 'Spectrum_ACC_VEA_...png' ) and although we do not see the sideband however we see a peak very near to where the sideband is at the 60Hz fundamental. This peak exist also when the burst is not there, so most probably is unrelated, but worth noticing.
Tomorrow morning we will go to EndY with portable magnetometers to see if we can notice anything. And to predict what time the spikes should occur I looked at the most recent data I have from 2015-11-16 21:15:49 to 2015-11-17 05:15:50 shown on the attached plot 'Latest_60Hz_bursts.png', we notice that the spacing is 85minutes, therefore we should spects spikes at about:
UTC LHO - Local
2015-11-17 16:02:00 8:02 am
2015-11-17 17:27:00 9:27 am
2015-11-17 18:52:00 10:52 am
2015-11-17 20:17:00 12:17 pm
2015-11-17 21:42:00 1:42 pm
This is a classic tale of IM1-3 woes - IM1-3 are very likely to move when the HAM2 ISI trips, so need to be checked every time.
The IMs come from IOO, so they are unlike any other optics we have, and so behave in a very different way, and are suscetable to changing alignment when they experience shaking, like they do when the ISI trips.
The IM OSEM values are consistant, and when the optic alignment shifts, it is consistantly recovered by driving the optic back to previous OSEM vlaues, regardless of slider values. The OSEM values, when restored, consistantly restore the pointing onto IM4 Trans QPD.
IM4 Trans QPD reads different values for in-lock vs out-of-lock, so it's necessary to trend a signal like OMC DC A PD to correctly compare times.
IM4 does sometimes shift it's alignment after shaking, but because it's moved around by the IFO, choosing a starting value can be difficult. In the case of IM4, restoring it's alignment to a recent out-of-lock value should be sufficient to lock, but ultimately IM4 needs to be pointing so that we can lock the X arm in red.
I've tracked the alignment changes for the IM1-3 since 9 Nov 2015, and they are listed below.
These alignment changes are big enough to effect locking, and it's possible that the IFO realignment that was necessary last night was in part a response to IM pointing changes.
I've attached plot showing the IM alignment channels.
Armed with those channels, and the knowledge that the IM OSEM values are trustworthy, and the knowledge that under normal running conditions IM1-3 only drift 1-2urad in a day, checking and restoing IM alignemt after a shaking event (ISI trip, earthquake) should be a fairly quick process.
Thanks for the write-up here, Cheryl!
General Statement:
Honestly, when it comes to gross misalignments (those which CANNOT be fixed with an Initial Alignment; usually caused by something catastrophic [i.e. power outage, huge earthquake, etc]), I don’t have an idea of where to start.
For example, what specific channels does one check for misalignments (i.e. specific channel name, is it same for all optics? What about for ISIs/HEPI, do we need to check them for misalignment?). This is a more specific question for IO, SUS, SEI, & TMS.
Specific Statement/Question:
It sounds like you are finding that the Input Mirrors (IMs) are more susceptible to “shakes” from SEI; whereas since SUS’s are so much different and bigger, they aren’t as susceptible. This is a big thing, and we should pay attention to changes to the IMs.
Side question: Are the IMs similar to the Tip Tilts?
For input pointing misalignments, what is the cookbook/procedure for checking & fixing (if needed) alignment? Sounds like we:
All of this can be done in the control room, yes? Do we ever have to go out on an IO table?
I’d like something similar for SUS, TMS, & SEI. What signals (specific channels) is best to look at to check for alignment of each suspension or platform?
Anyway, thank you for the write-up and helping to clarify this!
Another 5.4M earthquake from Greece coming through...... Pause the initial alignment.
We also just had a wind gust at ~50 mph. EQ band seismic is on the rise. Relocking is going to be a while......
First try relocking. Lockloss at REDUCE_CARM_OFFSET (11:02:17 UTC). Wind speed ~30 mph.
Wind reaching 40 mph. Locking seems hopeless right now. Just lost lock again at CARM_ON_TR (11:08:2 UTC). Put ISC_LOCK to DOWN.
I noticed SRC_CAGE_SERVO user message tells me to "Align SR3 to offload M2 actuators". I'm not what action to take but I've attached the cage servo output below. I also noticed OMC shutter was opened while locking DRMI couple of times. Could that have intefered with DRMI locking? I don't remember the shutter being opened that early.
Tidal still high. Relocking still unsuccesful.
Wind speed decreased to ~20mph. So I tried again. Lockloss while locking PRMI. Looks like ETMX ran away.
Wind speed is back to ~30 mph.
The forecast lied to me saying the wind speed would decrease towards the morning. The wind has been 30-40 mph since 10 hours ago with no sign of decreasing. LLO has been down also because of wind. I'm calling it a night (or morning?).