Title: 1/3 Eve Shift 0:00-8:00 UTC (16:00-24:00 PST). All times in UTC.
State of H1: Observing
Shift Summary: Started the shift with an unlocked IFO due to EQ. After EQ ringdown, aligned and locked IFO. Since going to Observing, we have been watching the slow degradation of power via POP_A_LF, much like last EVE. At one point we attempted to reverse the trend by adjusting ETMy pitch, but were thwarted by WFS control loops.
Activity log:
11:30 Lockloss probably from whatever is causing POP to drift down
13:15 Back to observing, shortly thereafte I notice that POP is once again drifting down. No clear cause, environment is quiet, RF45 seems to be behaving, ASC foms etc all look nominal.
14:45 Bubba starts clearing the parking lot. This has a noticeable effect on the IFO.
Starting at about 14:30, Bubba started cleaning the parking lot. He started with shoveling, which had no effect. He's now running the tractor near the OSB and it's having some effect on the IFO. The range has gotten pretty ratty and POP (which is still trending down after re-locking 2 hours ago) is occasionally getting knocked around. Think we'll survive, but analysis people should be aware when looking at this mornings data.
Can't say I'm suprised, but we just lost lock. The hours long decline in POP had looked like it was stabilizing. Apparently not. Time to see how re-locking goes.
Title: 1/3 Eve Shift 0:00-8:00 UTC (16:00-24:00 PST). All times in UTC.
State of H1: Observing
Shift Summary: Started the shift with an unlocked IFO due to EQ. After EQ ringdown, aligned and locked IFO. Since going to Observing, we have been watching the slow degradation of power via POP_A_LF, much like last EVE. At one point we attempted to reverse the trend by adjusting ETMy pitch, but were thwarted by WFS control loops.
Incoming operator: Jim
Activity log:
0:33 Vinny +1 to LVEA and Betsy to Optics Lab
1:08 Vinny +1 out
1:22 Adjusted ALS fiber polarization for both X and Y
1:27 Switched ETMx ISI X blend to 90 to stop an oscillation and began initial alignment
1:29 Gerardo to CP3
1:30 Betsy out
1:46 Betsy to LVEA for cables
1:56 Gerardo done
2:00 Betsy out
2:50 Observing
6:28 Commissioning mode to adjust ETMy pitch in attempt to save lock
6:40 Back to Observe
7:16 H1:FMC-CS_LVEA_REHEAT_4_DEGF alarm
We are once again experiencing the slow degradation of power as we saw during my shift yesterday. At Kissel and Betsy's suggestion, we briefly went out of Observing in order to attempt to save the lock as we are approaching the "point-of-no-return". I adjusted ETMy in pitch since Betsy found a correlation between ETMy drifting down in pitch and the POP power dropping. This worked temporarily, until the WFS caught wind of my treason and fought back. We decided to surrender and let it play out.
The ETMy has been slowly sagging over the last few months. Recently, the sag rate has increased. While the temperature control at EY appears to be noisier during this December's cold spell, the overall temp has not changed by a large (greater than a deg C) amount, so I doubt that is the full cause of the drift. However, there have been a few pressure changes that the ETMy chamber has witnessed, according to the VE pressure channels I picked off and plotted (attached). I can't correlate all of the vertical motion humps exactly to these pressure steps, but the direction is correct: lower pressure means a a lower QUAD - that said, the sag usually happens much faster than weeks when the pressure or temp is changed, so wt-heck?. The most recent step in pressure shown on the plot was Dec 11th when the VE crew was doing some work on an IP bakeout at Y2-8. I'll poke them to see what the nature of the pressure step is. I worry a bit that there is something wrong with the channel after the step feature since it goes very quiet past that point.
The corresponding ETMy pitch motion change is much more subtle, which is good from an alignment standpoint. The BOSEMs still have plenty of signal, so we shouldn't have any issues with drive due to this drift.
Another day, another mystery. Keep H1 weird.
This pressure change is too small to account for the vetical drift. In log 15887, ETMY is measured to sag by 120 microns during pumpdown, a pressure change of 760 torr. For a pressure change on the order of 1e-8 torr, as seen here, you would therefore expect a vertical change of
(1e-8 / 760) * 120 microns = 1.6e-9 mincrons or 1.6e-15 meters. This is well into the noise of the OSEMs.
I think it is more likely the in-vacuum temperature is different from the VEA temperature sensor. Are the any in-chamber temperature sensors on the ISI table?
A consistency check you can do is see if it is just the suspension, or really a change in the whole chamber is to see if the ISI is sagging as well; or rather trying to sag since it has active position control. So, if the vertical ISI actuators are pushing up harder proportional to how much the quad is sagging, then there is at least some consistency throughout the chamber.
Since the suspension pitch isn't changing (which is great) that suggests that either the suspension is super well balanced, or it is just the top springs at the ISI table that are sagging. The temperature of these springs likely follows the temperature of the ISI much more closely than the ones lower down.
Just because...
While watching the IFO come back up, it was noticed that the live trend of ISS Diffracted Power was wiggling alot like the live trends of POP_A and IMC_TRANS. Attached is a trend for the curious.
J. Kissel, B. Weaver I'm the curious. I did a bunch more starring at these channels, trying to come up with a theory for all of this ETMX oscillation business. My best theory at this point: this wiggling is a mixing of tidal + length to angle coupling of the QUADs against power fluctuations arising from the ISS 2nd loop's feedback to the 1st loop, which is then fed back to the AOM's signal that we stare at for drift. Check out my attachment. All of the channels in the attachment, representing DHARD, Tidal, the IMC, the ISS... are all wiggling at roughly the same frequency. Here're some things I've notice while staring: (1) Notice that the Tidal Error signal (as shown by H1:IMC-F_OUT16 in DARK Green) is perfectly anti-correlated with the DHARD Pitch control signal (H1:ASC-DHARD_P_OUT16 in Purple). This is no surprise to me because I know that Tidal is fed to the UIM stages of both QUADs, which are known to have plenty of UIM-length-to-TST-angle imperfections. You could also claim some correlation between the DHARD yaw control signal (H1:ASC-DHARD_Y_OUT16) as well, but it's not so obvious, so I segregated it. (2) Along the bottom are the DC component of 1st / Inner loop ISS PDs (H1:PSL-ISS_PD[A/B]_CALI_DC_OUT16), these lag just a smidge (5-10 [sec]), but perfectly behind the IMC TRANS PD (H1:IMC-TRANS_OUT16). This PD is watching the same light the 2nd ISS array is watching, whose control signal is fed back to the -- you guessed it -- the error point of 1st loop of the ISS, whose actuator -- you guessed it -- is the AOM. The same AOM who's diffracted power we monitor for drift (which is why Betsy had it up). When you actuate on the diffracted power, you're directly changing the input power into the IFO. Not so stable at low frequency, this intensity stabilization loop... (3) The cavity powers (PRC and ARM, as measured by H1:LSC-POP_A_LF_OUTPUT and H1:LSC-TR_[X/Y]_NORM_INMON, respectively), also show the same "oscillation" as the ISS Inner Loop PDs, but they're look like a mix between the IMC TRANS / ISS Inner Loop PDs's fluctuations (i.e. input power fluctions into the PRC) and the Tidal / DHARD Pitch fluctuations (i.e. arm power fluctuations because of inadvertent pitch from tidal control). Some times they follow one, other times they follow the other. I suspect this is why the POP_LF trace (shown up on the front wall in bright yellow, so very obvious) seems to show slow oscillation, with and even slower envelope. I have a feeling that when you add a gain peaky-very-low-frequency blend filter into the equation (ETMX, when in the 45 [mHz] blends), it exacerbates the whole problem. (Note, I attach the correct configuration of the BSC ISIs's blend filters) Hurumph! It seems like the source of all of these symptoms appears to be the IMC -- both tidal control and PRC input power fluctuations. Maybe we should take a serious look at the low-frequency performance of the IMC? My deepest darkest fear is that it all comes from the HAM3 0.6 [Hz] problems, but I think that's too fast this issue. Anyways as I've claimed all along, I think at least these sets of traces convince me that this is a particularly complex interaction of *many* plants and anima*ahem* control loops. I think we might be at the point in LIGO's life time that we can't keep getting away with just assuming that everything below 100 [mHz] won't interact with each other, and I expect this will only get worse with higher power. Since we'll shortly have the oppurtunity to play again, I suggest we try a few things and see if it helps: (1) Change / improve the isolation configuration on HAM2/HAM3 (2) Try improving the UIM L to P (3) Change the UGF of the DHARD loops (4) Rejigger the ISS loop (or at least characterize it better) such that the low-frequency loop shape is different. P.S. The template for this StripTool lives in /ligo/home/jeffrey.kissel/2016-01-03/PowerFluctionations.stp Betsy's pulling up DTT on this, check for coherence, but just by looking at the time-series, I'd have a tough time mathematically proving that this is a linear coherence. We'll post if we get any clues there.
Note the coherence of many things with the HAM3 0.6Hz... (spectra taken from this lock stretch).
Dear DETCHAR - Since we can't seem to get the data from 10 days ago in our own controlroom, can you please look into coherence with IFO channels during that time when the 0.6Hz SEI HAM3 peak wasn't around?
Thanks-
Betsy, Kissel
Sad that I have to tag SEI on this... Perhaps in the O1/O2 interim we need a concerted campaign of replacing electronics in HAM3.
The low frequency gain of the ISS should be large enough: in the normal configuration there is a low frequency boost and an integrator. However, in the past it was clear that the low frequency (below ~20 Hz) error signal was spolied by some additional noise, most likely scatetred light. Indeed, when the beam on the external table was not properly dumped, we saw a lot of increased noise due to back scattering.
So, it's also possible that the ISS loop has enough gain at low frequency, but the error signal is polluted by some scattered light, which might be correlated with the motion of some suspended object.
Once the EQ band calmed back to normal levels, I did an initial alignment and brought us back to Observing at 2:50 UTC. I adjusted the ALS fiber polarization for both arms while we were waiting for the EQ to ring down.
After notifying the operator I drove past the CS @ 01:31 and arrived at Y-Mid station @ 01:35.
Started filling CP3 @ 01:36, opened the exhaust check valve bypass valve, then opened LLCV bypass valve 1/2 turn, started to notice flow 29 seconds later. however I closed the LLCV bypass valve after allowing flow for a total of 2 minutes.
6 minutes later I closed the exhaust check valve bypass valve.
Started driving back from Y-Mid @ 01:47, arrived at the CS @ 01:52.
Activity Log: All Times in UTC (PT) 00:00 (16:00) Take over from TJ 18:02 (10:02) High temp alarm on CS Zone 4 duct. Temperature hovering around 100F. Will monitor 23:24 (15:24) Lockloss – Mag 6.8 EQ in India. 23:38 (15:38) Put IFO in DOWN until EQ ring up passes 23:47 (15:47) Reset timing error on H1SUSETMY 00:00 (16:00) Turn over to Travis End of Shift Summary: Title: 01/03/2015, Day Shift 16:00 – 00:00 (08:00 – 16:00) All times in UTC (PT) Support: Vinny, TVo Incoming Operator: Travis Shift Detail Summary: Mag 6.8 EQ in India, R-Wave 9.6um/s, seismic a little over 2.0um/s. Put IFO in DOWN until seismic settles down. Otherwise, it was a good observing shift.
23:24 (15:24) Lockloss due to mag 6.8 EQ in India. Seismic rung up to over 2.0um/s.
Reset H1SUSETMY timing error.
IFO locked and observing for past 9 hours. Seismic and microseism are flat and mostly within normal range bounds. Microseism is near the top of the range bounds. Winds are up to a light breeze (up to 7mph), outside temps are 18F for all reporting stations. There have been several ETM-Y saturations, however the remains around 80Mpc.
It is currently snowing at the site, a dry light snow with about .25" accumulation. The roads and parking lot do not appear to be slippery. However visibility is somewhat limited.
Our cdsfs1 backup machine once again crashed, was not accessible via the network. I was able to remotely power cycle it via IPMI.
O1 day 107
model restarts logged for Sat 02/Jan/2016 No restarts reported
Looking at the spectrograms and glitchgrams on the summary pages for January 2nd there seems to be some interesting behaviour from about 10 - 35 Hz at LHO. The noise drops in this band over the course of the day and glitches in this band become less frequent as well. It does not seem to correlate to wind speed or ground motion, at least not in an obvious way. Looking at the ground motion BLRMS page of the summary pages, there do seem to be some spikes between 4 - 6 UTC, in multiple bands. However these spikes are relatively short-lived and the noise in the spectrograms starts before that. I've attached a few images showing noise and glitches from the band of interest.. The plots are also visible on the summary pages.
The last lock loss on 1 Jan was due to PSL issues (see alog). The low frequency noise reported here seems to be correlated with unusual noise in PSL ISS and also shows up in IMC frequency noise. The attached pdf shows apparently normal behavior on 1 Jan for PSL ISS, IMC frequency noise, and strain followed by excess noise in all three on 2 Jan. While things appear to have calmed down on 3 Jan (so far), there appears to be a PSL glitch that shows up in the strain channel.