I reset the PSL 35W FE watchdog this morning at 18:34 UTC (10:34 PST). I also noticed the crystal chiller was a little low so added 150 mL of water to top it off.
Standard weekly test of the SNEWs and G-wave alarms did not come through as expected, and are also absent from the GraceDB page.
WP 5620; SEI Log 841; DCC T1500549
I've SVN UP'd userapps/release/isi/common/guardian getting:
Title: Ops Day Shift, 16:00-00:00UTC (8:00-16:00PT), all times in UTC
State of H1: unlocked, Down, maintenance started around 16:20UTC with PSL REF cav realignment
Outgoing Operator: Nutsinee
Site status:
- freezing rain and snow last night, roads are passable but not clear
- temperatures are around 28F across the site
- useism is increasing, with the corner station at about 0.6um/s, and EY and EX at 0.4um/s
- winds are 5-15mph across all stations
TITLE: Nov 24 Owl Shift 08:00-16:00UTC (00:00-08:00 PST), all times posted in UTC
STATE Of H1: Lock Acquisition
SUPPORT:
INCOMING OPERATOR: Cheryl
END-OF-SHIFT SUMMARY: High useism and wind speed is on the rise. The IFO was locked through out the night and lost lock just before the maintenance time. Relocking attempt has been unsucessful for me. Got to LOCK DRMI once and the other time it just lose lock at FIND IR. I touched TMSY to maximize the ALSY transmission. I hit ETMY diag reset to clear the timing error. I switched blends on both end stations to be 90 mHz and left the corner station ISI at 45 as Jim suggested. Handed off the IFO to Cheryl.
ACTIVITY LOG:
08:05 Kyle on site
09:20 Kyle left
14:41 Bubba driving the tractor around to clear snow off the drive way.
Haven't investigated the cause but the ALS beams don't look too good.
Quiet night so far. Useism is on the rise. A HPI-ETMX_IPS location mon was increasing the beginning of the shift but now seems to flatten out.
Added ~ 5 gallons of fuel to generator -> found blown fuse for 1 of 2 variacs supplying heat to 1 1/2" tubes + gate of 10" gate valve -> replaced 5A fuse -> OK now -> T/C for ion pump indicates 188C -> adjusted output (decreased) for both variacs supplying heat to ion pump (150C is target)
Title: 11/23 Eve Shift 24:00-8:00 UTC (8:00-16:00 PST). All times in UTC.
State of H1: Observing
Shift Summary: EX ISI is causing troubles again, but some changes to ASC seem to be keeping things under control for now
Onsite Support: Evan, Jeff, Jenne
Activity log:
IFO was unlocked when I arrived. Evan and Jeff were trying to diagnose EX.
1:00 I start initial alignment, this is hampered by AlignIFO and IMC nodes not cooperating. IMC node seemed to hang and refuse to lock occasionally, I had to re-request Lock. Lock X-arm IR was also uncooperative.
2:40 Finally to NLN. Except for EX slowly ringing up and then down, all's quiet.
A few details on the Xarm locking, in case it's helpful for someone else in the future:
We had the XARM gain turned up to 0.15 (which many people have been doing lately, when the Xarm is being fussy (nominal is 0.05)), however when it would catch lock, it would oscillate significantly. I think this was because the optical gain was too low (highest flashes and average transmission were about 0.8), and so we couldn't afford the 2 integrators that come on immediately after the lock is acquired. I had Jim disable the filter module triggering (by setting FM1 and FM2 of the XARM to "Man" rather than "Trig"), and we were easily able to acquire a smooth lock. After a few seconds the ASC brought the arm power to just above 1.0, and we switched the triggering settings back to normal. Alternatively, we probably could have turned the XARM gain up even higher, to make sure we weren't so close to the low edge of the phase bubble, but there didn't seem to be a reason to explore this more thoroughly.
The IFO just got to NLN, so I switched to Observe. Shortly after, the ring EX ring up reappeared, so we have dropped out of Observe again, and are tuning some ASC to try to settly the IFO down.
Getting ready to go back into Observe, Jenne had me reduce the gains on PRC1 Y with a -20dB filter, and reduce the gains on DHARD P (from 10 to 7) and Y(from 15 to 10). The low frequency ground motion is coming up, so we may be getting an earthquake.
Not a cause for alarm. Just noting for commissioners interest that after doing initial alignment tonight, I went straight to Lock PRMI, which acquired quickly, but the transition to DRMI failed. This is just the first round of trying to relock for tonight. DRMI just acquired for a second time, then promptly dropped the whole IFO out. Looking to be a long night.
J. Kissel While trying to characterize problems with HEPI / TIDAL / ISI / ASC (LHO aLOGs 23676 and 23676), I tried taking a swept-sine transfer function with three data points between 0.001 and 0.005 to charaterize the HEPI / UIM cross-over frequency (recall that the TIDAL crossover frequencies were never calibrated beyond "roughly," LHO aLOG 16255). However, upon starting the excitation, DTT gave the IFO a good kick on lots of ASC and LSC DOFs. It did *not* break the lock, but it certaintly came close, and any higher an excitation amplitude would have certaintly lost it. From a look at the time serious, it looks like DTT *tried* to ramp the signal, but chose a much faster ramp than is necessary for such a waveform. Attached are the start of the waveform, a zoom in on the ramp, and then a screenshot of the DTT params.
After the maintenance of GraceDB this afternoon (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23665) I restarted the ext_alert process on h1fescript0 using Monit.
Evan, Hugh
As seen by JimW and others, the ETMX was moving ALS excessively especially wrt ETMY. Attached are the in line outputs from the Isolation loop for EX and EY. This two hour trend plot shows an EQ that took the IFO out of lock around 1/3 of the wasy through. The ring up of the ISO X at ETMX is very clear in the second have of the plot. The second attachment shows uncalibrated spectra of the X Drive two hours ago and during the ring up where it appears to be elevated in the 40mHz region. Oh what the heck, I'll put them on the same snap.
We deisolated the DOF and then reisolated. This action seems to have addressed the problem for now. Well not for long. Rung up again with IFO locked. Took it out of Observing and tried turning of boosts and adjusting gains.
E. Hall, J. Kissel, T. Sadecki, H. Radkins, J. Warner
The IFO had lost lock again from the same problem about ~20 minutes later. We noticed LLO was down, so we tried pushing buttons hoping to rectify the problem.
We tried several things, all on hunches we'd had from all signals we saw oscillating at ~30-40 [mHz], which included HEPI / ST1 ISI / PRC1 Yaw and TIDAL -- so this is NOT just an "ISI ETMX Loop goes unstable" problem, its a nasty, slow, cross-coupled interaction of which the ISI is only one of the players.
Things we "know" that informed our hunches and button mashing:
- This only shows up when the ISIs are on 45 [mHz] blends.
- Symptoms point us toward ETMX, because it stays rung-up after lock-loss
- The above mentioned channels are visible wobbling at these 30-40 [mHz] frequencies
- There seems to be a 6-8 minute period envelope to the oscillations
Our hunches include
(1) Bad IMC_F-to-UIM or UIM-to-HEPI tidal cross-over when ISIs are in 45 [mHz] blends, i.e. the tidal offload is at a higher frequency than the ISI's inertial blend frequency, so the ISI's isolation loops are fighting actuation from tidal
(2) PRC1 Yaw ASC loop (which keeps PRM aligned to the PRC's cavity axis) unstable, which modulates the power into the arms. This modulates the classical radiation pressure on the ETMs in common, which excites CARM / Tidal. This then puts excess motion into the Tidal feedback and offload.
(3) Excess motion on HEPI from tidal is causing excess tilt, because HEPI tilts when you request it to drive in longitudinal. With the ISI ST1 in 45 [mHz] blends, and the RX&RY blend still in the 250 [mHz] blend configuration, the excess tilt from HEPI is coupling into the ST1 CPS, tilting the ISI, and fooling the X&Y DOFs into thinking there's excess X&Y motion, and drives excess X&Y. This pushes the test mass in X&Y, exciting some DARM / CARM action, which then spills back to HEPI.
Note that these are all sort of half-coupling mechanisms that, if I waved my hands hard enough, you might believe. But, we have no measurements (i.e. Sys ID, Loop Transfer Function measurements) to prove any of these statements.
Based on these hunches, we twiddled the following knobs:
- Reducing UIM to HEPI offload UGF (H1:LSC-${X,Y}_TIDAL_CTRL_UGF) from 0.002 to 0.001 [Hz] --> no noticable effect, but we were watching a 10 [minute] strip tool, so out patience may not have been high enough. The Oscillation was present, but not ringing up.
- Reducing UIM to HEPI offload UGF by half again to 0.0005 [Hz] --> same result
- Returning UIM to HEPI offload UGF to 0.001 [Hz], and reducing the IMC-F to UIM offload UGF (H1:LSC-${X,Y}_COMM_CTRL_UGF) --> We might have started to see a reduction in amplitude.
- Reducing the PRC1 loop gain (via the POP A DC to PRC1 ASC input matrix H1:ASC-INMATRIX_Y_3_21) from 1.0 to 0.5 --> This seems to reduce the oscillations in all signals.
While Evan and Jim coninued to twiddle knobs, I and LLO was down, I suggested we try to characterize some of these low frequency loops. However, a third of the way through my initial characterization of the UIM to HEPI tidal offload (which had already had a bad start, see LHO aLOG 23680), the oscillations had come back to a point of no return. Unclear whether they were because we just didn't reduce the original oscillation fully, or whether the bad start to the excition kicked everything enough to restart a bigger oscillation, or if it was my measurement itself slowly driving things around that re-triggered the oscillation enough to eventually saturate Tidal / the UIM. However, it was apparent that it was the 6-8 minute envelope that brought things over the edge, not the 30-40 [mHz].
After that lock-loss, Jim had found IR in the arms to be pretty bad, so he has since gone into initial alignment.
Stay tuned...
I attach some visual aides that I'd used to discuss the hunches with Hugh / Evan / Travis in case it helps anyone else. (THC = Tilt Horizontal Coupling).
LLO had called and said they would be down for awhile. I'm going to finish doing a swept-sine through the INJ_HARDWARE filterbank from 5-500Hz. Previously I had only finished 500-2000Hz. The IFO intent mode has been turned off. This is WP#5595.
Previous swept-sine measurement was: aLog 23307 The following was done: * psinject turned off at 5:20 UTC * swept sine measurement started at 5:21 UTC * swept sine measurement ended at 5:48 UTC * psinject turned on at 5:49 UTC I've attached a plot of the transfer function and coherence between INJ_CW and DELTAL_EXTERNAL_DQ. WP#5955 is now complete.
Attached are results of computing a transfer function from H1:CAL-INJ_CW_OUT to H1:GDS-CALIB_STRAIN, using the same method used for the higher-frequency swept sine on Nov 11. The transfer function is generally flat and well behaved below 500 Hz and above ~35 Hz. The feature at 167 Hz appears to be an artifact of the swept sine measurement at that point being artificially split into several points. Figure 1 - Spectogram of H1:CAL-INJ_CW_OUT during swept sine Figure 2 - Spectogram of H1:GDS-CALIB_STRAIN during swept sine Figure 3 - Transfer function and coherence More detailed results can be found here.
The DTT template for doing the 5 to 500Hz swept-sine measurement is in the SVN here: template_cw_inj_sweptsine_5to500_20151121.xml The DTT template for doing the 500 to 2000Hz swept-sine measurement is in the SVN here: template_cw_inj_sweptsine_500to2000_20151111.xml