We have just set the observation bit, we're currently at 75Mpc.
The alignment the interferometer found seems to be a bit better this lock. RF18, RF90 and PRG are not looking great though.
Plot of our 51 locklosses from the last 20 hours from Dan.The lockloss tool stopped working 4 hours ago. Problems today, in addition to 45998: NOMINAL_LOW_NOISE has 8 locklosses, but it's not clear why we are so unstable here. I mention blaming ASC last night, so I wrote a cross-correlation tool to look at which LSC/ASC error signal fluctuations were most correlated with the dips in PR Gain, POPAIR 18 and POPAIR 90. It seemed that SRC1 and PRC1 were the most correlated with POPAIR, along with CSOFT, so we opened the SRC loops and found offsets for SRC1 and SRC2 which minimized POPAIR 90 fluctuations. We were in the process of walking PRC1 and PRC2 yaw when we lost lock in CHARD_BLEND.
H1:ASC-SRC1_P_OFFSET -0.4054 H1:ASC-SRC1_Y_OFFSET -0.0767 H1:ASC-SRC2_P_OFFSET 0.1109 H1:ASC-SRC2_Y_OFFSET -0.0083Still unclear if our NOMINAL_LOW_NOISE locklosses are actually ASC, but our fluctuations in POPAIR 18 and 90 can be as large as 20% and are slow, with dips lasting about 2-3 seconds. If these fluctuations exceed 20% we tend to lose lock. See Pic 3. ENGAGE_DRMI_ASC locklosses, turning off the ASC MICH FM1 (a -20dB gain) seems to be happening a bit early and kicking MICH in pitch pretty hard. EDIT: I changed the ramp time of FM1 for MICH_P and MICH_Y from 4 seconds to 10 seconds. TR_CARM locklosses from glitching from the LSC IN2GAIN lowering. ALS XARM is glitching pretty badly sometimes, causing several ALS locklosses.
Recovery from the PZT mount swap last week has pushed the ISS Second Loop to 0.88 in Yaw, and lowered the power sum of the ISS Second Loop's 8 PDs. It seems likely this is contributing to ISS issues, so recommended to recenter QPD and maximized power on the DC PDs.
The PZT swap moved the beam in yaw on the ISS Second loop QPD from -0.12 to I'm not sure, it looks like the QPd was centered, and then from that position to 0.88 when IM3 and 4 were used for recovering IM4 Trans and POP QPDs.
Plots attached show the PD inner sum decrease, QPD yaw alignment, IM3 alignment, and the input power, which shows that fpr requested power of 20W, the measured power went from 19.6W to just shy of 20W (19.98).
Yesterday we changed ITMY lens by -5uD with the ring heater. We've now undone that change. We have got the ITMY HWS leveled off to value we were going into DRMI locking with before the RH change yesterday (~30uD). DRMI has locked the last several times in a minute or less.
Alignment still doesn't look great, We are currently losing locks around DRMI_LOCK_CHECK_ASC and OFFLOAD_DRMI_ASC. Adjusting alignment to get the error signals zeroed better before going up.
DRMI_LOCKED_CHECK_ASC Locklosses:
Caused by turning on the ISS second loop, the only thing of import that happens in that stage (pic 1). I noticed ISS turn-on DRMI locklosses several times over the last couple of weeks appearing in ASC-MICH control, but didn't know the cause until now.
EDIT: Pic 3 shows a DMRI lock, then a DRMI-only lockloss from the ISS turning on, then a DRMI reacquistion.
OFFLOAD_DRMI_ASC Locklosses:
When switching MICH control from AS_B_RF45_Q to AS_A_RF36_Q. Could be because of the smooth limiter limiting the oscillatory error signal too much (pic 2). I disengaged the smooth limiter in this state for now, we'll see if this helps.
EDIT: This seem to have worked once at least.
--------------------------------------------------------------------------------------------------------------------------------
EDIT2: I turned off the ISS second loop in guardian after it caused two DRMI locklosses in a row.
IMC_LOCK.py Line 513: ezca['PSL-ISS_SECONDLOOP_ENABLE'] = 0
The other thing that is happening during these guardian states is that DRMI length control is being switched over to the 3F sensors.
The attached plot shows the LSC signals for the lockloss that Craig is plotting. In this case it looks ambiguous to me which thing causes the lockloss. (The ISS is enabled after the MICH ASC control signal gets large and the LSC error signal has a large excursion.)
The second attachment shows a very similar lockloss to the one that Craig plotted last night where we waited in the DRMI states (and the ISS was commented out of the guardian). At around -25 seconds, the DRMI control is switched to 3F and all the LSC signals become noisier. Immediately after the transition to 3F, there is a large glitch in the LSC signals when the coil driver state switches at -23 seconds (see 4th plot in top row). After this, the M2 gain (which is really just an input gain on the signal going to M1 since we don't send the length control to M2) is increased on PRM and SRM, with a 10 second ramp. The large distubance in MICH seems to start 10 seconds after the M2 gain ramp should be finished, so I am not sure what caused it.
For now I have done 2 things, neither of which I expect to solve the problem: I added the ISS second loop back into the guardian because we don't think it's the problem. I also edited the wait in the DRMI guardian that was intended to wait for the gain ramping to finish, but was shorter than the ramp time.
TITLE: 12/16 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Struggled to get locks longer than a few minutes in Low Noise. DRMI would lock quickly, but not with any robustness. Bringing the ASC error signals may help a bit. Currently Dan Brown and Danny Vander-Hyde are changing the ring heater settings back from the changes they made last night. The most recent lock loss we had just as we reached NLN, Verbal gave us a LSC tidal X error. I trended the channel that verbal looks at, and it did seem to go into error a few times around the lock loss. The last error that the channel has seen was around 14:32 UTC today, and there have been many more before that. The Verbal logs dont seem to have these show up though. Perhaps this is somewhat common during lock losses, but Verbal just picked it up this time?
LOG:
Danny and I did like I have done in the past and tried to minimize the ASC error signals as best we could before turning on the DRMI ASC. Hopefully this will hold a bit better.
Lock Loss 2256 UTC
Back it back by slowly walking through DRMI again.
Lockloss at 2044 UTC. It looks like maybe DHARD or DSOFT started to become unstable?
Although according to one of our lockloss tools, it seems that the LSC started to see overflows before the ASC. I have no idea what 30 & 31 are though.
I walked through the DRMI states slowly, one by one, and it made it all the way up without issue. I've accepted the following SDF settings in the attached screen shots.
Well that was short. Lock loss at 19:40:57 UTC
I'm not sure if it is the high useism, or something else, but I am having issues getting much past DRMI. I was able to make it past the CARM reduction one time, but since then I have been struggling with getting a stable DRMI. With DRMI locked, POP18 and POP90 have large, ~0.1Hz, period to them. This seems suspicious of the useism, but I cannot say for sure.
TITLE: 12/16 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
Wind: 8mph Gusts, 5mph 5min avg
Primary useism: 0.16 μm/s
Secondary useism: 1.12 μm/s
QUICK SUMMARY: useism is back up, ISC_LOCK state was in Locking_arms_green when I arrived. The ALS_YARM node was managed by USER?
Seems like there was a good amount of triple coincident data last night, awesome!
Dan Brown, Cheryl, Craig All three observatories have switched on the OBSERVING bit, and are acquiring triple coincident data. (pic 1) #ER13 Some notes from Hanford Saturday Night Locking: - I fixed the INCREASE_POWER state so that is actually always goes to the power set in lscparams.input_power['NLN'], which is 20 watts currently. Previously it would stop anywhere between 10 and 20 because of a race condition within the state. We lost lock once while trying to get through LOWNOISE_ASC with only 10 watts. - Dan and I tried engaging the DRMI ASC straight after locking DRMI (INP1, PRC1/2, SRC1/2) using the settings that are in there for full IFO. PRC1/2 were able to close, but made the alignment worse. SRC1/2 killed the DRMI lock straightaway. - MICH ASC sensor transitions are extremely rough, and have caused some locklosses right after acquiring DRMI. - We keep having FAST SHUTTER DID NOT CLOSE guardian warnings, even though the fast shutter appears to be working fine. This causes us to linger in DOWN forever rather than continuing to lock. This seems to be happening because the LOCKLOSS_SHUTTER_CHECK guardian is always unhappy with the results of its shutter check test. - Dan changed the ITMY ringheater set value from 0.89 W to 1.15 W for both upper and lower heaters at 2018/12/16 1:43:54 UTC. Waiting for things to thermalize to see if this was a move in the right direction. EDIT: - We've had two locklosses tonight, neither of which we understand. There was nothing too suspicious in any of the ASC or LSC signals, the microseism is super high right now, 1 μm/s. I checked the drift monitors for locks so far, nothing seems to out of the ordinary, maybe SR2 is drifting more than the other optics during the course of the locks, could the significant differential heating of our ITMs be responsible? (pic 3, 4). - After locklosses, the FSS seems to be losing lock and remaining down for ~4 minutes or so. The laser crystal temperature seems to be jumping around a lot when it gets close to the correct values (pic 2). I don't understand enough about the FSS autolocker to change anything, but it used to reacquire in under a minute. - We just recovered the IFO in 28 minutes without us touching it. EDIT 2: - We lost lock again after about an hour. It's possible that this lockloss was due to INP1, PRC1, and PRC2 pitch, but I'm not certain. It seems like the locklosses tonight have been due to ASC, because the PR gain and POPAIR 18 and 90 respond slightly before the lockloss, and have been "dippy", (i.e. slow fluctions in power) all night. I accuse PRC1 and PRC2 pitch because they all happened to hit local extrema at the same time as the maximum dip in the power monitors (pic 5). The problem is these dips don't seem particularly egregious by themselves, especially considering the previous time series of these error signals.
H1 entered Observe: 2018-12-16 5:21:50 UTC, 1228972928
SDFs cleared by accepting changes (roughly 22:58:00 UTC to 23:06:22 UTC), snapshots of all diffs attached, excitation testpoint was open, H1:LSC-EXTRA_AO_2_EXC in a diaggui, but no excitation was active, and the DIAG_EXC guardian cleared when the diaggui was closed, plot showing OBSERVATION_READY and H1:LSC-EXTRA_AO_2_EXCMON attached.
ALS Y has been glitching again tonight
After touching base with Keita, I have turned on CW hardware injections into H1 (scheduled to begin at GPS 1228959604). CW injections are now running at both observatories.
Here are time series showing the turn-on of H1 injections with L1 injections in background (CW_OUT and HARDWARE_OUT)
I discovered an error in my configuration of the CW injection files for ER13 that led to a large frequency offset in all of the injected signals. That error has now been fixed and the injections restarted. CW injections made before GPS 1229023116 (19:18:18 UTC) should be ignored. The attached figure shows the latest turn-on point
