Not sure what happen here. ASC loop won't engage.
I followed ISC_DRMI graph and re-request OFFLOAD_DRMI. That did the trick. Now the Guardian has moved on.
Cause not obvious. I don't see anything harmful looking in the environment channels except for the traffic is picking up on Rt.10.
LLO called to let me know that they will be starting the Maintainence activity in an hour since they've recently recovered from a lockloss.
LLO's down for Maintennance
The cause of lockloss was a 5.2M earthquake in Vanuatu btw. Terramon updated that a little slow.
Still Observing and going strong. Low wind (near 0mph). Useism is stabilizing at 50th percentile. 501.749 Hz violin mode is high but not ringing up as far as I can tell (now ~3 order of magnitudes above its noise floor).
TITLE: 12/14 EVE Shift: 00:00-08:00UTC (16:00-00:00PDT), all times posted in UTC
STATE of H1: NLN @ 80Mpc.
Incoming Operator: Nutsinee
Support: Sheila On Phone at beginning of shift
Quick Summary:
Shift Log:
Corey have asked me to followup the lockloss at 6:23 UTC which appeared to have no obvious cause. I ran the generic template lockloss tool and DHARD pitch seems to be the first of the control signal to go awry (Fig1 and 2). Hang's lockloss tool suggested OMC_A/B_SUM_OUT seems to be the first to go (Fig3). Andy's saturation tool suggested ETMY L3 to have gone first (Fig4). Spectrogram from the summary page didn't show anything obvious right before the lockloss (Fig5). Even Omicron didn't seem to notice anything out of ordinary. However, GND STS channel seems to have noticed something at ETMY in the EQ band around the time of the lockloss (Fig 6 and 7). There were no reports from either USGS or Terramon of any earthquakes prior to this lockloss. GND STS at HAM2, hAM5, and ITMY are seeing this too.
C. Cahillane I have produced the total LHO uncertainty budget, now including Pcal sensing information at 1 kHz and 3 kHz. (Thanks Shivaraj!) C01 means the kappas were left uncorrected in the uncertainty propagation. C02 is the "perfect" calibration model, with kappas corrected. C00 is the "nominal" model output by the front end, which has uncorrected kappas AND static frequency dependent systematics left in. PDFs 1 and 2 each contain nine plots. Plots 8 and 9 are the total magnitude and phase uncertainty in strain. Plots 6 and 7 are the components that make up the uncertainty. Plot 1 is the C00 response divided by either the C01 or C02 response to show the frequency dependent systematic errors that were corrected. PDF 3 is the sensing measurements and fits. (We have a nonlinear fit for the sensing function magnitude, which results in this dominating the uncertainty budget.) PDF 4 is the actuation measurements and fits. PDF 5 is the direct comparison of C01 and C02. This basically shows the total effect correcting the kappas has on the response function. LHO has a total uncertainty under 6% and 3.5 degrees (See plots 8 and 9 from PDF 2) For the LLO budget please see aLOG 23573
This is per the Ops Daily Check Sheet for a task to be done by a Monday Operator.
Laser Status:
SysStat is good
Front End power is 30.93W (should be around 30 W)
Frontend Watch is GREEN
HPO Watch is RED
PMC:
It has been locked 20.0 days, 13.0 hr 48.0 minutes (should be days/weeks)
Reflected power is 1.751Watts and PowerSum = 24.67Watts.
FSS:
It has been locked for 0.0 days 1.0 h and 21.0 min (should be days/weeks)
TPD[V] = 1.429V (min 0.9V)
ISS:
The diffracted power is around 8.255% (should be 5-9%)
Last saturation event was 0.0 days 1.0 hours and 21.0 minutes ago (should be days/weeks)
Intentionally dropped from Observing Mode to address SR3 Cage Servo, but then accidentally had a lockloss while addressing this servo. Out of Observing for roughly 3hrs. Here's timeline:
For Initial Alignment noticed the ALS arms were off in pitch and ugly, but after taking care of them, only other noticeable variance was the Dark Michelson was also off in pitch. Restored this by looking at the AS Camera and the AS_C medm window's QPD, and steering the SR2 in pitch quite a bit (from 1938 to 2022); but this is what was needed to bring the spot back down in the center of the camera. From here on, it was normal work.
Locking was usual time sink when we have high useism (but it's dropping). During one drop, Guardian was in the middle of DC_READOUT_TRANSITION & giving OMC not locked error. I re-requested READY_FOR_HANDOFF, but H1 dropped. Think next time, we should select DC_READOUT_TRANSITION on the ISC_LOCK guardian (instead of having NOMINAL_LOW_NOISE), so that we don't fight the Guardian while locking the OMC.
At this point, we could have opted to try the new FULLY_ISOLATED state for the BS ISI, but we had been out of Observing for so long, that we decided to hold this off until Maintenance tomorrow.
H1 made it to NLN on the next lock. There were a few known SDF diffs: BS ISI time ramps (which I returned to 0sec), and the new filters Evan added for the test mases (I ACCEPTED these).
Over last 12hrs, useism has been trending down (and maybe flattening for the last 2hrs) and is at ~0.4um/s for the LVEA & winds are hovering below 10mph.
There are two quick tasks we want to do on the way to a full lock, isolate BS ISI ST2, and measure PRCL and MICH gains in DRMI.
Isolated BS ISI stage 2, (WP 5652). We did this because of the upconversion that Robert saw in DARM when shaking the BS ISI, 24194. We did this without any problems, then the IFO broke lock durring an epics freeze while locking the OMC, so we didn't get a chance to see if it had an impact on DARM.
I also grabbed some OLG measurements with DRMI locked on 1F, as described in WP5654. (motivated by alog 24136)
The PRCL gain isn't very different, the main difference is a boost that is on in DRMI but not PRMI. The MICH gain is about 10dB high though, resulting in a UGF just above 15 Hz for DRMI and around 5.5 Hz for PRMI. Maybe we should try lock DRMI with a lower MICH gain and see if it catches better.
Since we occasionally need to adjust the ALS comm offset in order to have IR resonating in the arms, I have (finally) added the comm offset slider to the ALS overview screen. Operators are familiar already with the diff offset slider that has been there, but now if comm has an issue, we no longer have to go hunting to remember which is the right slider.
Despite turning off all violin mode damping in full lock, some of the fundamental modes appear to not ring down over long locks.
There is a very slight drive to the PUM coils from the ASC signals, which I have now notched out at 500 Hz (using FM5 in the SUS-[E|I]TM[X|Y]-L2-LOCK_[P|Y] filter modules). It's not clear that this drive is the culprit, however.
TITLE: 12/14 EVE Shift: 00:00-08:00UTC (16:00-00:00PDT), all times posted in UTC
STATE of H1: 21+hr lock?
Outgoing Operator: TJ
Quick Summary:
H1 was doing fine, but TJ/Jenne mentioned a railed value for the SR3 Servo. So we wanted to proactively address this, whenever L1 dropped out of Observing. Doug L eventually called to let us know about them dropping out to do A2L work, and we took that opportunity to address the SR3 Cage Servo......unfortunately, when we changed an OFFSET value to -16k, we immediately dropped out of lock (I'm letting Jenne address specifics in her alog). Before dropping out of lock, Jenne loaded new code to the SR3 Cage Servo Guardian to be better at notifying us when we are railed.
OK, going to look at alignment now (because the ALS arms both look ugly in pitch).
Attached are 7-day pitch, yaw, and sum trends for all active H1 optical levers.
TITLE: 12/14 Day Shift: 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC"
STATE Of H1: Observing at 75Mpc for 20.5hrs
SUPPORT: Patrick, Vern, Jenne
SHIFT SUMMARY: Quiet except for an EX ISI oscillation, the blends were changed and seemed to have made it better. The blends for EX only are now in the Quite_90s.
INCOMING OPERATOR: Corey
ACTIVITY LOG:
The SR3 cage servo has gone beyond its limit, and is saturating the output of the SR3 M2 coils.
Back in November (aLog 23131) I put a check into the SR3_CAGE_SERVO guardian to let the operator know when it's close to the limit, and that it needed to be offloaded to the M1 actuators. However, I only put in a check for if the servo is getting close to the positive limit. As of 12 Dec around 18:00 UTC, we've been railed at the negative limit. Ooops.
Right now, SR3 is about 13 urad from the usual value in pitch (currently the OSEMs read 935, rather than the 922 that is the cage servo's setpoint), but the ASC seems to be fine with it.
At the next opportunity, we'll offload SR3's alignment, and fix the guardian code so that it checks for both the positive and negative limits.
In the attached figure, you can see that SR3 is no longer at its nominal pointing, and that the cage servo's setting is no longer within +/-15,000 counts (which corresponds to +/-130k counts at the DAC).
Trying to fix this broke the lock :(
Even though Corey and I typed a number (-16,000) into the SR3_M2_TEST_P_OFFSET (to get the cage servo closer to zero) that should have left the coil actuators railed, 2 of the coils seem to have flipped sign. This broke the lock. I'm still looking into why typing -16,000 didn't do what I expected.
I have been looking into the lockloss that I caused earlier when Corey and I were offloading the SR3 cage servo, because I don't think that it should have caused a lockloss.
I think that I have found a mysterious sign flip in the DAC output, which I'm hoping that someone will have an idea about. Somehow, the LR and UR DAC outputs switched sign a few days ago, even though the MASTER_OUTs did not flip sign. See first attachment for a 3 day trend showing this.
When Corey typed -16,000 into the SR3_M2_TEST_OFFSET, the signs of the LR and UR DAC outputs flipped to correct themselves. See second attachment for a 20 minute trend showing this. The COILOUTFs are the same as the MASTEROUTs, and they stay whatever sign they started (LL and LR positive, UL and UR negative). However, the DAC_OUTPUTs 6 and 7 (6=UR, 7=LR) flip sign. This undoes the mysterious sign flip that happened a few days ago.
Why is the DAC output flipping sign, when the digital signals going to them are not?!?!