jason.oberling@LIGO.ORG - posted 09:56, Tuesday 15 December 2015 (24235)
H1 PSL FE Watchdog Reset
I reset the 35W FE watchdog today at 17:34 UTC (9:34 PST).
I reset the 35W FE watchdog today at 17:34 UTC (9:34 PST).
We have lost lock a few times when the SR3 cage servo becomes railed, and sometimes while trying to releave.
Since there is more range, I moved the actuation to M1 (WP 5657). Since we use M1 TEST P for alingment offsets, I used M1 DITHER instead. I reduced the gain from 300 to 1, this seems to be working fine.
TITLE: 12/15 OWL Shift 08:00-16:00UTC (00:00-08:00 PST), all times posted in UTC
STATE Of H1: Down for Maintenance
SUPPORT:
END-OF-SHIFT SUMMARY: One lockloss towards the end of shift due to an earthquake in Vanuatu. Otherwise it's been a quiet night. Sheila arrived early to measure some transfer function so Maintenance started early (LLO was already down). Mike wasn't very happy. Reminder to all the operators to check with him before making such decision.
Activity:
14:00 Richard driving down Y arm to check on some bright light.
14:40 Stopped at ENGAGE_ASC_PART3 for Sheila
15:55 Lockloss. Sheila's driving ETMs.
16:00 Jeff B. driving a folklift from LSB to warehouse.
Sheila arrived and has started working on WP#5563 (driving the ETMs). Since LLO is down I guess our Maintenance can start early.
I turned on the damping for 501.749 Hz (ITMY Mode6 - this also damp 501.811 Hz) since the higher harmonics started to show up. The mode became even higher after the previous lockloss.
Not sure what happen here. ASC loop won't engage.
I followed ISC_DRMI graph and re-request OFFLOAD_DRMI. That did the trick. Now the Guardian has moved on.
Cause not obvious. I don't see anything harmful looking in the environment channels except for the traffic is picking up on Rt.10.
LLO called to let me know that they will be starting the Maintainence activity in an hour since they've recently recovered from a lockloss.
LLO's down for Maintennance
The cause of lockloss was a 5.2M earthquake in Vanuatu btw. Terramon updated that a little slow.
Still Observing and going strong. Low wind (near 0mph). Useism is stabilizing at 50th percentile. 501.749 Hz violin mode is high but not ringing up as far as I can tell (now ~3 order of magnitudes above its noise floor).
TITLE: 12/14 EVE Shift: 00:00-08:00UTC (16:00-00:00PDT), all times posted in UTC
STATE of H1: NLN @ 80Mpc.
Incoming Operator: Nutsinee
Support: Sheila On Phone at beginning of shift
Quick Summary:
Shift Log:
Corey have asked me to followup the lockloss at 6:23 UTC which appeared to have no obvious cause. I ran the generic template lockloss tool and DHARD pitch seems to be the first of the control signal to go awry (Fig1 and 2). Hang's lockloss tool suggested OMC_A/B_SUM_OUT seems to be the first to go (Fig3). Andy's saturation tool suggested ETMY L3 to have gone first (Fig4). Spectrogram from the summary page didn't show anything obvious right before the lockloss (Fig5). Even Omicron didn't seem to notice anything out of ordinary. However, GND STS channel seems to have noticed something at ETMY in the EQ band around the time of the lockloss (Fig 6 and 7). There were no reports from either USGS or Terramon of any earthquakes prior to this lockloss. GND STS at HAM2, hAM5, and ITMY are seeing this too.
C. Cahillane I have produced the total LHO uncertainty budget, now including Pcal sensing information at 1 kHz and 3 kHz. (Thanks Shivaraj!) C01 means the kappas were left uncorrected in the uncertainty propagation. C02 is the "perfect" calibration model, with kappas corrected. C00 is the "nominal" model output by the front end, which has uncorrected kappas AND static frequency dependent systematics left in. PDFs 1 and 2 each contain nine plots. Plots 8 and 9 are the total magnitude and phase uncertainty in strain. Plots 6 and 7 are the components that make up the uncertainty. Plot 1 is the C00 response divided by either the C01 or C02 response to show the frequency dependent systematic errors that were corrected. PDF 3 is the sensing measurements and fits. (We have a nonlinear fit for the sensing function magnitude, which results in this dominating the uncertainty budget.) PDF 4 is the actuation measurements and fits. PDF 5 is the direct comparison of C01 and C02. This basically shows the total effect correcting the kappas has on the response function. LHO has a total uncertainty under 6% and 3.5 degrees (See plots 8 and 9 from PDF 2) For the LLO budget please see aLOG 23573
This is per the Ops Daily Check Sheet for a task to be done by a Monday Operator.
Laser Status:
SysStat is good
Front End power is 30.93W (should be around 30 W)
Frontend Watch is GREEN
HPO Watch is RED
PMC:
It has been locked 20.0 days, 13.0 hr 48.0 minutes (should be days/weeks)
Reflected power is 1.751Watts and PowerSum = 24.67Watts.
FSS:
It has been locked for 0.0 days 1.0 h and 21.0 min (should be days/weeks)
TPD[V] = 1.429V (min 0.9V)
ISS:
The diffracted power is around 8.255% (should be 5-9%)
Last saturation event was 0.0 days 1.0 hours and 21.0 minutes ago (should be days/weeks)
Intentionally dropped from Observing Mode to address SR3 Cage Servo, but then accidentally had a lockloss while addressing this servo. Out of Observing for roughly 3hrs. Here's timeline:
For Initial Alignment noticed the ALS arms were off in pitch and ugly, but after taking care of them, only other noticeable variance was the Dark Michelson was also off in pitch. Restored this by looking at the AS Camera and the AS_C medm window's QPD, and steering the SR2 in pitch quite a bit (from 1938 to 2022); but this is what was needed to bring the spot back down in the center of the camera. From here on, it was normal work.
Locking was usual time sink when we have high useism (but it's dropping). During one drop, Guardian was in the middle of DC_READOUT_TRANSITION & giving OMC not locked error. I re-requested READY_FOR_HANDOFF, but H1 dropped. Think next time, we should select DC_READOUT_TRANSITION on the ISC_LOCK guardian (instead of having NOMINAL_LOW_NOISE), so that we don't fight the Guardian while locking the OMC.
At this point, we could have opted to try the new FULLY_ISOLATED state for the BS ISI, but we had been out of Observing for so long, that we decided to hold this off until Maintenance tomorrow.
H1 made it to NLN on the next lock. There were a few known SDF diffs: BS ISI time ramps (which I returned to 0sec), and the new filters Evan added for the test mases (I ACCEPTED these).
Over last 12hrs, useism has been trending down (and maybe flattening for the last 2hrs) and is at ~0.4um/s for the LVEA & winds are hovering below 10mph.
The SR3 cage servo has gone beyond its limit, and is saturating the output of the SR3 M2 coils.
Back in November (aLog 23131) I put a check into the SR3_CAGE_SERVO guardian to let the operator know when it's close to the limit, and that it needed to be offloaded to the M1 actuators. However, I only put in a check for if the servo is getting close to the positive limit. As of 12 Dec around 18:00 UTC, we've been railed at the negative limit. Ooops.
Right now, SR3 is about 13 urad from the usual value in pitch (currently the OSEMs read 935, rather than the 922 that is the cage servo's setpoint), but the ASC seems to be fine with it.
At the next opportunity, we'll offload SR3's alignment, and fix the guardian code so that it checks for both the positive and negative limits.
In the attached figure, you can see that SR3 is no longer at its nominal pointing, and that the cage servo's setting is no longer within +/-15,000 counts (which corresponds to +/-130k counts at the DAC).
Trying to fix this broke the lock :(
Even though Corey and I typed a number (-16,000) into the SR3_M2_TEST_P_OFFSET (to get the cage servo closer to zero) that should have left the coil actuators railed, 2 of the coils seem to have flipped sign. This broke the lock. I'm still looking into why typing -16,000 didn't do what I expected.
I have been looking into the lockloss that I caused earlier when Corey and I were offloading the SR3 cage servo, because I don't think that it should have caused a lockloss.
I think that I have found a mysterious sign flip in the DAC output, which I'm hoping that someone will have an idea about. Somehow, the LR and UR DAC outputs switched sign a few days ago, even though the MASTER_OUTs did not flip sign. See first attachment for a 3 day trend showing this.
When Corey typed -16,000 into the SR3_M2_TEST_OFFSET, the signs of the LR and UR DAC outputs flipped to correct themselves. See second attachment for a 20 minute trend showing this. The COILOUTFs are the same as the MASTEROUTs, and they stay whatever sign they started (LL and LR positive, UL and UR negative). However, the DAC_OUTPUTs 6 and 7 (6=UR, 7=LR) flip sign. This undoes the mysterious sign flip that happened a few days ago.
Why is the DAC output flipping sign, when the digital signals going to them are not?!?!
