oli.patane@LIGO.ORG - posted 15:29, Sunday 06 August 2023 - last comment - 16:15, Sunday 06 August 2023(72010)
Lockloss
Everything looking normal now, we've been Locked for 1hr 25mins.
Sun Aug 06 10:11:29 2023 INFO: Fill completed in 11min 25secs
TITLE: 08/06 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: SEISMON_ALERT
Wind: 5mph Gusts, 3mph 5min avg
Primary useism: 0.07 μm/s
Secondary useism: 0.07 μm/s
QUICK SUMMARY:
Ryan S is heading over to the CER to restart the High Voltage from the lockloss(72000). Major dustmon alarm on for the PSL even though winds are low.
18:28UTC Reached Observing
I noticed that the dustmons for the PSL laser room spiked dramatically within 10 minutes of the power glitch even though wind was low at the time (attachment). The dust counts then went back down a few minutes after the NPRO and amps were restored in the PSL. We don't see how dust could jump after just a power glitch, and Austin suggested that it may just be some strange cross-coupling between the electronics in the PSL laser room and the dustmon in there.
When I found H1 this morning following the lockloss @ 13:34 UTC, the PSL was entirely down (NPRO off, verbal alarms mentioned "noise eater out of range" immediately prior to lockloss). I was able to restart the NPRO and both amps just fine, but when I got to relocking the PMC, I noticed the high voltage was dead (see first attachment). Thinking this may have been caused by a power glitch, I trended the mains voltage channels and sure enough that looks to be the case (see second attachment).
Jenne and I have turned on the PMC HV before following a power outage (alog 65026), so I'm headed to site and will start there for recovery.
I went to the upstairs CER rack C5 and indeed found the 375V high voltage supply off, so I restarted it. Once back in the control room, I was able to relock the PMC, FSS, ISS, and IMC without issue. Fortunately, it seems the PSL was the only thing impacted by the power glitch.
Oli has started relocking H1.
Still working on relocking; ISC_LOCK is not able to complete PREP_DC_READOUT_TRANSITION due to the OMC_LOCK guardian getting stuck at FIND_CARRIER many times and not being able to find the 45MHz carrier. OMC_LOCK log output for each time it goes through this cycle is attached as a txt.
DIAG_MAIN has a commented out error message that would alert us if OMC-PZT2_MON_DC_OUT is <1 and GRD-OMC_LOCK_STATE_N is not at 3 (OFF_RESONANCE), which we are at in this case (see attachment1). Due to the low voltage output and the old DIAG_MAIN error message that the problem is due to the PZT2 high voltage probably not being restored correctly.
OMC_DCPD_SUM_OUTPUT reads ~0.0055 at this state even though it should've started increasing once we got to ENGAGE_ASC_FOR _FULL_IFO.
Currently went back to DOWN and started the relocking process again, stopping ISC_LOCK at ENGAGE_ASC_FOR _FULL_IFO, but it looks like we are still having the same issue. Looking through alogs from previous times this happened.
No email from MSR UPS, but GC UPS reported (hour corrected for PDT)
Date : 08/06/2023
Time : 06:34:41
Warning - UPS: On battery power in response to an input power problem.
Date : 08/06/2023
Time : 06:34:45
Informational - UPS: No longer on battery power.
Corner station mainsmon plot attached. End station VFD units did not trip this time.
Reset High Voltage for the OMC and we are back on our way up
18:28UTC Reached Observing
H1 was out of observing from 08:04 UTC to 08:41 UTC because the SQZ ISS had saturated. Following the instructions in alog 70050, I was able to recover the squeezer and resume observing. The opo_grTrans_setpoint_uW within sqzparams.py is currently set to 50, and attached is the SDF diff I accepted for the adjusted OPO TEC temperature.
H1 back observing at 151Mpc.
TITLE: 08/05 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:
- Following a lockloss (possibly caused by an ASC ringup), I began relocking:
- EX saturations @ 3:21/4:30
- Lockloss @ 5:44
- Lost lock at MOVE SPOTS with suspiscious SRM/BS saturations upon relocking. Given all the instability tonight, I will be doing an IA
- H1 is currently relocking just fine (currently at ENGAGE ASC), I imagine with the IA, things should be much more stable now
LOG:
No log for this shift.
Lockloss @ 5:44, was seeing motion again on the ASC loops on the CR screen, attached scope, the SRC and SOFT loops look particularly unstable.
Following a lockloss, H1 just got back to NLN (mostly automated, had to touch ALS Y just a hair) and should be in OBSERVE within the next few minutes.
Lockloss @ 1:50 UTC, was noticing some instability in the ASC loops on the CR screen, particuarlly for the SRC and HARD loops, though can't say for certain if this caused the lockloss.
Anamaria's observation that LLO DARM RMS is affected by the SRCLFF made me look again at the same well known fact at LHO. In the past we reduced a large 1.34 Hz line in DARM by improving the SRLCFF highpass (LHO:71565). So I was curious to see what's the current situation. Spoiler: not great.
The first plot compares the control signals sent to DARM: DARM_OUT, SRCLFF1_OUT and MICHFF_OUT. One can see that DARM_OUT lines up with SRCLFF between 3 and 8 Hz, and is of similar amplitude as with MICHFF between 1 and 2 Hz. So it looks like most of the actuation to DARM is actually due to the SRCLFF, and the DARM loop is probably counteracting that force (indeed the transfer function between DARM_OUT and SRCLFF1_OUT is -1 between 1 and 8 Hz) . SRCLFF seems to be doing a good job of subtracting SRCL from DARM above 8 Hz, but doing nothing good below 8 Hz.
The second plot compares the DARM input and output signals during a brief time when the MICH and SRCL FF were off (yesterday during my measurements, so SRCLFF was actually off, while MICHFF was used for some noise injection above 10 Hz). DARM residual motion is lower between 1.5 and 8 Hz when the FF are off. The total RMS of DARM residual motion is increased by a factor 2.2 by the LSC FF (mostly SRCL)
How can we fix this?
Another question is of course: should we fix this? Or in other words, is it a problem? LLO saw evidence of noise modulation because of this effect (LLO:66597)
SRCL is highly coherent with SRM DAMP_L at the 1.3 Hz peak and below: SUS-SRM_M1_DAMP_L_IN1_DQ
And highly coherent with SR2 DAMP_P between 3 and 8 Hz: SUS-SR2_M1_DAMP_P_IN1_DQ
TITLE: 08/05 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
INCOMING OPERATOR: Austin
SHIFT SUMMARY:
Besides the one lockloss and a number of earthquakes that we were able to ride out, the shift was pretty quiet.
15:04UTC Detector Observing and Locked for 2hours
17:08:49 syscssqz diffs took us out of Observing but promptly disappeared as usual
17:09:00 I put us back into Observing
17:45 Red alarm for low temps at MY (71986)
17:55 Lockloss (71983)
18:58 Reached NOMINAL_LOW_NOISE
19:11 Observing
19:19 Earthquake mode activated due to incoming earthquake
19:29 Back to CALM
21:14:08 syscssqz diffs pushed out out of Observing
21:14:16 I put us back into Observing
21:36 Low temp alarm for MY again
LOG:
no log
TITLE: 08/05 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 145Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 9mph Gusts, 6mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.05 μm/s
QUICK SUMMARY:
- H1 has been locked for 4 hours
- CDS/DMs ok
- Few EQs today, but ground motion looks to be going down
Looking quickly at peakmon, lockloss occurred at crosshair mark on attachment. Spikes are relatively small but sharp.