TITLE: 05/18 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Lost lock at the end of the shift, DRMI alignment issues and PRMI was plagued by a ~20Hz buzz that kept killing it after 2 seconds, the mode cleaner was slow to lock in each attempt today. After the 3rd IA DRMI was finally in the mood to lock and I didn't have to go through PRMI and I returned to Observing at 21:15UTC. I was not able to take the calibration measurement this afternoon as planned.
15:25 UTC HAM3 ISI CPS trip while doing green arms in IA. I could not finish SRC align, as SRM was continually saturating. There was also a couple of IY saturations, and there was an earthquake shaking everything at the time.
16:02 UTC HAM3 ISI CPS trip during FIND_IR? I couldn't get PRMI to stay locked for more than 2 seconds.
17:00 UTC I restored the alignments to 05/18/24 03:00 UTC which was right before we locked last night for the 11 hour lock. After locking ALS I could immediately see the beatnotes were better, but DRMI only had small flashes. A few rounds of CHECK_MICH and still bad flashes, 13:31 I started a 2nd IA skipping GREEN_ARMs, there were no saturations during this IA and I was able to finish SRC_ALIGN.
I'm still not able to get PRMI to stay locked for more than 2 seconds, theres a 20ish Hz oscillation that kills it when the MICH1 filter is ramped on? I tried lowering the PRMI-MICH gain from 3.2 to 2.8 in lscparams as Sheila suggested the other day. It lasted longer but was still ultimately killed by the 20Hz buzzing, I lowered it more to 2.6 and it lasted even longer but was killed when PRCL1 turned on? AS AIR was still clearly misaligned so I went through CHECK_MICH which made things worse? I now had worse PRMI flashes and still a clearly misaligned AS AIR. During this I was also trying to help adjust PRM/BS while in PRMI to fix AS AIR.
19:37UTC After some more seemingly fruitless adjusting of PRM and BS I started a 3rd IA... it looks centered on AS-AS_C during SRC and PRC alignment?
Same behavior as before, bad DRMI then PRMI gets killed after a few seconds from the 20Hz, its also always right after the BS_M1_L filter is turned on. Lockloss after a few rounds of PRMI.
For the next attempt I tried turning down the PRMI-PRCL gain from 6 to 4 but then it was able to lock DRMI all of a sudden and we didn't even go through PRMI... so I'll revert these changes. I have no idea whats different this attempt from the last or why it decided now it wants to work but I'll take it. The alignment was just finally good enough, even though it wasn't on the first attempt after the IA finished?
21:15 UTC back to Observing
23:21 UTC lockloss from a HAM3 ISI trip, same as before, CPS sensor tagging SEI
Looking at trends, there were four trips today from HAM3 CPS glitches, none of which can be attributed to earthquakes, as far as I can tell. HAMs are usually not the first to get tripped, and the glitches Ryan posts don't look physical, they look like typical CPS glitches. Tony went out and power cycled the CPS parts of the HAM ISI interface chassis for HAM3, we'll just have to wait and see if that fixes it or if more invasive work is needed.
First attached trends are the high frequency blrms we have for monitoring for glitches, they seem like they would have provided some warning for this, there were glitches an hour or so before the first trip. I think we have some tests for this in DIAG_MAIN? We should think about how to get this to get some more attention.
There is indeed a test in DIAG_MAIN for noisy CPS sensors called "SEI_CPS_NOISEMON" and it was going off last night during the glitches and 2 trips (marked by the T cursors at 02:37 and 02:47 UTC), doing a "guardctrl log -n1000 -a "1400207700" -b "1400210000" DIAG_MAIN" of a time where the CPS was glitching last night yielded the following trace back. Maybe there should be verbal_alarms check for this as well?
2024-05-20_02:34:59.601882Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'V2')]
2024-05-20_02:36:49.102804Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'V2')]
2024-05-20_02:37:01.099752Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H2'), ('HAM3', 'V2')]
2024-05-20_02:37:01.353280Z DIAG_MAIN [RUN_TESTS.run] USERMSG 1: SEI_STATE: ['HAM3'] is not nominal
2024-05-20_02:37:05.341046Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: FAST_SHUTTER_HV: Fast Shutter HV not Ready
2024-05-20_02:37:05.353980Z DIAG_MAIN [RUN_TESTS.run] USERMSG 3: SEISMON_EQ: LSC-REFL_SERVO_SPLITMON > 8V
2024-05-20_02:37:05.354500Z DIAG_MAIN [RUN_TESTS.run] USERMSG 4: SHUTTERS: AS beam shutter open (nominal: closed)
2024-05-20_02:37:05.478436Z DIAG_MAIN [RUN_TESTS.run] USERMSG 3: SEISMON_EQ: IMC-REFL_SERVO_SPLITMON > 8V
2024-05-20_02:37:05.478660Z DIAG_MAIN [RUN_TESTS.run] USERMSG 4: SHUTTERS: Shutter A closed (nominal: open)
2024-05-20_02:37:05.605105Z DIAG_MAIN [RUN_TESTS.run] USERMSG 6: SQUEEZING: SQZ_MANAGER is not in the nominal state. Check we are Squeezing.
2024-05-20_02:37:05.907983Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: ESD_DRIVER: ESD X driver OFF
2024-05-20_02:37:08.857117Z DIAG_MAIN [RUN_TESTS.run] USERMSG 2: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'V2')]
2024-05-20_02:37:09.471870Z DIAG_MAIN [RUN_TESTS.run] USERMSG 2: PSL_FSS: PZT MON is high, may be oscillating
2024-05-20_02:38:30.391003Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:40:33.387179Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:40:41.389505Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:40:54.382921Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:41:06.394464Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:41:11.386640Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:41:34.390778Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:43:51.602046Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H2'), ('HAM3', 'V2')]
2024-05-20_02:44:18.387630Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:47:24.388026Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:47:37.597677Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H2'), ('HAM3', 'V2')]
2024-05-20_02:47:37.854048Z DIAG_MAIN [RUN_TESTS.run] USERMSG 1: SEI_STATE: ['HAM3'] is not nominal
2024-05-20_02:47:42.394967Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:48:21.393532Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:48:33.387284Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:50:27.386254Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:51:25.216172Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_FSS: PZT MON is high, may be oscillating
2024-05-20_02:52:30.383231Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:53:48.383839Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:54:03.858072Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H2'), ('HAM3', 'V2')]
2024-05-20_02:54:04.477258Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H1'), ('HAM3', 'H2'), ('HAM3', 'H3'), ('HAM3', 'V1'), ('HAM3', 'V2'), ('HAM3', 'V3')]
2024-05-20_02:54:05.232099Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H1'), ('HAM3', 'H2'), ('HAM3', 'H3'), ('HAM3', 'V1'), ('HAM3', 'V3')]
2024-05-20_02:54:05.475537Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H1'), ('HAM3', 'H3'), ('HAM3', 'V1'), ('HAM3', 'V3')]
2024-05-20_02:54:10.853656Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H1')]
2024-05-20_02:54:27.392176Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:54:47.394045Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:54:57.724598Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H1'), ('HAM3', 'H3'), ('HAM3', 'V1'), ('HAM3', 'V3')]
2024-05-20_02:55:00.292951Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H1'), ('HAM3', 'H2'), ('HAM3', 'H3'), ('HAM3', 'V1'), ('HAM3', 'V2'), ('HAM3', 'V3')]
2024-05-20_02:55:06.224234Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H2'), ('HAM3', 'H3'), ('HAM3', 'V1'), ('HAM3', 'V2'), ('HAM3', 'V3')]
2024-05-20_02:55:06.356844Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H2'), ('HAM3', 'V1'), ('HAM3', 'V2')]
2024-05-20_02:55:06.482500Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H2'), ('HAM3', 'V2')]
2024-05-20_02:55:08.976174Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: SEI_CPS_NOISEMON: Noisy HAM CPS(s): [('HAM3', 'H2')]
2024-05-20_02:56:27.394588Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
2024-05-20_02:56:56.390876Z DIAG_MAIN [RUN_TESTS.run] USERMSG 0: PSL_ISS: Diffracted power is low
Related: alog75134
