david.barker@LIGO.ORG - posted 10:32, Saturday 02 December 2023 - last comment - 12:06, Saturday 02 December 2023(74539)
ETMX Hardware Watchdog Tripped at 07:23 this morning
We are investigating why the ETMX HWWD saw high SUS RMS for 20 minutes which caused it to trip, whereas all the other HWWDs saw less than 1 minute of high RMS. It would appear that ETMX's high RMS stopped within one minute of the time the HWWD powered down the ETMX ISI coil drivers.
Jim, Fil, Tony, Dave:
Tony spoke with Jim on the phone, I spoke with Fil, we have a procedure to untrip the ETMX HWWD:
We are keeping the EX SWWD tripped at this point (no ISI or HEPI DAC drives)
Tony will go to EX with a CDS laptop, communicating via teamspeak
Tony to verify initial state: HWWD is tripped, and ISI coil drivers are powered down
Tony will untrip the HWWD via its front panel button, verify the coil drivers spring back to life. If any need a reset, they will be reset.
Wait for several minute to see if SUS ETMX RMS rings up
Untrip the SWWD so DAC drive is restored, wait several minutes to see if RMS rings up
Tony heads back to the control room.
If at any point the RMS rings up, Tony will power down the ISI coil drivers via their back panel power switch, one at a time to see if we can identify which unit(s) is doing this. At this point Jim and/or Fil will need to go to the site.
Timeline of ETMX watchdogs (all times local)
07:04:20 HWWD and SWWD detect high SUS motion RMS and start their countdowns
07:09:22 SWWD h1iopsusex 1st countdown completes, starts h1iopseiex countdown
07:14:21 SWWD h1iopseiex countdown completes, all ISI and HEPI DAC drives are zeroed. RMS continues
07:24:01 HWWD 20 minute countdown completes, all three ISI Coil Driver Chassis are POWERED DOWN
07:24:31 RMS motion of SUS ETMX below trip point for both SWWD and HWWD.
11:27 Tony untripped the ETMX HWWD, all came back with no problems.
After 2 minutes with no ring-up, I untripped the SUS and SEI SWWD, no problems.
Tony started both ISI and HEPI drives, again no problems
After 5 minutes Tony left EX and is heading back to the control room.
Here is a trend of SUS ETMX top OSEM RMS channels from 07:00 to 07:30.
All OSEMs ring up and exceed their 110mV trip level at 07:04:20. The F1, F2, F3 OSEMs keep the trip active throughout, the others (LF, RT, SD) ring back down. The trip+20minute time mark is shown, at which time the ISI Coil Drivers are powered down. The RMS ring down to below the trip level over the next 34 seconds.
While trying to figure out what eactly was wrong and the proper order to resetting all of these systems. I spotted a ring up happening. We have a before and an after of DCPDs.