Reports until 00:45, Tuesday 04 September 2018
H1 ISC (ISC)
craig.cahillane@LIGO.ORG - posted 00:45, Tuesday 04 September 2018 - last comment - 08:22, Tuesday 04 September 2018(43798)
ETMX ESD HV Driver Not Working
I was unable to lock today for several reasons:

Solved
-  DC centering caused RM2 was to rail both in pitch and yaw, which wrecked the PRM alignment while going through initial alignment.  It took me a while to figure out that RM2 was railed since there are no audible warnings and the suspensions monitor TV was frozen.  I reset everything to two days ago, seemed fine and I was able to get through initial alignment.

-The FSS was not locking for extended periods today.  The fast monitor was just sitting at 11 V while the loop rapidly shut itself on and off over and over.  I upped the oscillation threshold from 0.6 V to 3 V, this allowed the loop to close for a decent amount of time and actually lock the FSS.

Unsolved
- Something weird is going on with the fiber polarization: DIAG_MAIN reports ALS Y polarization is > 20%.  I haven't looked into this because of the next issue.

- At GPS 1220076426, I was locking arms green and X-arm transmission went to 0 and never returned.  The X-arm guardian gave messages "PDH; ReflPD A; Green input align": there is no green light coming back from ETMX.  It seems the HV driver for the ETMX ESD is off and cannot be reset.  The ISC_LOCK guardian in PREP_FOR_LOCKING is continually resetting the ETMX ESD HV, so I've paused the guardian for now.  I shut off control signals to ETMX L3 stage as well.


ISC_LOCK guardian repeated message
2018-09-04_07:33:42.929020Z ISC_LOCK [PREP_FOR_LOCKING.run] ETMX bias not at expected value
2018-09-04_07:33:42.929381Z ISC_LOCK [PREP_FOR_LOCKING.run] ETMX ESD Driver is OFF. Hitting RESET to turn it ON.
2018-09-04_07:33:42.929789Z ISC_LOCK [PREP_FOR_LOCKING.run] ezca: H1:SUS-ETMX_BIO_L3_RESET => 1
2018-09-04_07:33:42.930095Z ISC_LOCK [PREP_FOR_LOCKING.run] ETMX ESD Driver is being reset. Sleep for 2 sec.

Coming from Line 438 of ISC_LOCK:

else:  # Make sure ETMX ESD is on
    if abs(ezca['SUS-ETMX_L3_ESDAMON_DC_INMON']) < 20000: # ETMX ESD is not ready
        log('ETMX bias not at expected value')
        if -17000 < ezca['SUS-ETMX_L3_ESDAMON_DC_INMON'] < -15000: # Check if railed negative
            notify('ETMX ESD is potentially railed!')
        elif abs(ezca['SUS-ETMX_L3_ESDAMON_DC_INMON']) < 1000: # In case requested DC bias completely turned off
            if abs( ezca['SUS-ETMX_L3_LOCK_INBIAS'] ) < 5.0:
                log('SUS-ETMX_L3_LOCK_INBIAS too small. Check the requested epics value')
                self.timer['wait'] = 2
            else:
                log('ETMX ESD Driver is OFF. Hitting RESET to turn it ON.')
                ezca['SUS-ETMX_BIO_L3_RESET'] = 1
                log('ETMX ESD Driver is being reset. Sleep for 2 sec.')
                self.timer['wait'] = 2 # wait two seconds for the microcontroller to take action
        return # Exit and re-execute run to make check again

Images attached to this report
Comments related to this report
jenne.driggers@LIGO.ORG - 08:22, Tuesday 04 September 2018 (43800)

I agree with RichardM that we need to reboot EX's IOP model.  Hopefully that will fix things.  (He drove to the end earlier this morning, and all the hardware looks good).