I was unable to lock today for several reasons:
Solved
- DC centering caused RM2 was to rail both in pitch and yaw, which wrecked the PRM alignment while going through initial alignment. It took me a while to figure out that RM2 was railed since there are no audible warnings and the suspensions monitor TV was frozen. I reset everything to two days ago, seemed fine and I was able to get through initial alignment.
-The FSS was not locking for extended periods today. The fast monitor was just sitting at 11 V while the loop rapidly shut itself on and off over and over. I upped the oscillation threshold from 0.6 V to 3 V, this allowed the loop to close for a decent amount of time and actually lock the FSS.
Unsolved
- Something weird is going on with the fiber polarization: DIAG_MAIN reports ALS Y polarization is > 20%. I haven't looked into this because of the next issue.
- At GPS 1220076426, I was locking arms green and X-arm transmission went to 0 and never returned. The X-arm guardian gave messages "PDH; ReflPD A; Green input align": there is no green light coming back from ETMX. It seems the HV driver for the ETMX ESD is off and cannot be reset. The ISC_LOCK guardian in PREP_FOR_LOCKING is continually resetting the ETMX ESD HV, so I've paused the guardian for now. I shut off control signals to ETMX L3 stage as well.
ISC_LOCK guardian repeated message
2018-09-04_07:33:42.929020Z ISC_LOCK [PREP_FOR_LOCKING.run] ETMX bias not at expected value
2018-09-04_07:33:42.929381Z ISC_LOCK [PREP_FOR_LOCKING.run] ETMX ESD Driver is OFF. Hitting RESET to turn it ON.
2018-09-04_07:33:42.929789Z ISC_LOCK [PREP_FOR_LOCKING.run] ezca: H1:SUS-ETMX_BIO_L3_RESET => 1
2018-09-04_07:33:42.930095Z ISC_LOCK [PREP_FOR_LOCKING.run] ETMX ESD Driver is being reset. Sleep for 2 sec.
Coming from Line 438 of ISC_LOCK:
else: # Make sure ETMX ESD is on
if abs(ezca['SUS-ETMX_L3_ESDAMON_DC_INMON']) < 20000: # ETMX ESD is not ready
log('ETMX bias not at expected value')
if -17000 < ezca['SUS-ETMX_L3_ESDAMON_DC_INMON'] < -15000: # Check if railed negative
notify('ETMX ESD is potentially railed!')
elif abs(ezca['SUS-ETMX_L3_ESDAMON_DC_INMON']) < 1000: # In case requested DC bias completely turned off
if abs( ezca['SUS-ETMX_L3_LOCK_INBIAS'] ) < 5.0:
log('SUS-ETMX_L3_LOCK_INBIAS too small. Check the requested epics value')
self.timer['wait'] = 2
else:
log('ETMX ESD Driver is OFF. Hitting RESET to turn it ON.')
ezca['SUS-ETMX_BIO_L3_RESET'] = 1
log('ETMX ESD Driver is being reset. Sleep for 2 sec.')
self.timer['wait'] = 2 # wait two seconds for the microcontroller to take action
return # Exit and re-execute run to make check again
I agree with RichardM that we need to reboot EX's IOP model. Hopefully that will fix things. (He drove to the end earlier this morning, and all the hardware looks good).