Sheila, Ryan S, Ibrahim
Continuation of Transition from EX Failures - The Lockloss causing state
- Sheila came on in the afternoon to help - this time investigating EY L1 and L2 saturations/overflows.
- She found that during this state, which is known to be more delicate, the DAC values to get quite high, sometimes saturating and sometimes (the running idea is). Causing locklosses given already high microseism.
- Even during the last successful lock - at this stage, DAC2 values were still high.
- The filter for this drive is supposed to be a copy of the EX actuator, which does not see such high values, but maybe some difference contributing to instabilities.
- Screenshot: Sheila writes “This is a successful lock acquisition from the other day at the TRANSITION_FROM_ETMX state. The drive to EY L2 is large at the microseism frequency, not enough to saturate the 20 bit DAC though. When we complete the transition to ETMX, the drive is not as large. The EY + IX actuator is supposed to be a copy of the low noise EX actuator, but perhaps there is some difference in the filters. I will look for that now. In recent locklosses it looks like this L2 drive is even larger, probably because the ground is moving more.”
- This seems to support the hypothesis that something is slightly wrong, causing instabilities that would otherwise be invisible and unnoticed if not for the extremely high microseism and wind.
- What might be slightly wrong?
- There are filter differences between the so-called same filters (EY vs. EX L1)
- Sheila added a new filter (screenshot 2), which did successfully stop the counts from getting as high as they did in EY. Screenshot 3 comparison
- Sheila noticed that the EX L3 Cal lines were erroneously on during the transition, when it should have been off
- Further investigation by Ryan S found that L1, L2 and L3 Cal line were all on, incorrectly.
- Ryan S investigated this and found that it was wrongly accepted into SDF and therefore, when PREP_FOR_LOCKING turns it off, SDF_REVERT turns it back on. It should only be turned on during TURN_ON_CALIBRATION_LINES.
- We think these two issues may be the cause and we'll see during the next lock acquisition if they are
- Ryan S is investigating further.
Expanding on the point about the ETMX ESD calibration lines being on, first noticed by Sheila:
The ETMX cal lines should be off for all of lock acquisition up until the TURN_ON_CALIBRATION_LINES state (and turned off while in NLN_CAL_MEAS). However, these were being turned back on during SDF_REVERT after a lockloss as they had been incorrectly accepted as on in the h1susetmx SAFE.snap SDF table. Interestingly, it looks like these were not turned off for some reason when going to NLN_CAL_MEAS during the commissioning time yesterday (see attached ndscope; perhaps contributing to problems mentioned in alog75204?). During this lock acquisition, I set the ETMX L3 cal line gains back to zero and accepted them in SDF (screenshot attached). This appears to have fixed the problem with locklosses during TRANSITION_FROM_ETMX which have been plaguing us since yesterday, as H1 has just reached low noise and started observing.
Thanks to Ibrahim and Ryan for following up on these problems and aloging this.
Two things to add to what Ibrahim said about the EY filter: I plotted (and looked at) only one quadrant of the PUM drive in the screenshots, which was close to saturating the 20 bit DAC on EY L2, but not quite saturating. Since the PUM also gets ASC signals all 4 quadrants get somewhat different signals and probably one of the unplotted quadrants is what saturated and caused the locklosses.
The filter which was missing from EY in TRANSITION_FROM_ETMX is called muBoost, and is meant to offload the microseism drive from the PUM to L1. It is in FM4 in ETMY L1 LOCK and FM2 in ETMX L1 LOCK. This seems to have been an oversight, which has probably been this way for quite a while. FM4 is already turned on in the LOWNOISE_ESD_ETMY state. I've just now committed ISC_LOCK to the svn revision 27023.
This has probably been an ocassional problem for a long time, but the combination of more ground motion due to wind and microseism and having the calibration line on with the linearization on while the bias ramps down made this state not work at all.
The problem with the calibration line is that it passes through the linearization for the ESD which we only use in the high noise high range ESD configuration. As we ramp the bias to zero in order to switch the ESD driver to low noise, if the calibration line is on the linearization does some large and strange things, which have been a cause of locklosses in the past.