Reports until 12:05, Thursday 22 October 2015
H1 General
travis.sadecki@LIGO.ORG - posted 12:05, Thursday 22 October 2015 - last comment - 00:27, Friday 23 October 2015(22746)
Lockloss

Lockloss @ 18:46 UTC.  Cause is currently under investigation.

Comments related to this report
jenne.driggers@LIGO.ORG - 13:02, Thursday 22 October 2015 (22747)

This lockloss was caused by an EPICS freeze. 

We regularly run the SR3 "cage servo" instead of an oplev servo.  It is normally always on (even if the IFO is unlocked), but just in case it's not, the locking guardian requests that the SR3_CAGE_SERVO guardian turns on the servo sometime during the DRMI lock sequence.  As a reminder, this servo looks at the OSEM pitch value of the M3 stage, and actuates on the M2 stage to keep the OSEM values constant. This has been determined to be more stable over very long time scales than the optical levers. 

The SR3_CAGE_SERVO guardian was running along like normal, trying to write values to the SUS-SR3_M2_TEST_P_OFFSET channels. However, since EPICS was frozen, these values weren't actually being written.  The servo thinks that it just needs to push harder and harder, so starts significantly changing the offset value that it's trying to write (it's normally +-10 counts or so, but after this starts changing by about 1000 counts).  Once the EPICS freeze is over, it successfully writes one of these new, significantly different values.  This kicks SR3 pretty significantly, and we lose lock shortly afterward.

The attached lockloss plot shows POPAIR dropping at -1.75 seconds, which is when the actual lockloss happens.  Starting at -20 seconds, the OFFSET channel flatlines, which causes the M2_MASTER_OUT channels to also flatline.  You can see in 3 of the 4 NOISEMON channels that the RMS is significantly reduced temporarily.  Perhaps the M2 UL Noisemon is one of the ones that is broken, and that's why we don't see it there?  Anyhow, at around -6 seconds, the EPICS freeze is over, and the OFFSET comes back on with a very different value.  This causes a big glitch in the MASTER_OUTs and the NOISEMONs.  We lose lock about 4 seconds later.

Images attached to this comment
sheila.dwyer@LIGO.ORG - 18:06, Thursday 22 October 2015 (22754)

Dave, Evan, Sheila

We've added a few lines to the SR3 cage servo guardian to hopefully avoid this in the future. This will not update the offset unless the witness sensor value has changed. This may cause the cage servo to occasionally not run, but this doesn't seem like a problem. 

 

in the main we added 

        self.wit = ezca['SUS-SR3_M3_WIT_PMON']

 

    def run(self):
        if ezca['GRD-SUS_SR3_STATE_N'] == 100 and not self.not_aligned_flag:
            #if the value has not changed (possibly an epics freeze) skip running the servo
            if self.wit == ezca['SUS-SR3_M3_WIT_PMON']:
                pass
            else:
                self.servo.step()
                return True
        else:
            notify('SR3 not aligned!')
            return 'CAGE_SERVO_OFF'
    def run(self):
        if ezca['GRD-SUS_SR3_STATE_N'] == 100 and not self.not_aligned_flag:
            #if the value has not changed (possibly an epics freeze) skip running the servo
            if self.wit == ezca['SUS-SR3_M3_WIT_PMON']:
                pass
            else:
                self.servo.step()
                return True
        else:
            notify('SR3 not aligned!')
            return 'CAGE_SERVO_OFF'
sheila.dwyer@LIGO.ORG - 00:27, Friday 23 October 2015 (22764)GRD
Actually the code above won't do what was intended. We need to make sure self.wit gets updated each time run is executed. This code does no harm as is, so I will wait until morning to fix it.