We regularly run the SR3 "cage servo" instead of an oplev servo. It is normally always on (even if the IFO is unlocked), but just in case it's not, the locking guardian requests that the SR3_CAGE_SERVO guardian turns on the servo sometime during the DRMI lock sequence. As a reminder, this servo looks at the OSEM pitch value of the M3 stage, and actuates on the M2 stage to keep the OSEM values constant. This has been determined to be more stable over very long time scales than the optical levers.
The SR3_CAGE_SERVO guardian was running along like normal, trying to write values to the SUS-SR3_M2_TEST_P_OFFSET channels. However, since EPICS was frozen, these values weren't actually being written. The servo thinks that it just needs to push harder and harder, so starts significantly changing the offset value that it's trying to write (it's normally +-10 counts or so, but after this starts changing by about 1000 counts). Once the EPICS freeze is over, it successfully writes one of these new, significantly different values. This kicks SR3 pretty significantly, and we lose lock shortly afterward.
The attached lockloss plot shows POPAIR dropping at -1.75 seconds, which is when the actual lockloss happens. Starting at -20 seconds, the OFFSET channel flatlines, which causes the M2_MASTER_OUT channels to also flatline. You can see in 3 of the 4 NOISEMON channels that the RMS is significantly reduced temporarily. Perhaps the M2 UL Noisemon is one of the ones that is broken, and that's why we don't see it there? Anyhow, at around -6 seconds, the EPICS freeze is over, and the OFFSET comes back on with a very different value. This causes a big glitch in the MASTER_OUTs and the NOISEMONs. We lose lock about 4 seconds later.
Images attached to this comment
sheila.dwyer@LIGO.ORG - 18:06, Thursday 22 October 2015 (22754)
We've added a few lines to the SR3 cage servo guardian to hopefully avoid this in the future. This will not update the offset unless the witness sensor value has changed. This may cause the cage servo to occasionally not run, but this doesn't seem like a problem.
in the main we added
self.wit = ezca['SUS-SR3_M3_WIT_PMON']
def run(self):
if ezca['GRD-SUS_SR3_STATE_N'] == 100 and not self.not_aligned_flag:
#if the value has not changed (possibly an epics freeze) skip running the servo
if self.wit == ezca['SUS-SR3_M3_WIT_PMON']:
pass
else:
self.servo.step()
return True
else:
notify('SR3 not aligned!')
return 'CAGE_SERVO_OFF'
def run(self):
if ezca['GRD-SUS_SR3_STATE_N'] == 100 and not self.not_aligned_flag:
#if the value has not changed (possibly an epics freeze) skip running the servo
if self.wit == ezca['SUS-SR3_M3_WIT_PMON']:
pass
else:
self.servo.step()
return True
else:
notify('SR3 not aligned!')
return 'CAGE_SERVO_OFF'
sheila.dwyer@LIGO.ORG - 00:27, Friday 23 October 2015 (22764)GRD
Actually the code above won't do what was intended. We need to make sure self.wit gets updated each time run is executed. This code does no harm as is, so I will wait until morning to fix it.
This lockloss was caused by an EPICS freeze.
We regularly run the SR3 "cage servo" instead of an oplev servo. It is normally always on (even if the IFO is unlocked), but just in case it's not, the locking guardian requests that the SR3_CAGE_SERVO guardian turns on the servo sometime during the DRMI lock sequence. As a reminder, this servo looks at the OSEM pitch value of the M3 stage, and actuates on the M2 stage to keep the OSEM values constant. This has been determined to be more stable over very long time scales than the optical levers.
The SR3_CAGE_SERVO guardian was running along like normal, trying to write values to the SUS-SR3_M2_TEST_P_OFFSET channels. However, since EPICS was frozen, these values weren't actually being written. The servo thinks that it just needs to push harder and harder, so starts significantly changing the offset value that it's trying to write (it's normally +-10 counts or so, but after this starts changing by about 1000 counts). Once the EPICS freeze is over, it successfully writes one of these new, significantly different values. This kicks SR3 pretty significantly, and we lose lock shortly afterward.
The attached lockloss plot shows POPAIR dropping at -1.75 seconds, which is when the actual lockloss happens. Starting at -20 seconds, the OFFSET channel flatlines, which causes the M2_MASTER_OUT channels to also flatline. You can see in 3 of the 4 NOISEMON channels that the RMS is significantly reduced temporarily. Perhaps the M2 UL Noisemon is one of the ones that is broken, and that's why we don't see it there? Anyhow, at around -6 seconds, the EPICS freeze is over, and the OFFSET comes back on with a very different value. This causes a big glitch in the MASTER_OUTs and the NOISEMONs. We lose lock about 4 seconds later.
Dave, Evan, Sheila
We've added a few lines to the SR3 cage servo guardian to hopefully avoid this in the future. This will not update the offset unless the witness sensor value has changed. This may cause the cage servo to occasionally not run, but this doesn't seem like a problem.
in the main we added
self.wit = ezca['SUS-SR3_M3_WIT_PMON']