I have added a check to the LocklossShutterCheck guardian that will add to the guardian log a note whether the HAM6 GS13s saw a large kick or not. Note that this does not in any way change the functionality of this guardian, in that it still requires an Operator to run and look at the plot, then manually reset this guardian.
However, we should be able to look through the guardian log to ensure that each time we lose lock, the log notes that the GS13s saw a big kick. When we are satisfied that this is a good measure of the shutter having closed at the last lockloss, we can consider using it as an automation metric in lieu of Operators manually resetting the guardian.
I have set the threshold for "big kick" for the GS13 to be 30,000 nm/s (1nm/s / count, so 3e4 counts). When the shutter closes fast, the GS13 sees a signal as large as 90,000 nm/s, but when the shutter closes slowly it only kicks the GS13s to about 10,000 nm/s. The threshold of 30,000 nm/s should differentiate between a fast vs. slow closing of the shutter. Normal GS13 signal is under 100 nm/s peak.
Attached is the code, which is also checked into the svn. The idea is (once we're ready to make this automated) that if we see the GS13 kick, the CheckShutter state will automatically go to the LowArmPower state, and allow the IFO to relock. If there is any problem, including inability to fetch the data, the guardian will stay in the CheckShutter state, and will require human intervention. One could also consider adding light level checks to this, but I think that's probably overkill given the other checks that we already have in place.
At least twice today we've run into the situation where we're still locked at 2W but lose lock, and the power going to the AS port is not enough to warrant a fast close of the fast shutter. This makes things a bit more confusing, since you must know that it's okay that the shutter didn't close in this case.
So, I've added another level of checks to the LocklossShutterCheck guardian to check if the trigger PD saw more than the threshold. If not, it logs that the shutter should not have fired. If the trigger PD goes above threshold, then it looks at the GS13 data and logs whether it saw a kick or not.
Again, these changes only affect what goes into the guardian log. Manual intervention is still required at this time to reset the guardian and allow the IFO to relock.
A different option that we can also implement is to alter the thresholds for the power level that sends the guardian into the CheckShutter state. Sheila points out that maybe we only should be running these checks when there's a chance that the shutter will have closed, so at more than 2W.
After talking with Daniel this morning I made a few more changes to this guardian. The problem that Jenne was trying to address with the check on the AS power was that for 2 Watts most locklosses from 0 CARM offset will trip the shutter, but not all of them should since sometimes not as much of the power goes to the AS port. We think that the second option Jenne mentioned, only requiring a check of the shutter after high power locklosses in which we nearly always expect it to fire, will let us avoid relying on the trigger PD in this logic.
This morning I redid the arm power checking a little bit, to include the calibration into watts, and increased the threshold for requiring a shutter check to 14 kWatts of circulating power in an arm. (This is just over 4 Watts input power). I also took out the check on the triger PD, and made the logic that Jenne had logging results last night be used to reset the state of the guardian. Now the gaurdian will rest itself to LOW_ARM_POWER if it sees that the shutter went off and the GS13s see a kick. If the GS13s do not pass the test it will go to a new SHUTTER_FAIL state.
We will test this when we get to relocking this afternoon.