On Wednesday TJ and I tripped the ISI several times executing the HAM3 ISI GS13 gain switch in various ways. I have several examples to look at but here is a comparison of one trip to a successfull switch of the gains using the SEI COMMAND perl script.
The first attachment has 20 seconds of the trip example next to the switch that does not trip. It looks like there isn't anything different going on before the two cases: There are larger swings in X & Y in both and the other DOFs all look fairly similar too. There is a larger glitch 5+ seconds before the trip in Z but that seems to be settled by the switch/trip time; these would be related to Reference Position ISO servoing.
In the second attachment, again side by side are the two cases zoomed in a bit closer (10 sec.) The two stage switch by the perl script is evident with the gain switch doing more glitching and the whitening stage doing nothing. There is a delay of the sensors wrt to the switching by the guardian which seems weird but I'm not looking at outputs from the filter bank as we have no DQ channels here... These channels are the inputs to the blend bank after transforming to cartesian basis. Again, nothing stands out as to why there is a problem with the guardian script. We (TJ & I) did adjust the guardian code to separate the FMs switching like the perl script does but it still tripped the ISI.
The third attachment is a close zoom in on the ISI trip switch. It is what it is... Given that the Y and RZ are the dofs that ultimately go big, I'll suspect the horizontals but I could be fooled.
4 trips of the ISI with Guardian. Three are H3 and one is V3. Problem with corner 3?
I thought I had a trend going seeing the only excursion hitting the trip level on H3 first (see first three attachments.) Watch the channel colors as I added a channel but they are all H3. The fourth plot shows that the V3 sensor goes to the rail and causes the trip. If I saw a bunch more instances and they all hapened on corner3 I'd be very suspicious of bio switching or something; as it is I'm just suspicious.
The fifth plot compares the switching sequence on the six sensors between the COMMAND perl (left) and the Guardian (right.) Yes, the perl script does manage to hit the switches all at once whereas the guardian is clearly spaced in time. Of course, guardian manages to switch these on all the other platforms (except HAM2) without problem. Hmmmm. I'd like to work HAM2 over too and see wwhat that tells us.
Actually, we, TJ & I, are getting confused as to which of the above is guardian or perl as we mucked with the guardian code.
We think the one on the right is guardian with sleeps between the switches trying to mimic the perl Command script shown on the left.