Earlier today, I wrote a couple out of loop feedforward filters to the BS ISI foton file using Foton. When I hit the load coefficients button (while the ISI was isolated, the ff paths were off, so it shouldn't have done anything), the ISI tripped, hard. It rang up the T240's pretty bad and I couldn't isolate the ISI for several minutes after. Worried I had inadvertantly written some other filter I ran a diff on the most recent archived file and the file created yesterday when Jeff restarted the models. This showed a whole bunch of filter coefficient differences, which shouldn't have been there (as reported by a diff of the two archive files, I don't know exactly what changed, see attached). Talking to Jim, Dave and Jeff, it sounds like the glitch was probably caused by my having used Quack recently (June 22nd) to load some blend filters. Jeff's model restart (and even a prior model restart on June 30) simply inherited that quack-written file. Today was the first time the BS's foton file was opened and saved in Foton. Quack can apparently load coefficients with higher precision than Foton will accept, so when you open and save a "too high" precision filter with Foton, it rounds the coefficients off. Sudden change in precision of SOS coefficients in the blend filters = bad for isolation loops = bad trip.
We've seen this Foton vs. Quack Rounding problem before -- see e.g. https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=3553 -- and it's still biting us.
This sounds like a relatively easy thing to control for, I can think of two ways:
- getting Quack to check and do the rounding on it's own before writing to the Foton file.
- have the post-build script run a "foton -c" on all filter files before the model gets restarted.
Is there someone in the CDS group who can fix this? Maybe it has been? There are several versions of Quack running around, June was my first attempt with it, maybe I used the wrong one.
I used /ligo/svncommon/SeiSVN/seismic/Common/MatlabTools/autoquack.m
https://svn.ligo.caltech.edu/svn/seismic/Common/MatlabTools/autoquack.m
Last Changed Author: brian.lantz@LIGO.ORG
Last Changed Rev: 7939
Last Changed Date: 2014-02-14 15:38:15 -0800 (Fri, 14 Feb 2014)
Text Last Updated: 2014-02-14 15:48:16 -0800 (Fri, 14 Feb 2014)
we should use the readfoton script to read and plot the installed filter, i can do that
I suspect that the problem appears because a change (however small) in the filter coefficients causes the filters to reset (clear history, start over) and reset of the filter history = glitch in the output. It is easy to image this glitch being quite large for a ISI loop which is holding a static offset. I am working on an update to autoquack which will have it automatically call foton -c so that the filter updates happen in a deterministic way, and there is a log file telling you which filters have been touched.
filed ECR https://services.ligo-wa.caltech.edu/integrationissues/show_bug.cgi?id=1077 testing of possible solution, see https://alog.ligo-la.caltech.edu/SEI/index.php?callRep=789 -Brian