Jeff wrote the more detailed alog of the problems with the interferometer and how we tried to track down the issue, but short version, I installed a bad blend filter that has been causing an offset to accumulate in the ISI RZ locations for all of the HAMs, except HAM8. The accumulations take about 2 weeks to reach 1 urad levels. I designed the filter for HAM8 and it has been running there with no problems since last year, so I installed on HAM4-5 on Dec 11 last year, and March 15 on HAM2-3 this year. Unfortunately, there are some issues with the foton implementation, either because I missed some cleanup in the filter or because I got bit by some issue with writing the filters from Matlab to the foton file. Attached screenshot shows some of the zpk coefficients from Foton. There are 3 poles at, or near 0hz, and there are two negative zeros at 10^-6 hz. I certainly didn't use those in my design in matlab, but I guess I missed a step or two in cleaning up the filters before install. I think I've got a fix for that now, but will wait for Tuesday to try. They shouldn't accumulate much offset over the long weekend.

I've also added a test to DIAG_MAIN, to warn if the errant integration reaches 1urad on any of the HAM chambers. This should take 2 weeks or more at the current rate, but if any of the HAMs reach 1urad offset, DIAG_MAIN will report:

"ISI {} RZ CPS residual bad"

The test in DIAG_MAIN is:

@SYSDIAG.register_test
def HAM_RZ_RESIDUAL():
    """HAM RZ residual test

    """
   
    chambers = ['HAM2', 'HAM3','HAM4', 'HAM5', 'HAM6', 'HAM7','HAM8']
    bscs = chambers
    tots = []
    for chamber in bscs:
        if ezca['ISI-' + chamber +'_CPS_RZ_RESIDUALMON'] > 1000:
            tots.append(chamber)
    if tots:
        yield "ISI {} RZ CPS  residual bad".format(tots)