J. Kissel, A Pele, S. Ballmer, M. Barton

For once, I'm not gunna go into great detail. 

We wasted today trying to chase down why the ETMY's damping loops had gone unstable. The reason was that, after a reboot of the ETMY front-end model (the reboot was necessary because I found the model frozen when I got in this morning -- perhaps because of whatever weather event took down the whole IFO over the weekend), it has acquired a gigantic time delay. It was cleared by a reboot of the IOP front end process and ETMY frontend process.

After blaming 
- bad burt restores
- bad filter definitions
- broken analog electronics
- the new(ish) filter designs
- DAQ down sampling filters
and discovering (with out measuring much) a 5.5 [Hz] slow instability in Roll (a loop we hadn't touched since 2013-05-01), we took all sorts of trial measurements to try to narrow down the problem. Finally I measured the open loop gain transfer functions of a few degrees of freedom, and compared them against the model (see attached plots).

The ratio of the two revealed the ridiculous time delay of 0.023 [s]. That's 376, 16 [kHz] cycles. I glanced a the ETMY GDP_TP screen, which indicated that the CPU_MAX had at some point hit 191 [us] (though was currently green). Since we hadn't changed anything else, and we'd confirmed everything SUS related was OK, we resigned to restart the IOP front end process as well as the ETMY front end process. Unfortunately, at that point, I was too frustrated to remember to look and see if anything was red in the IOP GPD_TP screen (though Jim later informed me that there's no good channel what would error on cycle mismatch).

Anyways, the bootfest cured the problem. *sigh* See ya tomorrow.