RyanS told me yesterday that he could re-isolate the ETMX ISI. I finally got to looking at it today and something was off with the CPS. Using the weekly CPS spectra script I got ASDs for the CPS from last night and a bunch of the the CPS spectra were way off, first image.

I went to EX and looked at racks, SEI chassis all looked fine. I unplugged a couple of CPS probes at the chamber and did not get the expected behavior. One CPS went to the near limit rail (-32k) the others hardly changed at all. This made me suspect timing, so I traced out the timing sync wires, found the 71Mhz CPS timing fanout and the lights were all off. Looked at the power strip and only the -18v led was on. Turned off the fanout chassis, took some pics and found Marc and Fil.

Marc came to the end and we looked at power supplies in the front of the building and the +18 for the rack was dead. We swapped both power supplies for spares, and CPS timing seems to have come back, I was able to isolate the ISI. Marc said the fans seemed a bit stiff and smelled weird, so maybe another fan failure.