As Hugh noted in his July 19th alog (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=28525) when we reprogrammed the site-wide timing systems on 19th May 2015 and 19th July 2016 the HEPI pump controller units saw a bogus drop in hydraulic fluid pressure. On 19th July of this year the drop at EY was sufficient for the PID controller to attempt to correct, resulting in an excess of presssure which caused problems.

At these times we also saw a bogus increase in cold cathode gauge readings from the vacuum controls system (Kyle put in an alog). Initially I thought more systems were being affected, but today I did some more in depth investigation and the problem appears to be limited to hepi pressure and vacuum cold cathode signals.

At EY I trended all the raw ADC channels in the Beckhoff Vacuum system (terminals M12-16 chans 1-4). The only channels which saw an increase were M11-chan2, M12-chan1 and M12-chan3. These are the input signals for the three cold cathode gauges. When we trend these same signals from May 2015 (back when the computer was the orignal VME system) these same signals show bogus deviations. I looked at the slow-controls Beckhoff system and the weather station, but could not see the problem there.

There are some exceptions to the rule (for example in 2015 hepi-EX did not see a change) but generally when we reprogrammed the timing system the Cold Cathode and Pump Controllers saw a change of input voltage. The question is why only timing reprogramming causes the problem and not the many reboots we have done over the years. I suspect the powering of the 12V units (fanout, comparator, RF Amp, IRIG-b) which we rarely do as opposed to the more frequent powering of the 24V IO Chassis may be a clue.

Tomorrow, under WP 6035, we will go to EY and repeat the power cycling sequence we followed on 19th July. We will repeat everything except for the actual reprogramming. We will be closely monitoring the ADC inputs for these signals. At the end of the excercise Ansel will upgrade the FPGA on the EY fanout using Daniel's latest code.