Kyle will write a separate alog with the vacuum details of this event.

I rebooted h0veex to fix the ion pump epics database to match the new hardware. After the IOC was rebooted and the PID control of the LN2 level in CP8 was enab led the pump quickly overflowed because the fill valve was incorrectly set by the software to 92% open. I tested the EPICS code and found that if the IOC was soft booted, or reset by the watchdog card, the fill valve was always opened to the LN2 level set point (nominally 92%). I went to EX and power cycled the VME crate twice, the first time the fill valve started at 41% (which happened to be half the setpoint at the time) and the second time the IOC started correctly with 0% valve opening (because CP8 was 112% full at this time).

I did an initial investigation of the database and no immediate problem could be found. Having the output of the PID record be essentially the setpoint indicates that the controlled PV is zero on its first run, but it is not obvious how this could be. Also the different starting behavior between a reboot and a power cycle may indicate the erasure of mv162 RAM may be an issue.

This is the first time we have encountered this error on any PID controlled LN2 level, so aging hardware may also be a factor. We will continue the investigation this week and would like to reboot this system again next Tue maintence to test its robustness.

We are also discussing adding software limits to prevent this from happening again.