After John Worden noticed multiple gauges were flatlined, we discovered that pulling the fuse to depower PT180 gauge for bake-out inadvertently flatlined LX beckhoff vacuum system this past weekend, including CP2 PID loop. CP2 LLCV locked at 60% open (1.7x its nonimal value) due to integrator high limit. Strange that the % full dipped from ~92% to 91.7% (Kyle noted a step change in PT120 (cold cathode gauge) after pulling PT180 fuse). Bubba noticed significant ice build up on exhaust line this morning. Now we know why. There was significant LN2 consumption as a result of overfilling for four days. Dewar level fell from 49% to 32%. The exhaust pressure was also not reading actual values. Thanksfully we opened all CP exhaust bypass valves as new nominal state last week to prevent overpressurization!
There was no indication of faults from the vacuum overview MEDM screen. PT180 appeared to be powered with a reasonable pressure value. The LVEA subscreen, however, did show a fault for PT180. I filed FRS 6772 to have faults moved to front overview page. We will work with Patrick to create an alarm handler for future events where perhaps a fuse blows on its own.
PT180 was manually disabled in beckhoff this morning which caused the other components to read as-is, showing PT180 reading red and zero and CP2 and LX gauges to reflect their faults. Richard will write an instruction manual so we can do this in the future for planned events.
Once again I must profess my gratitude for this amazing team!