Summary--HEPI is running and the platform is isolated but, the Pump Station pump could be on its last legs. The motor is now turning at 3x the previous speed to achieve nominal pressures; and, based on the PID controller, the plant is different.
Cronological Details [edits in brackets]
1531utc (0831pdt) HEPI Tripped [there was activity in the LVEA at this time]; JeffB entered the EndX Mech Room, from the VEA at 1533 (percomm RichardM.) Jeff recorded HEPI Reservoir Fluid level (after doing Dust Monitor Pump checks) after the HEPI had tripped based on the level he recorded. It is very noisy in this space and he did not realize when he read the reservoir level that the pump was not running. Based on conditions later found, this first trip was a fluid level trip.
Almost an hour later, JeffB and JimW initiated restarting the pump station. This was not successful mainly likely because the fluid level was still low and would trip before reaching nominal pressures. After these operators had reached their maximum cpu output, I was called in. Observation and test revealed the fluid level was in fact low and this was mitigated by adding ~1/4" more fluid to the revervoir. [The accumulator pressures were checked and a thorough look for fluid leaks was performed--accumulators were fine and no leaks were found.]
Once fluid level was comfortably above trip level, the drive was manually run up to achieve nominal differential pressure. At this point, in hind site with trends to review, the return pressure was running low almost 10% and the supply pressure was ~nominal but much noisier. Further, at this point the output of the pump station was running almost 15psi higher than normal while the pump was running at close to nominal speed! Does this mean that something was obstructing the flow bewteen the pump station and the chamber pressure sensors?
After running like this for about 10 minutes and meanwhile having swept the VEA, JeffB and I were about the drive away, Patrick informed us the Platform had tripped again. The trends reveal that at this time§, the supply pressure (at the chamber) spiked almost 15 psi (return pressure did as well.) Of course this spiked the differential pressure and the PID drove the output towards zero. Within 5 seconds, the pressures drop and the PID turns the output around and it now maxes out at long established output limits. blah blah...
At this point RichardM checked the power supply confirming we still had all phases and we also checked that the rotational speed of the motor matched the VFD output. So it seems the pump is not now able to pressure the system as before. Once we figured out Patrick's PID controller we got the output pressures nominal and got the SEI platforms isolated. But, to achieve this pressure, the motor/pump is now spinning +3x faster than before. Pressures are ~nominal and no more or less noisy than they were before, except, the first two pressure points after the pump is down 0.4 and 0.2psi suggesting with a stretch that some obstruction cleared...
Attached are two graphs, the first is 6 hours from before the trip to final operation state and the second is 20 seconds around the big change point discussed above §. I'll take some time to annotate these graphs but for now I'll get these into the log.
[While the motor/pump here is now running 3x faster than before, it is still only running at 2/3 maximum. For reference, the corner station pumps all run at ~1/2 maximum. So, there should be no issues until the pump falls apart. Comforting, right!]
1:35UTC7:35UTC Stand Down expired as per Keita's instructions7:51UTC INJ_TRANS returned to nominal. I got involved in aLogging a couple of things and forgot to actually perform the action. d'oh!