Last week I tripped the HEPI pump controller valving out the BSC2 actuators, it took me until today to try to recover it. In the process of investigating this I found that one of the 4 corner station HEPI pump carts has been more or less off since a power outage in December. The remaining pump stations have been running at about 95% drive to cover for the lost cart. I am working on some revisions to the famis and asking to Tony to add a verbal notification if there is a mismatch in the manifold pressures for the 4 corner pump stations.
Nice to know that we have overhead to lose one pump station and still supply the corner HEPI with pressure. Drive is much lower now that all 4 are running again.
Attached image shows how long we have been running with only 3 pumps. Crosshair in the upper left plot shows approximate time of the power outage. Bottom right shows we have been running at ~1900 cts drive out of 2048 total available since then. We are back around 1100 cts drive now.
I've added a message to DIAG_MAIN and to VerbalAlarms to "Check HEPI pump station pressures" if any of the four CS pump stations manifold pressures are more than 5% different than any other (this threshold may need to be updated in the future if we find it's too tight, but right now it's okay). All updates committed to svn.
The channels being used for this comparison are H1:HPI-PUMP_L0_PS{1,2,3,4}_PRESS1