The CS HEPI Pump Servo program froze ~1258utc today. Recovery was easy and typical. Details:
Following the OPS WIKI, https://cdswiki.ligo-wa.caltech.edu/wiki/SEI, wiki text is verbatum in blue and my comments are in red:
HEPI Pump Stations
You are here likely because an Alarm on the HEPI Differential Pressure. Couple things: is the Alarm Invalid or other? Is it Momentary, or Ongoing?
- Momentary: The pressure sensors can be noisy but Alarm parameters should make these alarms rare. Make a log and get back to better things.
-
Ongoing: This indicates a trend in the pressures which will be caused by the servo likely being stalled. If not white boxed, there is likely a fluid level shutdown: follow the Other than Invalid Alarm guidance below.
Invalid Alarm This was our situation this morning
If the alarm is Invalid, the Pump Controller medm is likely white boxed; look for this under the Sitemap HEPI pulldowns. A restart of the servo is required. Don't panic though, the system may still actually be running and might be able to be restarted without losing the HEPI Platform's Isolation state. If the HEPI Platforms have not tripped, this is likely the situation. If power to the servo computer wasn't interrupted, the output from the servo to the VFD may be steady and keeping the Pressure near Okay.
Again, exactly the case this morning. Went to the pump stations and the dial gauge on the pumps were reading ~80psi, normal. The computer appeared to be running but the heartbeat was flatlined. I did not restart the computer.
Restarting the White Boxed Servo: If your servo is not running (white boxed medm) restart it with the following sequence:
If the HEPI platforms are still isolating (not tripped,) trend the VOUT channel: H1:HPI-PUMP_L0/EX/EY_CONTROL_VOUT for use in restarting. The conditions would indicate the servo computer has latched its output to the VFD and the Pump Motors are running 'open loop' but with the servo dead, you don't know the drive level. The trend will tell you what it was when it stopped, and when. Now
The trends showed that the last output from the PID to the VFD was ~1085
continue with the following sequence to restart the servo:
1. ssh controls@h1hpipumpctrll0 For the Corner Station
-
ssh controls@h1hpipumpctrlex For the EndX Station
-
ssh controls@h1hpipumpctrley For the EndY Station See Secrets for passwords or ask Hugh
2. cd target-new For the Corner Station
-
cd target-ex For the EndX Station; cd target-ey For the EndY Station
3. sudo ./run
Logging in was trivial but the medm took a good while to come completely alive but it did. During this period, the IFO dropped out of observing as two fields changed: 1) the PID Setpoint level, I could hardcode that into the dB, and 2) the Servo Control Auto/Manual button. This must come up in manual mode so the program does not write anything to the VFD. This is what allows the HEPI system to keep running in this situation. If it were in Auto, the PID would push a zero before it had a chance to calculate and process a reasonable value. Maybe a smart dB programmer could work this out but for now, this is what we (I) do.
This should restart the servo and the medm should come alive. Do not push any buttons, you don't want the servo to process any records. Note the pressures, these are valid and hopefully not too far from nominal. However, the servo will not have processed any records yet and the 0 counts output to the VFD is not valid. The servo starts in a safe state, most importantly Manual Control and no Burt Restore is requied. Now using the VOUT value from the trends before the drop out, put this value into EPICS:
caput H1:HPI-PUMP_L0/EY/EX_CONTROL_VOUT value. caput the 1085 gleaned from the trends above.
-
This will put control back to the servo setting the VOUT to that value and things should carry on as before.
Now, using the Manual Output[SetPoint, Output is incorrect, I'll fix the WIKI] Tweak, adjust the differential pressure to 70 psi and[this actually doesn't apply, will remove from WIKI guiadance.] Using the SetPoint Tweak, set the SetPoint to 70 psi; and finally, change Servo Control to Auto. The UGF is quite low so this is pretty harmless. Return the Tweak to 0 and things should be good. This is what was done and the results were good. The PID changed the output to the VFD a count or two but it is always doing that. The restart was perfect, a perfect ping.
Okay, that's it. Sure, look at the 6 hour trend plot attached.
