VACSTAT, the vacuum glitch detection system, is now online after a soft rollout. This is a stand-alone EPICS IOC which monitors vacuum gauge pressures and reports if it has detected a vacuum pressure glitch.
Its overview medm can be launched from the sitemap VE pull down, or from the CDS Overview vacuum section.
The attached image shows the VACSTAT Overview MEDM, the system details MEDM and a trend plot MEDM for one of the pressure gauges.
For each pressure gauge, the system maintains lookback FIFOs of the pressure trends. Currently the system is looking back 60, 300 and 600 seconds. Every 10 seconds, the slopes of the data are calculated using a linear regression algorithm. If the absolute value of a slope exceeds its trip value, a glitch detection is raised.
In addition, the total change in pressure over each trend is calculated (Pdelta = Pmax - Pmin). If the delta pressure exceeds a trip value, a glitch detection is raised.
The trend plots have three rows, one for each of the lookback time. The left column is a live plot which continuously streams. The right column latches if a glitch is detected, stopping when the glitch is at the half-way mark.
If a glitch is detected, cell phone alarms are sent to the vacuum team. In addtion, the VACSTAT block on the CDS Overview MEDM will go RED.
All parameters of the system are defined by a configuration yaml file. I'm in the process of writing the user guide.
prod.yaml currently in use in production:
---
ifo: H1
glitch_monitor:
lookback_times: [60, 300, 600]
proc_period_secs: 10
vacuum_gauges:
default:
glitch_press_rate: [1.0e-10, 1.0e-10, 1.0e-10]
glitch_delta_press: [1.0e-08, 1.0e-08, 1.0e-08]
valid_value_min: 1.0e-10
valid_value_max: 1.0e-04
H0:VAC-LY_X0_PT100B_PRESS_TORR:
description: "Corner Station HAM1"
H0:VAC-LY_Y1_PT120B_PRESS_TORR:
description: "Corner Station BSC2"
H0:VAC-LX_Y8_PT132_MOD2_PRESS_TORR:
description: "Corner Station BSC3"
H0:VAC-LX_Y0_PT110_MOD1_PRESS_TORR:
description: "Corner Station HAM6"
H0:VAC-LY_Y3_PT114B_PRESS_TORR:
description: "Corner Station CP1"
H0:VAC-LX_X3_PT134B_PRESS_TORR:
description: "Corner Station CP2"
H0:VAC-LY_Y4_PT124B_PRESS_TORR:
description: "Corner Station Y-Arm"
H0:VAC-LX_X4_PT144B_PRESS_TORR:
description: "Corner Station X-Arm"
H0:VAC-MY_Y1_PT243B_PRESS_TORR:
description: "Mid-Y Y1 Beam Tube"
H0:VAC-MY_Y5_PT246B_PRESS_TORR:
description: "Mid-Y Y2 Beam Tube"
H0:VAC-EY_Y1_PT423B_PRESS_TORR:
description: "End-Y Beam Tube"
H0:VAC-EY_Y3_PT410B_PRESS_TORR:
description: "End-Y BSC10"
H0:VAC-MX_X1_PT343B_PRESS_TORR:
description: "Mid-X X1 Beam Tube"
H0:VAC-MX_X5_PT346B_PRESS_TORR:
description: "Mid-X X2 Beam Tube"
H0:VAC-EX_X1_PT523B_PRESS_TORR:
description: "End-X Beam Tube"
H0:VAC-EX_X3_PT510B_PRESS_TORR:
description: "End-X BSC9"