Reports until 18:14, Tuesday 22 March 2011
LHO FMCS
kyle.ryan@LIGO.ORG - posted 18:14, Tuesday 22 March 2011 (652)
CP8 alarm, IP12 fix followup
CP8 overfilled following repeated reboot/restore cycles today.  

Dave B. needed to reboot the VE at the X-end to incorportate new parameters into the database.  These new parameters were to correct for the similar but different output (to CDS) of the newly replaced and updated ion pump controller for IP12.  Dataviewer data indicates that the PID output controlling CP8's liquid level control valve (LLCV) was erroneous following the burt restore and not based upon the PID's reading of CP8's actual level.  The result was a rapid overfilling of CP8 following the manual enabling of the instrument air to the LLCV at some point after the restore.  This overfilling resulted in exhaust vapor pressure building and GN2 leaving the 80K pump via three paths:  

1) the normal exhaust line to atmosphere
2) backwards flow through the reneneration line and out to atmoshpere through a preexisting broken weld at the ambient air vaporizer 
and 3) into the VEA through the diaphram of the newly installed magnehelic differential presure gauge which had ruptured due to the high pressure (diaphram rated for 15 psig, nominally 1.5 psig).  

-> I valved-out the broken magnehelic gauge and will replace it later.  
-> I opened the exhaust line check-valve bypass to to reduce exhaust line back pressure.  
-> I placed my hand on the Burst Line pipe (it was warm) and confirmed that the pump's burst disk hadn't ruptured.  

Dave B. was able to repeat the events and to observe the bogus PID output following a subsequent reboot of the VE rack (remote reboot?).  He will continue his investigation next Tuesday to better understand the events and to propose a fix.  

I am leaving the instrument air "not enabled" to prevent any further filling of CP8 overnight and will restore PID control after the pump level receeds back below alarm levels (tomorrow).