I closed the check-valve bypass valve for CP8 this morning and enabled the instrument air to the LLCV.  The PID is back in control of CP8's LN2 level.

CP8 overfilled following repeated reboot/restore cycles today.  

Dave B. needed to reboot the VE at the X-end to incorportate new parameters into the database.  These new parameters were to correct for the similar but different output (to CDS) of the newly replaced and updated ion pump controller for IP12.  Dataviewer data indicates that the PID output controlling CP8's liquid level control valve (LLCV) was erroneous following the burt restore and not based upon the PID's reading of CP8's actual level.  The result was a rapid overfilling of CP8 following the manual enabling of the instrument air to the LLCV at some point after the restore.  This overfilling resulted in exhaust vapor pressure building and GN2 leaving the 80K pump via three paths:  

1) the normal exhaust line to atmosphere
2) backwards flow through the reneneration line and out to atmoshpere through a preexisting broken weld at the ambient air vaporizer 
and 3) into the VEA through the diaphram of the newly installed magnehelic differential presure gauge which had ruptured due to the high pressure (diaphram rated for 15 psig, nominally 1.5 psig).  

-> I valved-out the broken magnehelic gauge and will replace it later.  
-> I opened the exhaust line check-valve bypass to to reduce exhaust line back pressure.  
-> I placed my hand on the Burst Line pipe (it was warm) and confirmed that the pump's burst disk hadn't ruptured.  

Dave B. was able to repeat the events and to observe the bogus PID output following a subsequent reboot of the VE rack (remote reboot?).  He will continue his investigation next Tuesday to better understand the events and to propose a fix.  

I am leaving the instrument air "not enabled" to prevent any further filling of CP8 overnight and will restore PID control after the pump level receeds back below alarm levels (tomorrow).  

Kyle will write a separate alog with the vacuum details of this event.

I rebooted h0veex to fix the ion pump epics database to match the new hardware. After the IOC was rebooted and the PID control of the LN2 level in CP8 was enab led the pump quickly overflowed because the fill valve was incorrectly set by the software to 92% open. I tested the EPICS code and found that if the IOC was soft booted, or reset by the watchdog card, the fill valve was always opened to the LN2 level set point (nominally 92%). I went to EX and power cycled the VME crate twice, the first time the fill valve started at 41% (which happened to be half the setpoint at the time) and the second time the IOC started correctly with 0% valve opening (because CP8 was 112% full at this time).

I did an initial investigation of the database and no immediate problem could be found. Having the output of the PID record be essentially the setpoint indicates that the controlled PV is zero on its first run, but it is not obvious how this could be. Also the different starting behavior between a reboot and a power cycle may indicate the erasure of mv162 RAM may be an issue.

This is the first time we have encountered this error on any PID controlled LN2 level, so aging hardware may also be a factor. We will continue the investigation this week and would like to reboot this system again next Tue maintence to test its robustness.

We are also discussing adding software limits to prevent this from happening again.

[Corey, Dave, Greg, Hugh, Mark, Mick, Scott]

Support Tube/Crossbeam Centering

Scott, Dave, & Hugh determined that the BSC7 Crossbeams/Support Tubes (a "big metal square") Assembly was roughly out of place (i.e. the ends of the Support Tubes were not centered within their nozzles & hence the Bellows were also flexed a little...not sure how these guys got pushed around).

So with measurements by the crew above, this morning we went about the process of getting the "big square" centered.  For the most part it went well.  Closing the final gap was a little tough in that we could get the assembly where we wanted, but it would spring back to a place where it wanted to be.  So, we manhandled the assembly into place, and then lowered the load via the crane.  The crossbeams were then torqued in place onto the Support Brackets.  We now had the Support Tubes/Crossbeams at a much better place, and wanted to make sure the HEPI are positioned wrt the Crossbeams.

HEPI Swap

Greg & I have been pre-positioning HEPI Foot Assemblies on the floor in preparation for the installation.  Now that BSC7 Crossbeams are better positioned, we wanted to install these pre-positioned HEPIs (instead of adjusting the feet of the HEPIs already in place...would have been a little tough to do this 10' up next to the chamber).

So we pulled out the NE HEPI assembly and replaced it with SN 54 HEPI assembly.  We positioned this HEPI such that the four bolts on top for the Crossbeam/HEPI Foot connection could be made.  This was a VERY TIGHT FIT.  Most of the interference came with the Support Bracket (seemed like there were some spots where we had less than a 1mm of clearance [see photo]). 

Once the HEPI Foot was in place, we selected a Shim to fit between the Foot & Crossbeam (previously, a 0.5" shim was used).  This time the 0.25" shim looked the best (vs. the 1/8" shim).  To make this shim fit, we had to the lower the HEPI Foot a little.  We were able to lower the Foot a little via the Vertical Foot Clamping Bolts, but we also had to loosen the "top" bolts of the Spring.

ISSUE:  now that we have 0.25" less space here, the bolts we were using could not be used---they were too long.  Two of them would be ok, but the other two would run into the heads of the vertical HEPI Foot positioning bolts (see photo of how much these bolts run through the HEPI Foot top plate).  So, we need to find new bolts for this connection.

For the time being, we torqued down the two bolts which had no interference.  Once the foot was clamped to the Crossbeam, we backed off all of the HEPI Foot Clamping Bolts and then moved the HEPI Assy to line up with bolt holes on the Pier.  We want to move as little as possible here so we don't jeopordize any range of the Foot.  The two carbon steel 1/2-13 HoloKrome bolts on the inside of the HEPI Assy which bolt to the Pier technically have torque values of 150ft-lbs (!), but since they are going thu such a big hole/with a wide washer, we didn't want to use this value (went with 45ft-lbs).  (Is this ok??)

This HEPI Assy is close to being complete, but we still have a few issues we'd like to go over with Hugh.  We'll then move onto the other HEPIs.

A reboot of the EX vacuum system front-end this morning caused the overfilling of CP8 and some damage to equipment. Kyle resolved the immediate problem and other repairs will be necessary. Dave rebooted several times and eventually resolved the issue in the code.
HAM8 was removed from its usual location, and work is underway to prepare its new spot behind HAM7.
Preparations were made for preparing to prepare the grout preparation at H2PSL.
Oscar baled tumbleweeds.

Ran well pump to fill tank.

Linoleum was removed inside the grout dams and table legs were set inside, and leveled. We plan on grouting the legs into place tomorrow with hydrostone.

LDAS have replaced their aging 3511 disk systems (the ones fb0w uses) with a new Sataboy. The disk capacity has gone down to 18TB (from 30TB) but the new system will be immune to the disk crashes of S6. We copied the minute trend files across from the old disks, these are the only old DAQ data we will preserve. All other data (full frames, second trends) are starting from scratch.

So from nds0 which mounts the new disks, minute trends have a data gap from 08:46 Mar 16 to 11:50 Mar 21. All other data started from scratch at 11:50 today. If you need access to minute or second trends in this data gap, please use nds2 which mounts the fb1w system and which ran throughout this upgrade.

~9:00 – Michael and contractors began leveling the PSL table

~9:00 - Richard and contractor working at Y-End.

~9:30 – Kyle to begin venting RGA volume and remove RGA components from BSC6

Work throughout the day:
bins being moved around in the LVEA
HEPI installation continuing on BSC 7

I installed the ECD blocks on the R0 Top Mass/Tablecloth.  Attached is a plot showing the TF taken just before the addition, as a reference, and with the ECDs set to within a mm of touching the mass.  The green and black peaks are with the ECDs, and show significant attenuation of peaks.  We need to determine if the placement of the blocks is acceptable, or if they need some in or out adjustment. - Plot to be uploaded when test stand power is recovered.

With the new Sokkia Autolevel, I rechecked the level of the Assembly Test Stand sighting on the Upper Structure base surface which is a referenced flat in the suspension build.  (Recall that we re-built the solid stack which the suspension hangs from in order to open up assembly real estate and mount a cross flow HEPA.)  On Friday, I determined that the structure was out by 0.008" over a 17" length along the Y direction (perp to beam line of the QUAD).  This equates to 0.5mRad.  The Test Stand was designed to be level to within 0.2mRad as per T050113, so today I adjusted the Test Stand feet to bring it in to 0.13mRad.  A quick check on the beam-line direction which looked better than than 0.13mRad, so no adjustments made for that direction.  The BOSEM signals showed a little bit of swing due to this releveling (~100 counts), but no bigger than the daily temperature drift of the signals (where a few degrees soften(?) the blades over the course of the day such that BOSEM LEFT and RIGHT change in the same directions on both chains by ~500 counts).

Using the new HEPI alignment numbers BSC HEPIs 77 54 51 45 and HAM HEPIs 03 05 are now complete.

Grout dams were caulked into place, based off the layout Mick, Nicole and I set yesterday. These are only held into place by a bead of caulk, so take care not to kick them loose. Caution tape was set around the area. On Monday we will cut out the linoleum in the center of the dams and level our table legs.

Replaced IP12's non-working MultiVac ion pump controller with a DualVac controller yesterday and modified signal cabling pinouts to match.  Pumped X-end with MTP for awhile then switched over to IP12 (note IP12 status is still "RED" in CDS displays -> Dave B. will correct database parameters to match new controller's signal outputs before the next reboot at which point IP12 status will change to "GREEN" in CDS displays)

Spun down MTP at X-end and shut down QDP80 pump.  

Noticed IP6's high voltage output "FIXED" at 7000V -> Corrected to "STEP" output.  

Linoleum work was finished today. Contractor came on site and welded the seams. There were some concerns about welding the new linoleum to the old linoleum, but he was able to do it no problem.

Yesterday Robert and I found that the pump in the mechanical room that pulls air for the dust monitors that have their internal pump removed was not pulling air. Today I read the flow on the attached flow meter as .15. This means that the data from locations 1 (ISCT4), 7 (H2 racks) and 10 (H1 PSL) has been invalid for an unknown period of time. Also possibly location 8 (H1 racks), which had been replaced with one with an internal pump earlier. So today I went and replaced locations 1, 7 and 10 with ones that have an internal pump. Note also that dust monitor 3 was moved yesterday into a clean room next to where the jack hammering is taking place. Also for a period yesterday, location 4 was moved onto the floor by the H1 PSL, but has been moved to location 8 today.

Locations of LVEA dust monitors:
1 ISCT4
3 Clean Room over BSC4
5 Beer garden
6 Diode room
7 previous location of H2 racks
8 H1 racks
10 H1 PSL