- CDS tutorial began today. - De-grouting work continues under H2 input optics tube area. - EPICS channels from chiller yard aren't functioning. Apparently, this has been the case for a while, but I didn't realize it. - David Nolting convenes weekly safety meeting.
Ran well pump twice today. King Soft Water out for monthly service and water check.
- Filiberto out to MX/MY to grab boards for Daniel Sigg. - HEPI installation continues in Biergarten. - Projector2 (itself, not the server) is out of operation. Patrick Thomas is investigating. - A Windows machine on the CDS network was errantly sending many repeated requests to the DCC server at Caltech, inviting attention from admins at Caltech and LHO. - (269) 978-1229 called twice, purporting to be from "Google" about updating our site listing. It seems to be telemarketers who are not affiliated with Google. - Swagelok delivery for Hugh Radkins. - Crew begins to arrive for CDS aLIGO training later this week. - Ochoa Middle School (Pasco, WA) tours the site.
BSC 7 now has all of the HEPI Assemblies installed. The South West corner saw the need for the HEPI foot position to be adjusted slightly relative to the HEPI frame. The bottom clamps that had the bent screws have all been replaced with the HoloKrome Carbon steel screws. New torque values were used as well for all of the 1/2-13 and set screws.
Summary: On Thursday Mar 24th, we made an attempt to fully wire-brush and sample inside iLIGO HAM12 (aLIGO HAM11). We were partly successful in that 2/3 of the chamber is now wire brushed, and we believe the procedure outlined in T1000725 will form the baseline for cleaning fully the H2 diagonal envelope, and later, L1 and H1 volumes. However in the process we erroneously made use of a screw-type air/oil compressor which may have contaminated some interior surfaces of the chamber. FTIRs were taken and are due back Thu Mar 31. They will inform of a solution for how to proceed with this chamber. Some details: Setting out to wire brush the iHAM12 interior, we defined a matrix for interior surfaces of the chamber and began wire brushing the roof. This work is ergonomically taxing, requiring sufficient person-power in-chamber, and short shifts of about 5 minutes each, particularly when overhead. The original (clean) compressor supplying the pneumatic tool was found to decay in air-pressure, inducing delays while the compressor regenerated, so we opted to acquire another, larger capacity unit. This second compressor (Atlas Copco XAS 185) is a screw compressor in which the pump unit employs oil in the screws and then separates it out, and cannot be employed for our clean in-chamber work. Running the pneumatic tool with the new compressor for 20-30min yielded a failure of the tool (not unexpected as that tool had integrated ~1hr worth of activity). When inspected, oil mist contamination was noted in the tool, the operation was shut down, FTIR samples were taken of both the areas that were brushed with the clean compressor, and the two areas (approx 4 square feet each) that were cleaned with the dirty compressor. We have ceased operations on iHAM12 and will do no more tests until any further mitigation steps are decided upon; additional assays may be required. Hoses and garb from Thursday's cleaning operation are quarantined, as are the vacuum. More details and pictures in a follow-up alog.
[Corey, Greg]
SW Location (s/n 77)
A job we hope would be a one hour task turned into a day long endeavor (which is still ongoing). Yesterday afternoon Greg drilled and tapped holes on the Pier (after the iLIGO Actuation Stack was removed). This morning we hoisted the new HEPI Assy onto the Pier and discovered that the Support Bracket holding it was an unmodified one, and we had interference issues(!).
So we lugged the HEPI Assy down to the ground and for safety/alignment reasons, re-installed the iLIGO Actuation Stack.
We then went to the NE HEPI, confirmed all the HEPI Foot clamping bolts were securely holding the Foot, and also checked that all the Base clamps were torqued down to the Pier. The Support Bracket here was then removed (note, this one only has the vertical slot modifications, not the cut out on the top of the Baseplate).
The SW Support Bracket was removed and handed over to Bubba for mods, and the NE one was installed at the SW corner. Vertical Dial Indicators were installed at these two corners to monitor changes.
Once again, we hoisted the HEPI Assy up onto the Pier and we cleared the interference on top, but just as we were getting ready to drop bolts into the Foot on top, we discovered ANOTHER INTERFERENCE! This time the base of the HEPI Assy ran into the top of the Bottom Plate of the Support Bracket (see photo). {Crap!}
Greg then checked the position of the Foot just to confirm we didn't make a mistake with pre-positioning of the Foot (we were a little off and in a way which helped us, but still off). So we now need to wait until we get a fully modified Support Bracket before we can continue anything at BSC7.
Bottom Clamps & Bent Bolts
At one point it was noticed that non-called out bolts were used for Bottom Clamps (see photo #3) for the NW & SE HEPIs. As these Clamps were loosened up and the bolts were swapped it was noticed that it was hard to remove some of the bolts. In fact, there was one bolt which was noticeably bent! (see photo #2)
Looks like the bend in the bolt occurs at the interface between the bottom of the Bottom Clamp & the top of the Pier. Apparently the 1/2"-20 set screws and the 35ft-lbs we'd been taking them too is just too much for the 1/2"-13 bolts here. Unless someone else chimes in we will do the following in the future:
-Torque 1/2"-13 bolt to 67ft-lbs
-Torque top 1/2"-20 set screws to 25ft-lbs
-Tighten the side 1/2"-20 set screws enough so they won't loosen up on their own.
For what its worth the stainless 1/2"-13 bolts were swapped out with the black HoloKrome bolts. (Need to do this bolt swap on the SE HEPI, too).
I cleaned up the vacuum alarm handler to remove those ion pump channels which are always in alarm. All vacuum alarms from now onwards should be treated as real alarms, and hopefull they will be infrequent. I also added the cyropump fill valve alarms, if these go off a pump overfill is imminent. New alarms have guidance text attached.
I changed the alarm startup script (called from the medm site map) to only start the relevant alarm handlers (relevant to the status of the IFO and site at this time).
Table legs were grouted into place today. We attached fans to them to help them dry.
PSL table legs grouted Wire brushing in HAM 12 Dirty room moved over new location of HAM 8 to drill mounting holes Sprague arrived BSC7 HEPI work
[Corey, Greg]
Had another late start today as we had to give up our crane till late morning (If we had the crane to ourselves, I'm sure we could have easily finished installing HEPIs today).
NW Location
Today, we pulled out the NW aLIGO HEPI (S/N 51) and swapped it with one which had a correctly-positioned HEPI Foot. As with the SE, this one dropped in nicely and required no horizontal positioning of the Housing (preserving the HEPI Foot range). As for vertical position, we went with a 0.25" Shim & had to drop the HEPI Foot slightly to accomodate.
Note: This one is missing a "Right" Housing Clamp. We have one ready to go and will install it tomorrow.
SW Location
Here we had a iLIGO Actuation Stack in place. This was removed and staged for "trash/recycling". The iLIGO Adapter Plate was traced on the Pier. But for the Drill Template, instead of lining up to the Adapter Plate mark, we set the Template to the (4)-corner Holes on top of the Pier. Greg then drilled and tapped the holes. I gathered up clamping hardware.
This HEPI (which will be S/N 77) will be installed in the morning to minimal fanfare.
Q3 - The final ECD blocks were added to the Main Chain. TFs were taken before and after the ECD blocks were installed, for comparisons. We currently do not have a spec on where to set the ECD blocks, so we'll need someone to look at the plots to discuss. Unfortunately, there is a problem with DTT's ability to re-display saved REF traces in saved files, so I don't have good plots to upload here. Jim Batch is looking into this problem which apparenly has been a problem in the controlroom as well. Q2 has been re-hung in the LSAT, refitted as an ITM with weights as per the following: PenRe = 59,289 g TCP dummy = 20,032 g PEN = 39,710 +/-20 g TM = 39,610 +/- 20 g The blades of the UIM and Top Masses had been reset to the new lower blade tip parameter set on the bench, so the hang from the UIM down on the LSAT was quite easy - very minimal adjustments made to the blade tip heights and lateral positions.
(Corey, Greg)
BSC7 NE, SE, & NW HEPI Installation Progress
After consulting with Hugh about NE HEPI work from yesterday, we figured out torque values to work with (1/2" bolts we took to 67ft-lbs & set screws for the base clamps we took to only 35ft-lbs). We also decided to put (3) thick washers under the two Crossbeam->HEPI Foot bolts which interfered with the HEPI Foot vertical Adjustment Screws. (Attached is a photo of these bolts---their heads are seen inside the Crossbeam).
In the afternoon, we moved onto doing a full swap at the SE location. The original HEPI was removed, and then a new one was craned up (this one had a correctly pre-positioned Foot). The Foot was mated up with the Crossbeam (basically make sure the four bolts on top fit through holes).
This swap was smooth in a few ways. For Vertical, we were very close to having a 0.25" gap. So we used the 0.25" Shim and only had to lower the Foot very slightly. The Foot was then torqued to the Crossbeam. At this point we checked horizontal positioning of the Assembly (yesterday, the Assembly was slightly off so we had to move the HEPI Housing structure). Today, our holes lined up, so we didn't have to make any horizontal moves (thus preserving HEPI Foot range---a good thing!).
The HEPI Assy which was installed here (South East) was S/N 45.
We are now staged to do a HEPI swap on the NW Pier in the morning.
HAM HEPI Foot Positioning
We moved to Foot Positioning in the morning since the WEST crane was tied up for most of the morning (HAM12 doors). Two HEPI Feet were positioned.
Throughout the day: HEPI install work on BSC7 scrub cleaning prep on HAM 12 – door removed first ~10:15 – contractor on-site for End-Y work ~4:30 – Grant and Robert out to LVEA to crane mounting on the H1 PSL for the squeezer work. Other: Dani and Richard modified the Card Access control so the readers will allow exiting doors without an exit swipe. The enter swipe is still active. This will reduce alarms in the control room, allow contractors and others to leave and enter without being “locked out”, and not pose any hazards, because the LVEA is “laser safe”.
I closed the check-valve bypass valve for CP8 this morning and enabled the instrument air to the LLCV. The PID is back in control of CP8's LN2 level.
CP8 overfilled following repeated reboot/restore cycles today. Dave B. needed to reboot the VE at the X-end to incorportate new parameters into the database. These new parameters were to correct for the similar but different output (to CDS) of the newly replaced and updated ion pump controller for IP12. Dataviewer data indicates that the PID output controlling CP8's liquid level control valve (LLCV) was erroneous following the burt restore and not based upon the PID's reading of CP8's actual level. The result was a rapid overfilling of CP8 following the manual enabling of the instrument air to the LLCV at some point after the restore. This overfilling resulted in exhaust vapor pressure building and GN2 leaving the 80K pump via three paths: 1) the normal exhaust line to atmosphere 2) backwards flow through the reneneration line and out to atmoshpere through a preexisting broken weld at the ambient air vaporizer and 3) into the VEA through the diaphram of the newly installed magnehelic differential presure gauge which had ruptured due to the high pressure (diaphram rated for 15 psig, nominally 1.5 psig). -> I valved-out the broken magnehelic gauge and will replace it later. -> I opened the exhaust line check-valve bypass to to reduce exhaust line back pressure. -> I placed my hand on the Burst Line pipe (it was warm) and confirmed that the pump's burst disk hadn't ruptured. Dave B. was able to repeat the events and to observe the bogus PID output following a subsequent reboot of the VE rack (remote reboot?). He will continue his investigation next Tuesday to better understand the events and to propose a fix. I am leaving the instrument air "not enabled" to prevent any further filling of CP8 overnight and will restore PID control after the pump level receeds back below alarm levels (tomorrow).
Kyle will write a separate alog with the vacuum details of this event.
I rebooted h0veex to fix the ion pump epics database to match the new hardware. After the IOC was rebooted and the PID control of the LN2 level in CP8 was enab led the pump quickly overflowed because the fill valve was incorrectly set by the software to 92% open. I tested the EPICS code and found that if the IOC was soft booted, or reset by the watchdog card, the fill valve was always opened to the LN2 level set point (nominally 92%). I went to EX and power cycled the VME crate twice, the first time the fill valve started at 41% (which happened to be half the setpoint at the time) and the second time the IOC started correctly with 0% valve opening (because CP8 was 112% full at this time).
I did an initial investigation of the database and no immediate problem could be found. Having the output of the PID record be essentially the setpoint indicates that the controlled PV is zero on its first run, but it is not obvious how this could be. Also the different starting behavior between a reboot and a power cycle may indicate the erasure of mv162 RAM may be an issue.
This is the first time we have encountered this error on any PID controlled LN2 level, so aging hardware may also be a factor. We will continue the investigation this week and would like to reboot this system again next Tue maintence to test its robustness.
We are also discussing adding software limits to prevent this from happening again.
Yesterday Robert and I found that the pump in the mechanical room that pulls air for the dust monitors that have their internal pump removed was not pulling air. Today I read the flow on the attached flow meter as .15. This means that the data from locations 1 (ISCT4), 7 (H2 racks) and 10 (H1 PSL) has been invalid for an unknown period of time. Also possibly location 8 (H1 racks), which had been replaced with one with an internal pump earlier. So today I went and replaced locations 1, 7 and 10 with ones that have an internal pump. Note also that dust monitor 3 was moved yesterday into a clean room next to where the jack hammering is taking place. Also for a period yesterday, location 4 was moved onto the floor by the H1 PSL, but has been moved to location 8 today. Locations of LVEA dust monitors: 1 ISCT4 3 Clean Room over BSC4 5 Beer garden 6 Diode room 7 previous location of H2 racks 8 H1 racks 10 H1 PSL
3 moved into clean room over HAM 12 4 added outside clean room over HAM 12 9 added at H2 PSL area
