TITLE: 12/21 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC

STATE of H1: Observing at 74.4192Mpc

INCOMING OPERATOR: Ed

SHIFT SUMMARY: Been Observing for 4 hours and counting. Quiet shift so far. Lockloss a little bit after Robert went in to LVEA (I forgot to log this one), coincide with a 6.7M earthquake in East Timor but the earthquake didn't show up that high on the FOM (only 1um/s max). 

LOG:

0:27 Kyle back from mid Y

0:30 Kyle going back out

0:44 Robert out doing opportunistic injection in PSL and LVEA (LLO is down).

1:55 Kyle back

02:54 Observe

3:11 Out of Observe. LLO is down so Jeff is doing an opportunistic calibration work.

03:14 Lockloss. ASC signals went unstable. Jeff suspect high useism.

04:12 Observe

Carlos, Jim B, Dave, Nutsinee

Follow up of WP6382

I took advantage of the earlier down time to test the HWS stream images script and swapping CLink around. The original set up after Carlos installed the second PCI-e card was HWSY CLink was plugged into the new card and HWSX CLink remained the same (plugged to old card). HWSX streamed live images without problem while HWSY can't get any sensible image at all (see attachment). So I tried a few more configurations:

• HWSX CLink plugged to the new card, HWSY plugged to the old card
  • Result: HWSY worked, HWSX didn't
• HWSX and HWSY CLink both plugged into the new card
  • Result: None worked
• HWSX and HWSY CLink both plugged into the old card
  • Result: Both worked (just like how they were prior to adding the second card)

We have come to the conclusion that the spare card might be busted. The next thing to try is to order a new card and try again. This way we will know for sure that the card actually works. We don't know where the spare card came from and whether or not it has been tested.

This has not been attempted previously to CP3 or CP4 though it is not a new or original idea.  

Both CP3 and CP4 are thought to be experiencing the same issue, i.e. an obstructed level sensing line.  This could be due to the accumulation and clumping of "snow flakes" of water ice.  It may be that the relative buoyancy in LN2 of these "snow flakes" is concentrating them near the bottom of the LN2 pump which is also where the lower sensing line enters the pump and samples the LN2.  With an increasingly dense concentration over time it is easy to imagine these "snowflakes" interlocking and forming clumps which eventually restrict the ID of the sensing line (the "cholesterol" theory!) to the point of complete obstruction.    

In the past we had applied UHP N2 (various pressures/durations -> Max. of 180 psi) to these clogged lines in an attempt to "push" this obstruction back into the 80K pump.  These attempts gave no indication of a resulting movement of the obstruction nor flow of the applied UHP N2.  Today, while changing the most recently depleted cylinder of UHP LN2 (leak into the room), I decided to reconnect the sensing line to the local differential pressure gauge.  I noticed that it indicated a differential pressure of 5" of water.  I repeated the process of decoupling the line from the gauge and exposing it to atmosphere for a period then reconnecting it to the gauge.  Each time the pressure difference increased by ~0.5" of water.  This got me thinking..... 

Considering the relatively large mass of water that can be extracted from food in the freeze drying process while exposed to vacuum and in such a relatively short amount of time, I discussed with John W. the risks and possible benefits of applying vacuum to our clogged CP4 line.  The hope is to encourage the sublimation of the "ice plug".  It was decided that applying a vacuum to the sensing line would be low risk at this point.   

So, late in the day, I hooked up a small diaphragm pump and pumped on the sensing line for a few minutes.  After reconnecting it to the differential pressure gauge and CDS level transducer I found that the indicated pressure difference had increased.  Repeated cycles produced repeated benefit (see attached image).  Of particular note is that an audible release of built-up gas could be heard each time the sensing line connection was loosened to do the next round of vacuum pumping.  This suggests that the CDS indicated pressure difference is due to a real pressure difference and that, perhaps, the vacuum pumping has established a flow path through the plug!  Or, that the ice plug has been pulled into a warm area.

Gerardo and I will continue tomorrow.  Chandra R. will be back after the new year. 

Resumed pumping 2K IMC annulus ion pump volume with pump carts during the maintenance period today.  Energized ion pump controller after an hour or so of pump cart-only pumping and the combination got the ion pump current down to 5ma -> Left ion pump controller on but had to shut down the assisting pump carts at the end of the maintenance period.  As of this writing, it looks like the ion pump current is just now coming back on scale so, perhaps, additional assisted pumping won't be required.

WP6399 cdsfs0 raid firmware update

Carlos:

cdsfs0 raid controller card firmware was updated. The /ligo file system was offline for a short period of time

WP6382 add second camera card in h1hwsmsr

Carlos, Nutsinee:

The spare HWS camera pci-e card was installed in h1hwsmsr and one of the camera was moved over to it (see photo)

WP6407 start robo-alog code

Jonathan, Ryan, Dave:

The new robo-alog entry system was tested and is ready for use by the vacuum autofill process.

Attached:

• HEPI pumps - 45 days
• Temperatures - 45 days
• HEPI pumps - 10 days
• Temperatures - 10 days

Some drop in presure - possibly due to temperature.

Now the test timing system in the EE lab is powered down, h1ecatc1plc1 reported a difference for the master on port 7. Since this is a test system, we unmonitored this channel.

This is a test of posting from script0 using robot credentials.

This is a test of posting from script0 using robot credentials.
Have a very nice day.

After a little fix, moving cables to their correct block assignment (II cables were swapped at the box by the controller), the new cable was landed for ion pump IP12.

We are connected to the EI/II522A channels (all others not connected because they are not used anymore), work done per WP 6408.

Next, code will need to be updated, both in Beckhoff and MEDM world, to show upgrade to the new LPC Gamma type controller (positive polarity).

The IM alignments were adjusted many times in the commissioning ramp up to O2, so IMs have not had nominal values for some time, and DIAG_MAIN values were not consistant with the current IM values.

I adjusted IM1-3 alignments after taking some measurements during maintenance today, and in looking at the IM alignment history.

I saw the IM1 pitch had bee restored to the DIAG_MAIN value yesterday, and this was not a correct value.

The current nominal values for IM1-3 OSEM alignments, as deemed good on Dec 8th at 09:00UTC, after a discussion with Jenne, are:

These are also listed in the IM Nominal Alignment medm (IOO sitemap list), and are updated in the DIAG_MAIN code.

To check IM alignment:

• use the procedure in the "Recovering IM Alignment After ISI Trip" wiki page

Today, I performed the LVEA sweep after everyone finished their Tues maintenance work.  I confirmed everything on the list, except for 2 of the 3 crane parking placements - we didn't use any cranes today and they have been parked in the same place for over a week.  I also only glanced at the electronics racks, since we also had done a nice sweep to disconnect miscellaneous stuff from racks prior to the run starting.  I did not see any new dangling pieces of test equipment or other interesting looking stuff.

I did unplug the West bay phone along the wall - likely on since the start of O1 since it's a somewhat unfamiliar phone to people.  Here is the full list of LVEA phones I found and their status as of now (which should be added to the sweep checklist):

1) Inside H1 PSL enclosure - OFF

2) On tool box or table near H1 PSL enclosure - OFF

3) West bay along wall near crane parking sign - now OFF

4) On table near HAM7 East door - OFF

5) Along wall near HAM6 North - OFF

6) In Squeezer Bay on table - OFF

7) Back corner of electronics room, on floor - OFF

8) I suspect possibly one is inside H2 PSL enclosure - called it, didn't see it on HAM7 mic, unconclusive if it exists...

This morning, Jason, Mark and I swapped the assumed-to-be failing TCSY flow sensor which has been showing epochs of glitching and low readout (while other indicators show normal flow, alogs 32712 and 32230).  The process to do this was such:

1) Key laser off at control box in rack, LVEA

2) Turn RF off at mezzanine rack, Mech room

3) Turn chiller off on mezzanine, Mech room

4) Turn power off on back of controller box in rack, LVEA (we also pulled the power cable to the sensor off the front of the controller, but it was probably overkill)

5) Close in-line valves under BSC chamber near yellow sensor to-be-swapped, LVEA

6) Quick-disconnect water tubes at manifold near table, LVEA

7) Pulled yelow top off of yellow sensor housing under BSC at the piping, LVEA

8) Pulled the blue and black wires to the Power recepticles inside the housing (see pic attached).  Pulled full grey cable out of housing.

9) While carefully supporting blue piping*, unscrewed large white nut holding housing/sensor to piping (was tough, in fact so tough that we later removed all of the teflon tape which was unneeded in his join)

10) Pull* straight up on the housing (hard) and it comes out of the piping.

11) Reverse all above steps to insert new housing/sensor, wires and turn everything back on.  Watch for rolled o-rings on the housing and proper alignment of the noth feature when installing the new sensor.  Verify mechanical flow sensors in piping line show ~3-4 G/m readout when flow/chiller is restored to functionality.

12) Setup new flow sensor head with Settings:  Go to the other in-use sensor, pull off the top and scroll through the menu items (red and white buttons on the unit (shown in pic).  Set the new head to these values.

13) Verify the new settings on the head are showing a ~3 G/m readout on the medm screen.  If not, possibly there is setting on the sensor that needs revisited.

14) Monitor TCS to see that laser comes back up and stabilizes.

* Blue piping can crack so be careful to always support it and avoid torque torque

Note - with the sensor removed, we could see alot of green merk in the blue piping where the paddle wheel sits.  Still suffering green sludge in this system...