800 - Michael Rodruck restarting PSL

900 - Betsy/Travis working in HAM2

930 - SnoValley working on chillers

1030 - Michael R engaged H1 PSL enclosure chiller to bring temperature down

1040 - HEPI pump station restarted by Hugh. Leak checking.

1250 - HFD burning tumbleweeds along RT 10 between LHO and Wye barricade

1337 - PNNL called to say vibroseis "shaker" truck now active at old Nike missile site. (HAWA--- old "Big G" experiment site)

1450 - Cheryl transitioning LVEA to laser hazard.

Before starting messing up with IO alignments, Cheryl asked my to check if any major drift was visible in he HAM2 and 3 suspensions after the outage.

For the IMs, I restored the offsets that were in effect before the outage, and compared the OSEMs readings. No significant drift was observed. Results are summarized in the following table (the columns on the right are just a visual representation of the difference in log scale).

-------------------------------------------------------------------------------------------------------------
OSEMs readings comparison between gps = 1048629616 (reference) and gps = 1049482936
-------------------------------------------------------------------------------------------------------------
Channel name                      Value at T1     Value at T2        Differ.   1       10       100      1000
-------------------------------------------------------------------------------------------------------------
H1:SUS-IM1_M1_OSEMINF_UL_OUT_DQ   +003.73+-5.18   +004.11+-1.16      +000.38   |        |         |         |
H1:SUS-IM1_M1_OSEMINF_LL_OUT_DQ   -082.58+-5.41   -080.89+-1.22      +001.70   |>       |         |         |
H1:SUS-IM1_M1_OSEMINF_UR_OUT_DQ   +017.31+-4.85   +018.25+-1.16      +000.94   |        |         |         |
H1:SUS-IM1_M1_OSEMINF_LR_OUT_DQ   -084.14+-5.18   -081.39+-1.28      +002.75   |>>>     |         |         |
-------------------------------------------------------------------------------------------------------------
H1:SUS-IM2_M1_OSEMINF_UL_OUT_DQ   -290.53+-2.85   -289.29+-1.90      +001.24   |        |         |         |
H1:SUS-IM2_M1_OSEMINF_LL_OUT_DQ   -242.12+-3.14   -236.62+-2.46      +005.50   |>>>>>>  |         |         |
H1:SUS-IM2_M1_OSEMINF_UR_OUT_DQ   +236.39+-3.13   +238.20+-3.44      +001.81   |>>      |         |         |
H1:SUS-IM2_M1_OSEMINF_LR_OUT_DQ   +132.75+-2.92   +138.67+-2.79      +005.92   |>>>>>>> |         |         |
-------------------------------------------------------------------------------------------------------------
H1:SUS-IM3_M1_OSEMINF_UL_OUT_DQ   +171.96+-3.88   +170.05+-0.93      -001.90   |>>      |         |         |
H1:SUS-IM3_M1_OSEMINF_LL_OUT_DQ   -090.16+-3.86   -086.71+-0.90      +003.45   |>>>>    |         |         |
H1:SUS-IM3_M1_OSEMINF_UR_OUT_DQ   +141.78+-3.64   +139.41+-0.89      -002.37   |>>>     |         |         |
H1:SUS-IM3_M1_OSEMINF_LR_OUT_DQ   -138.79+-4.38   -135.01+-1.05      +003.77   |>>>>>   |         |         |
-------------------------------------------------------------------------------------------------------------
H1:SUS-IM4_M1_OSEMINF_UL_OUT_DQ   -075.93+-4.06   -063.20+-7.54      +012.74   |>>>>>>>>|>        |         |
H1:SUS-IM4_M1_OSEMINF_LL_OUT_DQ   +283.91+-4.13   +295.35+-8.44      +011.44   |>>>>>>>>|>        |         |
H1:SUS-IM4_M1_OSEMINF_UR_OUT_DQ   -210.78+-3.92   -199.14+-5.11      +011.64   |>>>>>>>>|>        |         |
H1:SUS-IM4_M1_OSEMINF_LR_OUT_DQ   +123.32+-4.36   +134.96+-7.23      +011.64   |>>>>>>>>|>        |         |

I repeated the check for MC1, MC3 and PRM suspension. For technical reasons I had to keep the offsets off, so I compared the readings with a moment before the outage in which the watchdogs were tripped. Again, no major drift was observed.

-------------------------------------------------------------------------------------------------------------
OSEMs readings comparison between gps = 1048658416 (reference) and gps = 1049464816
-------------------------------------------------------------------------------------------------------------
Channel name                      Value at T1     Value at T2        Differ.   1       10       100      1000
-------------------------------------------------------------------------------------------------------------
H1:SUS-MC1_M1_OSEMINF_T1_OUT_DQ   -034.70+-0.03   -034.09+-0.03      +000.61   |        |         |         |
H1:SUS-MC1_M1_OSEMINF_T2_OUT_DQ   -036.14+-0.05   -036.29+-0.03      -000.15   |        |         |         |
H1:SUS-MC1_M1_OSEMINF_T3_OUT_DQ   -003.59+-0.04   -003.57+-0.03      +000.02   |        |         |         |
H1:SUS-MC1_M1_OSEMINF_SD_OUT_DQ   +006.22+-0.20   +006.87+-0.05      +000.65   |        |         |         |
H1:SUS-MC1_M1_OSEMINF_LF_OUT_DQ   +010.79+-0.23   +010.52+-0.05      -000.27   |        |         |         |
H1:SUS-MC1_M1_OSEMINF_RT_OUT_DQ   +028.01+-0.23   +028.59+-0.05      +000.58   |        |         |         |
H1:SUS-MC1_M2_OSEMINF_UL_OUT_DQ   +012.16+-0.39   +011.65+-0.07      -000.51   |        |         |         |
H1:SUS-MC1_M2_OSEMINF_LL_OUT_DQ   +257.26+-0.39   +256.64+-0.06      -000.62   |        |         |         |
H1:SUS-MC1_M2_OSEMINF_UR_OUT_DQ   -134.76+-0.34   -135.18+-0.06      -000.42   |        |         |         |
H1:SUS-MC1_M2_OSEMINF_LR_OUT_DQ   +141.28+-0.60   +141.81+-0.08      +000.53   |        |         |         |
H1:SUS-MC1_M3_OSEMINF_UL_OUT_DQ   -046.67+-0.62   -047.22+-0.09      -000.55   |        |         |         |
H1:SUS-MC1_M3_OSEMINF_LL_OUT_DQ   +260.52+-0.57   +259.95+-0.07      -000.57   |        |         |         |
H1:SUS-MC1_M3_OSEMINF_UR_OUT_DQ   -067.45+-0.57   -067.82+-0.08      -000.37   |        |         |         |
H1:SUS-MC1_M3_OSEMINF_LR_OUT_DQ   +198.45+-0.65   +198.63+-0.08      +000.19   |        |         |         |
-------------------------------------------------------------------------------------------------------------
H1:SUS-MC3_M1_OSEMINF_T1_OUT_DQ   -008.19+-0.04   -008.33+-0.04      -000.14   |        |         |         |
H1:SUS-MC3_M1_OSEMINF_T2_OUT_DQ   -015.25+-0.07   -015.22+-0.03      +000.03   |        |         |         |
H1:SUS-MC3_M1_OSEMINF_T3_OUT_DQ   -020.96+-0.06   -021.16+-0.03      -000.19   |        |         |         |
H1:SUS-MC3_M1_OSEMINF_SD_OUT_DQ   -006.14+-0.07   -005.79+-0.05      +000.35   |        |         |         |
H1:SUS-MC3_M1_OSEMINF_LF_OUT_DQ   +008.46+-0.31   +008.51+-0.06      +000.05   |        |         |         |
H1:SUS-MC3_M1_OSEMINF_RT_OUT_DQ   -057.79+-0.33   -056.71+-0.06      +001.08   |        |         |         |
H1:SUS-MC3_M2_OSEMINF_UL_OUT_DQ   +143.88+-0.58   +144.30+-0.09      +000.42   |        |         |         |
H1:SUS-MC3_M2_OSEMINF_LL_OUT_DQ   +073.77+-0.91   +074.45+-0.11      +000.68   |        |         |         |
H1:SUS-MC3_M2_OSEMINF_UR_OUT_DQ   +079.45+-0.53   +080.13+-0.07      +000.68   |        |         |         |
H1:SUS-MC3_M2_OSEMINF_LR_OUT_DQ   +044.42+-0.73   +045.32+-0.08      +000.90   |        |         |         |
H1:SUS-MC3_M3_OSEMINF_UL_OUT_DQ   +092.77+-0.97   +093.25+-0.12      +000.48   |        |         |         |
H1:SUS-MC3_M3_OSEMINF_LL_OUT_DQ   +115.28+-1.26   +116.35+-0.16      +001.07   |        |         |         |
H1:SUS-MC3_M3_OSEMINF_UR_OUT_DQ   +082.64+-0.83   +083.89+-0.09      +001.25   |        |         |         |
H1:SUS-MC3_M3_OSEMINF_LR_OUT_DQ   +024.52+-1.46   +026.39+-0.16      +001.87   |>>      |         |         |
-------------------------------------------------------------------------------------------------------------
H1:SUS-PRM_M1_OSEMINF_T1_OUT_DQ   -010.93+-0.05   -011.00+-0.04      -000.07   |        |         |         |
H1:SUS-PRM_M1_OSEMINF_T2_OUT_DQ   +018.33+-0.06   +017.93+-0.04      -000.40   |        |         |         |
H1:SUS-PRM_M1_OSEMINF_T3_OUT_DQ   -011.63+-0.07   -011.87+-0.04      -000.24   |        |         |         |
H1:SUS-PRM_M1_OSEMINF_SD_OUT_DQ   -050.86+-0.21   -050.43+-0.06      +000.42   |        |         |         |
H1:SUS-PRM_M1_OSEMINF_LF_OUT_DQ   +001.20+-0.29   +001.34+-0.03      +000.14   |        |         |         |
H1:SUS-PRM_M1_OSEMINF_RT_OUT_DQ   -013.73+-0.29   -013.04+-0.03      +000.69   |        |         |         |
H1:SUS-PRM_M2_OSEMINF_UL_OUT_DQ   +064.17+-0.60   +064.34+-0.06      +000.17   |        |         |         |
H1:SUS-PRM_M2_OSEMINF_LL_OUT_DQ   +057.88+-0.88   +058.35+-0.07      +000.47   |        |         |         |
H1:SUS-PRM_M2_OSEMINF_UR_OUT_DQ   +021.27+-0.56   +021.52+-0.06      +000.25   |        |         |         |
H1:SUS-PRM_M2_OSEMINF_LR_OUT_DQ   +027.12+-0.81   +028.15+-0.06      +001.04   |        |         |         |
H1:SUS-PRM_M3_OSEMINF_UL_OUT_DQ   +023.28+-0.94   +024.00+-0.08      +000.72   |        |         |         |
H1:SUS-PRM_M3_OSEMINF_LL_OUT_DQ   -002.46+-1.23   -000.09+-0.09      +002.37   |>>>     |         |         |
H1:SUS-PRM_M3_OSEMINF_UR_OUT_DQ   +027.42+-0.91   +027.77+-0.07      +000.35   |        |         |         |
H1:SUS-PRM_M3_OSEMINF_LR_OUT_DQ   +066.64+-1.28   +068.56+-0.09      +001.92   |>>      |         |         |

Finally, MC2 had to be compared with offsets off, but from a different period. Again, no major drifts or shifts in alignment.

-------------------------------------------------------------------------------------------------------------
OSEMs readings comparison between gps = 1048939216 (reference) and gps = 1049464816
-------------------------------------------------------------------------------------------------------------
Channel name                      Value at T1     Value at T2        Differ.   1       10       100      1000
-------------------------------------------------------------------------------------------------------------
H1:SUS-MC2_M1_OSEMINF_T1_OUT_DQ   -032.01+-0.04   -028.31+-0.06      +003.70   |>>>>>   |         |         |
H1:SUS-MC2_M1_OSEMINF_T2_OUT_DQ   -076.77+-0.04   -074.11+-0.06      +002.66   |>>>     |         |         |
H1:SUS-MC2_M1_OSEMINF_T3_OUT_DQ   +007.41+-0.04   +010.39+-0.06      +002.98   |>>>>    |         |         |
H1:SUS-MC2_M1_OSEMINF_SD_OUT_DQ   -001.64+-0.06   -001.30+-0.12      +000.34   |        |         |         |
H1:SUS-MC2_M1_OSEMINF_LF_OUT_DQ   +025.48+-0.11   +024.70+-0.20      -000.78   |        |         |         |
H1:SUS-MC2_M1_OSEMINF_RT_OUT_DQ   +021.43+-0.08   +020.58+-0.14      -000.85   |        |         |         |
H1:SUS-MC2_M2_OSEMINF_UL_OUT_DQ   +027.67+-0.16   +026.48+-0.32      -001.20   |        |         |         |
H1:SUS-MC2_M2_OSEMINF_LL_OUT_DQ   +022.56+-0.22   +020.81+-0.45      -001.75   |>       |         |         |
H1:SUS-MC2_M2_OSEMINF_UR_OUT_DQ   +033.66+-0.17   +032.37+-0.32      -001.29   |        |         |         |
H1:SUS-MC2_M2_OSEMINF_LR_OUT_DQ   +052.13+-0.20   +050.76+-0.42      -001.37   |        |         |         |
H1:SUS-MC2_M3_OSEMINF_UL_OUT_DQ   +029.85+-0.25   +028.22+-0.54      -001.63   |>       |         |         |
H1:SUS-MC2_M3_OSEMINF_LL_OUT_DQ   -010.71+-0.29   -012.82+-0.61      -002.12   |>>      |         |         |
H1:SUS-MC2_M3_OSEMINF_UR_OUT_DQ   -020.21+-0.24   -021.43+-0.51      -001.22   |        |         |         |
H1:SUS-MC2_M3_OSEMINF_LR_OUT_DQ   +038.40+-0.34   +035.94+-0.73      -002.46   |>>>     |         |         |

The RF source signals became dirty. This is currently under investigation.

In the past Friday I was checking the electronics at the field racks as a part of the power recovery activity. And I found that the RF source signals from the RF distribution amplifiers had become dirty -- there are lots of harmonics lines and cross-talks (see below). If I remember correctly those were not seen when we tested the RF stuff in the last Month [1][2] and therefore something must have happened since then. The raw data files are attached as well.

Figures: output from the 24 MHz RF amp (figure 1), 9 MHz RF amp (figure 2) and 45 MHz RF amp (figure 3).
 

[1] LHO alog 5854 "Installation and Checkout of RF signal chains on ISCT1 and HAM1"
[2] LHO alog 5849 "The 9MHz amplifier box is back"

After fixing any loose dog clamps, and retorquing, Arnaud and I took B&K hammer measurements of the HAM2 PR3, PRM, and MC1 this morning.  We still need to do PR2 and MC3 (which has a temp IO alignment mirror nestled up against it's structure).  We took PR3 and PRM data with vibration absorbers on and off.  We only took MC1 data with the vibration absorbers on since they were already on and are difficult to remove and then replace.  Plots and statements to come.

The HEPI Pump Stations were not restarted after the outage.  They will not restart automatically at this time.
The fluid level relay box must be reset and the epics must be restarted manually.  Additionally, the Pump Station Controller VFD must be set to fwd/run to operate.  Once that is completed, the motors may be ramped up and the pressure servo engaged.
I did a 100% inspection of all lines.  Main line pressure is servoing at 28psi with the motor speed at ~2000/2048.

I documented all the BIOS settings for the EY computers. The h1hwsey BIOS was modified to prevent auto-power-on on power restoration. All the front ends were then restarted. We had to reconnect h1seih45 to the Dolphin network (its cable had been removed during initial roll out) to complete the set and get every Dolphin'ed front end going.

No major changes other than: the overflowing h1pslfss DAC channels as before (Rick said it would be ok to continue running this model); the h1susmc3 model would not auto-start, but we could manually start it; and h1susquadtst is showing timing errors (nothing connected to this system).

The DAQ was restarted. The EDCU is missing the PSL Beckhoff controller channels, presumably this computer is powered down.

The dust monitors and weather stations have been restarted and burt restored to the files in /ligo/cds/lho/h0/burt/2013/04/01/00:00.
I also burt restored FMCS to the same time.

The dust monitor at end Y location 2 has been swapped out for the one labeled 'S'. The previous one was removed on Tuesday (calibration failure). I removed the dust monitor in the LVEA at location 1 (calibration failure for a long time). The dust monitor at location 5 in the LVEA was moved out from under the clean room over BSC2 and into the beer garden.

The dust monitors in the PSL enclosures still need to be restarted.

- Apollo installed a couple of viewports for OL, and removed a couple of doors from HAM chambers, see Bubba's entry for more info.
- outside Tech serviced the FM200 system, system is back now.
- Dave continued working with bringing back the CDS system, more green now than this morning.
- Hertz in to take back their generators, all gone now.
- Patrick bringing back dust monitors.
- PSL still down, no light.
- We still have a 2 CCs (VE) down (red) from the power outage, one less than this morning, yes they were toggled now we wait (424B & 524B).

We removed the east door from ham 3 and both doors from ham 2. We are ready to remove the west door on ham 3, however C 3 ham door covers are rare, therefore we subsituted with dome covers and used all of those. We will replace the subsitute covers with the appropriate covers when they arrive, hopefully Monday. We also installed viewports (serial #s) 23&26 on the H 1 MC input tube.

In addition to other work that went on during the power outage.  We cleaned the Switchgear cubicles and cycled the breakers in key areas.  The MCC in the Carpenter shop. The MCC that feeds the chillers. This unit looks as if condensation in forming in the bottom a fair amount of rust has formed.  It also had dead mice in the gear.  The main switches in the Mechanical Room.  The MCC in the LSB.  The main panels in the End Stations.  We did not get to the Mid stations as we can do them at anytime and not bother the IFO.

Robert ran some seismic tests in the LVEA.  One problem was the Generator running by the carpenter shop.  Large peak in the data.

We also checked the status of the Emergency lights across the site.  We had a few immediate failures and some that did not last the full 90minutes required.  We will be addressing these issues.  A few actually lasted 3 hours.

Of particular concern were the exit lights in the OSB.  I will get with John to discuss a path forward.
Also the FM200 panel did not survive the outage.  We lost a power board around 8am

All in all the outage went smoothly and the return to normal only had some minor glitches.

Summary of Thursday CDS recovery:

All networking switches were started. hanford1 and hanford2 were started. hanford1 needed several restart attempts.

The vacuum controls workstation in the control room was started. The powered down Vacuum controls VME systems were restarted (h0veex, h0veey, h0velx, h0vely, h0vemr) along with their local workstations. All VE systems were burt restored to 23:00 Monday 1st April.

DC power was restored to the H1CER. h1psl0 was started to permit startup of the DAQ system. h1dc0 was started. We noticed a check led light on the Fujitsu 10GE switch, this was cleared before h1fw1 was started. A problem with the SATABOYs was tracked to solaris 11 not auto NFS sharing. Dan upgraded the MSR ldasgw machines to sol11.1 and patched, but still had to manually share the file systems. Also the MSR DAQ SATABOY performed an auto parity scan.

The cdswiki machine took several restarts before it booted. The MEDM snapshot program was started on script0 and the internal EPICS gateways were started. The FMCS EPICS IOC was started. The NAT router was started. At this point the VE and FMCS MEDM screens are available on the CDS web page.

The DAQ system was restarted, the VE and FMCS channels are being trended.

The autoburt system was started.

We noticed that the PSL FSS model was overdriving some DAC channels, so we stopped the PSL models. We started the h1 susaux front ends.

The front end and DAQ computers' BIOS settings were documented.

Following the site wide power outage from 9am Wed 3rd to 11am Thurs 4th April we are presently recovering the CDS systems.

At this point, we have started all the vacuum controls racks, the networking systems and the FMCS systems. A full recovery procedure will be posted in the DCC when we have completed the startup.

I'll be manually topping 80K pump levels and checking pressures etc. regularly during the outage.

Last week Chris and Kiwamu told me the slow controls weren't working for the low noise VCOs.  They are now working. 

Some of the issues were:
The corner 3 beckhoff chassis middle rail was not powered correctly, Filiberto fixed this.  
The inputs labeled RF mon on the tester go to pins that the wiring diagram indicates are temp mons, and the rf mon inputs go to pins the wiring diagram indicates are RF mons.  If the tester is right and the w 
The cable for the comm vco was unplugged at the beckhoff chassis
there were no links in the system manager for the diff vco

The Beckhoff modules used as ADCs on the IMC VCO (EL3102) are different from the ones used for the ALS VCOs (EL3104) but the conversion is identical for the different modules, so no changes the TwinCAT library are required

The RF detector used in the VCO (Minicurcuits ZX47-50LN-S+) has a temperature sensor, which the data sheet says has a linear slope of 2mV/degree C, and gives 0.6V at 20C, with the factor of ten gain in the schematic D100064, it seems like the equation used by the library (T=20+50*(V-6)) is right although the values displayed for temperatures do not make sense.  Looking at the test procedure for the VCO it doesn't seem like this part was tested.  

I turned the HEPAs and AC units off in both the H1 and H2 rooms. I've left the make up air on at 100%.

Last full day of power - tomorrow the site will have it's power shut down - DOE making changes.

Site activity:
- lots of activity with shutting things off in every building
- crane activity this morning - all are currently locked out
- two loads of Praxair
- crate inspection at MY for PSL parts
- people at EY to get the fiber working
- people looking at storage
- dust monitors moved around and staged in strategic places - some will be recording data and people will be coming in to swap out batteries
- cryopump work shimming on LN2 transfer lines
- all optics have their Master Switches set to "OFF"
- doors are on the HAM chambers
- class-A hardware in the H2 PSL is covered
- a snapshot of the vacuum overview medm is attached - unusual with all the red