The dust monitors and weather stations have been restarted and burt restored to the files in /ligo/cds/lho/h0/burt/2013/04/01/00:00.
I also burt restored FMCS to the same time.

The dust monitor at end Y location 2 has been swapped out for the one labeled 'S'. The previous one was removed on Tuesday (calibration failure). I removed the dust monitor in the LVEA at location 1 (calibration failure for a long time). The dust monitor at location 5 in the LVEA was moved out from under the clean room over BSC2 and into the beer garden.

The dust monitors in the PSL enclosures still need to be restarted.

- Apollo installed a couple of viewports for OL, and removed a couple of doors from HAM chambers, see Bubba's entry for more info.
- outside Tech serviced the FM200 system, system is back now.
- Dave continued working with bringing back the CDS system, more green now than this morning.
- Hertz in to take back their generators, all gone now.
- Patrick bringing back dust monitors.
- PSL still down, no light.
- We still have a 2 CCs (VE) down (red) from the power outage, one less than this morning, yes they were toggled now we wait (424B & 524B).

We removed the east door from ham 3 and both doors from ham 2. We are ready to remove the west door on ham 3, however C 3 ham door covers are rare, therefore we subsituted with dome covers and used all of those. We will replace the subsitute covers with the appropriate covers when they arrive, hopefully Monday. We also installed viewports (serial #s) 23&26 on the H 1 MC input tube.

In addition to other work that went on during the power outage.  We cleaned the Switchgear cubicles and cycled the breakers in key areas.  The MCC in the Carpenter shop. The MCC that feeds the chillers. This unit looks as if condensation in forming in the bottom a fair amount of rust has formed.  It also had dead mice in the gear.  The main switches in the Mechanical Room.  The MCC in the LSB.  The main panels in the End Stations.  We did not get to the Mid stations as we can do them at anytime and not bother the IFO.

Robert ran some seismic tests in the LVEA.  One problem was the Generator running by the carpenter shop.  Large peak in the data.

We also checked the status of the Emergency lights across the site.  We had a few immediate failures and some that did not last the full 90minutes required.  We will be addressing these issues.  A few actually lasted 3 hours.

Of particular concern were the exit lights in the OSB.  I will get with John to discuss a path forward.
Also the FM200 panel did not survive the outage.  We lost a power board around 8am

All in all the outage went smoothly and the return to normal only had some minor glitches.

Summary of Thursday CDS recovery:

All networking switches were started. hanford1 and hanford2 were started. hanford1 needed several restart attempts.

The vacuum controls workstation in the control room was started. The powered down Vacuum controls VME systems were restarted (h0veex, h0veey, h0velx, h0vely, h0vemr) along with their local workstations. All VE systems were burt restored to 23:00 Monday 1st April.

DC power was restored to the H1CER. h1psl0 was started to permit startup of the DAQ system. h1dc0 was started. We noticed a check led light on the Fujitsu 10GE switch, this was cleared before h1fw1 was started. A problem with the SATABOYs was tracked to solaris 11 not auto NFS sharing. Dan upgraded the MSR ldasgw machines to sol11.1 and patched, but still had to manually share the file systems. Also the MSR DAQ SATABOY performed an auto parity scan.

The cdswiki machine took several restarts before it booted. The MEDM snapshot program was started on script0 and the internal EPICS gateways were started. The FMCS EPICS IOC was started. The NAT router was started. At this point the VE and FMCS MEDM screens are available on the CDS web page.

The DAQ system was restarted, the VE and FMCS channels are being trended.

The autoburt system was started.

We noticed that the PSL FSS model was overdriving some DAC channels, so we stopped the PSL models. We started the h1 susaux front ends.

The front end and DAQ computers' BIOS settings were documented.

Following the site wide power outage from 9am Wed 3rd to 11am Thurs 4th April we are presently recovering the CDS systems.

At this point, we have started all the vacuum controls racks, the networking systems and the FMCS systems. A full recovery procedure will be posted in the DCC when we have completed the startup.

I'll be manually topping 80K pump levels and checking pressures etc. regularly during the outage.

Last week Chris and Kiwamu told me the slow controls weren't working for the low noise VCOs.  They are now working. 

Some of the issues were:
The corner 3 beckhoff chassis middle rail was not powered correctly, Filiberto fixed this.  
The inputs labeled RF mon on the tester go to pins that the wiring diagram indicates are temp mons, and the rf mon inputs go to pins the wiring diagram indicates are RF mons.  If the tester is right and the w 
The cable for the comm vco was unplugged at the beckhoff chassis
there were no links in the system manager for the diff vco

The Beckhoff modules used as ADCs on the IMC VCO (EL3102) are different from the ones used for the ALS VCOs (EL3104) but the conversion is identical for the different modules, so no changes the TwinCAT library are required

The RF detector used in the VCO (Minicurcuits ZX47-50LN-S+) has a temperature sensor, which the data sheet says has a linear slope of 2mV/degree C, and gives 0.6V at 20C, with the factor of ten gain in the schematic D100064, it seems like the equation used by the library (T=20+50*(V-6)) is right although the values displayed for temperatures do not make sense.  Looking at the test procedure for the VCO it doesn't seem like this part was tested.  

I turned the HEPAs and AC units off in both the H1 and H2 rooms. I've left the make up air on at 100%.

Last full day of power - tomorrow the site will have it's power shut down - DOE making changes.

Site activity:
- lots of activity with shutting things off in every building
- crane activity this morning - all are currently locked out
- two loads of Praxair
- crate inspection at MY for PSL parts
- people at EY to get the fiber working
- people looking at storage
- dust monitors moved around and staged in strategic places - some will be recording data and people will be coming in to swap out batteries
- cryopump work shimming on LN2 transfer lines
- all optics have their Master Switches set to "OFF"
- doors are on the HAM chambers
- class-A hardware in the H2 PSL is covered
- a snapshot of the vacuum overview medm is attached - unusual with all the red

"ISC Table" has been covered up.  iLIGO SUS Towers have been covered.  I saw Greg address TCS stuff on a flow bench. 

On Patrick's request, I powered off the three Dust Monitors in these rooms.

Cleanroom lights are OFF, and fans are ON.

XBM pressure as indicated by PT170B will be increasing -> No action in required

Want MTP rotor to be stopped and parked before loss of power tomorrow

Prep work continued yesterday in the West Bay. Signage was generated and attached to pallets and storage boxes, including the BS/FM at the end of the shelving units. BS/FM container ratchet was sealed. I found that the ICS paperwork for many parts could not be completed because of shipments not being received so I worked on that for a while this morning. I also participated in a telecon with the PSL folks WRT storage issues. We made some progress in giving the PSL crates at Y Mid unique identifiers and correlating existing info (Benno's sketch and spreadsheet, RS and MR inspection info) with the new identifiers. Please see attached documents for further details. I applied a few signs in the LVEA. I will continue to work ICS issues this afternoon.

[Sheila, Kiwamu]

 For some reason the issue of the picomotor controller box #3 [1] disappeared spontaneously.
We have no idea of what exactly was going on. Keep your eyes on it.

Picomotor Tests
  As planned [2] we tested the slow digital system this morning by plugging a spare controller box directly to the Beckoff chassis with short DB25 cables. With this setup we could drive an in-air picomotor and it didn't show the problem. Then we thought the original long cables that connects the Beckoff chassis and the controller box might be doing something funny. So we tested it by hooking up the in-air picomotor at the front of the original controller box (which is on the top of the IOT2L enclosure) and driving it from the Beckoff through the original cables and controller box. So this is the same configuration as at the time when we had discovered the issue [1] (except we inserted extension DB25 cables to reach the box as ITO2L is put afar). However today we didn't see the issue at all even in this configuration. We tried to reproduce the problem by loosening the DB25 cables (CAB_178 and CAB_179) intentionally but it only made the drive either completely functioning or completely not-moving and hence we were unable to reproduce the issue.

So the issue is gone spontaneously and we quit investigating it for now.

[1] LHO alog 5939 Comments on "HAM3 ISC IR QPDs are likely broken, in-vac cabling error, IO QPD should be OK"
[2] LHO alog 5960 "Test of picomotor controller box #3 : healthy"

New vacuum workstations have been installed in all of the outbuildings and in the LVEA, replacing the aging Sun workstations in those locations.  These workstations are not dependent on any other systems, and may be rebooted at any time without a loss of functionality.  All of the files necessary to monitor the vacuum system are included on the local disk.  The exception is DataViewer, which requires the NDS server (and the /ligo filesystem to run the binaries).  Note that the vacuum VME crates themselves still require hanford1/2 to boot, and should NOT be restarted during a power outage.

In addition, the network connections to the vacuum racks in the LVEA have been reconfigured such that all components are on UPS.  This means we should be able to monitor these systems from the control room as long as UPS power lasts.  The outbuilding switches are in varying states of UPS readiness, and their operation through a power outage is not guaranteed - the local workstations are available for monitoring in these locations (again, for as long as the UPS batteries last).

I've attached a flowmeter to the vent from the CP1 dewar. I'll be using this to monitor the exhaust flow for the next couple of weeks. One concern is this may ice up and block the flow, I'll be keeping an eye on it to make sure this doesn't happen. There is an overpressure vent upstream of the flowmeter as well.

[by everyone involved in IMC commissioning]

This post explains how the two IMC-X and IMC-F channels are generated and calibrated. As of the date of this post, this has been made uniform between the two sites.

NOTES:
- here the term "IMC length" is used to indicate the round trip length (~32 m). Because the IMC is a very squeezed triangle, there is sometimes a tendency to refer to half of this value (~16 m) as "the length of the IMC" (assimilating it to a linear cavity); this has been a source of misunderstanding in the past. Here, the change in half-round trip length is called IMC-X and is equivalent to the displacement of the MC2_M3 optic.
- there may or may not be other filters present in the filter banks cited below. This additional filters are temporary, for testing purpose, or leftovers from old tests, and are supposed to be disabled. Only the filters explicitly mentioned in this post should be engaged for the calibration to work.


********************************************************************************************************
            IMC-X (DAQ channel IMC-X_DQ)
********************************************************************************************************
The displacement of MC2_M3 is read by the OSEMs and recorded as MC2_M3_L_OUT_DQ; however, the sensitivity of the OSEMs is not sufficient to measure the motion of the IMC when locked, so IMC_X needs to be estimated from the lock feedback signal sent to the various MC2 stages. Note that this is independent from how the lock signal is filtered and distributed to the various stages of the suspension.

M1:
M1_LOCK_L goes into the IMC_X_M1 filter bank, where the following filters are enabled:
    "dc_cal": converts counts into displacement at DC (thus including counts->force calibration and the dc value of the force->displacement TF for M1). LLO measured value: 8.2e-13. LHO is using the same value.
    "white": whitening filter to compensate for rounding errors of the digital system. Double zero at 0.2 Hz, double pole at 1 kHz, dc gain = 1.
    "wresg1" and "wresg2": transfer function from "force on M1" to "displacement on M3", assuming the resonant gain damping filter is enabled (as it should be), and normalized to 1 at dc (the dc gain is absorbed in "dc_cal"). Two filters are necessary because the expression is too complex to be represented by a single foton filter.
    "to_um": converts the channel into um for convenience.

M2:
Same as M1 (appropriately replacing M1 with M2), except:
    "dc_cal" value is 1.75e-11 (measured at LLO, copied at LHO)
    "wresg1" and "wresg2" are replaced by a single "wresg" filter, as the expression is simple enough that it doesn't need to be split.

M3:
Same as M2 (appropriately replacing M2 with M3), except:
    "dc_cal" value is 1.75e-9 (measured at LLO, copied at LHO)

IMC-X is obtained by summing the outputs of these three filter banks (IMC_X_M1/M2/M3).

NOTES:
- it is calibrated in um
- remember that it needs to be "de-whitened"


********************************************************************************************************
            IMC-F (DAQ channel IMC-F_OUT_DQ)
********************************************************************************************************
The "fast output" of the common mode board used for IMC locking is digitized and sent into the IMC-F filter bank, where the following filters are enabled:
    "cts2V": to convert ADC counts into volts. Both sites use the value of 0.000610016 V/cts.
    "InvGenFilt":  invert common mode board "generic filter". It is a double pole at 10 Hz, double zero at 100 Hz, 0.5 DC gain (to account for differential output fo the board).
    "VCO": control signal --> frequency shift calibration; it accounts for the VCO gain, an internal filter and the double pass into the AOM. Zero at 40 Hz, pole at 1.6 Hz. DC gain (2*measured VCO gain) is 496900 V/Hz at LLO, and 536604 V/Hz at LHO.
    "tokHz": converts the output in kHz for convenience.

To be "calibrated" in meters and plotted together with IMC-X (MC2 displacement = delta(IMC length)/2), IMC-F needs to be multiplied by 5.85e-14 m/Hz, according to the formula:

delta(IMC-X) = delta(IMC-F)*lambda/(2*FSR)