Summary of Thursday CDS recovery:

All networking switches were started. hanford1 and hanford2 were started. hanford1 needed several restart attempts.

The vacuum controls workstation in the control room was started. The powered down Vacuum controls VME systems were restarted (h0veex, h0veey, h0velx, h0vely, h0vemr) along with their local workstations. All VE systems were burt restored to 23:00 Monday 1st April.

DC power was restored to the H1CER. h1psl0 was started to permit startup of the DAQ system. h1dc0 was started. We noticed a check led light on the Fujitsu 10GE switch, this was cleared before h1fw1 was started. A problem with the SATABOYs was tracked to solaris 11 not auto NFS sharing. Dan upgraded the MSR ldasgw machines to sol11.1 and patched, but still had to manually share the file systems. Also the MSR DAQ SATABOY performed an auto parity scan.

The cdswiki machine took several restarts before it booted. The MEDM snapshot program was started on script0 and the internal EPICS gateways were started. The FMCS EPICS IOC was started. The NAT router was started. At this point the VE and FMCS MEDM screens are available on the CDS web page.

The DAQ system was restarted, the VE and FMCS channels are being trended.

The autoburt system was started.

We noticed that the PSL FSS model was overdriving some DAC channels, so we stopped the PSL models. We started the h1 susaux front ends.

The front end and DAQ computers' BIOS settings were documented.

Following the site wide power outage from 9am Wed 3rd to 11am Thurs 4th April we are presently recovering the CDS systems.

At this point, we have started all the vacuum controls racks, the networking systems and the FMCS systems. A full recovery procedure will be posted in the DCC when we have completed the startup.

I'll be manually topping 80K pump levels and checking pressures etc. regularly during the outage.

Last week Chris and Kiwamu told me the slow controls weren't working for the low noise VCOs.  They are now working. 

Some of the issues were:
The corner 3 beckhoff chassis middle rail was not powered correctly, Filiberto fixed this.  
The inputs labeled RF mon on the tester go to pins that the wiring diagram indicates are temp mons, and the rf mon inputs go to pins the wiring diagram indicates are RF mons.  If the tester is right and the w 
The cable for the comm vco was unplugged at the beckhoff chassis
there were no links in the system manager for the diff vco

The Beckhoff modules used as ADCs on the IMC VCO (EL3102) are different from the ones used for the ALS VCOs (EL3104) but the conversion is identical for the different modules, so no changes the TwinCAT library are required

The RF detector used in the VCO (Minicurcuits ZX47-50LN-S+) has a temperature sensor, which the data sheet says has a linear slope of 2mV/degree C, and gives 0.6V at 20C, with the factor of ten gain in the schematic D100064, it seems like the equation used by the library (T=20+50*(V-6)) is right although the values displayed for temperatures do not make sense.  Looking at the test procedure for the VCO it doesn't seem like this part was tested.  

I turned the HEPAs and AC units off in both the H1 and H2 rooms. I've left the make up air on at 100%.

Last full day of power - tomorrow the site will have it's power shut down - DOE making changes.

Site activity:
- lots of activity with shutting things off in every building
- crane activity this morning - all are currently locked out
- two loads of Praxair
- crate inspection at MY for PSL parts
- people at EY to get the fiber working
- people looking at storage
- dust monitors moved around and staged in strategic places - some will be recording data and people will be coming in to swap out batteries
- cryopump work shimming on LN2 transfer lines
- all optics have their Master Switches set to "OFF"
- doors are on the HAM chambers
- class-A hardware in the H2 PSL is covered
- a snapshot of the vacuum overview medm is attached - unusual with all the red

"ISC Table" has been covered up.  iLIGO SUS Towers have been covered.  I saw Greg address TCS stuff on a flow bench. 

On Patrick's request, I powered off the three Dust Monitors in these rooms.

Cleanroom lights are OFF, and fans are ON.

XBM pressure as indicated by PT170B will be increasing -> No action in required

Want MTP rotor to be stopped and parked before loss of power tomorrow

Prep work continued yesterday in the West Bay. Signage was generated and attached to pallets and storage boxes, including the BS/FM at the end of the shelving units. BS/FM container ratchet was sealed. I found that the ICS paperwork for many parts could not be completed because of shipments not being received so I worked on that for a while this morning. I also participated in a telecon with the PSL folks WRT storage issues. We made some progress in giving the PSL crates at Y Mid unique identifiers and correlating existing info (Benno's sketch and spreadsheet, RS and MR inspection info) with the new identifiers. Please see attached documents for further details. I applied a few signs in the LVEA. I will continue to work ICS issues this afternoon.

[Sheila, Kiwamu]

 For some reason the issue of the picomotor controller box #3 [1] disappeared spontaneously.
We have no idea of what exactly was going on. Keep your eyes on it.

Picomotor Tests
  As planned [2] we tested the slow digital system this morning by plugging a spare controller box directly to the Beckoff chassis with short DB25 cables. With this setup we could drive an in-air picomotor and it didn't show the problem. Then we thought the original long cables that connects the Beckoff chassis and the controller box might be doing something funny. So we tested it by hooking up the in-air picomotor at the front of the original controller box (which is on the top of the IOT2L enclosure) and driving it from the Beckoff through the original cables and controller box. So this is the same configuration as at the time when we had discovered the issue [1] (except we inserted extension DB25 cables to reach the box as ITO2L is put afar). However today we didn't see the issue at all even in this configuration. We tried to reproduce the problem by loosening the DB25 cables (CAB_178 and CAB_179) intentionally but it only made the drive either completely functioning or completely not-moving and hence we were unable to reproduce the issue.

So the issue is gone spontaneously and we quit investigating it for now.

[1] LHO alog 5939 Comments on "HAM3 ISC IR QPDs are likely broken, in-vac cabling error, IO QPD should be OK"
[2] LHO alog 5960 "Test of picomotor controller box #3 : healthy"

New vacuum workstations have been installed in all of the outbuildings and in the LVEA, replacing the aging Sun workstations in those locations.  These workstations are not dependent on any other systems, and may be rebooted at any time without a loss of functionality.  All of the files necessary to monitor the vacuum system are included on the local disk.  The exception is DataViewer, which requires the NDS server (and the /ligo filesystem to run the binaries).  Note that the vacuum VME crates themselves still require hanford1/2 to boot, and should NOT be restarted during a power outage.

In addition, the network connections to the vacuum racks in the LVEA have been reconfigured such that all components are on UPS.  This means we should be able to monitor these systems from the control room as long as UPS power lasts.  The outbuilding switches are in varying states of UPS readiness, and their operation through a power outage is not guaranteed - the local workstations are available for monitoring in these locations (again, for as long as the UPS batteries last).

I've attached a flowmeter to the vent from the CP1 dewar. I'll be using this to monitor the exhaust flow for the next couple of weeks. One concern is this may ice up and block the flow, I'll be keeping an eye on it to make sure this doesn't happen. There is an overpressure vent upstream of the flowmeter as well.

[by everyone involved in IMC commissioning]

This post explains how the two IMC-X and IMC-F channels are generated and calibrated. As of the date of this post, this has been made uniform between the two sites.

NOTES:
- here the term "IMC length" is used to indicate the round trip length (~32 m). Because the IMC is a very squeezed triangle, there is sometimes a tendency to refer to half of this value (~16 m) as "the length of the IMC" (assimilating it to a linear cavity); this has been a source of misunderstanding in the past. Here, the change in half-round trip length is called IMC-X and is equivalent to the displacement of the MC2_M3 optic.
- there may or may not be other filters present in the filter banks cited below. This additional filters are temporary, for testing purpose, or leftovers from old tests, and are supposed to be disabled. Only the filters explicitly mentioned in this post should be engaged for the calibration to work.


********************************************************************************************************
            IMC-X (DAQ channel IMC-X_DQ)
********************************************************************************************************
The displacement of MC2_M3 is read by the OSEMs and recorded as MC2_M3_L_OUT_DQ; however, the sensitivity of the OSEMs is not sufficient to measure the motion of the IMC when locked, so IMC_X needs to be estimated from the lock feedback signal sent to the various MC2 stages. Note that this is independent from how the lock signal is filtered and distributed to the various stages of the suspension.

M1:
M1_LOCK_L goes into the IMC_X_M1 filter bank, where the following filters are enabled:
    "dc_cal": converts counts into displacement at DC (thus including counts->force calibration and the dc value of the force->displacement TF for M1). LLO measured value: 8.2e-13. LHO is using the same value.
    "white": whitening filter to compensate for rounding errors of the digital system. Double zero at 0.2 Hz, double pole at 1 kHz, dc gain = 1.
    "wresg1" and "wresg2": transfer function from "force on M1" to "displacement on M3", assuming the resonant gain damping filter is enabled (as it should be), and normalized to 1 at dc (the dc gain is absorbed in "dc_cal"). Two filters are necessary because the expression is too complex to be represented by a single foton filter.
    "to_um": converts the channel into um for convenience.

M2:
Same as M1 (appropriately replacing M1 with M2), except:
    "dc_cal" value is 1.75e-11 (measured at LLO, copied at LHO)
    "wresg1" and "wresg2" are replaced by a single "wresg" filter, as the expression is simple enough that it doesn't need to be split.

M3:
Same as M2 (appropriately replacing M2 with M3), except:
    "dc_cal" value is 1.75e-9 (measured at LLO, copied at LHO)

IMC-X is obtained by summing the outputs of these three filter banks (IMC_X_M1/M2/M3).

NOTES:
- it is calibrated in um
- remember that it needs to be "de-whitened"


********************************************************************************************************
            IMC-F (DAQ channel IMC-F_OUT_DQ)
********************************************************************************************************
The "fast output" of the common mode board used for IMC locking is digitized and sent into the IMC-F filter bank, where the following filters are enabled:
    "cts2V": to convert ADC counts into volts. Both sites use the value of 0.000610016 V/cts.
    "InvGenFilt":  invert common mode board "generic filter". It is a double pole at 10 Hz, double zero at 100 Hz, 0.5 DC gain (to account for differential output fo the board).
    "VCO": control signal --> frequency shift calibration; it accounts for the VCO gain, an internal filter and the double pass into the AOM. Zero at 40 Hz, pole at 1.6 Hz. DC gain (2*measured VCO gain) is 496900 V/Hz at LLO, and 536604 V/Hz at LHO.
    "tokHz": converts the output in kHz for convenience.

To be "calibrated" in meters and plotted together with IMC-X (MC2 displacement = delta(IMC length)/2), IMC-F needs to be multiplied by 5.85e-14 m/Hz, according to the formula:

delta(IMC-X) = delta(IMC-F)*lambda/(2*FSR)

 It turned out that the picomotor controller box #3 (S1107569) was healthy.
We thought this wasn't correctly functioning [1], but it actually works good.
This means that the slow digital system which has been controlling the box is more suspicious at this point.

 We knew that this controller box was not correctly working when it was controlled from the slow digital system (see the details in [1]). This can mean that either/both the box is not functioning or/and the slow digital system is not. Today I removed the box from the top of IOT2L and tested the box itself without the slow digital system by controlling it with the tester board. With this setup I drove a picomotor which was hooked up to the front panel in order to see if the box can drive it with no problems. I tested all eight channels by sequentially connecting the picomotor to each of them. All of them worked without any problems : the drive speed seems correct and both CW and CCW rotation can be performed as designed. Both x and y rotation made sense. The test was done at 500 Hz most of the times. Also I tested channel 1 in a frequency range from 100 Hz to 500 Hz by changing the drive frequency and the rotational speed still seemed (in fact even sounded) healthy to me. I concluded that this box was healthy and put it back on the top of the IOT2L enclosure.

 We will check both hardware and software of the slow digital system at the H1 electronics room.

[1] LHO alog 5938 Comments on " HAM3 ISC IR QPDs are likely broken, in-vac cabling error, IO QPD should be OK"
 

Attached are plots of dust counts requested from 5 PM March 31 to 5 PM April 1.

A big thank you to Jim B. for his help with this!

Following Keita's aLog 5843 describing the issue we have with our EPICS channel names -iop model vs user model DAC output-, I started modifying the macros defining those channels (following the method he used for TMSy)

The text file attached describes the values asssigned to the arguments of the sitemap for each suspension of SUSHAM. Since there is a 256 character limit in the medm editor, we will have to wait until that limit is raised to 2048, as it has been done at LLO, to implement the changes.

[...] to be continued

 

[Corey, Kiwamu]

  We replaced the two broken infrared QPDs that were in the POP QPD sled at HAM3 with the new ones.
After the replacement we confirmed that the new QPDs works fine. From the ISC point of view HAM3 is ready for pitting the doors.

QPD swapping
 The broken QPDs (SN#39 in QPD1 and SN#53 in QPD2) [1] were replaced with the spares. The list below shows which QPD mount has which diode after the replacement.

  QPD1 = SN#20 (the one close to the west door)
  QPD2 = SN#24 (the one close to the north door)

We accessed the HAM3 table from the west and north side. We followed the same in-situ replacement technique as we did at EY [2] so that we don't have to remove the whole thing but we just swap only their QPDs. The replacement operation went quite smooth and was much easier than the last time when we performed it at EY where the optical table was suspended and freely swung. After the replacement we tested the QPDs by three ways : photo-current check with a hand-held QPD tester, diode voltage drop test and signal chain test with dataviewer. The QPDs signal made sense when we used the hand-held QPD while the QPD was illuminated with a visible red laser pointer. The voltage drop across each segment was approximately 0.36 V which seemed healthy. We were able to see signals at the IOP of the ASC front end with dataviewer although the signals were spiky since there was no anti-whitening filter for them. (In fact there was even no ASC front end model which takes care of these signals). From these tests we are confident that the new QPDs work fine and conclude that this mission is done.


QPD inspection
 After the in-vac work we inspected the broken QPDs at the EE shop by inspecting the appearance and electrical characteristic. Filiberto looked at the QPDs with a magnifier and noticed that some of the wires were physically cut. The picture below shows a front view of one of the two broken QPDs. You can see that two wires, which are pointed by blue arrows in the picture, are cut. This is probably due to too much voltage (current) going through them when we were accidentally sending the picomotor driving signal which was more than 100 V due to the cabling error [1].




 According to the electrical tests with a multi-meter, segment 2 and 4 of both QPDs are completely open as expected from the magnifier inspection. The rest of two segments are fine : they show 0.36 V voltage drop and the correct polarity. Note that this broken characteristic is different from that of the one we extracted from EY which was shorted [3].

Some notes
 We fully removed the ISC-related tools from the HAM3 table so that we don't have to return to this chamber any more. However there was a thin metal plate left on the table which looked some kind of guard for the suspended mirrors. As we had no idea of what it was we left it in the chamber.

[1] LHO alog 5937 "HAM3 ISC IR QPDs are likely broken, in-vac cabling error, IO QPD should be OK"
[2] LHO alog 5718 "Comments on 'A new PZT mirror installed at ISCTEY and red in-vac QPD swapped on TMSY' "
[3] LHO alog 5942 " Broken infrared QPD of TMSY inspected "