This morning I found the servo driving the pumps at maximum voltage and only achieving 26.3psi rather than the 28 psi setpoint. I backed off the setpoint to the usual past 26psi and the drive dropped to ~2000/2048. Probably a viscosity change from temperature increase. Seems normal.
[Cheryl, Giacomo]
The IMC is flashing again.
We did restart alignment almost from scratch, through the following main steps:
- Beam centered on the top periscope mirror on the PSL table, usign the bottom periscope mirror and the mirror before that
- Beam centered on the HAM2 periscope, using the top periscope mirror. This did NOT recover the original input beam pointing, as the spot reflected off of the input viewport into the external PSL wall are slightly offset. However, it did look nice as input beam alignment.
- Beam centered on the irises in front on MC1 and MC2 using the actuated and non-actuated steering mirror, respectively.
- Beam centered on the iris in front of MC3 using MC2
- Round trip closed using MC3 and MC1... and IMC flashing!
The MCx offsets are as follows (in um on the alignment offset sliders):
| MC1 | MC2 | MC3 | |
| Pitch | 500 | -125 | 395 |
| Yaw | -790 | 200 | -955 |
It's good to see that the big (and a bit worrying) offsets in yaw of MC1 and MC3 are gone...
Attached are plots of dust counts requested from 5 PM April 7 to 5 PM April 8. The plots for dust monitor 4 in the LVEA are interesting. Perhaps the change was from turning on the clean room over the test stand?
In prep for IAS alignment upcoming, I uncovered the BS suspension and resuspended it. I also removed the corner cube mount in case SEI wants to do any balancing/checkouts/etc. before we get onto alignment (i.e. payload is correct).
The laser was restarted today without any trouble. I took the chance to realign the PMC as well and gained about 400 mW. A quick measurement of the visibility gave 97%.
Also, I was able to start the VB spectrum analyzer program on the Beckhoff computer and we finally have an all green status screen.
While investigating lost frames, Tx/Rx errors and a non zero CRC for the Corner MSR L8_9 Vertex (EK1521) (the terminal for the fiber connection to the H1 electronics room), the h1ecatc1 computer froze and restarted. I had looked into the chassis housing this terminal in the MSR and had not seen anything obviously wrong with it. However, the error LEDs on the Corner MSR L2 (EL3202-0010) (the temperature probe readout) terminal were both lit. After the computer restarted I restarted the system manager and deleted the configured terminal for the connection to end X (it was disabled because it is not connected), and checked that scanning the modules gave an identical configuration. It did, and I put it in run mode and started PLC1 and PLC2. The previous errors with the EK1521 are gone, but the error LEDs are still red on the EL3202. I have not restarted the EPICS IOCs.
The lost frames, Tx/Rx errors and nonzero CRC for the Corner MSR L8_9 Vertex (EK1521) fiber coupler terminal are back this morning.
The error LEDs on the EL3202 appear to just be an indication that a temperature probe is not connected.
The chassis in the MSR that this is referring to is S1103490. Yesterday I swapped both the EK1521 terminal and the internal connecting multimode fiber. This and the other fibers are bent over fairly sharply by the chassis cover. So far this seems to have fixed the lost frames, Tx/Rx errors and non zero CRC. It is unclear at this point if the problem was with the fiber or the terminal.
800 - Michael Rodruck restarting PSL
900 - Betsy/Travis working in HAM2
930 - SnoValley working on chillers
1030 - Michael R engaged H1 PSL enclosure chiller to bring temperature down
1040 - HEPI pump station restarted by Hugh. Leak checking.
1250 - HFD burning tumbleweeds along RT 10 between LHO and Wye barricade
1337 - PNNL called to say vibroseis "shaker" truck now active at old Nike missile site. (HAWA--- old "Big G" experiment site)
1450 - Cheryl transitioning LVEA to laser hazard.
Before starting messing up with IO alignments, Cheryl asked my to check if any major drift was visible in he HAM2 and 3 suspensions after the outage.
For the IMs, I restored the offsets that were in effect before the outage, and compared the OSEMs readings. No significant drift was observed. Results are summarized in the following table (the columns on the right are just a visual representation of the difference in log scale).
------------------------------------------------------------------------------------------------------------- OSEMs readings comparison between gps = 1048629616 (reference) and gps = 1049482936 ------------------------------------------------------------------------------------------------------------- Channel name Value at T1 Value at T2 Differ. 1 10 100 1000 ------------------------------------------------------------------------------------------------------------- H1:SUS-IM1_M1_OSEMINF_UL_OUT_DQ +003.73+-5.18 +004.11+-1.16 +000.38 | | | | H1:SUS-IM1_M1_OSEMINF_LL_OUT_DQ -082.58+-5.41 -080.89+-1.22 +001.70 |> | | | H1:SUS-IM1_M1_OSEMINF_UR_OUT_DQ +017.31+-4.85 +018.25+-1.16 +000.94 | | | | H1:SUS-IM1_M1_OSEMINF_LR_OUT_DQ -084.14+-5.18 -081.39+-1.28 +002.75 |>>> | | | ------------------------------------------------------------------------------------------------------------- H1:SUS-IM2_M1_OSEMINF_UL_OUT_DQ -290.53+-2.85 -289.29+-1.90 +001.24 | | | | H1:SUS-IM2_M1_OSEMINF_LL_OUT_DQ -242.12+-3.14 -236.62+-2.46 +005.50 |>>>>>> | | | H1:SUS-IM2_M1_OSEMINF_UR_OUT_DQ +236.39+-3.13 +238.20+-3.44 +001.81 |>> | | | H1:SUS-IM2_M1_OSEMINF_LR_OUT_DQ +132.75+-2.92 +138.67+-2.79 +005.92 |>>>>>>> | | | ------------------------------------------------------------------------------------------------------------- H1:SUS-IM3_M1_OSEMINF_UL_OUT_DQ +171.96+-3.88 +170.05+-0.93 -001.90 |>> | | | H1:SUS-IM3_M1_OSEMINF_LL_OUT_DQ -090.16+-3.86 -086.71+-0.90 +003.45 |>>>> | | | H1:SUS-IM3_M1_OSEMINF_UR_OUT_DQ +141.78+-3.64 +139.41+-0.89 -002.37 |>>> | | | H1:SUS-IM3_M1_OSEMINF_LR_OUT_DQ -138.79+-4.38 -135.01+-1.05 +003.77 |>>>>> | | | ------------------------------------------------------------------------------------------------------------- H1:SUS-IM4_M1_OSEMINF_UL_OUT_DQ -075.93+-4.06 -063.20+-7.54 +012.74 |>>>>>>>>|> | | H1:SUS-IM4_M1_OSEMINF_LL_OUT_DQ +283.91+-4.13 +295.35+-8.44 +011.44 |>>>>>>>>|> | | H1:SUS-IM4_M1_OSEMINF_UR_OUT_DQ -210.78+-3.92 -199.14+-5.11 +011.64 |>>>>>>>>|> | | H1:SUS-IM4_M1_OSEMINF_LR_OUT_DQ +123.32+-4.36 +134.96+-7.23 +011.64 |>>>>>>>>|> | |
I repeated the check for MC1, MC3 and PRM suspension. For technical reasons I had to keep the offsets off, so I compared the readings with a moment before the outage in which the watchdogs were tripped. Again, no major drift was observed.
------------------------------------------------------------------------------------------------------------- OSEMs readings comparison between gps = 1048658416 (reference) and gps = 1049464816 ------------------------------------------------------------------------------------------------------------- Channel name Value at T1 Value at T2 Differ. 1 10 100 1000 ------------------------------------------------------------------------------------------------------------- H1:SUS-MC1_M1_OSEMINF_T1_OUT_DQ -034.70+-0.03 -034.09+-0.03 +000.61 | | | | H1:SUS-MC1_M1_OSEMINF_T2_OUT_DQ -036.14+-0.05 -036.29+-0.03 -000.15 | | | | H1:SUS-MC1_M1_OSEMINF_T3_OUT_DQ -003.59+-0.04 -003.57+-0.03 +000.02 | | | | H1:SUS-MC1_M1_OSEMINF_SD_OUT_DQ +006.22+-0.20 +006.87+-0.05 +000.65 | | | | H1:SUS-MC1_M1_OSEMINF_LF_OUT_DQ +010.79+-0.23 +010.52+-0.05 -000.27 | | | | H1:SUS-MC1_M1_OSEMINF_RT_OUT_DQ +028.01+-0.23 +028.59+-0.05 +000.58 | | | | H1:SUS-MC1_M2_OSEMINF_UL_OUT_DQ +012.16+-0.39 +011.65+-0.07 -000.51 | | | | H1:SUS-MC1_M2_OSEMINF_LL_OUT_DQ +257.26+-0.39 +256.64+-0.06 -000.62 | | | | H1:SUS-MC1_M2_OSEMINF_UR_OUT_DQ -134.76+-0.34 -135.18+-0.06 -000.42 | | | | H1:SUS-MC1_M2_OSEMINF_LR_OUT_DQ +141.28+-0.60 +141.81+-0.08 +000.53 | | | | H1:SUS-MC1_M3_OSEMINF_UL_OUT_DQ -046.67+-0.62 -047.22+-0.09 -000.55 | | | | H1:SUS-MC1_M3_OSEMINF_LL_OUT_DQ +260.52+-0.57 +259.95+-0.07 -000.57 | | | | H1:SUS-MC1_M3_OSEMINF_UR_OUT_DQ -067.45+-0.57 -067.82+-0.08 -000.37 | | | | H1:SUS-MC1_M3_OSEMINF_LR_OUT_DQ +198.45+-0.65 +198.63+-0.08 +000.19 | | | | ------------------------------------------------------------------------------------------------------------- H1:SUS-MC3_M1_OSEMINF_T1_OUT_DQ -008.19+-0.04 -008.33+-0.04 -000.14 | | | | H1:SUS-MC3_M1_OSEMINF_T2_OUT_DQ -015.25+-0.07 -015.22+-0.03 +000.03 | | | | H1:SUS-MC3_M1_OSEMINF_T3_OUT_DQ -020.96+-0.06 -021.16+-0.03 -000.19 | | | | H1:SUS-MC3_M1_OSEMINF_SD_OUT_DQ -006.14+-0.07 -005.79+-0.05 +000.35 | | | | H1:SUS-MC3_M1_OSEMINF_LF_OUT_DQ +008.46+-0.31 +008.51+-0.06 +000.05 | | | | H1:SUS-MC3_M1_OSEMINF_RT_OUT_DQ -057.79+-0.33 -056.71+-0.06 +001.08 | | | | H1:SUS-MC3_M2_OSEMINF_UL_OUT_DQ +143.88+-0.58 +144.30+-0.09 +000.42 | | | | H1:SUS-MC3_M2_OSEMINF_LL_OUT_DQ +073.77+-0.91 +074.45+-0.11 +000.68 | | | | H1:SUS-MC3_M2_OSEMINF_UR_OUT_DQ +079.45+-0.53 +080.13+-0.07 +000.68 | | | | H1:SUS-MC3_M2_OSEMINF_LR_OUT_DQ +044.42+-0.73 +045.32+-0.08 +000.90 | | | | H1:SUS-MC3_M3_OSEMINF_UL_OUT_DQ +092.77+-0.97 +093.25+-0.12 +000.48 | | | | H1:SUS-MC3_M3_OSEMINF_LL_OUT_DQ +115.28+-1.26 +116.35+-0.16 +001.07 | | | | H1:SUS-MC3_M3_OSEMINF_UR_OUT_DQ +082.64+-0.83 +083.89+-0.09 +001.25 | | | | H1:SUS-MC3_M3_OSEMINF_LR_OUT_DQ +024.52+-1.46 +026.39+-0.16 +001.87 |>> | | | ------------------------------------------------------------------------------------------------------------- H1:SUS-PRM_M1_OSEMINF_T1_OUT_DQ -010.93+-0.05 -011.00+-0.04 -000.07 | | | | H1:SUS-PRM_M1_OSEMINF_T2_OUT_DQ +018.33+-0.06 +017.93+-0.04 -000.40 | | | | H1:SUS-PRM_M1_OSEMINF_T3_OUT_DQ -011.63+-0.07 -011.87+-0.04 -000.24 | | | | H1:SUS-PRM_M1_OSEMINF_SD_OUT_DQ -050.86+-0.21 -050.43+-0.06 +000.42 | | | | H1:SUS-PRM_M1_OSEMINF_LF_OUT_DQ +001.20+-0.29 +001.34+-0.03 +000.14 | | | | H1:SUS-PRM_M1_OSEMINF_RT_OUT_DQ -013.73+-0.29 -013.04+-0.03 +000.69 | | | | H1:SUS-PRM_M2_OSEMINF_UL_OUT_DQ +064.17+-0.60 +064.34+-0.06 +000.17 | | | | H1:SUS-PRM_M2_OSEMINF_LL_OUT_DQ +057.88+-0.88 +058.35+-0.07 +000.47 | | | | H1:SUS-PRM_M2_OSEMINF_UR_OUT_DQ +021.27+-0.56 +021.52+-0.06 +000.25 | | | | H1:SUS-PRM_M2_OSEMINF_LR_OUT_DQ +027.12+-0.81 +028.15+-0.06 +001.04 | | | | H1:SUS-PRM_M3_OSEMINF_UL_OUT_DQ +023.28+-0.94 +024.00+-0.08 +000.72 | | | | H1:SUS-PRM_M3_OSEMINF_LL_OUT_DQ -002.46+-1.23 -000.09+-0.09 +002.37 |>>> | | | H1:SUS-PRM_M3_OSEMINF_UR_OUT_DQ +027.42+-0.91 +027.77+-0.07 +000.35 | | | | H1:SUS-PRM_M3_OSEMINF_LR_OUT_DQ +066.64+-1.28 +068.56+-0.09 +001.92 |>> | | |
Finally, MC2 had to be compared with offsets off, but from a different period. Again, no major drifts or shifts in alignment.
------------------------------------------------------------------------------------------------------------- OSEMs readings comparison between gps = 1048939216 (reference) and gps = 1049464816 ------------------------------------------------------------------------------------------------------------- Channel name Value at T1 Value at T2 Differ. 1 10 100 1000 ------------------------------------------------------------------------------------------------------------- H1:SUS-MC2_M1_OSEMINF_T1_OUT_DQ -032.01+-0.04 -028.31+-0.06 +003.70 |>>>>> | | | H1:SUS-MC2_M1_OSEMINF_T2_OUT_DQ -076.77+-0.04 -074.11+-0.06 +002.66 |>>> | | | H1:SUS-MC2_M1_OSEMINF_T3_OUT_DQ +007.41+-0.04 +010.39+-0.06 +002.98 |>>>> | | | H1:SUS-MC2_M1_OSEMINF_SD_OUT_DQ -001.64+-0.06 -001.30+-0.12 +000.34 | | | | H1:SUS-MC2_M1_OSEMINF_LF_OUT_DQ +025.48+-0.11 +024.70+-0.20 -000.78 | | | | H1:SUS-MC2_M1_OSEMINF_RT_OUT_DQ +021.43+-0.08 +020.58+-0.14 -000.85 | | | | H1:SUS-MC2_M2_OSEMINF_UL_OUT_DQ +027.67+-0.16 +026.48+-0.32 -001.20 | | | | H1:SUS-MC2_M2_OSEMINF_LL_OUT_DQ +022.56+-0.22 +020.81+-0.45 -001.75 |> | | | H1:SUS-MC2_M2_OSEMINF_UR_OUT_DQ +033.66+-0.17 +032.37+-0.32 -001.29 | | | | H1:SUS-MC2_M2_OSEMINF_LR_OUT_DQ +052.13+-0.20 +050.76+-0.42 -001.37 | | | | H1:SUS-MC2_M3_OSEMINF_UL_OUT_DQ +029.85+-0.25 +028.22+-0.54 -001.63 |> | | | H1:SUS-MC2_M3_OSEMINF_LL_OUT_DQ -010.71+-0.29 -012.82+-0.61 -002.12 |>> | | | H1:SUS-MC2_M3_OSEMINF_UR_OUT_DQ -020.21+-0.24 -021.43+-0.51 -001.22 | | | | H1:SUS-MC2_M3_OSEMINF_LR_OUT_DQ +038.40+-0.34 +035.94+-0.73 -002.46 |>>> | | |
The RF source signals became dirty. This is currently under investigation.
In the past Friday I was checking the electronics at the field racks as a part of the power recovery activity. And I found that the RF source signals from the RF distribution amplifiers had become dirty -- there are lots of harmonics lines and cross-talks (see below). If I remember correctly those were not seen when we tested the RF stuff in the last Month [1][2] and therefore something must have happened since then. The raw data files are attached as well.
Figures: output from the 24 MHz RF amp (figure 1), 9 MHz RF amp (figure 2) and 45 MHz RF amp (figure 3).
[1] LHO alog 5854 "Installation and Checkout of RF signal chains on ISCT1 and HAM1"
[2] LHO alog 5849 "The 9MHz amplifier box is back"
After fixing any loose dog clamps, and retorquing, Arnaud and I took B&K hammer measurements of the HAM2 PR3, PRM, and MC1 this morning. We still need to do PR2 and MC3 (which has a temp IO alignment mirror nestled up against it's structure). We took PR3 and PRM data with vibration absorbers on and off. We only took MC1 data with the vibration absorbers on since they were already on and are difficult to remove and then replace. Plots and statements to come.
Arnaud and I snuck in and got B&K hammering done on PR2 before the transition to laser hazard.
Betsy and I hammered the last suspension missing in our list: MC3.
The results are coming soon
Results are at alog 6014
LHO aLOG 6014 are different results -- from PR3, and they don't even include the results for that measurement. Arnaud and I will be processing and post all the B&K data from the HAM2, HAM3, and and BSC3 (ITMX) next week.
The HEPI Pump Stations were not restarted after the outage. They will not restart automatically at this time. The fluid level relay box must be reset and the epics must be restarted manually. Additionally, the Pump Station Controller VFD must be set to fwd/run to operate. Once that is completed, the motors may be ramped up and the pressure servo engaged. I did a 100% inspection of all lines. Main line pressure is servoing at 28psi with the motor speed at ~2000/2048.
I documented all the BIOS settings for the EY computers. The h1hwsey BIOS was modified to prevent auto-power-on on power restoration. All the front ends were then restarted. We had to reconnect h1seih45 to the Dolphin network (its cable had been removed during initial roll out) to complete the set and get every Dolphin'ed front end going.
No major changes other than: the overflowing h1pslfss DAC channels as before (Rick said it would be ok to continue running this model); the h1susmc3 model would not auto-start, but we could manually start it; and h1susquadtst is showing timing errors (nothing connected to this system).
The DAQ was restarted. The EDCU is missing the PSL Beckhoff controller channels, presumably this computer is powered down.
The dust monitors and weather stations have been restarted and burt restored to the files in /ligo/cds/lho/h0/burt/2013/04/01/00:00. I also burt restored FMCS to the same time. The dust monitor at end Y location 2 has been swapped out for the one labeled 'S'. The previous one was removed on Tuesday (calibration failure). I removed the dust monitor in the LVEA at location 1 (calibration failure for a long time). The dust monitor at location 5 in the LVEA was moved out from under the clean room over BSC2 and into the beer garden. The dust monitors in the PSL enclosures still need to be restarted.
- Apollo installed a couple of viewports for OL, and removed a couple of doors from HAM chambers, see Bubba's entry for more info.
- outside Tech serviced the FM200 system, system is back now.
- Dave continued working with bringing back the CDS system, more green now than this morning.
- Hertz in to take back their generators, all gone now.
- Patrick bringing back dust monitors.
- PSL still down, no light.
- We still have a 2 CCs (VE) down (red) from the power outage, one less than this morning, yes they were toggled now we wait (424B & 524B).
We removed the east door from ham 3 and both doors from ham 2. We are ready to remove the west door on ham 3, however C 3 ham door covers are rare, therefore we subsituted with dome covers and used all of those. We will replace the subsitute covers with the appropriate covers when they arrive, hopefully Monday. We also installed viewports (serial #s) 23&26 on the H 1 MC input tube.
In addition to other work that went on during the power outage. We cleaned the Switchgear cubicles and cycled the breakers in key areas. The MCC in the Carpenter shop. The MCC that feeds the chillers. This unit looks as if condensation in forming in the bottom a fair amount of rust has formed. It also had dead mice in the gear. The main switches in the Mechanical Room. The MCC in the LSB. The main panels in the End Stations. We did not get to the Mid stations as we can do them at anytime and not bother the IFO. Robert ran some seismic tests in the LVEA. One problem was the Generator running by the carpenter shop. Large peak in the data. We also checked the status of the Emergency lights across the site. We had a few immediate failures and some that did not last the full 90minutes required. We will be addressing these issues. A few actually lasted 3 hours. Of particular concern were the exit lights in the OSB. I will get with John to discuss a path forward. Also the FM200 panel did not survive the outage. We lost a power board around 8am All in all the outage went smoothly and the return to normal only had some minor glitches.
Summary of Thursday CDS recovery:
All networking switches were started. hanford1 and hanford2 were started. hanford1 needed several restart attempts.
The vacuum controls workstation in the control room was started. The powered down Vacuum controls VME systems were restarted (h0veex, h0veey, h0velx, h0vely, h0vemr) along with their local workstations. All VE systems were burt restored to 23:00 Monday 1st April.
DC power was restored to the H1CER. h1psl0 was started to permit startup of the DAQ system. h1dc0 was started. We noticed a check led light on the Fujitsu 10GE switch, this was cleared before h1fw1 was started. A problem with the SATABOYs was tracked to solaris 11 not auto NFS sharing. Dan upgraded the MSR ldasgw machines to sol11.1 and patched, but still had to manually share the file systems. Also the MSR DAQ SATABOY performed an auto parity scan.
The cdswiki machine took several restarts before it booted. The MEDM snapshot program was started on script0 and the internal EPICS gateways were started. The FMCS EPICS IOC was started. The NAT router was started. At this point the VE and FMCS MEDM screens are available on the CDS web page.
The DAQ system was restarted, the VE and FMCS channels are being trended.
The autoburt system was started.
We noticed that the PSL FSS model was overdriving some DAC channels, so we stopped the PSL models. We started the h1 susaux front ends.
The front end and DAQ computers' BIOS settings were documented.
