J. Oberling, E. Merilh, J. Bartlett
PSL recovery from the power outage is complete. We had 1 issue when bringing the laser up: the power meter that reads the power output from the PSL appears to have failed.
After turning everything (power supplies, control boxes, etc.) we began by bringing up the chillers; no issue here. When bringing up the PSL for the first time we noticed all the power appeared to be in the backwards direction, with nothing in the forward the direction. We killed the laser using the kill button on the table, then ran into the problem of the interlock not resetting. After much messing around with the interlock button and the control box, we found that you REALLY have to pull the button out to reset the interlock (when it feels like the button is about to break, pull harder). That taken care of, we turned the laser on and this time opened the external shutter; there was power being measured on the PDs and power meters external to the HPO box (PMC REFL, HPO external PD) so everything with the laser was functioning properly. Tried power cycling the questionable meter and saw no change. In the interest of getting the PSL up and running, and as this power meter is not used during normal operation of the PSL (only used when the HPO external shutter is closed), I decided to not swap the power meter. I did confirm that we do have a spare, but after a quick survey I could not find water fittings for it (it is a water cooled power meter). We will swap this out at the next opportunity.
All other PSL systems came online without issue. System is currently on and running, and has been warming up for ~1 hour. Will continue to monitor.
It occured to me that the hysteresis of the pzt might be something I could overcome by dithering the pzt around the values that were restored when we brought the system up.
I started the dither at +/-5000, then dithered at +/-1000, then +/-100, then +/-10, then +/-1, with 20 second sleeps between each change.
I dithered both pitch and yaw.
When the beam from the PSL was restored, the IMC REFL beam was completely on the camera, and the IMC locked (though at low value, WFS are off).
I've uploaded the executable file that will automatically dither the pzt to the OPS wiki, under the page name "pzt dither to recover alignment," and copied the executable into the userapps/release/isi/h1/scripts directory, file name 20160606_pzt_dither.txt.
This will be a running log of activities as they occur for bringing back CDS after the planned outage over the weekend. Refer to Richard's alog about what happened over the weekend for prep.
We started the shift with lights, front gate operation, wifi, etc.
Step 6.3: LVEA Network Switches
This box appeared to be running (green lights & no errors).
Step 6.4 Work Station Computers
Work Stations powered in Control Room-----> But BEFORE these were powered up, The NFS File Server should have been checked in the MSR before powering anything up. This needs to be added to the procedure.
We have found several issues due to computers being started before the NFS File Server was addressed. These items had to be brough up again:
----->And the items above allow us to run Dataviewer now and also bring up the Vacuum Overview
Step 6.5 Wiki & alog & CDS Overview MEDM
They are back. Overview on the wall (thanks, Kissel).
Update: The wiki was actually NOT back since it was started before the NFS File Server was started. So the wiki was restarted.
Sec 8: EX (Richard & TJ)
They have run through the document & are moving on to other outbuildings. On the overview for EX, we see Beckhoff & SUSAUX are back.
Sec 10: MX (Richard & TJ)
Haven't heard from them, but we see that PEMMX is back on the Overview.
Sec 9: EY (Richard & TJ)
This is backonline. So now we can start front ends in the Corner Station!
They are now heading to MY now.....
Sec 11: MY (Richard & TJ)
...but it looks like MY is already back according to the CDS Overview.
STATUS at 10AM (after an hour of going through procedure):
Most of the CDS Overview is GREEN, -except- the LSC. Dave said there were issues with bring the LSC front end back up and will need to investigate in the CER.
End Stations: (TJ, Richard)
7.5 Front Ends (Updated from Kissel)
Everything is looking good on the CDS Overview
Sec 7.8: Guardian (updated from Kissel)
Working on bringing this back. Some of these nodes need data from LDAS (namely, ISC); so some of these may take a while.
BUT, basic nodes such as SUS & SEI may be ready fairly sooner.
7.1 CORNER STATION DC POWER SUPPLIES
These have all been powered ON (Richard).
(Still holding off on End Stations for VAC team to let us know it's ok.)
EY (Richard)
High Power voltage & ESD are back (this step is not in recovery document).
EX (Richard)
High Power voltage & ESD are back (this step is not in recovery document).
There appear to be some order issues here. (Jeff, Nutsinee, and others are working on fixing the order in the document.)
1) We held off on addressing DC high power because of wanting to wait for Vacuum System Team at the LVEA (for vacuum gauge) and at the End Stations (for vacuum gauge & the ESD).
2) We held off on some Corner Station items, because of them being on the Dolphin Network. So to address End Stations FIRST, Assigned Richard & TJ to head out to start the End Station sections of the document & get their Dolphin Network items online. Once they were done, Dave started on cornter station Front Ends on the Dolphin network.
Extraneous items:
Sec 7.6 LVEA HEPI Pump Station Controller
After Hugh's HEPI Maintenance, Jim brought this back.
Sec 8.5 EX HEPI Pump Station Controller
After Hugh's HEPI Maintenance, Jim brought this back.
Sec 9.5 EY HEPI Pump Station Controller
After Hugh's HEPI Maintenance, Jim brought this back.
UPDATE:
At this point (11:42am), we appear to be mostly restored with regard to the CDS side. Most of the operations subsystems are back (we are mostly green on the Ops Overview). The VAC group's Annulus Ion Pumps are back to using site power.
Lingering CDS/Operations items:
Filiberto, Ed, Rich Installation is complete for the ITM ESD Driver for ITMX and ITMY. A short was found on the bias connection to ITMX (see attached sketch to clear up the pin numbering of the legacy connector). The shield was cut and insulated on the 2ft section of cable up near the vacuum feedthrough for this connection. All nominally 24V connections were HIPOT tested to 500V, and the high voltage bias connections were tested to 700V. An ADC (ADC07) was added to SUS-C5-14 to permit the ITM ESD readback channels to be implemented per the latest wiring diagram (D1500464). At time of writing, no aspects of the installed system have been verified in situ. This is the next item on the checkout. Some useful system constants (total for both drivers): +/- 18VDC Nominal Quiescent Current -> +460mA, -420mA +/- 24VDC Nominal Quiescent Current -> +40mA, -40mA +/- 430VDC Nominal Quiescent Current -> +/- 5.6mA Serial Numbers: ITMY ESD Driver (D1600092) (installed in SUS-R6) -> S1600266 ITMX ESD Driver (D1600092) (installed in SUS-R5) -> S1600267 ITM PI AI Chassis (D1600077) (installed in SUS-C6 U23) -> S1600245
As noted by Dave B. We began the orderly shutdown of the site at 1500 local time on Friday. The power actually went out at 1715 and we were almost ready. Vacuum system with the generators setup did not go as smoothly as it could. 1. Generator output was too low for the UPS to operate on. UPS looking for 115V and we were at 112V. We bypassed the UPS and ran the 24V dc supplies directly from the Generator. 2. The GFI outlet on EY generator would not function so it was replaced By 1930 we were comfortable leaving the site. Sat. 0800 Site was dark all buildings, generators, Vacuum system in good shape 1800 New switch at substation installed. Sun. 0800 Site was dark all buildings, generators, Vacuum system in good shape 1200 Testing of switch complete (hicups included) 1300 Site power restored. Facility cooling began. converted Vacuum system to building power generators turned off. 1500 left for day. Other Vacuum details by Vacuum group.
State of H1: systems are off
Weekend Prep activities:
* indicates there was some small glitche in our procedure
Filed FRS 5642 for the failed power meter.