Turned Kobelco ON at 7:45 am local. Working through vent procedure turning off/valving out equipment to prepare for slow vent today.

To verify that front ends could be started, I first started all the non-dolphin FECs (except for the mid station PEMs), there were no problems. Before starting h1psl0 I consulted with Peter King to see if there would be any PSL issues when I do so, he said there would be none.

I noticed a timing error in the corner station and found it to be the IO Chassis for h1seib3 (ITMX) which was powered down. Its front panel switch was in the OFF position. Richard and I think this was accidentally switched much earlier, and only caused a problem after the computer was power cycled. Before powering this IO Chassis up, I first powered down the two AI chassis because of the issue with the 16bit DACs outputting voltage when the IO Chassis is ON and the FEC is OFF (after h1seib3 was later operational I powered the AI chassis back on).

I restarted all the dolphin FECs, using IPMI for the end stations and front panel switches for the MSR. Many FECs started with IRIGB errors, both in the positive and negative directions. We know from experience some of these take many minutes to clear, so I continued with the non-FEC restarts.

I consulted with Jim Warner on starting the end station BRS computers (it was OK to do so) and the HEPI Pump Controllers (we will leave them till Monday).

I went to EY, the first of many trips as it turned out. I powered up h1ecaty1 and h1hwsey. I noticed that h1brsey was already powered up, but its code is not running?

Back in the control room, I noticed all EY FECs had the same large positive IRIG-B error, indicating a problem with the IRIG-B Fanout. Back at EY I confirmed the IRIG-B fanout was reporting the date as mid June 1999. After some issues, I power cycled the IRIG-B chassis and rebooted the FECs. The front ends were now running correctly.

At this point I ran out of time. There is timing issue with h1iscex, and the Beckhoff timing fanout is reporting an error with the fourth Duotone slave which I suspect is h1iscex.

To be done:

Start mid station PEM FECs.

power up h1ecatx1 and h1hwsex

Start Beckhoff slow controls code on h1ecat[x,y]1. Start HWS code on h1hwse[x,y]

Investigate Duotone Timing error at EX, get h1iscex running.

Start hepi pump controllers.

Start digital video servers.

Start BRS code.

Start PSL diode room Beckhoff computer.

At Approx. 8:12 this morning the site had a power outage.  The GC and CDS UPS have mains shutting off and returning to normal in a 16 second span.  This of course means anything not on UPS shut down.   Bubba was able to log in and verify FMCS was functioning fine.  I came to the site and verified no Fire Pumps were running as indicated by the alarms we were recieving. I have restarted the EPICS interface to FMCS to try to eliminate the text alarms.  

Also started the 2fa machine so others could log in and monitor their systems.

There was a fire near the substation that provides our site power.  I do not know if the cause of the power outage also caused the fire or if the fire perhaps caused our outage or they were unrelated.  Not likely that they were unrelated.  I will attempt to contact electrical dispatch Monday morning to see if they know. 

Kyle is on site checking vacuum.  Robert is crunching data and everything seems stable. 

Dave will probably come in later today to get the front ends running again.

It looks like the only "hiccuup" experienced by the vacuum system following the site wide power outage was that IP6's controller defaulted to a "standby" state, i.e. High Voltage was disabled -> I re-enabled it.  

The vacuum bake ovens isolated from the roughing pumps at the loss of power but the Turbo pumps all spun down.  As such, the bake load contents were exposed to a Torr*L or two of turbo exhaust but at least reversed viscous flow was prevented.  Both ovens remained above 100C and the parts should clean up with the Turbos now restarted.  The RGAs for the VBOs were isolated at the time and should not be affected by the spun down turbos.  

1100 hrs, local -> Kyle leaving site now.  Robert S. and Richard M. still on site.

Richard and Jonathan have fixed remote login to CDS.

I am able to see FMCS from my computer at home and all FMCS systems have restarted and are working towards back to normal.

We had what appears to be a site-wide power glitch at 08:12 PDT this morning.  Both CDS and GC UPS systems report being on battery power for about 6 seconds. Systems on UPS power rode through this, all other systems are down.

Vacuum controls is OK, MEDM overview screen is attached. Also DAQ is OK.

Systems down includes end station Beckhoff, all front ends, FMCS. We are getting multiple text alarms due to the systems being INVALID.

J. Oberling, E. Merilh, P. King

A rather frustrating day with little progress.  We started in the morning by turning down the diode current for the NPRO and the MOPA to get 35 W out of the MOPA.  After trying several different settings, at Peter's suggestion we set the NPRO to output ~1.8 W; for this the diode current was set to 2.221 A, which yielded an output power of 1.81 W, which in turn gave 1.56 W at the entrance to the MOPA.  Before adjusting the MOPA diode currents too much we wanted get a look at the beam profile, as adjusting the diode current by too much can change the beam profile (this is where things began to fall apart...).  We intially set the WinCam behind the shutter for the 35W DBB path, as this was the only spot with enough room for the WinCam; the MOPA diode currents were set at 44.0 A for D1 and D2, and 45.0 for D3 and D4 (a leftover from earlier adjustments).  The beam profile here looked horrendous.  It was at least round, but there was a significant shoulder that ringed the bottom half of the beam; changing the MOPA diode currents up and down only served to make the profile worse.  Seeing this, we decided to go ahead and skip the MOPA diode current adjustment (leaving the currents at the previously stated level) and move straight into mode matching.

From section 4.3.4 of the 35W FE laser user manual (T0900646), the MOPA wants a 150 µm diameter beam approximately 30 mm inside the MOPA enclosure.  Using the leakage beam on mirror FE_M4, we measured out the required distance and set up the WinCam.  The beam diameter at this point was ~330 µm horizontal and ~377 µm vertical.  We then proceeded to adjust the mode matching lenses to improve this.  We moved lens FE_L3 towards the NPRO by a few cm (had to loosen the pedestal the lens was attached to in order to make the move) and lens FE_L4 towards the NPRO by ~4 mm. This gave us a beam diameter of ~290 µm horizontal and ~310 µm vertical; still not where it should be, but better than it was.  We then installed the 200W power meter at the exit of the FE and turned on the MOPA.  At the same pump diode currents (44.0 and 45.0 A respectively), the laser was outputting 28.5 W.  Ed tweaked the beam alignment into the MOPA and we ended up with 36.0 W output from the FE.  Unfortunately, the beam profile was unchanged (I did not have the presence of mind to get a picture of the beam here).

Thinking that maybe we were clipping on something, we checked the FE alignment through the HPO irises to get an idea of how misaligned the beam was.  These actually looked pretty good; horizontal alignment through the irises was as before the swap, vertical alignment was slightly high.  Peter was thinking that there could be something in the optical path from the FE to the WinCam down the DBB path that could be contributing to the beam profile issue (some back reflection causing interference maybe?).  To check this, the polarizing beam splitter cube PBS01 was removed from the table and the WinCam was installed in its place (this is the pick-off right behind the turning mirror M02 that directs the FE beam into the HPO, see first attachment); in this way we eliminated the entirety of the DBB path.  At this point in the optical path the beam looked much better, round and almost Gaussian.  I tweaked the horizontal alignment into the MOPA to get the beam symmetric, the results are shown in the 2nd attachment (beam profile is rotated by +90°, i.e. UP is on the right side of the profile).  At this point we called it a night and will continue mode matching and HPO recovery on Monday.

This morning at 16:09 UTC, IP14 on HAM6 turned off for some unknown reason. Turned back on this evening.

Diagonal vacuum volume now left unpumped until further notice.  Also, the 10" turbo isolation valve closed and did not gall while partially open.  However, it has to be replaced soon and can not be relied upon anymore.  I'll de-energize the turbo levitation next week.