Step 6.3:  LVEA Network Switches 

This box appeared to be running (green lights & no errors).

Step 6.4 Work Station Computers

Work Stations powered in Control Room----->  But BEFORE these were powered up, The NFS File Server should have been checked in the MSR before powering anything up.  This needs to be added to the procedure.

We have found several issues due to computers being started before the NFS File Server was addressed.  These items had to be brough up again:

• EPICS Gateway
• NDS

----->And the items above allow us to run Dataviewer now and also bring up the Vacuum Overview

Step 6.5  Wiki & alog & CDS Overview MEDM

They are back.  Overview on the wall (thanks, Kissel).

Update:  The wiki was actually NOT back since it was started before the NFS File Server was started.  So the wiki was restarted.

Sec 8:  EX  (Richard & TJ)

They have run through the document & are moving on to other outbuildings.  On the overview for EX, we see Beckhoff & SUSAUX are back.

Sec 10:  MX (Richard & TJ)

Haven't heard from them, but we see that PEMMX is back on the Overview.

Sec 9:  EY (Richard & TJ)

This is backonline.  So now we can start front ends in the Corner Station!  

They are now heading to MY now.....

Sec 11:  MY (Richard & TJ)

...but it looks like MY is already back according to the CDS Overview.

STATUS at 10AM (after an hour of going through procedure):

Most of the CDS Overview is GREEN, -except- the LSC.  Dave said there were issues with bring the LSC front end back up and will need to investigate in the CER.

End Stations: (TJ, Richard)

• ALS lasers powered on (which wasn't mentioned in procedure).
• Mentioned issues with high power voltage power supplies are intentionally left off so VAC Team can check their system first.

7.5 Front Ends (Updated from Kissel)

Everything is looking good on the CDS Overview

Sec 7.8:  Guardian (updated from Kissel)

Working on bringing this back.  Some of these nodes need data from LDAS (namely, ISC); so some of these may take a while.  

BUT, basic nodes such as SUS & SEI may be ready fairly sooner.

7.1 CORNER STATION DC POWER SUPPLIES

These have all been powered ON (Richard).

(Still holding off on End Stations for VAC team to let us know it's ok.)

EY (Richard)

High Power voltage  & ESD are back (this step is not in recovery document).

EX (Richard)

High Power voltage  & ESD are back (this step is not in recovery document).

There appear to be some order issues here.  (Jeff, Nutsinee, and others are working on fixing the order in the document.)

1)  We held off on addressing DC high power because of wanting to wait for Vacuum System Team at the LVEA (for vacuum gauge) and at the End Stations (for vacuum gauge & the ESD).

2)  We held off on some Corner Station items, because of them being on the Dolphin Network.  So to address End Stations FIRST, Assigned Richard & TJ to head out to start the End Station sections of the document & get their Dolphin Network items online.  Once they were done, Dave started on cornter station Front Ends on the Dolphin network.

Extraneous items:

• Conlog was brought up (Patrick)

Sec 7.6 LVEA HEPI Pump Station Controller

After Hugh's HEPI Maintenance, Jim brought this back.

Sec 8.5 EX HEPI Pump Station Controller

After Hugh's HEPI Maintenance, Jim brought this back.

UPDATE:

At this point (11:42am), we appear to be mostly restored with regard to the CDS side.  Most of the operations subsystems are back (we are mostly green on the Ops Overview).  The VAC group's Annulus Ion Pumps are back to using site power.

Lingering CDS/Operations items:

• NDS may still be down
• PSL restoration is still underway (Jason, Ed)

There is some noise on the LL quadrant of the ITMX readback.  H1:IOP-SUS_AUX_B123_MADC7_TP_CH4.   Appears at ~ 8kHz 3000 counts peak.    
 

• 22:47 - Controll Room screens off
• 22:48 - Fil to mids and ends to power off
• 22:56 - Control Room machines off
• 22:57 - OPS workstation / aux computers off

Evan wondered if the ~5% increase in total injection amplitude was dominated
by the highest frequency injection or one at lower frequencies. I took a look 
for this time interval and found that the total amplitude is dominated by
the injection at ~1220.5 Hz. Simply comparing spectral line strengths 
before and after the changeover turned out not to be a robust way to
estimate the frequency-dependent ratio of the new to the old inverse actuation 
function, because some pulsar injections (especially the highest frequency one)
are going through rapid antenna pattern modulations during this period.

But comparing the new to the old spectral line strengths at the same
sidereal time several days later (after power outage recovery) gives
robust measures for a sampling of low-, medium- and high-frequency
injections:


Freq (Hz)
"Old" amplitude (before switchover)
New amplitude (4 sidereal days later)
Ratio (new/old)
190.90
0.32292
0.32211
1.00
849.00
60.502
62.344
1.03
1220.5
299.37
318.70
1.06
1393.1
207.50
224.37
1.08
1991.2
28.565
32.788
1.15


These results seem in reasonable agreement with Evan's expectation of 
a new/old ratio rising with frequency, reaching 15% at 2 kHz.

Plots and text files of 4-minute spectral averages are attached 
for June 3 before the switchover and for June 7 with the newer
inverse acuation function.

To be more clear regarding the HEPI task I will perform Monday morning, see WP 5910.

This is the Capacitive Accumulator Pressure checking which requires the HEPI Pumps off.  This is done only every 3 to 6 months.

SR3 is not the problem after all. 

KIwamu and I unplugged the cables from the Sat amp to the chamber for both M2 and M3, and the locklosses and glitches still happened.  The good news is that Kiwamu seems to have found a good clue about the real culprit.

Our current theory is that locklosses are due to the ISS which shuts itself off for some reason at random times at a rate of once in 10 minutes or so. This causes a glitch in the laser intensity. Before a lockloss, there was a fast glitch (~milliseconds) in PRCL, SRCL and CARM error signals. That made us think that the laser field may be glitching. Indeed, we then found that the ISS had gone off automatically at the same time as the glitch and seemingly had caused the subsequent lockloss. We then tested the stability of ISS in a simpler configuration where only IMC is locked. We saw glitches of the same type in this configuration too.

In order to localize the issue, we are leaving the ISS open overnight to see if some anomaly is there without the ISS loop.

Conclusion: it was the ISS which had a too low diffraction power.

According to the overnight test last night, I did not find a glitchy behavior in the laser intensity (I looked at IMC-MC2_TRANS_SUM). This means that the ISS first loop is the culprit. Looking at trend of the recent diffraction power, the diffraction power kept decreasing in the past few days from 12-ish to almost 10% (see the attached). As Keita studied before (alog 27277), a diffraction power of 10% is about the value where the loop can go unstable (or hit too low diffraction value to shut off the auto-locked loop). I increased the diffraction power to about 12% so that the variation in the diffraction power looks small to my eyes.

Note that there are two reasons that the diffracted power changes, i.e. intentional change of the set point (left top) and the HPO power drift (right bottom). When the latter goes down, ISS doesn't have to diffract as much power, so the diffraction goes lower.

In the attached, at the red vertical line somebody lowered the diffraction for whatever reason, and immediately the ISS got somewhat unhappy (you can see it by the number of ISS "saturation" in right middle panel).

Later at the blue vertical line (that's the same date when PSL air conditioning was left on), the diffraction was reduced again, but the HPO power went up, and for a while it was OK-ish.

After the PSL was shut down and came back, however, the power slowly degraded, the diffraction went lower and lower, and the number of saturation events sky-rocketed.