State of H1: laser is off

Summary of Weekend issues:

• alog 30328 - locking but can't get past DRMI to ENGAGE_ASC
• alog 30335 - something happened to the PMC
• alog 30347 - plot showing drop in all signals coming from the PMC, but not PMC REFL
• alog 30348 - signals in DBB are changing - deviations of 0-8, though not sure if that's power in Watts
• alog 30349 - laser trip data
• alog 30351 - Beckhoff issues?
• alog 30353 - electronic "dummy" installed on PSL plumbing
• alog 30356 - PSL servos - some ok, some not
• alog 30360 - laser trips - diagnostics on chiller flow
• Saturday, laser trips 3 times during day shift, and during the night after swing shift
• Sunday, laser trips at least 3 times

The weekend has not been good for the laser and I didn’t get to investigate the coupling at HAM2 more, complete the PEM injections I wanted to make, or even look through the viewport. I attach a plot from Friday’s work showing that shaking by about a factor of 10 at HAM2 produces a couple of features 3 or 4 times above the DARM noise floor. The second figure shows the results of heavy shaking.

03:35 UTC Connecting to ~100,000 channels from conlog-test-master.

Corey has cancelled Ed's owl shift and the rest of mine.

23:04 UTC reset HAM2 HEPI watchdog
23:08 UTC reset HAM2 ISI watchdog
23:12 UTC MC locked. Leaving in down state for Robert.
23:19 UTC PSL tripped off.
23:20 UTC Peter to LVEA to remove PSL test equipment
23:27 UTC Peter back
23:30 UTC Peter back to LVEA
23:32 UTC Peter back
23:40 UTC reset noise eater
23:47 UTC taking power to 50W with just IMC locked for Robert.
23:53 UTC Taking power back to 2W
23:54 UTC Attempting to lock
00:23 UTC PRMI not locking. Starting initial alignment.
00:50 UTC Initial alignment done.
00:53 UTC ETMs are misaligned. Taking input power to 50W for Robert.
01:12 UTC Back to 2W. Attempting to lock.
01:28 UTC reset noise eater
01:39 UTC reset noise eater, back to CHECK_IR
01:51 UTC locked on PRMI, adjusted BS and PRM. PRMI to DRMI transition held, but signals in striptool very ratty. Lost lock on TURN_ON_BS_STAGE2.
Not making it through DRMI locked. I'm suspecting the ISS is oscillating.
02:38 UTC PSL tripped off. Just when I got the ISS and FSS relocked (had to turn random knobs and reset the noise eater a number of times).

The laser tripped out again.  Attached are two plots of the flow rate signals around the time of the trip.
Bear in mind that the minimum head flow rate was set to 0.5 lpm.  The flow rate for head 4 had a small
excursion close to 0.5 lpm.  Zooming in on the low point shows the flow rate to be above this limit.
It might be that the flow rate may have dipped below this level faster than the sampling time between
TwinCAT/OPC and EPICS.

    Assuming that the flow rate in head 4 did not trip the laser out, then the problem lies with the
crystal chiller.  Not sure why at this stage.  One thing to try the next time is to swap where the
chiller(s) get their 3-phase power, ie interchange where the crystal chiller plugs in to where the
diode chiller plugs in and vice versa.  The chiller is relatively new, so it's hard to believe that there's
something wrong with it but at this point everything points in that direction - in my opinion.

    Most of the laser trips have been associated with the crystal chiller.  Even with the previous two
chillers that were used.  The only thing we did not change was where the crystal chiller was plugged in.
Swapping its power socket with the diode chiller might fix the problem or switch the problem over to the
diode chiller.

    The "electronic" flow sensors have been removed and the flow rate values have been forced within
TwinCAT.

    Should the laser trip out again, please do not bring it back.

Attached are the head flows for the past 2 hours.

state of H1:  IMC was lockednow unlocked

Help: Jason, Peter

Ditails:

• laser tripped last night
• I restarted with Jason's help
• I filled both chillers with about 100ml
• Peter was on site when I returned to the CR
• the laser tripped out again
• Peter restarted and went into the PSL to realign
• IMC locked about 30 minutes ago and has stayed locked
• NPRO_NoiseEater is complaining that it's out of range
• diffracted power is coming up on DIAG_MAIN as HIGH at 6%
• Peter is going out to fix
• Robert tripped MC3 with shaking, IMC is out of lock

Should the laser trip off later on this evening, please do not attempt to bring it back.
There are a couple of things that are cobbled in that I would like removed before the laser is
restored.

The pre-modecleaner had some problems re-acquiring after the last laser trip.  The symptoms were that one could
see resonant flashes but the servo never really grabbed.  The pre-modecleaner high voltage signal instead of
looking like a ramp signal, looked like three parallel lines which is probably indicative of the high voltage
oscillating.  This is also an indication that the beam is not well centred on the locking photodiode.

    I re-aligned the beam both into the pre-modecleaner and onto the locking photodiode.  Unlocked the DC output
was ~1V.  Locked, ~0.26V.  Crudely that's a visibility of 74% with the ISS off.  That is consistent with the
power throughput of the pre-modecleaner as calculated from the values displayed on the MEDM screen.

    In bring the FSS back up using the autolocker, I noticed that the common gain kept going between 0, 2 and
16 dB even though the autolocker status did not indicate the gain was being ramped.  I assume this was guardian
doing its thing.

    The ISS however seems to be exhibiting signs of one of the switches being touchy because turning the loop
on/off at times seems to have no effect.  Unless, of course, the corresponding DAC channel is intermittent.

  Hamlet 1.4

Peter has been in the PSL aligning, and is now in the CR attempting to restore the FSS and ISS and at this point they have not come back.

Installed the "electronic", ie dummy, flow sensors into both the power meter and front end circuits.
So the two readouts of those flow rates will be bogus for a while.  Currently the front end circuit
reads 4.5 lpm and the power meter circuit reads 0.5 lpm.  Even though the current sources were set
to mimic the real flow rate.

    The Martel calibrator always complains that it is "OPEN LOOP".  The other source didn't have
any error messages.

    The minimum flow rate trip point was increased from 0.2 lpm to 0.4 lpm.

1. I can't use the stand up desk because OPSWS12 won't bring up dataviewer and moving screens is so delayed that I can't put them where I want them
2. on ther OPS workstation, dtt has an error I've neer seen before, I've manually set the range for the X axis on two plots and get different actual range on the coherance plot
   1. the top plot starts at 121 and ends at 700 as I've requested
   2. the bottom plot starts below 121, and ends at what looks like 760, when I asked for 121 to 700

I had to set the bottom plot to "transfer function" and back to coherance to get the X axis to start at 121 and end at 700

Restarted the laser a few minutes ago.  No configuration changes per se.  This time the crystal chiller was
explicitly power cycled, ie turned off and then back on again.

    Next attempt will include installation of the "electronic" flow sensor unless something more obvious
comes to the fore.

OverallView.png is the top level signals related to the cooling system.  Clearly there is an oscillation with the flow in the
power meter circuit that is not apparent in the front end cooling circuit.

PowerMeterCircuitFlow.png shows the laser output and the flow in the power meter circuit.

Flows.png shows the flows in the power meter circuit and the front end circuit.

CrystalChillerFlow.png shows the flow in the power meter circuit and the flow from the crystal chiller.

    Unfortunately the laser was being restarted when I got in, so I cannot tell if the error condition on the Beckhoff terminal
that reads the flow rate was latched or not.  In this particular case it appears that the problem lay with the flow rate in the
power meter circuit and not the chiller, at least at first glance.

    At this point the prime candidate for installation of the "electronic" flow sensor would be in the power meter cooling
circuit.

Something happened early in the morning today (about 2016-10-08 12:50 UTC +- 10 min) and the transmission is only 78W as of now while the reflection is 38W or so.