In this case, the flow rate for head 4 clearly dropped below 0.5 lpm causing the watchdog to trip.
Corey has cancelled Ed's owl shift and the rest of mine. 23:04 UTC reset HAM2 HEPI watchdog 23:08 UTC reset HAM2 ISI watchdog 23:12 UTC MC locked. Leaving in down state for Robert. 23:19 UTC PSL tripped off. 23:20 UTC Peter to LVEA to remove PSL test equipment 23:27 UTC Peter back 23:30 UTC Peter back to LVEA 23:32 UTC Peter back 23:40 UTC reset noise eater 23:47 UTC taking power to 50W with just IMC locked for Robert. 23:53 UTC Taking power back to 2W 23:54 UTC Attempting to lock 00:23 UTC PRMI not locking. Starting initial alignment. 00:50 UTC Initial alignment done. 00:53 UTC ETMs are misaligned. Taking input power to 50W for Robert. 01:12 UTC Back to 2W. Attempting to lock. 01:28 UTC reset noise eater 01:39 UTC reset noise eater, back to CHECK_IR 01:51 UTC locked on PRMI, adjusted BS and PRM. PRMI to DRMI transition held, but signals in striptool very ratty. Lost lock on TURN_ON_BS_STAGE2. Not making it through DRMI locked. I'm suspecting the ISS is oscillating. 02:38 UTC PSL tripped off. Just when I got the ISS and FSS relocked (had to turn random knobs and reset the noise eater a number of times).
The laser tripped out again. Attached are two plots of the flow rate signals around the time of the trip.
Bear in mind that the minimum head flow rate was set to 0.5 lpm. The flow rate for head 4 had a small
excursion close to 0.5 lpm. Zooming in on the low point shows the flow rate to be above this limit.
It might be that the flow rate may have dipped below this level faster than the sampling time between
TwinCAT/OPC and EPICS.
Assuming that the flow rate in head 4 did not trip the laser out, then the problem lies with the
crystal chiller. Not sure why at this stage. One thing to try the next time is to swap where the
chiller(s) get their 3-phase power, ie interchange where the crystal chiller plugs in to where the
diode chiller plugs in and vice versa. The chiller is relatively new, so it's hard to believe that there's
something wrong with it but at this point everything points in that direction - in my opinion.
Most of the laser trips have been associated with the crystal chiller. Even with the previous two
chillers that were used. The only thing we did not change was where the crystal chiller was plugged in.
Swapping its power socket with the diode chiller might fix the problem or switch the problem over to the
diode chiller.
The "electronic" flow sensors have been removed and the flow rate values have been forced within
TwinCAT.
Should the laser trip out again, please do not bring it back.
Attached are the head flows for the past 2 hours.
state of H1: IMC was lockednow unlocked
Help: Jason, Peter
Ditails:
Should the laser trip off later on this evening, please do not attempt to bring it back. There are a couple of things that are cobbled in that I would like removed before the laser is restored.
The pre-modecleaner had some problems re-acquiring after the last laser trip. The symptoms were that one could
see resonant flashes but the servo never really grabbed. The pre-modecleaner high voltage signal instead of
looking like a ramp signal, looked like three parallel lines which is probably indicative of the high voltage
oscillating. This is also an indication that the beam is not well centred on the locking photodiode.
I re-aligned the beam both into the pre-modecleaner and onto the locking photodiode. Unlocked the DC output
was ~1V. Locked, ~0.26V. Crudely that's a visibility of 74% with the ISS off. That is consistent with the
power throughput of the pre-modecleaner as calculated from the values displayed on the MEDM screen.
In bring the FSS back up using the autolocker, I noticed that the common gain kept going between 0, 2 and
16 dB even though the autolocker status did not indicate the gain was being ramped. I assume this was guardian
doing its thing.
The ISS however seems to be exhibiting signs of one of the switches being touchy because turning the loop
on/off at times seems to have no effect. Unless, of course, the corresponding DAC channel is intermittent.
Hamlet 1.4
Peter has been in the PSL aligning, and is now in the CR attempting to restore the FSS and ISS and at this point they have not come back.
Installed the "electronic", ie dummy, flow sensors into both the power meter and front end circuits.
So the two readouts of those flow rates will be bogus for a while. Currently the front end circuit
reads 4.5 lpm and the power meter circuit reads 0.5 lpm. Even though the current sources were set
to mimic the real flow rate.
The Martel calibrator always complains that it is "OPEN LOOP". The other source didn't have
any error messages.
The minimum flow rate trip point was increased from 0.2 lpm to 0.4 lpm.
The Beckhoff terminal for the flow rates reports the following:
- an under-range and error flag for the power meter circuit
- an over-range and error flag for the front end circuit
The laser was still running under these conditions.
I decided to try force writing these variables in TwinCAT to match the previous values.
The "electronic" flow sensors are still attached however.
I had to set the bottom plot to "transfer function" and back to coherance to get the X axis to start at 121 and end at 700
Restarted the laser a few minutes ago. No configuration changes per se. This time the crystal chiller was
explicitly power cycled, ie turned off and then back on again.
Next attempt will include installation of the "electronic" flow sensor unless something more obvious
comes to the fore.
Attached is what I could glean from the Beckhoff terminal that monitors the flow rates. The error bit
was not set in either case. Unfortunately I do not recall if this terminal latches when an error condition
is detected.
The Beckhoff terminal concerned is a EL3054 4 channel analogue input, 4 - 20 mA.
Checked, the EL3054 does latch if an error condition presents itself.
OverallView.png is the top level signals related to the cooling system. Clearly there is an oscillation with the flow in the
power meter circuit that is not apparent in the front end cooling circuit.
PowerMeterCircuitFlow.png shows the laser output and the flow in the power meter circuit.
Flows.png shows the flows in the power meter circuit and the front end circuit.
CrystalChillerFlow.png shows the flow in the power meter circuit and the flow from the crystal chiller.
Unfortunately the laser was being restarted when I got in, so I cannot tell if the error condition on the Beckhoff terminal
that reads the flow rate was latched or not. In this particular case it appears that the problem lay with the flow rate in the
power meter circuit and not the chiller, at least at first glance.
At this point the prime candidate for installation of the "electronic" flow sensor would be in the power meter cooling
circuit.
Kiwamu is heading out, and so am I.
Passed Kiwamu on route 10 and turned around. Kiwamu has reset the laser.
PMC does not lock. It seems as if the servo board is not working. The pzt ramping is still functional. This needs some help from experts.
Initially PMC, FSS and ISS were OK. However approximately 30 minutes after the PSL restart, suddenly the PMC unlocked when we were running initial alignment and did come back to resonance itself.
It appears that the PSL may have tripped after the last lock loss. The NPRO output power is 0.
There is a flow sensor error on the top chiller and water on the front of it. The fill cap is still in place.
Talked with Peter. He talked me through turning off the auto locking and laser. Since no one else is here I am going to head out and he will come in tomorrow morning.
PSL restarted without no issue. The errors were xtal chiller and heads 1-4 flow error. We added 200 ml water.
Something happened early in the morning today (about 2016-10-08 12:50 UTC +- 10 min) and the transmission is only 78W as of now while the reflection is 38W or so.
It looks like the ISS diffracted power jumped up at the lockloss. I started changing the H1:PSL-ISS_REFSIGNAL to bring it back, but then trended it and saw it hadn't changed, so put it back.
More about changes in PMCsignals at lock loss: PMC Trans drops, FSS and ISS increase in power
