Dan, Kiwamu, Evan

Summary

Tonight we worked on getting the interferometer back to its low-noise state. We are stable at 10 W, but there is some instability at higher powers.

Details

ITM steering

First, at 3 W we manually steered the ITMs to a good recycling gain (38 W/W), and then updated the TMS QPD offsets. We also locked the arms in green, adjusted the green QPD offsets for maximum buildup, and then updated the ITM camera references. Then we re-enabled the ITM loops in the guardian. This allowed us to power up all the way to 21 W without significant degredation of the recycling gain.

After that, we were able to consistently engage the ASC with the guardian.

Power-up issues

However, we found that at 21 W the interferomter suddenly unlocks in a matter of minutes. There seems to be no instability in the arm or sideband buildups before the lockloss. We looked at OMC DCPD signals for signs of PI, but we did not see anything ringing up during any of our short high-power locks. Some times to look at are 02:29:50, 02:59:50, 04:57:30, 06:55:00, all 2015-05-26 UTC. But any of the other 21 W locklosses in the past 12 hours follow this pattern.

We measured the OLTFs of PRCL, MICH, SRCL, and DARM before and after powering up, but they all look fine and did not change with power. For CARM, we start at 3 W with a UGF of 14 kHz with 47° of phase. Then during power-up, the electronic gain is automatically adjusted to compensate for the increased optical gain. The algorithm for this was shooting a little high, so after power-up the UGF was more like 27 kHz with 30° of phase. This is probably fine, but we adjusted the algorithm anyway, so that the UGF is placed at 19 kHz, with 45° of phase. Anyway, this did not solve the lockloss issue.

We also tried locking at some lower powers. At 15 W the interferometer lasted for about 15 minutes before unlocking. At 10 W, the lock time seems to be indefinite (at least 90 minutes).

DARM crossover

Using FM9 in ETMY L1 LOCK L (zero at 2 Hz, pole at 5 Hz), we were able to push the L1 crossover from <1 Hz to 1.7 Hz by adjusting the filter gain from 0.16 to 0.31. Measurement attached, showing before and after. This is not included in the guardian. By pushing up the crossover, the rms drive to L2 decreases from >10000 ct to about 6000 ct or so.

Other

For the record, we did not notice any kicks to the yaw of IMC REFL tonight.

The Symptoms
============

The laser is off, the chillers are running and the status screen shows either
"Interlock OK" as red, or both "Interlock OK" and "Epics Alarm" as red.



Possible Causes
===============

a) One of the safety relays in the interlock box is faulty.

b) There is a problem with the TwinCAT code controlling the chillers.

c) There is a problem with the TwinSafe code implementation.

d) Network delays in the PSL EtherCAT network accumulate to a point where
TwinSafe initiates a shutdown.

e) The turbine based flow sensor in the chiller gets stuck, registering a drop
in flow which in turn triggers the laser interlock.



The Evidence
============

a) The Dold LG5929 safety relay extension module has a mechanical life time of
20 million switching cycles and a mean time to dangerous failure of 144.3 years.
The contacts are normally open.

    Safety relays fail, they do not fail intermittently.

    Simulating an interlock box failure by switching off the interlock box
results in a sea of red on the status screen.

b) I have gone through the TwinCAT code for both chillers.  The only difference
is that there is an extra variable declared for the diode chiller.  However this
variable is never invoked in the chiller code.  The instruction sets for both the
diode and crystal chiller are the same.

c) I have gone through the TwinSafe function blocks and have not found anything
wrong with it.

d) This possibility game up because each time the laser had tripped, Patrick and
I noticed that there was 1 lost frame in the TwinCAT datastream.  However with
the laser running last week I noticed that something like 201 frames were lost
and the laser was still running.  So network delays were ruled out.

    Simulating a network delay by removing either the Rx or Tx fibre from the
Ethernet switch, results in a sea of red on the status screen.

e) The output of the flow rate sensor goes to the chiller controller via a normally
open switch.  If the flow rate is within the allowed range, the switch remains closed.
If it goes out of range, it opens and remains open until the flow rate is restored.

    For the crystal chiller, opening the flow rate switch does indeed switch off the
chiller.  Closing the flow rate switch does not switch the chiller back on automatically.
A number of fields on the status screen go red and can only be cleared by switching
the chiller back on and pressing the reset button on the status screen.

    For the diode chiller, opening the flow rate switch does switch off the chiller.
Closing the flow rate switch, turns the chiller back on.  What's more all the red
flags on the status screen go green automatically except "Interlock OK".



Conclusion
==========

Of the 5 cases outlined above, the one that seems to reproduce the observed events
is the last one.  In particular it seems that it is the flow rate sensor in the diode
chiller.

The MC2 suspension occasionally ran into a situation where some part of the suspension was kicked intermittenly for some unknown reason. I noticed this behavior this morning. I could not identify what was kicking.

The worst part is that the kicks (or glitches) seem to be gone since sometime around 14:14 local. Very bad.

The symptoms are:

• Once it runs into a glitchy period, the suspension glitches mostly in yaw. This is visible in the witness sensors and even in the IMC REFL camera.
  • Usually not big enough to unlock the IMC. However, very bad for lock acquisition which relies on the stability of the laser power.
• According to the witness sensors at the bottom stage, glitches are as large as 20 urad in yaw in pk-pk and 5 urad in pitch.
• This happened even when all the active loops were off (LSC, ASC, and damping loops).
  • Note that I left the static alignment biases on -- so there were some finite constant DAC counts at the top stage.
• When it came back to the nominal situation where no glitches, the actual alignment changed roughly by 5 urad in yaw.
• Evan also noticed a similar behavior yesterday evening for the first time.
  • Maybe this issue is new and started happening very recently ?

(Checking various channels)

Apparently this behavior is different from the two DAC issues we have seen in this past week (alog 18453 outputting a high constant voltage, alog 18569 outputting a nonlinear signal). As mentioned above, I disabled ail the active loops including the LSC, ASC and the top stage damping loops. Even in this condition, the suspension still kept being kicked interminttently. I looked at VOLMONs on all the stages, but did not find any suspicious activities. Also, I checked motions of the HAM3 ISI in L, P and Y using the suspension-coordinate-witness-signals (i.e. SUS-MC2_M1_ISIWIT_L(P, Y)MON), but did not  find such a fast signal. In addition, I checked the bottom stage witness sensors of PR2 as well to see if this is some kind of table motion, but the PR2 was very quiet and no glitches were found at the time when MC2 was glitching.

(A test without LSC or ASC signals)

I attach a second trend of some relevant channels. This is a two-hours trend, with the top stage damping loops fully on and with no LSC or ASC signals. You can see that the wintess sensors of the bottom stage showed glitches in the first 1/3 of the trend and suddenly stopped. Since the damping loops were engaged during this periode, the VOLMONs showed some reaction of the loops and I think they are just reactions and not the cause of the glitches.

Actually, now the top stage RT OSEM makes me think this is it. Its VOLTMON showed a noticable discrete jump (by 14 counts) in the attached trend right around the time when glitches stopped.

Thank you, Dave, Richard, John, Gerardo and those who helped us today even though it was a holiday. These pictures are for you guys.

Richard, Kiwamu, Dave:

Following Richard's suggestion, Kiwamu powered down the IO Chassis and removed the lid. After about 10 minutes the IO Chassis and then the CPU were powered up. I had removed h1seih45 from the rtsystab file to prevent any models auto starting. After several minutes I verified the Gen Std cards were still visible on the bus. I then started the IOP model. It came up with a minor IRIG-B excursion, which came back down in a few minutes. The Gen Std cards were still with us. I then started the user models, one at a time, with no problems.

Kiwamu and Evan are now untripping the watchdogs and starting isolation. Hopefully this will get us through to tomorrow.

I've put h1seih45 back into rtsystab. Kiwamu noted that the ADC and DAC Gen Std cards have LEDs which are on when powered up, and the DAC cards LEDs go out when the IOP is started. We should see what these LEDs mean.

SudarshanK, DarkhanT, RickS

We performed a routine calibration of Photon calibrator photodioides (both transmitter module PD-TxPD and receiver module PD-RxPD) at Xend on 20th May and Yend on 22nd May. We will post the results of the calibration soon.

Richard, Kiwamu, Dave:

the IOP model on h1seih45 has failed, it cannot see any General Standards cards in the IO Chassis when doing a software bus scan (lspci), but it can see the Contec Binary IO cards:

root@h1seih45 ~ 1# lspci -v |grep 3101

root@h1seih45 ~ 1# 

root@h1seih45 ~ 1# lspci -v |grep 3120

root@h1seih45 ~ 1# 

root@h1seih45 ~ 1# lspci -v |grep -i contec

13:00.0 Multimedia controller: Contec Co., Ltd Device 8682 (rev ff) (prog-if ff)

21:00.0 Multimedia controller: Contec Co., Ltd Device 8682 (rev ff) (prog-if ff)

2c:00.0 Multimedia controller: Contec Co., Ltd Device 8632 (rev ff) (prog-if ff)

Kiwamu went into the CER and confirmed that the h1seih45 chassis is powered up, the ADC interface cards have their LEDs lit, fans are on. We powered down the CPU and the Chassis (keeping the latter down for a minumum of 30 seconds) and then powered them back up.

This is where it gets strange. As soon as I could log back into h1seih45 I could see the General Standards cards on the PCI bus (six ADCs, two 16bit DACs). But, when the IOP model started, they disappeared from the bus and once again all I could see are the Contec cards.

controls@h1seih45 ~ 0$ lspci -v|grep 3101

Subsystem: PLX Technology, Inc. Device 3101

Subsystem: PLX Technology, Inc. Device 3101

Subsystem: PLX Technology, Inc. Device 3101

Subsystem: PLX Technology, Inc. Device 3101

Subsystem: PLX Technology, Inc. Device 3101

Subsystem: PLX Technology, Inc. Device 3101

controls@h1seih45 ~ 0$ lspci -v|grep 3120

Subsystem: PLX Technology, Inc. Device 3120

Subsystem: PLX Technology, Inc. Device 3120

controls@h1seih45 ~ 0$ lspci -v|grep -i contec 

13:00.0 Multimedia controller: Contec Co., Ltd Device 8682

Subsystem: Contec Co., Ltd Device 8682

21:00.0 Multimedia controller: Contec Co., Ltd Device 8682

Subsystem: Contec Co., Ltd Device 8682

2c:00.0 Multimedia controller: Contec Co., Ltd Device 8632

Subsystem: Contec Co., Ltd Device 8632

< about here the models try to startup >

controls@h1seih45 ~ 0$ lspci -v|grep 3101

controls@h1seih45 ~ 1$ 

Richard suggested we power the sytem down for an extended time (10 minutes) to cool everything down and run with the lid off. Kiwamu is doing that at the moment.

I'll disable the autostartup of the models, so we can manually step through the startup process.

Kiwamu and Dan reported that there is no potable water in the OSB. It looks like the RO system went into fault on May 22 around 5:30pm and this morning the potable water tank went dry.

Kiwamu visited the water room for me and reset the RO system so it now appears to be operating normally and making water. However, the building is not yet pressurized.

I suspect the Naimco water skid will need to be reset.

Dan, Kiwamu, Evan

List of things done today:

• The ETMX UIM coil driver had tripped off sometime yesterday, and we drove down to the end station to turn it back on.
• In ETMY L3 drivealign, there is now compensation for the LVLN driver summing network. The compensation is a zero at 152 Hz and a pole at 3.2 kHz. This brings the high-frequency DARM phase response back to its previous shape. (See attachment.)
• In the DARM_OFFSET state, the DARM gain is now increased by a factor of 2 and the low-frequency boost (FM7) is engaged. This makes DARM more stable during the transition to dc readout. However, this throws off the procedure for determining the scale factor between OMC DC and AS45Q, so sometimes the transition causes the DARM gain to decrease.
• A high fifth-harmonic violin mode was causing the ETMY ESD to saturate, so there is now a stopband filter in the ETMY L3 LOCK filter bank (since DARM is full).
• LOWNOISE_ESD_ETMY has been rewritten for the new transition procedure. It has been tested (once) and works.
• We took a crossover measurement of ETMY L2. We tried to get ETMY L1 as well, but it seems currently that it's quite low (below 1 Hz). The gain in the L1 LOCK filter can be pushed up from 0.16 to 0.22 without issue (and maybe even further).

The CW hardware injection code psinject now runs without crashing at LHO.  (Thanks, Peter Shawhan, for debugging the launch script.)  The injection pathway is turned off, but the injection code has been sending 13 standard CW signals into the CAL model since approximately 21:30 PDT.  Psinject is still running many hours later, so the code seem stable.  Additionally, all injection code has now been migrated to svn here:

https://svn.ligo.caltech.edu/svn/dac/hwinj/Details

I've tested the svn version of tinj (for transient injections) at LHO and it seems to work properly, though, additional tests are planned.  The latest version includes additional interaction with EPICS channels.  During the course of today's debugging, I determined that GRB alerts do not appear to be recorded in the channel H1:CAL-INJ_EXTTRIG_ALERT_TIME.  I have contacted Andrew Williamson to sort out why.

Kiwamu, Peter, Dan, Evan

We made it back to dc readout tonight. 3 W PSL power, DARM feedback to ETMX.

We are in the process of trying to engineer the new transition to ETMY. We were hampered by the intrusion of a large amount of 1/f³ noise in DARM, with ASD 1×10⁻¹⁴ m/Hz^1/2 at 10 Hz. The initial portion of the lock was fine, and then the noise came. We trended various channels and looked at coherences, but could not nail down the cause. The noisy time starts at 2015-05-23 9:41:00 UTC. After about an hour, the noise went away again.

The ASC was a bit touchy. It could be engaged by hand, but with the guardian it seemed that engaging the PRM pointing loop caused several of the other loops to go unstable. Also, we gave up on the ITM pointing loops. They increased the recycling gain, but made the arm and sideband buildups less stable.

We found that there was no tidal signal being sent to ETMY. At first blush it appears to be an IPC issue.

Peter, Calum, Ben, Rich

Took transfer functions (1Hz to 10kHz, 255 points in to the DAC Drive Input on the front panel of the ESD driver, out of the respective SHV connector going to the ETM) of all the quadrant paths of the newly installed LN Driver.  The results are stored in:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Measurements/ElectronicsMeasurements/
under the following file suffixes

UR -> TFSR785_22-05-2015_102054.txt
LR -> TFSR785_22-05-2015_102804.txt
UL -> TFSR785_22-05-2015_103546.txt
LL -> TFSR785_22-05-2015_105053.txt

Everything looks good as compared to the Spice model.

Also, we measured the output noise of each channel at 20Hz with and without DC bias.  Results are all consistent with the design (<40nV/rtHz at 20Hz and above) and Spice model.

0V bias/-7.4V bias
UR -> 21nV/rtHz/32nV/rtHz
LR -> 22nV/rtHz/30nV/rtHz
UL -> 23nV/rtHz/37nV/rtHz
LL -> 28nV/rtHz/37nV/rtHz

Analyzer Noise Floor 10nV/rtHz @ 20Hz

Typical output noise at higher frequencies is:
20nV/rtHz @ 50Hz, 22nV/rtHz @ 100Hz, 20nV/rtHz @ 150Hz

Noticed that there is still a discrepancy in the DAC monitors for the user model vs. the IOP.  Richard has the details and Jeff will likely follow up next week.

Once Richard sorts out the somewhat touchy vacuum interlock (he's working on that now and knows the problem), the entire system is now functional and ready for use.