Evan, Sheila,
Many things were tripped this morning after the earthquake in Nepal, (ISIs, HEPIs, and suspensions). most of this went smoothly. I had trouble untripping HAM5, the screen shot I attach is the collection of screens that I was looking at when I was confused, from these I thought that the watchdog was untripped, but that was untrue, in the ISI watchdog screen the small box aroud the rouge watchdog was red. Maybe this could be more obvious, say if the watchdog indicator on the ISI overview screen was red when the rouge watchdog was tripped, and if the reset all button really reset all the watchdogs. That might have save Jeff and Jim from a flurry of saturday morning texts in the future. Jeff logged in and reset the rouge watchdog.
Also, the PSL was tripped this morning about 16 UTC (not at the same time as the earthquake), it might have been a flow sensor problem. After turning the laser on we weren't able to open the HPO external shutter from the beckhoff computer, we came to the control room and found that the flow sensor was red, we reset that but still weren't able to open the shutter from the control room. We went back into the diode room and were able to reset the shutter from there.
After the PSL came back SYS_DIAG said that the NPRO noise eater was oscillating, but we think that it is not. The attached screenshot shows 100 days of the NPRO ROO readback, this number is around -500 when the noise eater is osciallating, Each time that the laser tripps and is turnd on again this readback is high for a while before settling down. We changed the tolerance on the readback to 50 counts from 10 counts in the SYS_DIAG test.
At one point the ETMX ESD driver tripped off, and we could not reset it by toggling the hi/lo binary outputs.
We drove down to the end station, toggled the binary output, but could not hear the driver relay switching. So we unplugged the remote control dsub, activated the driver via the front panel button, and then plugged the cable back in. Toggling then worked as expected.
[Edit: this trip is happening after pretty much every lock loss. It can be untripped by bringing BO_4 low momentarily, and then toggling BO_3 as usual.]
The PSL tripped again. It was restarted in the same fashion.
Some more notes:
- I got tired of the intermittent 0.4 Hz instability breaking the lock, so I closed some weak oplev damping loops around the pitch of the ITMs (FM1, FM4, FM7, and FM8 in the L2 oldamp filter banks, with a gain of -300). These suppress the motion around 0.4 Hz by a factor of 3 or so with the interferometer unlocked (these loops could probably be made stronger with some inversion of the suspension TF). Anyway, after turning these on I did not have any more problems with the instability. Of course, it could just be a coincidence, since the instability is intermittent.
- On that note, I saw the instability sometimes appear even if the interferometer came into resonance with a mediocre recycling gain (30 W/W or so).
- The ITM camera ASC loop was causing the Y arm to drop lock in green, so I turned it off (by zeroing the input matrix elements) and steered the ITM by hand in order to bring the camera spot close to the saved position.
- I reverted to the old transition point (and gain) for the handoff of DARM from ALS DIFF to AS45Q. The ETMX ESD was badly saturating, even with linearization off. If we really feel it is necessary to do the transition at a higher CARM offset, we need to reexamine the DARM loop shape and the crossover with L1.
The PSL tripped again. I called Rick, who said it was OK to restart it.
These trips appear to be correlated with momentary trips in the diode chiller flow bit (the most recent trip is attached). The computer in the diode room shows the flow holding steady at 20 lpm or so.
These are approximately the three trip times (UTC):
- 2015-04-25 16:29:14
- 2015-04-26 05:40:55
- 2015-04-26 20:33:54
The PSL tripped once more, around 2015-04-26 22:57:47 UTC. I reset it.