Evan, Sheila,
Many things were tripped this morning after the earthquake in Nepal, (ISIs, HEPIs, and suspensions). most of this went smoothly. I had trouble untripping HAM5, the screen shot I attach is the collection of screens that I was looking at when I was confused, from these I thought that the watchdog was untripped, but that was untrue, in the ISI watchdog screen the small box aroud the rouge watchdog was red. Maybe this could be more obvious, say if the watchdog indicator on the ISI overview screen was red when the rouge watchdog was tripped, and if the reset all button really reset all the watchdogs. That might have save Jeff and Jim from a flurry of saturday morning texts in the future. Jeff logged in and reset the rouge watchdog.
Also, the PSL was tripped this morning about 16 UTC (not at the same time as the earthquake), it might have been a flow sensor problem. After turning the laser on we weren't able to open the HPO external shutter from the beckhoff computer, we came to the control room and found that the flow sensor was red, we reset that but still weren't able to open the shutter from the control room. We went back into the diode room and were able to reset the shutter from there.
After the PSL came back SYS_DIAG said that the NPRO noise eater was oscillating, but we think that it is not. The attached screenshot shows 100 days of the NPRO ROO readback, this number is around -500 when the noise eater is osciallating, Each time that the laser tripps and is turnd on again this readback is high for a while before settling down. We changed the tolerance on the readback to 50 counts from 10 counts in the SYS_DIAG test.
At one point the ETMX ESD driver tripped off, and we could not reset it by toggling the hi/lo binary outputs.
We drove down to the end station, toggled the binary output, but could not hear the driver relay switching. So we unplugged the remote control dsub, activated the driver via the front panel button, and then plugged the cable back in. Toggling then worked as expected.
[Edit: this trip is happening after pretty much every lock loss. It can be untripped by bringing BO_4 low momentarily, and then toggling BO_3 as usual.]
The PSL tripped again. It was restarted in the same fashion.
Some more notes:
The PSL tripped again. I called Rick, who said it was OK to restart it.
These trips appear to be correlated with momentary trips in the diode chiller flow bit (the most recent trip is attached). The computer in the diode room shows the flow holding steady at 20 lpm or so.
These are approximately the three trip times (UTC):
The PSL tripped once more, around 2015-04-26 22:57:47 UTC. I reset it.