Reports until 17:41, Tuesday 07 July 2015
H1 SUS (CDS, ISC, SYS)
jeffrey.kissel@LIGO.ORG - posted 17:41, Tuesday 07 July 2015 - last comment - 09:55, Wednesday 08 July 2015(19480)
ETM ESD High Voltage Found OFF, ETMX resurrected, ETMY on, but Driver is Railed Negative Again
J. Kissel, L. Prokhorov, T. Shaffer

Another episode in the saga of the High Voltage ESD Drivers. Though we've not been able to identify why, the high-voltage for both end station's ESD High Voltage Drivers were found OFF. We've queried the usual suspects, and those who were around, and no one has any idea why they might have been *turned* off, so we suspect that they had tripped off from all of the vacuum and electronics work at the end-stations today (the best aLOGs for which are the sparse ops log, LHO aLOG 19462, and VE logs 19472, and 19471).

The ETMY Driver remains railed negative. We've tried all of the possible button mashing and cable plugging and unplugging we've either done before or heard has been tried before, with no success. We request the assistance of the CDS team in the morning.

---------

As you know, we're still in the odd state that the ETMY high-voltage driver is controlled by the new(ish) low-voltage driver, and the ETMX high-voltage driver is controlled via a temporary Beckhoff setup. Therefore, the start-up procedure for both is different.

ETMX
- Flip ON the high-voltage supplies' power switch in the racks near the entrance of the building (you'll hear a ~few second loud continuous beep, wait for the initialization procedure to complete)
- Hit black "V SET" button, followed by entering in 430 V (i.e. hit 4, 3, 0, then enter)
- Hit black "I SET" button, followed by entering in 80 mA (i.e. hit 8, 0, then enter)
- Hit red "OUTPUT ON/OFF" button
You should see the volts go to ~430 V, and the current go to ~3 mA

Now, head out to the driver in the XVEA. You should see both the 430 V and the (low voltage, I don't remember what it is 12 or 15 or something) lights on the front panel showing bright green. You can either turn on the driver manually by hitting the red "start" button, or using the temporary remote system. You'll know the driver is ON when the light above that button goes from OFF to bright RED.
To use the temporary remote system, via a terminal on a work station, toggle one of the remote switches:
caput H1:ISC-EXTRA_X_BO_4 0
caput H1:ISC-EXTRA_X_BO_4 1
then turn on the second,
caput H1:ISC-EXTRA_X_BO_3 1
Once you see all of the lights on on the front panel of the high voltage driver, make sure to confirm that you see a change in the readbacks when turning ON and OFF requested signals (i.e. turn the BIAS "DC" ON and OFF, and turn an OFFSET in each of the quadrants UL, LL, UR, and LR ON and OFF).

ETMY
(This is the same as ETMX)
- Flip ON the high-voltage supplies' power switch in the racks near the entrance of the building (you'll hear a ~few second loud continuous beep, wait for the initialization procedure to complete)
- Hit black "V SET" button, followed by entering in 430 V (i.e. hit 4, 3, 0, then enter)
- Hit black "I SET" button, followed by entering in 80 mA (i.e. hit 8, 0, then enter)
- Hit red "OUTPUT ON/OFF" button
You should see the volts go to ~430 V, and the current go to ~3 mA

(This is different than ETMX)
- Head out to the YVEA (so you can see the front panel of the ESD driver, because these first few steps rarely work), and open the BIO screen of the ETMY SUS.
- Hit the big beige RESET button on the bottom right corner. This *may* toggle the High Voltage Driver on and off, and you *may* regain functionality but likely not.
- Several times now, the driver has been found to be "railed negative" in which he primary symptom is that the readbacks for every channel show about -15700 [ct], and the channels are unresponsive to change in requested drive signal.
Here's what has been tried in the past (ranging from "that's probably it" to "there's no reason hat would have worked anyways, but we're grasping at straws"), and so what I've tried this evening:
- Manually turning ON and OFF the driver with the START button on the front, with the REMOTE RESET cable and INPUT cable still plugged in.
- Unplugging the REMOTE RESET cable and manually turning ON and OFF the driver with the START button (INPUT cable still plugged in)
- Plugging the REMOTE RESET cable back in, and trying both to remotely and manually turn ON and OFF the driver
- Unplugging REMOTE RESET and INPUT cables, manually turning ON and OFF the driver
- Toggling all of the low-voltage driver binary I/O switches (i.e. the HI/LO VOLTAGE and HI VOLT DISCONNECT switches)
None of which were successful this time around.
Comments related to this report
jeffrey.kissel@LIGO.ORG - 17:45, Tuesday 07 July 2015 (19481)
Opened FRS Ticket #3284 (https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=3284)
richard.mccarthy@LIGO.ORG - 07:24, Wednesday 08 July 2015 (19487)
This seems to be a problem when the HV is removed and the system is not powered down or reset.  Not sure why.  At EY I powered the unit off removed the DAC cable Powered the unit on re attached the DAC cable and all seems to be working.
jeffrey.kissel@LIGO.ORG - 09:55, Wednesday 08 July 2015 (19491)
Attached is an image of what Richard means by "the DAC cable," which, on the front panel is marked as "PREAMP" and "INPUT." As mentioned above in my debugging procedure, I had tried unplugging this cable, hitting the start/stop button, then replugging in the cable, but this did *not* work for me. But I *must* have done something different than Richard, because he says that this method is the most reliable method for fixing this problem. He did mention that he *did not* unplug the REMOTE RESET cable, where as I had already had the REMOTE RESET cable unplugged (from other attempts) when I performed the power cycle. Maybe the REMOTE RESET cable needs to be plugged in IN when unplugging the PREAMP INPUT cable and hitting the start/stop button. 
Non-image files attached to this comment