The double-edged sword of cutting-edge electronics science

A. Helmling-Cornell, J. Kissel, M. Pirello,  G. Wallace

While Marc, Adrian, and Gavin were out this morning setting up test equipment to continue their study of SQZ phase noise on the 3.125 MHz line, we found ourselves with another LSC front-end crash. Again zooming out there to try to identify what is causing these occasional glitches, they imagined that it might have been coincident with several activities:
    - Plugging in their test equipment (via surge protector) to the 4th, only available, socket of the wall-wort power hub above ISC R3. 
    - Turning off the temporary (though permanently mounted) test RF oscillator in ISC R4 (powered via a power strip via the same wall-wort power hub)
    - Jostling the timing fibers coming in/out of the timing distribution system 
 
As such, after Dave restored the lsc0 front-end models, we ran through *many* iterations of the same activity, hoping to find *something* the reproducibly crashes the front end.
    - With the test equipments' surge protector plugged in to the 4th socket, we turned OFF, then ON the protect ~5 times. No crash.
    - With the test equipments' protector in the ON configuration, we installed and pulled out the plug from the 4th socket ~5 times. No crash. 
    - Turned on / off the surge protect for the test oscillator ~5 times. 
    - Jostled all of the power plug is in the wall-wort power hub.  No crash. 
    - Gently fussed with all of the timing fibers at the timing distribution ports.  No crash. 
    - Gently fussed with the timing fiber going in to the lsc0 IO chassis.  No crash. 
    - Gently fussed with the DC power cables going in to the lsc0 IO chassis.  No crash. 
    - Connected / disconnected the test equipment from the 3.125 MHz TNC ports under test.   No crash. 

So we're back to superstition-level "just don't go near anything," but this is just infeasible to follow and should not be the case. Having no better course of action to take, I've taped off the 4th socket with a label that says "DO NOT USE FOR TEST EQUIP."

First attachment shows the wall-wort power hub above ISC R3 before equipment was unplugged.
Second attachment shows the power supplies for the timing slaves for both the ASC and LSC IO chassis on top of ISC R1
Third attachment shows the LSC IO chassis and the temporary power cable leading in to it.
Fourth attachment shows the wall-wort power hub above ISC R3 now in the "DO NOT USE" configuration I left it.
Fifth attachment shows the permanently installed test oscillator in ISC-R1.

This "temporary" configuration of power supplies has been in place since WELL before the start of the run, originally in Sep 2018 -- according to the very terse, and closed FRS ticket about it (see FRS Ticket 11452), and then revisited Mar 2019 (see LHO aLOG 47673) which -- in fact -- glitched the LSC system upon install (see LHO aLOG 47681 and FRS Ticket 12567).

The most recent, last, time such a glitch because "people were near the racks, but no obvious cause" happened was Aug 29 2019 -- see LHO aLOGs 51616 and 51630. Other occurrences will be a challenge to retrace, as these crashes often happen in the middle maintenance day and/or recovery, and in the chaos folks want to just get *something* written down, and that *something* is often inconsistent and terse. 

Finally -- this temporary configuration of power supplies for our most sensitive ISC IO chassis timing slaves and fans  are *still installed* because we know that new IO chassis are coming in the future, and [as far as I know] don't want to replace them during the run because it's a big operation. I'll confirm this representative parties.