The dolphin network was fixed (by switching the port into which the problematic card was plugged to another) by 2019-07-24 19:20 UTC.

I suspect the root cause is that the end-station Dolphin switch got 'wedged' due to power glitch.  In this case it can be in a state where you can't re-enable ports.  I had this happen on the LLO DAQ test stand See TST log 12353, TST log 12338.  The cable swap work because it was to a port that had not been disabled previously (default state is enabled)

Just a s a point of interest, you should be able to program the HV power supply so that all it takes is the press of a couple buttons for anyone to reset it (even though here its usually Radar or I who still reset it). Here at LLO the PMC HV supply is the thing that usually goes out first with any power glitch and is the thing we reset the most/have to deal with with any power glitch.

Ive attached pics of the instructions we have on the front of our supply that once the values programed in show what we have to do to reset.

The requested powers are stored in three different Beckhoff PLCs' and one Front-end's SDF system, which is why they've been poorly maintained:

Ring Heater    Name on SDF         Name of SDF            Front-End "Computer"
               Overview            EPICs server           Name Hosting EPICs       System Type
ITMX & ITMY    CS_ECAT_PLC3        H1SYSECATCPLC3SDF      H1:FEC-1026_SDF          Beckhoff

ETMX           EX_ECAT_PLC3        H1SYSECATXPLC3SDF      H1:FEC-1029_SDF          Beckhoff 

ETMY           EY_ECAT_PLC3        H1SYSECATY1PLC3SDF     H1:FEC-1032_SDF          Beckhoff

Filter Input
All RHs        H1TCSCS             H1TCSCS_SDF            H1:FEC-26_SDF          "Fast" Front-end

We've now updated ALL of these SDFs' safe.snap to be set at the nominal values described above.

FURTHER: We haven't / hadn't trended back far enough, but -- in short -- these ring heater guardians should be in the NOMINAL state at all times, not FILTER_RH_INPUT.
Likely, when we powered up to 37 W in May, we intentionally changed the ring heater settings, and *left* the guardian states in "FILTER_RH_INPUT" -- because using the predictive filtering takes time -- and thus, in the rush to get to observing we accepted this state, and it's been this way for the recent past that we've trended on *some* (namely the ETMs) guardians.

However, we're no longer intentionally changing the ring heater settings (for the forseeable future, at least -- the next attempt power up is likely "weeks" away) so we don't need to engage the predictive filtering system. Thus, we should have all ring heater guardians in NOMINAL.

ALSO, ALSO: Back in May, during the power up, and in the rush to get back to observe, we un-monitored these requested power and filter input channels. Why? Because when we're *using* the predicitive filtering system, the predictive filter is slowly but actively changing the requested power. Again, because we're no longer using the predicitive filtering, and the requested power should be stationary, we have now REMONITORED the following channels:
    H1TCSCS
        H1:TCS-ITMX_RH_INVERSE_FILTER_IN
        H1:TCS-ETMX_RH_INVERSE_FILTER_IN
        H1:TCS-ITMY_RH_INVERSE_FILTER_IN
        H1:TCS-ETMY_RH_INVERSE_FILTER_IN

    CS_ECAT_PLC3
        H1:TCS-ITMY_RH_SETUPPERPOWER
        H1:TCS-ITMY_RH_SETLOWERPOWER
        H1:TCS-ITMX_RH_SETUPPERPOWER
        H1:TCS-ITMX_RH_SETLOWERPOWER

    EX_ECAT_PLC3
        H1:TCS-ETMX_RH_SETUPPERPOWER
        H1:TCS-ETMX_RH_SETLOWERPOWER

    EY_ECAT_PLC3
        H1:TCS-ETMY_RH_SETUPPERPOWER
        H1:TCS-ETMY_RH_SETLOWERPOWER

Importantly -- the time to which one should restore slider values is *just before* a lock acquisition sequence, NOT when we're locked in nominal low noise.
In this case, I chose 2019-07-24 03:30 UTC (1247974218, Jul 23 2019 20:30:00 PDT).

Niko called this morning saying that the PSL was offline.  At that point in time, the laser was on but all the
servos were unlocked.  The following steps were taken:
 - restored the Kepco high voltage power supply (BHK 500-0.4MG) settings.  Output voltage 375V, current limit 50mA.  These values are
     on a label stuck to the front panel of the power supply
 - lock the pre-modeecleaner by enabling the lock button and ramp button
 - check the output DC values of PDA and PDB on the ISS MEDM screen.  Both should be ~10V.  Enable the autolock.  Check that the diffracted
     power is not gyrating wildly.  An indication of this is that the pre-modecleaner transmission fluctuates by many watts.
 - lock the FSS by enabling the autolocker.  Niko stated that it was taking some time for the FSS to acquire lock, so I told him
     how to lock the servo manually by entering manual mode and changing the laser crystal temperature, making note of the minimum and
     maximum temperature values listed in the lock acquisition range.  These limits are to minimise the beat note with the ALS laser.
     Once a mode was observed in the reference cavity transmission camera, press the loop on button to close the loop, then press the
     temploop button.  Enable the autolocker by pressing the ON button.

    The PSL was back online after completion of these steps.

    Next up was bringing the input modecleaner online.  The message about the WFS not being centred flashed up regularly.  Niko slowly
  brought the IMC mirrors back to their nominal positions.  Whilst this was happening the reflected beam improved.  After the restoration
  the IMC had no problems locking.                                                                                                                    

Still same situation, data goes to EY, but no data in the reverse direction.

Looking at dmesg on h1cdsrfm and comparing EX card and EY card, they are the same until the end. EX has:

[   56.381196] IXH Adapter 1 : NodeId is 100 - Serial no : 002344

[   56.382683] IXH Adapter 102 : Configured Adapter to nodeId=104 (GX nodeId=26)

[   56.487658] IXH Adapter 1 : Local adapter is connected to IXS600 switch, TOP, port 7

[   56.487708] IXH Adapter 1 : Link 0 is operational - Link width: x8 - PCIe Gen 2 -- Downtime: 0 sec

whereas EY has:

[   56.384263] IXH Adapter 2 : NodeId is 104 - Serial no : 002364

[   56.384937] IXH Adapter 2 : Set reachable remote node list.

Note that the cdsrfm system plugs into different IX-switch ports at the three locations (CS=5, EY=6, EX=7) so there is no ambiguity as to which is EX and EY.

DMESG Details:

EX:

[   51.202139] IXH Adapter 1 : Set min link width to x8

[   51.202140] IXH Adapter 1 : Set min link speed to Gen1

[   56.380546] IXH Adapter 1 : Driver attaching device 1

[   56.380639] IXH Adapter 1 : Local adapter type: IXH611

[   56.380641] IXH Adapter 1 : Number of local link ports enabled: 1

[   56.380699] IXH Adapter 1 : IDT Logan adapter chip revision: ZC

[   56.380749] IXH Adapter 1 : EEPROM version NTB mode: 8

[   56.380751] IXH Adapter 1 : Adapter serial number: IXH611-CE-002344

[   56.380798] IXH Adapter 1 : PCIe slot - x1, (gen2) 5 GT/s

[   56.380799] IXH Adapter 1 : BAR2 size: 256 MB

[   56.380801] IXH Adapter 1 : BAR4 size: 64 MB

[   56.380802] IXH Adapter 1 : BAR5 size: 8 MB

[   56.380804] IXH Adapter 1 : Prefetch memory BAR2 size: 256 MB

[   56.380852] IXH Adapter 1 : Logan chip temperature = 69 C

[   56.380994] IXH Adapter 1 : EEPROM dip-switch swmode[3:0]: 0010

[   56.380995] IXH Adapter 1 : Number of images in the EEPROM: 3

[   56.381043] IXH Adapter 1 : Clock mode slot (P0): Local

[   56.381044] IXH Adapter 1 : Clock mode link (P8): Local

[   56.381093] IXH Adapter 1 : GPIO data: 0x42

[   56.381094] IXH Adapter 1 : Multicast group size: 16 MB

[   56.381096] IXH Adapter 1 : Max payload size (MPS): 256 Bytes

[   56.381144] IXH Adapter 1 : Link[0] config: LINK_CONFIG_NTB

[   56.381192] IXH Adapter 1 : Adapter link 0 link width: x8

[   56.381194] IXH Adapter 1 : Cable link 0 PCIe link - x8, (gen2) 5 GT/s

[   56.381196] IXH Adapter 1 : NodeId is 100 - Serial no : 002344

[   56.382683] IXH Adapter 102 : Configured Adapter to nodeId=104 (GX nodeId=26)

[   56.487658] IXH Adapter 1 : Local adapter is connected to IXS600 switch, TOP, port 7

[   56.487708] IXH Adapter 1 : Link 0 is operational - Link width: x8 - PCIe Gen 2 -- Downtime: 0 sec

EY:

[   51.202144] IXH Adapter 2 : Set min link width to x8

[   51.202145] IXH Adapter 2 : Set min link speed to Gen1

[   56.383642] IXH Adapter 2 : Driver attaching device 2

[   56.383754] IXH Adapter 2 : Local adapter type: IXH611

[   56.383756] IXH Adapter 2 : Number of local link ports enabled: 1

[   56.383766] IXH Adapter 2 : IDT Logan adapter chip revision: ZC

[   56.383813] IXH Adapter 2 : EEPROM version NTB mode: 8

[   56.383815] IXH Adapter 2 : Adapter serial number: IXH611-CE-002364

[   56.383862] IXH Adapter 2 : PCIe slot - x1, (gen2) 5 GT/s

[   56.383864] IXH Adapter 2 : BAR2 size: 256 MB

[   56.383865] IXH Adapter 2 : BAR4 size: 64 MB

[   56.383867] IXH Adapter 2 : BAR5 size: 8 MB

[   56.383868] IXH Adapter 2 : Prefetch memory BAR2 size: 256 MB

[   56.383917] IXH Adapter 2 : Logan chip temperature = 69 C

[   56.384060] IXH Adapter 2 : EEPROM dip-switch swmode[3:0]: 0010

[   56.384061] IXH Adapter 2 : Number of images in the EEPROM: 3

[   56.384110] IXH Adapter 2 : Clock mode slot (P0): Local

[   56.384111] IXH Adapter 2 : Clock mode link (P8): Local

[   56.384160] IXH Adapter 2 : GPIO data: 0x42

[   56.384161] IXH Adapter 2 : Multicast group size: 16 MB

[   56.384163] IXH Adapter 2 : Max payload size (MPS): 256 Bytes

[   56.384211] IXH Adapter 2 : Link[0] config: LINK_CONFIG_NTB

[   56.384260] IXH Adapter 2 : Adapter link 0 link width: x8

[   56.384262] IXH Adapter 2 : Cable link 0 PCIe link - x8, (gen2) 5 GT/s

[   56.384263] IXH Adapter 2 : NodeId is 104 - Serial no : 002364

[   56.384937] IXH Adapter 2 : Set reachable remote node list.

For the lay-person
    IXH Adapter 2 = dolphin network switch (maybe?)
    IX611 card = long range dolphin network card in side the h1cdsrfm computer.
    EY adnaco chassis = where the dolphin to long range fiber conversion happens

Received a call from Niko.

EY problem -> Wait for Dave till the morning.

Corner station problems need to be worked on now. Niko called Jason and Peter King to ask how to reset PMC HV supply.

In the mean time Niko told me that he couldn't move some of the optics state to ALIGNED. That was mostly becaue of DC centering loops and SQZ ASC were outputting huge numbers, so I just disabled ASC (didn't have to do that for SQZ ASC) and cleared the history.

MC2 length output was big because the MC board boost was on when not in lock. I disabled it to prevent MC2 from tripping. Hopefully the guardian will take care of the rest of the MC board setting once PSL comes back and when Niko runs the down script. If Niko cannot lock IMC he'll call me again.

opened FRS13311

All front end can see their respective IO Chassis, I'm starting the model recovery.

All models restarted. Jonathan reports DAQ is in good health.

Long range dolphin is down, I'm investigating.

h1cdsrfm computer had restarted itself (it is a diskless front end, but the only one which got rebooted). I had to remember how to start the code since we haven't done so since Nov 2018.

Code is running, bad news is that we are not receiving anything from EY. This could be a possible problem with the EY remote Adnaco chassis. No, I can see all three Dolphin IX cards.

I've tried several reboots of h1cdsrfm, but I still get the problem that IPC from EY to the corner station is failing (CS to EY is working). 

A couple pictures.  The first is the front panel for the TCSy CO2 laser.  Notice that the IR Fault light is not lit, but the Temp alarm is.  I seem to remember that when the viewport IR sensor trips, the IR Fault light flashes but does not hold, but I don't think the Temp alarm accompanies this.  So I think this rules out the viewport IR sensor.

The 2nd attachment is the blank display on the TCSx chiller, while the 3rd is the active display on the TCSy chiller.

Entered FRS 13309 for this trip.

The chiller screen has been blank since January when Patrick reenabled the serial communication (alog46289).

The discrepencies in the tempuratures is interesting. I think this may be a good lead because these TCSY lock losses have become more frequent when we put the spare chiller in.

I think TJ might be on to something here.  I trended the power and temp of the TCSy CO2 laser, and the temp reported by the TCSy chiller, through the month of January 2019, bookending the chiller swap that occured on Jan 15 2019.  After the swap the laser temp increased from ~24.5°C to ~25 °C.  It hovered there for about 3 days or so, then jumped up to ~ mid-26 °C, with jumps up to over 27 °C.  In addition to this, the laser temperature became much more erratic after the swap, and that behavior has continued to today.  It should be noted that the TCSx CO2 laser sits at a laser temperature of ~24.3°C, around where the TCSy CO2 laser temp used to sit before the chiller swap.  Next step is to look back into 2018 to see if the stable pre-chiller-swap TCSy laser temperature behavior is consistent or a fluke.  Another interesting thing to check is if the TCSy CO2 PZT is moving more than the TCSx CO2 PZT.  I seem to recall seeing an alog about this, I'll hunt it down and link it here if it's relevant.

More data mining.

1st attachment is the laser power and temperature, as well as the chiller setpoint and the temp reported by the chiller itself, for the 4th quarter of calendar year 2018 (Oct-Dec 2018).  As can be seen, even though there are spikes, the temp is consistently in the low-24s °C, not bouncing around as seen after the chiller swap (in the plot I posted in the 3rd comment above).  Also of note, the discrepency between the temp reported by the chiller and the chiller setpoint is present even on the original TCSy chiller.  That said, this is further evidence that the spare TCS chiller doesn't seem able to hold the laser at a consistent operating temperature.

Also, I found the alog regarding PZT movement between the 2 TCS CO2 lasers; TJ noted this on July 10th in this alog.  Taking this a step further, I plotted out the PZT signal for each CO2 laser for the month of January 2019 (as I did with the laser temp yesterday), as well as from Dec 2018 to Feb 2019 (inclusive; the cursors indicate the TCSy chiller swap).  The CO2y laser PZT (PZTy from here on, easier to type) does move more than the CO2x laser PZT (PZTx), but the difference in PZTy movement pre- and post-chiller swap is less pronounced but noticable.  This also holds when looking at the longer 3 month trend (interesting note: at times, PZTx moves more than PZTy, but it's rare).  I then looked at the PZT movement from Jan 1 2019 to now (final attachment, chiller swap again indicated by the cursors).  PZTy in general is more active that PZTx and has been for some time.  There are also stretches after the chiller swap where PZTy is moving more than its new norm, but this does not appear to be getting worse as time goes on.

Went out and checked on this and found there was still a small air bubble in the chiller reservoir underneath the mesh filter (as opposed to the wire filter that preceeds the mesh filter).  I unsealed the mesh filter (the mesh filters create a seal around the reservoir opening when properly installed), which allowed the air bubble to clear, and reseated it.  The water level dropped from 10.1 to 9.2 (this is normal when clearing an reservoir air bubble, as it causes faulty readings in the chiller level.  This is a design "feature" of the chiller, and the reason why we unseal the mesh filter when topping off the reservoir (adding water with the filter still sealed is the most common cause of reservoir air bubbles)); I did not add water as even with this level change the chiller is still mostly full.  I will monitor this over the coming days to see if there are any changes in water level and/or formation of a new air bubble.

Also, the fix for the water leak Corey mentions in his alog is described here.

One of the reasons we have been keeping the the filters not completely seated is because they can create false readings, perhaps from too good of a seal. I've reccomended that the filters be cocked at a slight angle to keep this from happening, with the risk that it might not filter 100% of the water.