X end got better, now it's similar to Y. See the first plot (top) showing the past 18 hours or so, zero-power period corresponds to the maintenance. You need to look at peak to peak, ignoring the spikes.

Don't worry about spikes as far as fiber is concerned, that's the result of failed locking attempts after the maintenance (IMC loses lock, FSS is kicked). See 2nd attachment.

J. Bartlett, T. Shaffer, J. Oberling

The new fitting has been installed and the leak is tentatively fixed.  One of us will check on it near the end of the maintenance window to make sure there are still no drips.  We will also monitor the new fitting over the next couple days before declaring final victory, but so far we are leak-free.

With the new fitting in place, the leak seems to be fixed. We will monitor during today and the next few days to make sure before closing the FRS out.

I've attached some pictures of the table after I found the leak.

Jeff Bartlett is on the case this morning, hunting down the last remaining black plastic t-fitting which looks to be a replacement for what is currently leaking on the TCSX table.  

here is the CDS overview before we started the recovery process.

Rolf asked for some information on the cpu cycle count through the glitch.  I've two plots of the cpu meter from h1iopsusb123, h1iopsush2a, h1ioplsc0, h1iopasc0.  The first plot is in dataviewer and gives the mean round the glitch, the second is from ndscope is is raw data (cpu_meter + cpu_meter_max).

A summary from nds_query giving raw numbers:

Data for 5 seconds starting at GPS: 1222517040
Channel                                  type      nWords   units
H1:FEC-28_CPU_METER                      int_4         80 
Channel: H1:FEC-28_CPU_METER  type: int_4  nWords: 80
    0        5        5        5        5        5        5        5        5
    8        5        5        5        5        5        5        5        5
   16        5        5        5        5        5        5        5        5
   24        5        5        5        5        5        5        5        5
   32        5        5        5        5        5        5        5        5
   40        5        5        5        5        5        5        5        5
   48        5    20228    20228    20228    20228    20228    20228    20228
   56    20228    20228    20228    20228    20228    20228    20228    20228
   64    20228        6        6        6        6        6        6        6
   72        6        6        6        6        6        6        6        6

Jonathan, Dave:

Recovery process:

Took h1psl0 out of the Dolphin fabric before restarting the models. Disabled its IXS600 switch port and pulled the IX cable at the rear of the computer. This should insulate the PSL from any further corner station Dolphin issues.  We will re-attach when the system is more stable.

Replaced h1asc0's Dolphin cable (this machine reported a Dolphin Gen1 issue). Powered h1asc0 down, replaced 2m cable with new 1m cable temporarily ran directly from IX600 switch C0 port 8 to cpu.

Systematically restarted all  models on all computers with errors.

When h1iopseib1 was starting, the computer froze. SSH session was unresponsive, local console was dark. Disabled IXS600 switch port and power cycled computer.

There were DAC errors on h1seib2 and h1seih16, so these IOP models were also restarted.

By the end of the day, h1seib3 was the only Dolphin'ed corner machine which did not get anything restarted on it.

I've started a wiki page to overview these recent front end issues: https://cdswiki.ligo-wa.caltech.edu/wiki/FrontEndCrash

Note that there are expected IPC errors on h1psliss and h1lsc following the disconnection of h1psl0 from the Dolphin network.

h1psliss receives two IPC channels from h1lsc (TRX and TRY normalised photodiode signals). Used for ISS third loop, not needed in the short term.

h1lsc receives one IPC channel from h1psldbb (FF_JITTER). Used to add to OMC-DARM signal. Has not had a useful signal for some time.

Both X and Y end ALS WFS sensing matrices were changed on September 26th, as reported in 44178. 

The new values should be accepted.

ECAT settings don't come back up exactly the same after PLC restart.

See attached screenshots of ECAT CS PLC2 and PLC4 after Patrick finished his prep work for TwinCAT 3 this morning in which he had to restart all IFO-related PLCs. Compare these agains the above screenshots taken before the boot.

The TwinCAT system saves its values once minute. After a reboot it will restore to the last set point automatically—in most cases. If the code was changed significantly, it may choose to ignore the old settings and start fresh with all zeroes. The above screens indicate the former, where the settings were restored to the last settings, but probably the sdf was and is outdated. TwinCAT will never use an EPICS/SDF saverestore, this has to be done by a user.

Model attached.

Could the same missing 50% explain why the preloading was off?

Sorry Daniel. This 50% was due to a couple of ridiculous mistakes on my part. The first mistake was that I assumed that the initial step response was caused by a ring heater step of 1 watt when in reality it was caused by a step of 3.13 watts (please see attached image for updated plant zpk model with new gain).  The second, and most glaring mistake, was not thinking about the additive power from both of the upper and lower portions of the ring heaters so this left a factor of two missing in my model. I've attached time series of the measured data of the ring heater power compared to the time series from the updated model. I've attached a spherical power time series comparison as well. The final differential lens that the model suggests is 18.98 microdiopters.

In regards to pre-loading, I can update you on how these time series compare to the TCS simulation time series whose input parameters are used in the pre-loading estimates. 

The model of the pre-loading estimates use the calibration in diopters/watt on the TCS simulation page.

Spherical power (udiopters) = [ 2*(-9E-6) + 0.9E-6]  * P_RH = -17 udiopters for 1 Watt

The first coefficent in the equation is the substrate lens (the factor of 2 for double passing) and the second coefficient is the radius of curvature change.

Here is a comparison plot with the added TCS simulation time series. The simulation appears to be converging to a differential lens of 18 +/- 1 microdiopters.