Tagging @DetChar and SEI for data in seismic's sensor correction configuration transition studies.

We lost lock engaging the soft loops a few times this evening.

It looked like the corner ASC could not keep up when we switched on the arm dither loops, symptom: power recycling gain goes up, rf18 buildup drops, we lose lock.

I removed 10dB of gain from the arm dither loops (pit4, pit5, yaw4, yaw5) when they first come on (set in PREP_ASC_FOR_FULL_IFO), and added the 10dB back in later in ENGAGE SOFT LOOPS. This worked by hand, so I put it in the guardian.

Dave was thinking we could restart this ISCEY computer without affecting the SEI or the SUS platforms and we all agreed until we realized we shouldn't have.  There was no impact on the models of the SEI or the SUS but, ALS is bleeding the Tidal Drive from the ETM to the HEPI.  When the ALS came back on with zero Tidal Drive, the HEPI was jerked back to zero offset.  The HEPI tripped on this 6 seconds before the ISI tripped when the T240 tilted off or got slung sideways by the HEPI.  Attached is 4 hours showing the tidal drive on HEPI coming from the ALS (lower left) going flat and then jerking this way and that on its way to zero when the other channels, the ISI's T240s, start railing.

I find interesting that the T240 signals all go flat when the ALS does with the Kernel panic but maybe that would be expected too.

We'll endeavor to implement some procedure to hold this output on the HEPI and then bleed it off at an acceptable rate.

I have recovered h1iscey by remotely resetting it via its IPMI port. Procedure was:

• take h1iscey out of the dolphin fabric
• issue cpu reset via the IPMI management system

Once the mx_stream was restarted on h1iscey, the DAQ errrors on the other system cleared. 

Handing over to Jeff B for IFO recovery.

Details, taking h1iscey out of Dolphin fabric and verification:

david.barker@zotws6: ./dolphin_switch_port_status.sh h1iscey

h1iscey Switch_IP 10.101.0.94 Switch_port 3 is ENABLED

david.barker@zotws6: dolphin_disable_switch_port h1iscey

Disabling h1iscey Switch_IP 10.101.0.94 Switch_port 3

david.barker@zotws6: ./dolphin_switch_port_status.sh h1iscey

h1iscey Switch_IP 10.101.0.94 Switch_port 3 is DISABLED

Details, remote IPMI chassis reset

david.barker@zotws6: ipmitool -I lan -H 10.99.101.193  -U (user) -P (passwd) chassis status

FRS13078 opened to cover this issue.

Transitioned back to WINDY around 21:00.  Forgot to log the exact time since Hugh just fixed the SDF issue so we no longer drop out of Observe and I don't have a Verbal timestamp.

The SDF issue dropping us out of observing was the ETMX ISI Stage1 X Y Z CPS T240 & L4C BLND NXT filters.  ETMY/ITMY & ITMX all had these filter banks Not Monitored.  These ETMX channels were switched to Not Monitored.

For longer term study, I've opened FRS Ticket 13123.

Travis's 4-word aLOG claimed the 50000th aLOG and thus we honored this amazing achievement with the attached LHO Paper Plate Award!

"This award goes to Travis Sadecki for his 5 words in the 50,000th aLOG (LHO aLOG 50000)."

A truly astounding achievement.
Who will claim the 100000th aLOG???

This spectra is taken some number of hours before the 'run' time, eight or something hours I'd guess.  This puts this auto spectra right in the middle of a nearly four hour out of full IFO period where in the SEI BS will be in ISOLATED_DAMPED mode for potentially much of the time.

We lost the units around 13:20 UTC (06:20 PDT)

Connection between corner chassis 4 L0 and M0 has failed. The first red block in the topology screenshot is corner chassis 4 M0 (EK1100).

Patrick and Corey are working the problem.

The time for the error Dave mentions above coincides  with the H1 Lockloss (and issues with IMC) at 13:21utc (6:21amPDT).

Patrick has tracked this down to the Corner Chasis 4 in the CER and says this may need to get pulled.  Fil has been notified and will work with Patrick.

For something like this which might have taken H1 down, should this be a Verbal Alarm?  (only way an operator would know about this issue is to happen to catch the red box on the CDS Overview (which we both didn't notice).

[Attached is the RED box on the CDS Overview.]

FRS Ticket submitted:  #13049.  

https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13049

So far down time is 3.5hrs, but now I need to try locking.

Side Note:  Alog would not let me link the FRS ticket to the text of "13049" above.  

EK1100 EtherCAT Coupler failed and replaced. Unit was scanned and communication errors have cleared.

We did noticed a buzzing sound coming from one of the terminals. Tried various combinations of powering off different components to isolate noisy terminal(s), but decided to reinstall unit to get IFO back up.

Corner 4 Slow Controls Chassis S1107450

F. Clara, P. Thomas

Testing alog web link for FRS13049

On the CDS Overview MEDM, if there is a Beckhoff hardware error the slow controls banner will turn RED for better visibility. I simulated a C1PLC1 error to demonstrate (see attachment).

All in all this cost us about 5 hours. See e.g. https://ldas-jobs.ligo.caltech.edu/~detchar/summary/day/20190612/ .

This FRS ticket (13049) is now CLOSED/RESOLVED.

Looks like this sensor came back on Friday night at around 5:30pm PDT.

This  has two CLOSED/RESOLVED FRS tickets for it:

• https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13066
• https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13073

FRS 13075

FRS 13075

During Tuesday Maintenance, the BS was taken to DAMPED and the CPS Interface chassis was power cycled in the CER.  This power cycled the satellite racks as well.  I did not un- & re-seat the gauge board cards in the satellite rack which is one thing we do to mitigate the elevated noise on a CPS.

Attached is a comparison of the BS Stage2 CPS between the spectrum Corey alogged here from June 3 and one taken this morning at 2am. This noise wasn't really too elevated but it sure looks much quieter now or this morning at least.  Will update/close the FRS.

FRS 12867 is now listed as CLOSED/RESOLVED.