The FMCS EPICS IOC froze up late last night about two hours after I restarted it. Timeline is:

1. Sat 15:57 IOC froze
2. Sat 20:50 I restarted IOC
3. Sat 23:15 IOC froze again (2h25m run)
4. Sun 10:05 I restarted IOC

IOC console as found is attached.

TITLE: 01/14 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY:
When I walked in H1 was in OBSERVING for 4 hours, but we had reasonably large 500hz Violin spike on darm, and the DCPD's were diverging.
I quickly checked the Violins, and noticed that the ITMY Mode 5 seting was in it's Nominal Gain state of +0.02 and was rising again.
Changed the Gain back to -0.02 and it has already turned around and started to fall.

Im not sure what is going on with  temperature senors looks like there was a rest or something but the temps all seem to be "Frozen" or not changing like they normally do.
I dont see any CDS errors about this, and it doesn't seem to be mentioned in the alogs. So tagging FMCS and documenting the temps are they currently stand.

CS: Everything looks good in the LVEA all green. 
H0:FMC-CS_LVEA_ZONE1B_DEGF= 67.4
H0:FMC-CS_LVEA_ZONE2A_DEGF= 66.79
H0:FMC-CS_LVEA_ZONE3A_DEGF = 67.3
H0:FMC-CS_LVEA_ZONE4_DEGF =67.7
H0:FMC-CS_LVEA_ZONE5_DEGF =67.3

EX Some temp channels are RED:
H0:FMC-EX_VEA_202A_DEGF = 57.40 RED
H0:FMC-EX_VEA_202B_DEGF =68.5 green
H0:FMC-EX_VEA_202C_DEGF =68.3 green
H0:FMC-EX_VEA_202D_DEGF = 58.70 RED
H0:FMC-EX_VEA_AVTEMP_DEGF = 64.73 green

EY:
H0:FMC-EY_VEA_202A_DEGF = 61.7
H0:FMC-EY_VEA_202B_DEGF = 67.40
H0:FMC-EY_VEA_202C_DEGF = 67.6
H0:FMC-EY_VEA_202D_DEGF = 59.9 RED
H0:FMC-EY_VEA_AVTEMP_DEGF = 64.97

H1 called for assistance at 11:28 UTC since the filter cavity was having trouble locking again. Much like what Oli needed to do four hours ago (alog75384), I paused the SQZ_FC Guardian, locked the FC green VCO manually using the toggle switch on the common mode board, and optimized SQZ-FC_TRANS_C_LF_OUTPUT using FC2 P/Y sliders (and successfully bringing the signal to around 83 counts). After resuming the Guardian and re-requesting SQZ_MANAGER to FREQ_DEP_SQZ, the FC locked without issue, I accepted the new slider values in SDF (screenshot attached), and H1 was able to start observing at 12:03 UTC.

Contrary to the issue earlier tonight, I did not see the CLF DEMOD phase go to zero (it's currently hanging around 171), so I assume the ADF angle servo is working properly this time.

TITLE: 01/14 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Struggling with violins and relocking all day, but we were able to reach NOMINAL_LOW_NOISE twice, but lost lock the first time before we could go into Observing (75381). Currently in NLN and Vicky and Daniel are working on getting the SQZ angle to behave because it keeps trying to move towards 0.

Violin modes to keep an eye on: ITMY05/06, ITMY08, ETMY01, ETMY20 (Rahul's alog with some values that work: 75378)

LOG:

00:00UTC Detector Relocking and in MAX_POWER trying to damp violins

03:24 Continuing up towards NLN

03:36 Reached OMC_WHITENING
04:48 Went into NOMINAL_LOW_NOISE
05:10 Lockloss from NOMINAL_LOW_NOISE (Not Observing) (75381)
05:30 Lockloss from LOCKING_ALS
05:41 Lockloss from CHECK_MICH_FRINGES, starting an Initial Alignment
06:10 Initial Alignment completed, relocking

06:56 Reached MAX_POWER, pausing to damp violins
07:42 Reached NOMINAL_LOW_NOISE
    - Had to realign FC2 (72084)

08:11 OBSERVING!
08:15 Dropped out of Observing to troubleshoot SQZ Angle issue

Lockloss @ 01/14 05:10UTC while in NOMINAL_LOW_NOISE (not Observing).

We weren't able to get into Observing due to the SQZ FC being unable to lock, and so I was going through Vicky's procedure (72084), and we lost lock a few seconds after I adjusted FC2 in pitch, although that definitely would not have caused a lockloss. We were so close :'(

Patrick, Oli, Dave:

The FMCS EPICS IOC stopped updating at 15:57 PST this afternoon. The screen session output is shown below. We killed the process and started a new IOC, which has fixed the issue. I restored the alarm settings using the restore button on the top level FMCS MEDM.

Oli, Rahul

Given below are the latest settings for the run up violin modes, which looks to be working for this lock,

ETMY mode 01: FM1 + FM6 + FM10 gain -0.1

ETMY mode 20: FM1 + FM4 + FM10 Gain 1.0  (this mode was really rung up, so one should keep an eye on this one for the next few lock)

ITMY05/06 looks to be doing fine with the settings Oli/Tony found this morning.

ITMY 08: FM1 + FM2 + FM10 gain -0.1

None of the above settings are in the lscparams, we should try this tomorrow and check if they still work.

TITLE: 01/14 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
While holding in MAX_POWER ITMY Violin Mode 5 was going up while the gain of +0.02 was being applied.
Using -0.02 was bringing it down but slowly.
H1:SUS-ETMY_L2_DAMP_MODE1_GAIN was set to zero because the ETMY Mode1 was also rising.

LOG:                                                                                                                                                    

TITLE: 01/14 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 22mph Gusts, 16mph 5min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.32 μm/s
QUICK SUMMARY:

Detector relocking and sitting in MAX_POWER to hopefully bring the violins down. Violins are really high, wind is going up to ~25mph. Temps at EX and in the LVEA are still fluctuating.

STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 19mph Gusts, 16mph 5min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.26 μm/s
QUICK SUMMARY:

After the Lockloss that tripped the I held us in down until Jim and I could figure out what is going on and how to stop it.
We ended up calling Danielle. Who helped us out by putting in a filter.
There was an ISI ITMX trip that occured while we were discussing the filter change. Jim asked me to go power cycle the Top ITMX ISI Interface chassis D1002432.

I got back to the control room and got a green light to start relocking. I started an initial alignment and then started to relock again.
we are currently in  Check Violins and moving up.

When the outside temps went below 0F last night, the corner weather station (Beckhoff readout) jumped to ~6550. When the temps came back up above 0F, the reading was correct again.

The outbuilding weather stations are still be readout by the original code and do not have this problem.

Sat Jan 13 10:02:47 2024 INFO: Fill completed in 2min 45secs

Due to higher starting temps I increased the trip temps to -60C for today's fill. TCmins = -73C,-67C.

Lockloss from OMC_Whitening due to ETMX Watchdog trips
HEPI, ISI, and SUS has tripped again at 18:15 UTC. This is now a trend that has happened 3 times in the last 24 hours. It happened twice on Oli's shift.
The ground motion out at EX is very slightly elevated due to a slight increase in wind since this morning. But the wind doesn't seem anywhere near high enough to cause all 3 of the WD to trip like this.
The EX Wind Fence is still standing as well.

I've attached the Lockloss plots and the Log to this Alog.

Link to Ryan's post where he has found some more evidence of something weird going on.