Oli, Rahul

Given below are the latest settings for the run up violin modes, which looks to be working for this lock,

ETMY mode 01: FM1 + FM6 + FM10 gain -0.1

ETMY mode 20: FM1 + FM4 + FM10 Gain 1.0  (this mode was really rung up, so one should keep an eye on this one for the next few lock)

ITMY05/06 looks to be doing fine with the settings Oli/Tony found this morning.

ITMY 08: FM1 + FM2 + FM10 gain -0.1

None of the above settings are in the lscparams, we should try this tomorrow and check if they still work.

TITLE: 01/14 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
While holding in MAX_POWER ITMY Violin Mode 5 was going up while the gain of +0.02 was being applied.
Using -0.02 was bringing it down but slowly.
H1:SUS-ETMY_L2_DAMP_MODE1_GAIN was set to zero because the ETMY Mode1 was also rising.

LOG:                                                                                                                                                    

TITLE: 01/14 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 22mph Gusts, 16mph 5min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.32 μm/s
QUICK SUMMARY:

Detector relocking and sitting in MAX_POWER to hopefully bring the violins down. Violins are really high, wind is going up to ~25mph. Temps at EX and in the LVEA are still fluctuating.

STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 19mph Gusts, 16mph 5min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.26 μm/s
QUICK SUMMARY:

After the Lockloss that tripped the I held us in down until Jim and I could figure out what is going on and how to stop it.
We ended up calling Danielle. Who helped us out by putting in a filter.
There was an ISI ITMX trip that occured while we were discussing the filter change. Jim asked me to go power cycle the Top ITMX ISI Interface chassis D1002432.

I got back to the control room and got a green light to start relocking. I started an initial alignment and then started to relock again.
we are currently in  Check Violins and moving up.

When the outside temps went below 0F last night, the corner weather station (Beckhoff readout) jumped to ~6550. When the temps came back up above 0F, the reading was correct again.

The outbuilding weather stations are still be readout by the original code and do not have this problem.

Sat Jan 13 10:02:47 2024 INFO: Fill completed in 2min 45secs

Due to higher starting temps I increased the trip temps to -60C for today's fill. TCmins = -73C,-67C.

Lockloss from OMC_Whitening due to ETMX Watchdog trips
HEPI, ISI, and SUS has tripped again at 18:15 UTC. This is now a trend that has happened 3 times in the last 24 hours. It happened twice on Oli's shift.
The ground motion out at EX is very slightly elevated due to a slight increase in wind since this morning. But the wind doesn't seem anywhere near high enough to cause all 3 of the WD to trip like this.
The EX Wind Fence is still standing as well.

I've attached the Lockloss plots and the Log to this Alog.

Link to Ryan's post where he has found some more evidence of something weird going on.

I trended the ISI sensor that saw the trip from last night (08:11UTC). On stage 1 the T240s tripped first, for the stage 2 trip theres an interesting oscillation in the GS13_V1 sensor about 5 seconds before the trip where all the sensors go crazy.

TITLE: 01/13 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 13mph Gusts, 9mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.26 μm/s
QUICK SUMMARY:

H0:FMC-MY_AH_REHEAT_DEGF was alarming when I arrived. H0:FMC-MX_VEA_202A_DEGF is 59.40 Deg F.

Looks like EX has tripped it's Hepi, ISI, and Its SUS at a 14:23 UTC which is surprising since I dont see any ground motion on the ground motion FOM.
Im going to start looking in to untripping these.



 

Following the WD trips of ITMX ISI and ETMX SUS, ISI, and HEPI on the previous shift, I was able to recover by using the oplevs to bring the position of ITMX and ETMX back to after the last initial alignment and started locking. I paused at CHECK_VIOLINS_BEFORE_POWERUP to damp violins with 2W settings as they were still rung up. After an hour of damping, I continued lock acquisition and made it to OMC_WHITENING without issue. H1 has been here for just over 1.5 hours damping violins at full power and progress is slow. H1_MANAGER is active so H1 should move to observing once violins have been damped down far enough.

TITLE: 01/13 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Lockloss early on in my shift due to an ETMX WD trip (75358), and after running an initial alignment and getting past all of the lower state locklosses, we've been sitting in OMC_WHITENING for three hours waiting for the DCPD values to lower.

The violins are very high and ITMY5 and ETMY1 started rising, so I changed ITMY5 to be -0.02 (nominal = +0.01), which worked to lower IY5/6, and turned off the gain for ETMY1, which sort of worked only in that it made ETMY1 level off, but isn't damping.

Right as I was about to post my alog, we lost lock due to watchdogs tripping for ITMX ISI, and for ETMX ISI, SUS, and HEPI. Currently have Ryan S on Teamspeak to pass the ifo over to him/troubleshoot.
LOG:

00:00UTC Detector Relocking and in LASER_NOISE_SUPPRESSION
00:09 Reached OMC_WHITENING
00:31 NOMINAL_LOW_NOISE    - went into NLN during a low point in the DCPD's oscillation, so moving into NLN started causing saturations soon after
00:32 Started getting verbal warnings for PI 31, DCPD, and PI 24, and PI 28, so Jenne changed the OMC_LOCK state to REMOVE_WHITENING while we wait for the DCPDs to go down

01:20 Jenne fixed an error in the OMC_WHITENING waiting state that caused us to prematurely turn on whitening (75359)
    - Jenne manualed us back to OMC_WHITENING
    - Reloaded ISC_LOCK

01:31 Lockloss due to ETMX HEPI, SUS, and ISI WD trip (75358)

01:51 Started relocking by taking us to GREEN_ARMS_MANUAL

01:58 FMC-MY_AH_REHEAT_DEGF red alarm - FMC-MY_AH_REHEAT_DEG temp exceeded 119F
    - Looks like MY AH had gone up to 128F 9hours ago, and Austin had already tagged FMCs then (75325)

02:33 Lockloss from FIND_IR

02:44 Lockloss from CHECK_MICH_FRINGES

02:48 I started an INITIAL_ALIGNMENT

03:13 INITIAL_ALIGNMENT completed, starting relocking

03:36 Lockloss from DRMI_LOCKED_CHECK_ASC
    - We had been cycling through ACQUIRE_PRMI -> PRMI_LOCKED -> PRMI_ASC -> LOCKLOSS_PRMI -> back around to ACQUIRE_PRMI multiple times before getting through and locking DRMI, before losing lock a couple of seconds after getting into DRMI_LOCKED_CHECK_ASC
    
04:01 Paused at CHECK_VIOLINS_BEFORE_POWERUP to try damping the violins before increasing power

05:07 Reached OMC_WHITENING, waiting for violins/DCPD counts to lower

08:11 Lockloss due to WD trip for ITMX ISI, and for ETMX ISI, SUS, and HEPI

Earlier today Camilla took In-Lock SUS Charge Measurements (75347) since we haven't gotten data points for a bit. Attached are the plots for ITMX, ITMY, ETMX, and ETMY.

Closes FAMIS#26275, last checked 75256

Corner Station Fans (attachment1)
All fans are looking normal and within range.

Outbuilding Fans (attachment2)
All fans are looking normal and within range.

Still trying to relock and currently we are sitting in CHECK_VIOLINS_BEFORE_POWERUP and damping the violins at 2W. We had to do an Initial Alignment but still went through the ACQUIRE_PRMI cycle a few times and then had a lockloss from DRMI_LOCKED_CHECK_ASC before getting back to where we are now.

At 01:58UTC we had an alarm go off for FMC-MY_AH_REHEAT_DEGF due to the MY_AH_REHEAT temp exceeded 119F - Tagging FMCs
    - Looks like MY AH had gone up to 128F 9hours ago, and Austin had already tagged FMCs then (75325)

As they've been skipped the last few weeks 75271, today while ISC_LOCK was in OMC_WHITENING waiting for violins, we took in-lock charge measurements. 21:19UTC to 21:35UTC.

There was a glitch at 21:33:10UTC at the end of the ETMX injections/ transition, should check if that's due to a mistake in the code.