We are investigating why the ETMX HWWD saw high SUS RMS for 20 minutes which caused it to trip, whereas all the other HWWDs saw less than 1 minute of high RMS. It would appear that ETMX's high RMS stopped within one minute of the time the HWWD powered down the ETMX ISI coil drivers.

Just like gauge at X2-8, the gauge at Y2-8 is not reporting a pressure as of 1 hour ago, due to the lack of sunlight needed to charge batteries via solar panels. No action is needed at this time.

TITLE: 12/02 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Owl Canceled.
CURRENT ENVIRONMENT:
    SEI_ENV state: LARGE_EQ
    Wind: 15mph Gusts, 9mph 5min avg
    Primary useism: 5.06 μm/s
    Secondary useism: 0.66 μm/s
QUICK SUMMARY:

When I Walked in H1 was being Saturated by a 7.6 mag Earthquakes whos peak is above the top of our earthquake FOM scale, and of course it's gang of aftershocks of mag 6.4 -5 ish.
I tried resetting the watchdogs for ISIs and HEPIs , but the HEPI tripped again.
ISC_LOCK was in Initial_Aligment, took that to IDLE, which stopped verbals from letting us know that the "IFO OUT" Saturation limits are being met.

Current DIAG_MAIN messages:
ESD_DRIVER: ESD X driver OFF
IOP_DACKILL_MON: iopsusex (87) IOP DACKILL Tripped
IOP_DACKILL_MON: iopsusex (90) IOP DACKILL Tripped
SEI_IOP_WD: SEI IOP ['ETMX'] coilmon drop out
SEI_STATE: ['ETMX'] is not nominal

I will begin resolving these issues.
 

It's not pretty, but H1 is back in OBSERVING; range started at 143 (but it's been nosediving the last 15min to 100Mpc).  Surprised to see violin modes looking fairly normal (even after being out of lock for a while and with the high useism). 

SDF Diffs (see attach#1):  Accepted difs for SUS: ITMx, ITMy, & BS as well as SEI:  ETMx Sensor Correction

Also needed to restart the nuc30 ALIGO DARM dtt session because it Timed Out.

Microseism is still pretty high solidly on the 95th percentile mark.

Wireless Access Point in the LVEA was ON, so I turned it OFF.  I left the WAP in the MSR ON.SQZ ASC AS42 not on?? Please RESET_SQZ_ASC

DARM (see attach#2) is elevated from 10-70Hz (the reason for the low range) and also broadly at high frequency....since this looked Squeezer-ish, I checked the squeezer to see that the SQZ MANAGER had a notification about "SQZ ASC AS42 not on?? Please RESET_SQZ_ASC".  When I saw this, I was just about to post something in CHAT, but saw that Naoki was already on it and posted a CHAT asking if I could take H1 out of OBSERVING for him to reset SQZ_ASC, so I did and a few minutes later I took H1 back to OBSERVING.  Now our range should look better...it's already back above 140Mpc!  :)

Taking GRD-IFO to AUTO & H1-Manager to LOW NOISE.

Still see DARM higher from about 10-55Hz.

TITLE: 12/02 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Microseism
INCOMING OPERATOR: N/A
SHIFT SUMMARY:

See mid-shift summary for beginning of the shift. 

For the 2nd half of the shift ended up moving away from alignments and locking and with Sheila focusing on the PRMI lock---namely why POP18 & 90 were so low in power.  Toward the end of the shift Sheila noticed how POP18 had been slowly drifting down in power for PRMI_ASC.  So she started walking PR2 + PR3 (along with a little PRM) in yaw.  And she was able to increase the PRMI powers.  

Remember this is with several Optics with new Offsets!  

I continued walking PR2 + PR3 (along with a little PRM) in pitch (& also revisiting yaw), but I don't know if I improved the powers much more than what Sheila did....maybe a little.  After going in circles walking optics for PRMI w/ NO ASC (POP18 was touching 80 counts), went to PRMI_ASC---this took POP18 up to 90+!  (at the beginning of the shift it was down around 50).

Let ISC LOCK continue, and DRMI locked all on its own.

I will take ISC_LOCK to NOMINAL LOW NOISE and see how it goes.  I imagine there could be some possible violin mode issues, but we'll see.

I should say, more walking of PRMs could possibly improve PRMI's POP18, but basically we are back to PRM powers we had back around Nov17 (NOTE:  on Nov10, POP18 was up to almost 100).

H1 currently at MOVE_SPOTS.

LOG:

• 0318 Turning off amps in lvea (robert)

Corey, TJ, Sheila

Sumary: We've had some clipping in the PRC, which has been getting worse over the last couple of weeks.  This could easily be the cause of our bad noise, and locking difficulties.  We had some sucsess unclipping it by moving yaw, but we aren't yet back to the POP18 build ups we had in mid November. Sensors for looking at PR3 drifts are confusing.

Corey was struggling with PRMI locking this evening.  When I logged in he and others had been working through initial alignment steps, and Corey was able to lock PRMI but with low buildups.  Corey and I spent some time looking at some history of POP18 build ups when PRMI is locked over the last several weeks:

• 11/10 22:29utc 98cnts
• 11/17 10:57utc 91cnts
• 11/25 19:04 UTC 80 cnts (DMRI 96)
• 11/28 1:41 UTC 87 cnts (DRMI 100)
• 11/28 7:32 UTC 85 cnts (DRMI 96)
• 11/28 22:24 UTC 80 cnts
• 11/29 2:18 UTC (a bunch of PRMI locks) 73 counts (DRMI 82)
• 11/29 5:22 UTC 70 cnts (DRMI 80 cnts)
• 11/29 21:16 UTC 62 cnts
• 11/30 1:56 UTC 58 cnts (DRMI 82 cnts)
• 11/30 14:39 UTC PRMI 56 cnts
• 12/2 4:37 UTC PRMI 52 counts

As you can see, the POP 18 build up has been slowly decreasing, even in times when we were locking. We then ran PRMI ASC (REFL 9 to PRM, AS 45 to BS) while we moved PR2, and saw that this did have an impact on the build ups, which it shouldn't if we aren't clipping. (see first attachment for pitch move of PR2).  This pitch move made the build ups more stable, but we couldn't improve the build up with PR2 pitch alone.  We saw similar behavoir with yaw, and moved PR2 yaw in the negative direction.  The second attached screenshot shows a walk of moving both PR3 and PR2 in the negative yaw direction, which did increase the POP 18 build up to about 80 counts.  This was done with ASC off. 

At this time we had trouble keeping PRMI locked for a little bit, it looked like the "mitosis" glitches that Ibrahim saw Wed which we believe were caused by the BS oplev (BS optical lever damping runs while PRMI ASC runs, we tried to turn it off but that was unstable).  We tried a few more rounds of yaw walking with the ASC off, but we saw POP18 drift up to 86.  I think the next step would be to try to walk PR3 and PR2 in pitch to get back to the POP18 build ups close to 100 that we had a few weeks ago, Corey is trying that now.  We probably would be able to lock the IFO with the current value of 86, if all else fails (but if we are still clipping in the PRC that could be causing bad noise).

What's the drift?

The third attachment here is a trend of several sensors of PR3 alignment over the last few weeks, they do not agree with each other.  The optical lever would indicate that the drift of PR3 has drifted in the positive yaw direction, while the top mass osems would indicate that it has drifted in the negative direction.  When I moved PR3, both sensors agree that the sign of the move was negative yaw, so there is not a sign error in one of them.  This reminds me a lot of 70008, and the incident that inspired that alog where the ISI was drifting so the suspension osems didn't show the drift.  TJ points out this alog about the PR3 sensor calibrations: 70197

After the Suspension Offset changes from the end of the day shift, ran a Manual Initial Alignment (except for Input Align).  H1 made it to Acquire DRMI, but flashes did not look great.  Went with LSC setting chage RyanS used a few nights ago (alog74487), and this allowed for short locks.  Tried to touch it up with the SRM, but did not have luck keeping it locked enough to let ASC take over.

Revisited Input_Align, since we skipped this earlier.  And it completed, however...

Returned to locking, but the alignment looked horrible for PRMI.  

Returning to Manual Alignment and skipping Input Align.

(Jordan V., Gerardo M.)
Late entry.
Last Tuesday Jordan and I installed 4 o-ring valves on the following crosses, FC-C2, FC-C3, FC-C4 and FC-C5.
All o-ring valves were installed on the +X 2.75" CF port for all crosses, all valves were leak tested and all new joints passed.

TITLE: 12/02 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Microseism
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.47 μm/s
QUICK SUMMARY:

Austin just exited the LVEA after finishing BS oplev investigations.

Walked in to manual alignment by Tony & TJ, but they are currently troubleshooting the SR optics.

TITLE: 12/02 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Microseism
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
LOG:

MicroSeism is still pretty high early in the morning,
Ran an Initial Alignment which completed successfully.

Paused ISC_LOCK on Acquire_DRMI_1F
Because DRMI was in a PRMI - DRMI loop

Changed the following channels to try and replicate Ryans technique.
H1:LSC-PRCL2_GAIN ->1.2 -1.5
H1:LSC-SRCL2_GAIN ->1.4 -1.5 just trying different settings.

H1:LSC-MICH_TRIG_THRESH_ON -> 25
H1:LSC-SRCL_TRIG_THRESH_ON -> 25
Move SRM ,SR2, and the Beam splitter.
 

Eventually ISC_LOCK being paused caused an issue with ALS due to X arm getting unlocked while TJ and Austin changed the power on OPLEV out by HAM4.

I then took ISC_LOCK back to execute. This caused an Immdiate lockloss.
While relocking we made changes to ISC_LOCK.py to increase the time that we wait before going to Acquire_PRMI.
Lines 1480 and 1481 were changes to:  
self.timer['try_PRMI'] = 3600 # Was 600, reduce to 300 24Oct2019 JCD
self.timer['DRMI_POPAIR_check'] = 3600
This is a temporary  change to try to lock while the MicroSeism is Elevated.

Pushed around SRM, SR2 and the BS trying to get DRMI locked.
Eventualy I took ISC_LOCK back to initial alignment.
That completed and we got back to ACQUIRE_DRMI_1F and I increased the gains again but it still will not lock.

Checked the Whitening filters for changes over the last week that would affect Acquire_DRMI_1F.
I have Changed the gains
H1:LSC-PRCL2_GAIN ->1.2 -1.8
H1:LSC-SRCL2_GAIN ->1.4 -1.8 just trying different settings.

None of this has worked out well.

Trended all the Quads and noticed that they have all changed in their vertical axis with the outside air temp. Please See comment (74533)
ITMY Vertical Motion
image(2).png
image(1).png
image.png

Changed H1:SUS-ITMY_M0_TEST_V_OFFSET to 90,000
and H1:SUS-ITMX_M0_TEST_V_OFFSET to 80,000

Ran dither align script.
And starting Initial Alignment now.
While Initial Alignment, was running SRC_ALIGN ISI HAM4 was tripped likley due to the BS OPLEV re-adjustments
We are having problems aligning SRM.

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                

In an attempt to try and troubleshoot the BS oplev glitching, it was noted by Jason that a good way to help with this problem solving was to try and maximize the oplev laser power (if it wasn't maxed already). I went into the LVEA to try and adjust/maximize the laser power on the oplev box to see if the sum counts would go up, but they did not. After this adjustment, Jason noticed that the oplev sum channel on detchar was very noisy at this point - ss attached. He recommended lowering the power a bit, so I went back in to do so, but noticed upon doing this that the sum counts started dropping very rapidly. Not sure what's going on, but Jason and I are both thinking that we might need to swap the oplev laser so we can open up the current one and look into it. Ultimately, we decided to readjust the power output back to the second adjustment where the noise was prevalent on detchar (as the sum counts were consistent at this power) to get us through the weekend.

I think for my previous SUS rubbing check alog74493 I used a bad time while the SUSs were still ringing down from a lockloss so I ran it a few more times with check different times and the same reference.

The BS OPLEV is glitching mostly in PIT (a few in YAW but nowhere near as many) but this doesn't seem to be new behavior, Its been going on for at least the past year from the ndscope.

ASC loops at three times are compared are below in DTT, all 0-232Hz, 0.05Hz BW with 50 averages and 45% overlap for ~ 8 minutes of data used. Good DARM: 2023/11/23 6:45UTC; Fuzzy Darm: 2023/11/23 7:15UTC; Bad Low Freq DARM: 2023/11/30 7:45UTC (useism high) range 140MPc.

• DHARD P and DHARD Y can see factor of 2 increase in P and Y between 5-8Hz on 11/30, on DTT (doesn't look as bad on ipynb?!) Some peaks in DHARD Y.
• CHARD Y has some peaks around 7.7, 10.8, Hz
• CSOFT, DSOFT are similar.

Reused Jenne's 64760 rubbing script for these loops with 10minutes of data. PDFs attached. Comparing to 2023/11/23 6:45UTC good DARM time:

• Recent 2023/11/30 7:45UTC you can see the differences listed above and other low-freq differences
• Last weeks 2023/11/23 7:15UTC DARM fuzz, there is no differences

I ran Jenne's rubbing script using a reference time of us being in DOWN on the 22nd before the fuzziness started on the 23rd around 07:00UTC and a check time of 10:00UTC on the 23rd while also in DOWN, I didn't really see anything noteworthy, some noise increase in the 0.1Hz area (microseism)?

I also ran using the same reference time against a time this morning and there was a couple of suspensions that looked interesting, there were some spikes in the 1.0 - 10 Hz bandwidth for PR2, PR3, PRM, SR3, SR2, MC3, MC1, and OM3. In the 0.0 - 1.0 bandwidth the Quads saw an increase in noise, specifically the ITMs, TMSs, and ETMY. SRM and SR2 have some large spikes just under 1 Hz. The ITMs temperature has slightly increased, ~ 0.25 - 0.3 degrees F but these changes are less than 0.5 degrees so its probably not a worry.

Later in the day I ran the script a 3rd time using a time from yesterday (13:54UTC 11/29) with the same reference from the 22nd and most of the features I saw are no longer there, except for ITMX. Maybe the time I chose earlier was just a bad time?

I looked into the two locklosses from November 29th, 01:39UTC and 04:25UTC from during the time where the AS72 Gain was increased from 12dB to 21dB (74457), before it was reverted due to potentially having caused the next two locklosses.

It doesn't seem to me that both locklosses had the same cause, since there seems to be some discrepencies between the two locks as well as between the actual locklosses. Based on looking through the lockloss ndscopes, it looks like the first lockloss was probably caused by the AS72 gain, but I don't think the second one was, or at least wasn't the only cause. Here's a quick breakdown of some differences:

ASC Loops (ASC-01:39UTC, ASC-04:25UTC)
01:39 Lockloss - Signals look normal up to lockloss
04:25 Lockloss - Many signals are much larger than during the previous lock, and some are increasing in the last ~10s in the INMON channel.

LSC Loops (LSC-01:39UTC, LSC-04:25UTC)
01:39 Lockloss - LSC loops look much different than usual
04:25 Lockloss - Loops look pretty normal

SUS MASTER_OUTs (SUS-bothtimes, SUS-01:39UTC, SUS-04:25UTC)
01:39 Lockloss - MASTER_OUT channel values are more or less stable once in NOMINAL_LOW_NOISE
04:25 Lockloss - During last ~15minutes of lock, many MASTER_OUT values double and stay that large up to the lockloss