for the three locks since last night, h1 gets to 140Mpc and then starts to nosedive in range over the first 15-20min of lock.  For the 3rd lock, this happened---basically a big noise hump in DARM would grow and grow (from 10-60Hz).  I waited for the SQZ ASC to happen (it did, but I also think the squeezer lost lock---that was different from last night's lock and the lock at the beginning of the shift).  So I was waiting for the SQZ to relock so I could resest the SQZ ASC, but then there was a lockloss.

Naoki suggested for the next lock, to preemptively Reset AS42 (per 71083).  So I did this as soon as the Squeezer was Squeezing and all Guardians were GREEN.  To get to Observing I had to ACCEPT a list of OFFSET SDF diffs (attached), and then went to OBSERVING.

But as I was typing this alog, sadly H1 lost lock after only 10min (or 4min of Observing).

H1 was locked about 2.5hrs and then had a lockloss (similar to last night's lock of ~3hrs).  Just recovered from the lockloss with H1 back to OBSERVING in under one hour.  Will keep an eye on SQZ to see if the SQZ ASC turns OFF.  Range is just above 140Mpc.

L1 is now back up and OBSERVING.

Winds have calmed and microseism still high (@95th percentile).

As mentioned earlier, Tony (w/ Keita & Sheila on TeamSpeak) had H1 well on its way to NLN (after the big Phillipines EQ + Hardware Watchdog Recovery + the other "ongoing" issues all happening at the same time!).

Now that H1 is at NLN, it's been decided to keep H1 in OBSERVING at this lower BNS range (due to 20-50Hz noise seen on DARM, tagging DetChar) to get some single-detector-OBSERVING in ....BUT Fri night, Robert was allowed COMMISSIONING time if he wants it (H1 range is lower (~140Mpc), so H1 may possibly be too low for his work).

The other change is the Vertical Offset changes from yesterday have been reverted for ITMx, ITMy, & BS (see attached for SDF diffs which were ACCEPTED for this).  H1 also had Gaurdians for automated operation from me last night; IFO NOTIFY and H1Manager are now set to IDLE.

Finally, a new NOTE:  IMC_WFS has a notification:  "IMC WFS not centered" now flashing every few seconds.  I hadn't seen that yesterday.

TITLE: 12/03 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 17mph Gusts, 14mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.54 μm/s
QUICK SUMMARY:

Walking in to seeing H1 locked w/ ASC improving DRMI & ISC_LOCK currently starting to power up H1.  Keita was called in earlier (& he and Sheila are chatting on TeamSpeak).

Definitely see effects of an EQ-rung Earth (starting 9.5hrs ago!)  PLUS   Windy day (gusts mostly only up to 20mph but 40+mph at 8amPST  PLUS  Microseism continues to be steady right at the 95th percentile.

(ISC_LOCK currently at MOVE_SPOTS)

TITLE: 12/03 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
 

Removing offsets that were applied yesterday.

ITMX
H1:SUS-ITMX_M0_TEST_V_OFFSET was set back to 0.
H1:SUS-ITMX_R0_TEST_V_OFFSET was set back to 0.

ITMY
H1:SUS-ITMY_M0_TEST_V_OFFSET was set back to 0.
H1:SUS-ITMX_R0_TEST_V_OFFSET was set back to 0.

and Beam Splitter's H1:SUS-BS_M1_TEST_V_OFFSET was set back to 0.
H1:SUS-BS_M1_TEST_V_SW1 was turned OFF

I Accepted all the SDFs for these channels, screenshot1 and screenshot2

After this I put ISC_LOCK into Manual green arms and Started to search for Light on ASL-Y arm. Fought with that for a while, until I realized that the Beam splitter needs to be aligned FIRST, which made getting good flashed on the arms much simpler.
Once I got good Flashes on both arms I started an inital alignment.

Initial Alignment completed, and it did go through SR2_ALIGN.

Started Locking! Got Past AQUIRE_DRMI_1F and we are currently at ENGAGE_ASC_FOR_FULL_IFO!

LOG:
Tony went to EX to UnTrip the Hardware Watchdog
Mike was giving a tour today.

The attached plot shows the ST1 H1,H2,H3 DAC drives for, from upper to lower, ETMX, ITMX, ETMY, ITMY.

H1=blue, H2=orange, H3=green.

ETMY does not rail at all, ITMY all 3 rail for 6 seconds, ITMX H1,H3 rail for 7 seconds, ETMX all 3 rail for 12 seconds.

ETMX only differs from ITMY in the length of time the DAC drives are saturated at 10V (12 sec vs 6 sec).

Sat Dec 02 10:05:06 2023 INFO: Fill completed in 5min 3secs

Ok First thing I did is start trying to get things into a safe state.
ETMX was in a bad SEI State, due to ISI being tripped and HEPI being tripped.
Simply resetting the watchdog on the HEPI didnt work because the Hardware WatchDog was tripped:  H1:SUS-ETMX_HWWD_TTF_MIN
This also stopped me from untripping the following:
H1:IOP-SUS_ETMX_DACKILL_STATE
H1:IOP-SEI_ETMX_DACKILL_STATE

While talking with Dave, Jim, And Fil about the Sequence that these things need to be turned back on.
noticed that the DCPD screen was showing an increase in light on the PDs. I snapped a before picture of the DCPD screen.
I went down to the End station to Reset the Hardware Watchdog while on teamspeak with Dave.

I took pictures of the Hardware Watchdog and the CoilDriver chassis before I touched anything.
Simply pushing and holding the Fault Reset button resets the Hardware watchdog. 
From there, I could reset the ISI and then the HEPI.
Coming back to the control room the ring up had been stopped.


Over the course of the morning 12 earthquakes have rolled through. 3 of which have been over 6.0 Magnitude.
 

We are investigating why the ETMX HWWD saw high SUS RMS for 20 minutes which caused it to trip, whereas all the other HWWDs saw less than 1 minute of high RMS. It would appear that ETMX's high RMS stopped within one minute of the time the HWWD powered down the ETMX ISI coil drivers.

Just like gauge at X2-8, the gauge at Y2-8 is not reporting a pressure as of 1 hour ago, due to the lack of sunlight needed to charge batteries via solar panels. No action is needed at this time.

TITLE: 12/02 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Owl Canceled.
CURRENT ENVIRONMENT:
    SEI_ENV state: LARGE_EQ
    Wind: 15mph Gusts, 9mph 5min avg
    Primary useism: 5.06 μm/s
    Secondary useism: 0.66 μm/s
QUICK SUMMARY:

When I Walked in H1 was being Saturated by a 7.6 mag Earthquakes whos peak is above the top of our earthquake FOM scale, and of course it's gang of aftershocks of mag 6.4 -5 ish.
I tried resetting the watchdogs for ISIs and HEPIs , but the HEPI tripped again.
ISC_LOCK was in Initial_Aligment, took that to IDLE, which stopped verbals from letting us know that the "IFO OUT" Saturation limits are being met.

Current DIAG_MAIN messages:
ESD_DRIVER: ESD X driver OFF
IOP_DACKILL_MON: iopsusex (87) IOP DACKILL Tripped
IOP_DACKILL_MON: iopsusex (90) IOP DACKILL Tripped
SEI_IOP_WD: SEI IOP ['ETMX'] coilmon drop out
SEI_STATE: ['ETMX'] is not nominal

I will begin resolving these issues.
 

It's not pretty, but H1 is back in OBSERVING; range started at 143 (but it's been nosediving the last 15min to 100Mpc).  Surprised to see violin modes looking fairly normal (even after being out of lock for a while and with the high useism). 

SDF Diffs (see attach#1):  Accepted difs for SUS: ITMx, ITMy, & BS as well as SEI:  ETMx Sensor Correction

Also needed to restart the nuc30 ALIGO DARM dtt session because it Timed Out.

Microseism is still pretty high solidly on the 95th percentile mark.

Wireless Access Point in the LVEA was ON, so I turned it OFF.  I left the WAP in the MSR ON.SQZ ASC AS42 not on?? Please RESET_SQZ_ASC

DARM (see attach#2) is elevated from 10-70Hz (the reason for the low range) and also broadly at high frequency....since this looked Squeezer-ish, I checked the squeezer to see that the SQZ MANAGER had a notification about "SQZ ASC AS42 not on?? Please RESET_SQZ_ASC".  When I saw this, I was just about to post something in CHAT, but saw that Naoki was already on it and posted a CHAT asking if I could take H1 out of OBSERVING for him to reset SQZ_ASC, so I did and a few minutes later I took H1 back to OBSERVING.  Now our range should look better...it's already back above 140Mpc!  :)

Taking GRD-IFO to AUTO & H1-Manager to LOW NOISE.

Still see DARM higher from about 10-55Hz.

TITLE: 12/02 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Microseism
INCOMING OPERATOR: N/A
SHIFT SUMMARY:

See mid-shift summary for beginning of the shift. 

For the 2nd half of the shift ended up moving away from alignments and locking and with Sheila focusing on the PRMI lock---namely why POP18 & 90 were so low in power.  Toward the end of the shift Sheila noticed how POP18 had been slowly drifting down in power for PRMI_ASC.  So she started walking PR2 + PR3 (along with a little PRM) in yaw.  And she was able to increase the PRMI powers.  

Remember this is with several Optics with new Offsets!  

I continued walking PR2 + PR3 (along with a little PRM) in pitch (& also revisiting yaw), but I don't know if I improved the powers much more than what Sheila did....maybe a little.  After going in circles walking optics for PRMI w/ NO ASC (POP18 was touching 80 counts), went to PRMI_ASC---this took POP18 up to 90+!  (at the beginning of the shift it was down around 50).

Let ISC LOCK continue, and DRMI locked all on its own.

I will take ISC_LOCK to NOMINAL LOW NOISE and see how it goes.  I imagine there could be some possible violin mode issues, but we'll see.

I should say, more walking of PRMs could possibly improve PRMI's POP18, but basically we are back to PRM powers we had back around Nov17 (NOTE:  on Nov10, POP18 was up to almost 100).

H1 currently at MOVE_SPOTS.

LOG:

• 0318 Turning off amps in lvea (robert)

Corey, TJ, Sheila

Sumary: We've had some clipping in the PRC, which has been getting worse over the last couple of weeks.  This could easily be the cause of our bad noise, and locking difficulties.  We had some sucsess unclipping it by moving yaw, but we aren't yet back to the POP18 build ups we had in mid November. Sensors for looking at PR3 drifts are confusing.

Corey was struggling with PRMI locking this evening.  When I logged in he and others had been working through initial alignment steps, and Corey was able to lock PRMI but with low buildups.  Corey and I spent some time looking at some history of POP18 build ups when PRMI is locked over the last several weeks:

• 11/10 22:29utc 98cnts
• 11/17 10:57utc 91cnts
• 11/25 19:04 UTC 80 cnts (DMRI 96)
• 11/28 1:41 UTC 87 cnts (DRMI 100)
• 11/28 7:32 UTC 85 cnts (DRMI 96)
• 11/28 22:24 UTC 80 cnts
• 11/29 2:18 UTC (a bunch of PRMI locks) 73 counts (DRMI 82)
• 11/29 5:22 UTC 70 cnts (DRMI 80 cnts)
• 11/29 21:16 UTC 62 cnts
• 11/30 1:56 UTC 58 cnts (DRMI 82 cnts)
• 11/30 14:39 UTC PRMI 56 cnts
• 12/2 4:37 UTC PRMI 52 counts

As you can see, the POP 18 build up has been slowly decreasing, even in times when we were locking. We then ran PRMI ASC (REFL 9 to PRM, AS 45 to BS) while we moved PR2, and saw that this did have an impact on the build ups, which it shouldn't if we aren't clipping. (see first attachment for pitch move of PR2).  This pitch move made the build ups more stable, but we couldn't improve the build up with PR2 pitch alone.  We saw similar behavoir with yaw, and moved PR2 yaw in the negative direction.  The second attached screenshot shows a walk of moving both PR3 and PR2 in the negative yaw direction, which did increase the POP 18 build up to about 80 counts.  This was done with ASC off. 

At this time we had trouble keeping PRMI locked for a little bit, it looked like the "mitosis" glitches that Ibrahim saw Wed which we believe were caused by the BS oplev (BS optical lever damping runs while PRMI ASC runs, we tried to turn it off but that was unstable).  We tried a few more rounds of yaw walking with the ASC off, but we saw POP18 drift up to 86.  I think the next step would be to try to walk PR3 and PR2 in pitch to get back to the POP18 build ups close to 100 that we had a few weeks ago, Corey is trying that now.  We probably would be able to lock the IFO with the current value of 86, if all else fails (but if we are still clipping in the PRC that could be causing bad noise).

What's the drift?

The third attachment here is a trend of several sensors of PR3 alignment over the last few weeks, they do not agree with each other.  The optical lever would indicate that the drift of PR3 has drifted in the positive yaw direction, while the top mass osems would indicate that it has drifted in the negative direction.  When I moved PR3, both sensors agree that the sign of the move was negative yaw, so there is not a sign error in one of them.  This reminds me a lot of 70008, and the incident that inspired that alog where the ISI was drifting so the suspension osems didn't show the drift.  TJ points out this alog about the PR3 sensor calibrations: 70197