Closes FAMIS#26218, last checked 73966

Laser Status:
    NPRO output power is 1.819W (nominal ~2W)
    AMP1 output power is 68.01W (nominal ~70W)
    AMP2 output power is 137.0W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN

PMC:
    It has been locked 12 days, 3 hr 5 minutes
    Reflected power = 15.94W
    Transmitted power = 109.9W
    PowerSum = 125.8W

FSS:
    It has been locked for 0 days 1 hr and 57 min
    TPD[V] = 0.7357V

ISS:
    The diffracted power is around 2.6%
    Last saturation event was 0 days 1 hours and 57 minutes ago

Possible Issues: None

Everything looking good!

Been having trouble relocking all day. Wind and microseism have been high and increasing, and the ALSX beat note strength being below the minimum has been throwing ALS-XARM into error (74289). Lowering the beat note strength threshold has been working for getting us past LOCKING_ARMS_GREEN, but we have now lost lock at CARM_OFFSET_REDUCTION three times, each time having a lot of EX and other suspensions saturating and causing us to lose lock.

Luckily the wind is starting to lower a bit, so I'll let the detector sit in DOWN for a bit and then hopefully relocking will go much better.

I've been getting Guardian ALS_XARM node in error when trying to lock today, something that Austin also got overnight. It looks like ALS-X_FIBR_A_DEMOD_RFMON has been worming its way down(attachment1, attachment2) towards the minimum beat note strength of -10dB since Nov 17th 16:30ish UTC. I have temporarily lowered the beat note strength minimum to be -20dB so that we can lock, but going off of the last time this happened, 70951, the ALSX laser temp and current are going to need to be adjusted.

FRS ticket already opened on this issue: FRS#28430

Sun Nov 19 10:09:43 2023 INFO: Fill completed in 9min 39secs

TITLE: 11/19 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 23mph Gusts, 14mph 5min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.56 μm/s
QUICK SUMMARY:

Detector is relocking and just got to ACQUIRE_PRMI. Wind is increasingly jumping up past 30mph, and secondary microseism is also way up due to a large storm off the coast.

ALS X arm guardian went into error in relocking when it went into the CHECK CRYSTAL FREQ state during LOCKING ARMS GREEN. Attached is the screenshot of the guardian log. I reset the ALS X ARM guardian and tried relocking x, which looks to have solved the issue. ISC is now continuing back up.

TITLE: 11/19 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: Austin
SHIFT SUMMARY:
Since the mid shift report which contained all the fun locking. 
The DARM FOM has stopped working. I have tried removing channels and restarting the app, and the NUC.
The Diaggui I was using to watch Violin progress also started to act strange around the same time.

7:53 UTC superevent candidate S231119u

LOG:
No log

FAMIS 26264
No visible changes in the vibration of the HVAC fans in the last 6 days.

TITLE: 11/19 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 7mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.29 μm/s
QUICK SUMMARY:

Locking Notes:
H1 got stuck in DRMI_LOCKED_CHECK_ASC PRC1 Pit not converged, thresh=400
and SRC2 yaw not converged, thresh=200

I tried moving SRM2 a little but realized that Gaurdian logs stopped giving me information and I couldn't see any movement and couldn't tell if I was making it better or worse.

Current gps time 1384392748
GPS Time of last good Initial alignment 1384029371 (3 days ago)

I imagine that this is a problem with alignment due to thermal changes over the last few days.
Looking at the Misaligned tool, tried to see if I can glean any info out of that.

Took ISC_LOCK to a DOWN state.
Used the Restored ALL button on AlignIFO MEDM screen to send all optics back to 1384029371 (the tail end of the last good Initial Alignment right before a nice lock stretch ~ 3 days ago)
Took ISC_LOCK to Initial Alignment.
Then While in Initial Alignment was locking green I Restored just the X and Y arms back to 1384392748 (just a few minutes ago) so I dont have to redo green arms again & Made it through Initial Alignment again.

Locking attempt 3
Made it past DRMI_1F
Made it past DRMI_LOCKED_CHECK_ASC
2:44 UTC reached NOMINAL_LOW_NOISE
2:48 UTC made it to OBSERVING 

TITLE: 11/19 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 7mph Gusts, 3mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY:
H1 was in Initial Alignment when I arrived.
IA completed and H1 is not locking DRMI_1F, and going to  ACQUIRE_PRMI in a loop for some reason.
starting to troubleshoot now.
 

TITLE: 11/19 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Had a good amount of issues getting locked today both times we were relocking, but I was hoping that that was just because the Initial Alignment from when I came in was four hours old at that point? Lock lasted only a few hours before we lost lock again and had alignment/locking issues. As of now, we have just completed an initial alignment and is going through ACQUIRE_PRMI multiple times again, so maybe the issue from the beginning of my shift is still present.
LOG:

16:00UTC Detector down with H1_MANAGER needing assistance since 12:11UTC

RELOCKING
16:35 Detector kept jumping back down to ACQUIRE_PRMI from ACQUIRE_DRMI_1F, so I followed Difficulty_locking_DRMI_1F2F3F instructions
    - After a couple of tries, at 17:15 the ifo took us through CHECK_MICH_FRINGES, and after went up normally.
18:13 NOMINAL_LOW_NOISE
18:19 Observing

21:59 Lockloss (74278)

RELOCKING
22:32 Was going through CHECK_MICH_FRINGES twice, and since we were having some alignment issues the last lock, I decided to take us to DOWN to run an initial alignment
    - After DOWN and INIT, the IMC wouldn't lock, we got an IMC_WFS: WFS not centered message, and the laser power was still at 10W, so I took the laser power down to 2W, and tried relocking IMC by DOWN and INITing it
    - IMC still wouldn't lock, followed Jenne's troubleshooting document and cleared the IMC WFS, but I accidentally cleared them for dof 1-5 in pitch and yaw, when Jenne's document only points to dof 1-3 for pitch/yaw. Not sure if that means I accidently cleared things I shouldn't have.
    - After I cleared them all, we started getting constant IMC_WFS: WFS not centered messages
    - I saw the values of the PZT had changed, and since the IMC still wouldn't lock, I thought that it was because I had accidentally cleared the PZT history and so tried reverting the PZT output values, which I then reverted back again
    - I then just tried lowering the IMC WFS trigger threshold (73887), which worked and locked the IMC.

21:59UTC Lockloss

Observing at 154Mpc and Locked for 2.5 hours now.

Between 16:54 and 17:30UTC DIAG_MAIN was saying: PSL_CHILLER: Check PSL chiller, but it had stopped alarming by the time I was able to go and check on it around 18:25UTC. ndscope shows that H1:PSL-LASER_CHILLER1_ALARM had been only going off intermittently during that time period, so probably from the water level just barely dipping too low. However when I went to check on the water level, it wasn't dipping anywhere near the minimum and was staying pretty solidly 1/4-1/3 of the way above the minimum (I forgot to take a photo before filling it), but maybe some ground motion or something similar caused it to oscillate below the min during those ~30 minutes. Either way, I filled it to the max line, which ended up being 134 mL. The last time it had been topped off was Oct 16 (73504).

Sat Nov 18 10:03:03 2023 INFO: Fill completed in 3min 1secs

TITLE: 11/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.22 μm/s
QUICK SUMMARY:

Detector is down with ISC_LOCK in Initial Alignment - looks like IA completed at 12:19UTC but then it stayed there, with H1 MANAGER saying assistance required 8 minutes earlier due to the IA timer expiring, which is the same issue that had woken Austin up a couple of hours previously. I assume that he did not get the second call for whatever reason? Either way, I started locking.

H1 MANAGER:

2023-11-18_12:11:11.082073Z H1_MANAGER [RUNNING_ALIGNMENT.run] timer['initial_alignment'] done
2023-11-18_12:11:11.136738Z H1_MANAGER [RUNNING_ALIGNMENT.run] Timer expired, alignment taking too long
2023-11-18_12:11:11.197052Z H1_MANAGER JUMP target: ASSISTANCE_REQ
2023-11-18_12:11:11.198696Z H1_MANAGER [RUNNING_ALIGNMENT.exit]
2023-11-18_12:11:11.252860Z H1_MANAGER JUMP: RUNNING_ALIGNMENT->ASSISTANCE_REQ
2023-11-18_12:11:11.252860Z H1_MANAGER calculating path: ASSISTANCE_REQ->LOW_NOISE
2023-11-18_12:11:11.252860Z H1_MANAGER no path from ASSISTANCE_REQ->LOW_NOISE
2023-11-18_12:11:11.258125Z H1_MANAGER executing state: ASSISTANCE_REQ (50)
2023-11-18_12:11:11.258405Z H1_MANAGER [ASSISTANCE_REQ.enter]
2023-11-18_12:11:11.316931Z H1_MANAGER [ASSISTANCE_REQ.run] USERMSG 0: Human assistance required! Request INIT to clear

IFO_NOTIFY:

2023-11-18_11:31:21.378817Z IFO_NOTIFY executing state: WAITING (20)
2023-11-18_12:11:11.271937Z IFO_NOTIFY [WAITING.run] H1_MANAGER requires assistance. Sending
2023-11-18_12:11:11.380463Z IFO_NOTIFY EDGE: WAITING->ALERT_ACTIVE
2023-11-18_12:11:11.382711Z IFO_NOTIFY calculating path: ALERT_ACTIVE->ALERTS_ENABLED
2023-11-18_12:11:11.382711Z IFO_NOTIFY new target: ALERTS_ENABLED
2023-11-18_12:11:11.384404Z IFO_NOTIFY executing state: ALERT_ACTIVE (30)
2023-11-18_12:11:11.393434Z IFO_NOTIFY [ALERT_ACTIVE.enter]
2023-11-18_12:11:11.465917Z IFO_NOTIFY [ALERT_ACTIVE.run] USERMSG 0: See log for alert. Select INIT to reset

H1 contacted me this morning ~ 3:15 am this morning. Looks like the timer expired after the IFO lost lock and is attemtping to run an initial alignment twice. Will intervene if the current locking aquisition fails.