Sheila, TJ

In times of heavy commissioning or troubleshooting, we want full control of the IFO and don't want to fight the automation. Sheila suggested that we add a manual_control boolean to lscparams.py that ISC_LOCK will look at to decide whether it will automatically run states like Check_Mich_Fringes, PRMI, Increase_Flashes, etc. When this is set to True, it will avoid these automated state either through a conditional in a state's logic, or by weighting edges to force ISC_LOCK to avoid particular states.

For now we are setting manual_control = True while we troubleshoot the IFO.  We will need to remember to return it when we want fully automatic operation again.

Pre April 18

• Camera offsets made : 77139 : April 12
• Unknown issues began with FC IR unlocking : 77164 : April 14
  • Solved : high winds : April 16
• Wind induced lockloss : 77192 : April 15
  • Audible creak in Control Room
  • 5.9 EQ in S. Korea inbound
• Windy making lock difficult : 77188 : April 16
• Squeezing having lock issues 77219 : April 16
• Lockloss - IX saturation : 77224 : April 16 16:41
• Lockloss - reason unknown : 77233 : April 16
• Nominal SQZ angle changes lock-to-lock : 77238 : April 17
• Feedforward and squeezing problem solving

April 18

• Lockloss, unkown cause, Apr 18 01:02 UTC : 77275
• USGS reports 4.3 in Gulf of Cal Apr 18 04:22:45 UTC
• Lockloss, Apr 18 04:46
• USGS reports 5.1 off CDN coast Apr 18, 06:06:04 UTC
• Lockloss from EQ @ Apr 18 06:11UTC : 77259
• USGS reports 4.3 off CDN coast Apr 18, 06:41:45 UTC
• USGS reports 5.2 in Gulf of Cal Apr 18 07:39:39 UTC
• Still aligning : EQ 5.3 from Mexico April 18, 2024 07:39:45UTC : 77261
• USGS reports 4.1 off CDN coast Apr 18, 08:40:12
• USGS reports 3.5 off west coast of Cal Apr 18, 09:33
• USGS reports 4.2 in Gulf of Cal Apr 18, 11:14:04
• USGS reports 2.7 in N Cal Apr 18, 11:28
• USGS reports 2.6 in N Cal Apr 18, 13:21
• USGS reports 5.6 in Gulf of Cal Apr 18, 15:12:30
• USGS reports 3.6 in W Wyom Apr 18, 15:15
• USGS reports 4.3 in S Cal Apr 18, 19:19

Post April 18

• Locked and observing for 36 min : 77280
• Locked for >8.5 hrs : 77281 : April 19, 08:02
• Back to FF and SQZ troubleshooting
• Lockloss, errant gain change at 21:56 : 77283
• Locked for ~4 hours at Apr 19 20:02
• USGS reports 2.9 N of Badger Mt Community Park : Apr 20 04:18:57
  • Stayed locked : 77299 : Apr 20, 00:00
• Lockloss Apr 2024, 15:23 : 77301
• Lockloss at Apr 20, 22:25
• 2 relocks : Apr 21 23:58 (windy)
• Lockloss 26 min after mid-shift entry : 77318
  • Claim - 5.4 EQ from Mexico at 13:43 cause lockloss
  • USGS reports a 5.4 EQ from 19:05 or a 4.4 EQ at 07:28
  • I am probably getting time zones mixed up

• Locked and observing for 36 min : 77280
• Locked for >8.5 hrs : 77281 : April 19, 08:02
• Back to FF and SQZ troubleshooting
• Lockloss, errant gain change at 21:56 : 77283
• Locked for ~4 hours at Apr 19 20:02
• USGS reports 2.9 N of Badger Mt Community Park : Apr 20 04:18:57
  • Stayed locked : 77299 : Apr 20, 00:00
• Lockloss Apr 20, 15:23 : 77301
• Lockloss at Apr 20, 22:25
• 2 relocks : Apr 21 23:58 (windy)
• Lockloss 26 min after mid-shift entry : 77318
  • Claim - 5.4 EQ from Mexico or wind at 13:43 cause lockloss
  • USGS reports a 5.4 EQ from 19:05 or a 4.4 EQ at 07:28
  • I am probably getting time zones mixed up, but these times conflict

Naoki, Sheila, Camilla

To continue investigating alignment changes: 79105. More details in Sheila's 79127.

We put SRM back to before alignment shift, misaligned SRM2 and ITMs,  injected 1.28mW of SQZ SEED beam (as measured on SQZT7 OPO_IR_PD_DC_POWERMON). Followed as in 78115. Increased AS_AIR camera exposure form 10,000 to 100,000

Translated SQZ beam around last NLN spot, each direction the throughput was getting better but in both pitch directions we saturated ZM4 before moving far. Were able to get all expected 0.9mW throughput by moving ZM5 Pitch and yaw both positive.

• In starting alignment ASC-AS_C measured 0.1 to 0.2mW. Same after we centered beam on AS-C.  Time: 2024/07/15 17:52:14 UTC Plot. AS_AIR beam very bad.
• Moving ZM5 Positive YAW (recentering with ZM4 +Y), increased AS_C measured to 0.6mW before it started getting worse. Time: 2024/07/15 18:10:56 UTC Plot
• From this spot (ZM5 Positive YAW) we moved ZM5 Positive PITCH (recentering with ZM4 -P), increased AS_C measured to 0.9mW before it started getting worse Time: 2024/07/15 18:16:33 UTC Plot AS_AIR beam got better.
• We then went back to the original ZM4/5 alignment
• Moving ZM5 Negative YAW (recentering with ZM4 -Y) we were improving but saturating ZM4 before we could finish, AS_C measured to 0.44mW,  Time: 2024/07/15 18:55:00 UTC Plot
• Moving ZM5 Positive PITCH (recentering with ZM4 -P) we were improving but saturating ZM4 before we could finish, AS_C measured to 0.25mW,  Time: 2024/07/15 19:01:57 UTC Plot
• Moving ZM5 Negative PITCH (recentering with ZM4 +P) we were improving but saturating ZM4 before we could finish, AS_C measured to 0.22mW  Time 2024/07/15 19:12:57 UTC Plot
• Note the max we got when not aligned on AS_C was around 0.1mW, maybe caused by edge effects as expect 0.7% reflection off SRM HR surface = 0.7 x 1.28mW  = 0.896mW MAX.

The two attached histograms show the relock times for LIGO Hanford during the whole of O4a.

The average relock time has been 2 hours and 55 minutes, while the median was at 1 hour and 33 minutes.

The relock time was computed as the time for which the ISC_LOCK_N value was consecutively below 580.

The 580 value corresponds to inject squeezing, which is why there were some short "relocks" when we used below 600 for the lockloss threshold, because the squeezer unlocked and relocked automatically.

Link to the github repository with the employed code and data

Summary: We think that the sqz beam is seeing similar loss to the IFO beam in th alignment that we were running in until Friday, but double passed. The sqz beam transmission can be restored by moving the beam, and there should be enough power in this beam to be a useful diagnostic in chamber to distinguish between OFI and SRM damage.

Camilla and Noaki have measured 1.3mW coming from HAM7 towards HAM5 with the OPO locked on the seed. SRM transmission is 32.34%, ( 72680 ) so if nothing was damaged we should have something like 0.41mW of the OPO seed beam in transmission of SRM, which we should be able to measure with a power meter. 

The table in 79101 indicates that the single bounce transmission to AS_C has dropped to 64% of what it was last Thursday.  Here's a table of power predictions for the sqz beam arriving in HAM6:

With the alignment used for observing in recent weeks Naoki and Camilla measured 0.2 mW on AS_C QPD (not well centered on AS_C, but on the diode), which is calibrated into Watts arriving in HAM6.  They moved ZM4 +5 to improve the transmission of the sqz beam to 0.9mW when centered on AS_C.  These numbers don't agree super well with the predictions above, but they seem to suggest that the SQZ beam sees similar damage to the main IFO beam, but double passed.

Sheila asked me to look at the same trends we used to track the damage in the OFI over the 22nd and 23rd of April when it first happened. We think something like this might have happened again on Friday.

Attached you can see the optical gain, coupled cavity pole, range in Mpc, power at the anti-symmetric port, circulating power, and power reflected from the OMC all changed between April the 22nd, when the damage first started affecting the range causing it to decrease slowly.

In the next image we have the same trends from the last couple of days, there doesn't seem to be a slow degradation in range in this case, the range got worse between two locks on the 12th but nothing else seems to have degraded (unlike beforte when we observed a drop in optical gain, circulating power, output power and coupled cavity pole) before we got to the point where we couldn't relock.

FAMIS 26315, last checked in alog78896

No appreciable change in any fan's noise levels compared to last check.

FAMIS 21008

No major events of note for these trends except for the PSL work done last Tuesday (seen in almost every trend), and, concerningly, a slight rise in PMC reflected power of about 0.3W over 5 days. It seems to have leveled off in the past day or so, however.

at 08:50 PDT the MSR Atomic Clock lost sync with the timing system, the comparator is reading a steady 0.24S 1PPS difference. This has happened before, it requires a manual resync of the Atomic Clock to the timing-system 1PPS. This was last done 6th May 2024

Mon Jul 15 08:06:22 2024 INFO: Fill completed in 6min 18secs

Jordan confirmed a good fill curbside.

TITLE: 07/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 5mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:

• The IFO is still down, pending more investigations

There was a pressure glitch on PT100B HAM1 cold-cathode at 07:25:15 PDT this morning. There was no corresponding glitch on any other corner station gauge as far as I can tell.

Pressure increased from 3.3e-08 to 4.0e-08 Torr, recovered in 4 minutes.

I've gathered several trends comparing the before and after of the alignment shift the IFO saw in the past couple of days, likely on Friday the 12th.

On all of these plots, I use y-cursors to indicate the levels each signal was at the appropriate time leading up to the last lock H1 had fully up to Nominal Low Noise, which lasted from 10:14 to 16:45 UTC on July 12th, in order to compare to what was likely the last time H1 was behaving "well."

DRMI signals with OM1/2 OSEMs:

The last DRMI acquisition before NLN, POP18 was around 66 counts while POP90 was around 11 counts.

The most recent DRMI acquisition, POP18 has dropped to 42 counts while POP90 has risen to 16 counts. The only significant alignment difference I see is with OM1 Y being about 70 counts different.

Optic alignments during 10W single-bounce: (times all taken from when ALIGN_IFO was in the state SR2_ALIGN [58])

The last SR2 alignment before NLN (P/SRC optics; large optics), AS_C_NSUM signal was at 0.0227 counts.

The next time in this configuration (P/SRC optics; large optics), AS_C_NSUM signal had dropped to 0.018 counts. The most obvious alignment changes are with PR2Y, PRM, some SR2, and SRM.

The most recent time in this configuration (P/SRC optics; large optics), AS_C_NSUM signal had dropped again to 0.014 counts. The same optics are off in alignment as before, but PRM has now flipped to being off in the opposite directions for both pitch and yaw.

OFI and KAPPA_C:

I recreated trends like Camilla did in alog78399 to check the behavior of the OFI. See attached for multi-day trend, zoomed in on shakes during high-state locklosses, and zoom in on one of these shakes.

HAM6 vacuum pressure:

Out of an abundance of caution, I trended the pressure gauge on HAM6 since we've seen pressure spikes somewhat recently, and I don't see anything that hasn't been previously noted on a 21-day timescale (the pressure rise ~20 days ago was noted to be because of the OM2 thermistor in alog78829).

16:44 UTC lockloss. PRCL was oscillating at about ~3.6 Hz.

LHO control room crew-

We suspect that something about the output Faraday has changed. 

• One observation is that the powers in HAM6 dropped 5% in Monday night's lock (see first attachment). 
• We spent a lot of time in SRY and single bounce, and comparing today's single bounce times to times Jennie Wright found from the last few weeks ago 77204.  Jenne Driggers was able to restore the same throughput from input power to HAM6 powers in single bounce by moving SR3 and SR2 by 800 urads in yaw, 77388.  We checked many witness sensors for large alignment shifts, it doesn't seem that any suspension shifted enough to explain this.
• We then tried to reproduce a SQZ single bounce from 2 weeks ago OMC scans, by reseting all PSAMs and osems on ZM4/5/6 SRM, and OM1/2.  This resulted in much lower power on AS_A and AS_B than two weeks ago, indicating that something is also shifting this beam.  We were able to restore the power by moving in alignment into the OFI by ~200 urad in yaw or pitch of ZM5.

The only thing that seems common to these two is the OFI, a baffle in it's vicinity or something like a wire from the fast shutter impacting the beam on the way to OM1.  We'd actuated the fast shutter and see no change in the AS camera, so we don't think that the fast shutter is the issue, which points us to looking at the OFI.

Editing to add that Keita points out something like a bad spot on OM1 is also a possibility.