Sheila, TJ

In times of heavy commissioning or troubleshooting, we want full control of the IFO and don't want to fight the automation. Sheila suggested that we add a manual_control boolean to lscparams.py that ISC_LOCK will look at to decide whether it will automatically run states like Check_Mich_Fringes, PRMI, Increase_Flashes, etc. When this is set to True, it will avoid these automated state either through a conditional in a state's logic, or by weighting edges to force ISC_LOCK to avoid particular states.

For now we are setting manual_control = True while we troubleshoot the IFO.  We will need to remember to return it when we want fully automatic operation again.

Pre April 18

• Camera offsets made : 77139 : April 12
• Unknown issues began with FC IR unlocking : 77164 : April 14
  • Solved : high winds : April 16
• Wind induced lockloss : 77192 : April 15
  • Audible creak in Control Room
  • 5.9 EQ in S. Korea inbound
• Windy making lock difficult : 77188 : April 16
• Squeezing having lock issues 77219 : April 16
• Lockloss - IX saturation : 77224 : April 16 16:41
• Lockloss - reason unknown : 77233 : April 16
• Nominal SQZ angle changes lock-to-lock : 77238 : April 17
• Feedforward and squeezing problem solving

April 18

• Lockloss, unkown cause, Apr 18 01:02 UTC : 77275
• USGS reports 4.3 in Gulf of Cal Apr 18 04:22:45 UTC
• Lockloss, Apr 18 04:46
• USGS reports 5.1 off CDN coast Apr 18, 06:06:04 UTC
• Lockloss from EQ @ Apr 18 06:11UTC : 77259
• USGS reports 4.3 off CDN coast Apr 18, 06:41:45 UTC
• USGS reports 5.2 in Gulf of Cal Apr 18 07:39:39 UTC
• Still aligning : EQ 5.3 from Mexico April 18, 2024 07:39:45UTC : 77261
• USGS reports 4.1 off CDN coast Apr 18, 08:40:12
• USGS reports 3.5 off west coast of Cal Apr 18, 09:33
• USGS reports 4.2 in Gulf of Cal Apr 18, 11:14:04
• USGS reports 2.7 in N Cal Apr 18, 11:28
• USGS reports 2.6 in N Cal Apr 18, 13:21
• USGS reports 5.6 in Gulf of Cal Apr 18, 15:12:30
• USGS reports 3.6 in W Wyom Apr 18, 15:15
• USGS reports 4.3 in S Cal Apr 18, 19:19

Post April 18

• Locked and observing for 36 min : 77280
• Locked for >8.5 hrs : 77281 : April 19, 08:02
• Back to FF and SQZ troubleshooting
• Lockloss, errant gain change at 21:56 : 77283
• Locked for ~4 hours at Apr 19 20:02
• USGS reports 2.9 N of Badger Mt Community Park : Apr 20 04:18:57
  • Stayed locked : 77299 : Apr 20, 00:00
• Lockloss Apr 2024, 15:23 : 77301
• Lockloss at Apr 20, 22:25
• 2 relocks : Apr 21 23:58 (windy)
• Lockloss 26 min after mid-shift entry : 77318
  • Claim - 5.4 EQ from Mexico at 13:43 cause lockloss
  • USGS reports a 5.4 EQ from 19:05 or a 4.4 EQ at 07:28
  • I am probably getting time zones mixed up

• Locked and observing for 36 min : 77280
• Locked for >8.5 hrs : 77281 : April 19, 08:02
• Back to FF and SQZ troubleshooting
• Lockloss, errant gain change at 21:56 : 77283
• Locked for ~4 hours at Apr 19 20:02
• USGS reports 2.9 N of Badger Mt Community Park : Apr 20 04:18:57
  • Stayed locked : 77299 : Apr 20, 00:00
• Lockloss Apr 20, 15:23 : 77301
• Lockloss at Apr 20, 22:25
• 2 relocks : Apr 21 23:58 (windy)
• Lockloss 26 min after mid-shift entry : 77318
  • Claim - 5.4 EQ from Mexico or wind at 13:43 cause lockloss
  • USGS reports a 5.4 EQ from 19:05 or a 4.4 EQ at 07:28
  • I am probably getting time zones mixed up, but these times conflict

Naoki, Sheila, Camilla

To continue investigating alignment changes: 79105. More details in Sheila's 79127.

We put SRM back to before alignment shift, misaligned SRM2 and ITMs,  injected 1.28mW of SQZ SEED beam (as measured on SQZT7 OPO_IR_PD_DC_POWERMON). Followed as in 78115. Increased AS_AIR camera exposure form 10,000 to 100,000

Translated SQZ beam around last NLN spot, each direction the throughput was getting better but in both pitch directions we saturated ZM4 before moving far. Were able to get all expected 0.9mW throughput by moving ZM5 Pitch and yaw both positive.

• In starting alignment ASC-AS_C measured 0.1 to 0.2mW. Same after we centered beam on AS-C.  Time: 2024/07/15 17:52:14 UTC Plot. AS_AIR beam very bad.
• Moving ZM5 Positive YAW (recentering with ZM4 +Y), increased AS_C measured to 0.6mW before it started getting worse. Time: 2024/07/15 18:10:56 UTC Plot
• From this spot (ZM5 Positive YAW) we moved ZM5 Positive PITCH (recentering with ZM4 -P), increased AS_C measured to 0.9mW before it started getting worse Time: 2024/07/15 18:16:33 UTC Plot AS_AIR beam got better.
• We then went back to the original ZM4/5 alignment
• Moving ZM5 Negative YAW (recentering with ZM4 -Y) we were improving but saturating ZM4 before we could finish, AS_C measured to 0.44mW,  Time: 2024/07/15 18:55:00 UTC Plot
• Moving ZM5 Positive PITCH (recentering with ZM4 -P) we were improving but saturating ZM4 before we could finish, AS_C measured to 0.25mW,  Time: 2024/07/15 19:01:57 UTC Plot
• Moving ZM5 Negative PITCH (recentering with ZM4 +P) we were improving but saturating ZM4 before we could finish, AS_C measured to 0.22mW  Time 2024/07/15 19:12:57 UTC Plot
• Note the max we got when not aligned on AS_C was around 0.1mW, maybe caused by edge effects as expect 0.7% reflection off SRM HR surface = 0.7 x 1.28mW  = 0.896mW MAX.

The two attached histograms show the relock times for LIGO Hanford during the whole of O4a.

The average relock time has been 2 hours and 55 minutes, while the median was at 1 hour and 33 minutes.

The relock time was computed as the time for which the ISC_LOCK_N value was consecutively below 580.

The 580 value corresponds to inject squeezing, which is why there were some short "relocks" when we used below 600 for the lockloss threshold, because the squeezer unlocked and relocked automatically.

Link to the github repository with the employed code and data

Summary: We think that the sqz beam is seeing similar loss to the IFO beam in th alignment that we were running in until Friday, but double passed. The sqz beam transmission can be restored by moving the beam, and there should be enough power in this beam to be a useful diagnostic in chamber to distinguish between OFI and SRM damage.

Camilla and Noaki have measured 1.3mW coming from HAM7 towards HAM5 with the OPO locked on the seed. SRM transmission is 32.34%, ( 72680 ) so if nothing was damaged we should have something like 0.41mW of the OPO seed beam in transmission of SRM, which we should be able to measure with a power meter. 

The table in 79101 indicates that the single bounce transmission to AS_C has dropped to 64% of what it was last Thursday.  Here's a table of power predictions for the sqz beam arriving in HAM6:

With the alignment used for observing in recent weeks Naoki and Camilla measured 0.2 mW on AS_C QPD (not well centered on AS_C, but on the diode), which is calibrated into Watts arriving in HAM6.  They moved ZM4 +5 to improve the transmission of the sqz beam to 0.9mW when centered on AS_C.  These numbers don't agree super well with the predictions above, but they seem to suggest that the SQZ beam sees similar damage to the main IFO beam, but double passed.

Sheila asked me to look at the same trends we used to track the damage in the OFI over the 22nd and 23rd of April when it first happened. We think something like this might have happened again on Friday.

Attached you can see the optical gain, coupled cavity pole, range in Mpc, power at the anti-symmetric port, circulating power, and power reflected from the OMC all changed between April the 22nd, when the damage first started affecting the range causing it to decrease slowly.

In the next image we have the same trends from the last couple of days, there doesn't seem to be a slow degradation in range in this case, the range got worse between two locks on the 12th but nothing else seems to have degraded (unlike beforte when we observed a drop in optical gain, circulating power, output power and coupled cavity pole) before we got to the point where we couldn't relock.

FAMIS 26315, last checked in alog78896

No appreciable change in any fan's noise levels compared to last check.

FAMIS 21008

No major events of note for these trends except for the PSL work done last Tuesday (seen in almost every trend), and, concerningly, a slight rise in PMC reflected power of about 0.3W over 5 days. It seems to have leveled off in the past day or so, however.

at 08:50 PDT the MSR Atomic Clock lost sync with the timing system, the comparator is reading a steady 0.24S 1PPS difference. This has happened before, it requires a manual resync of the Atomic Clock to the timing-system 1PPS. This was last done 6th May 2024

Mon Jul 15 08:06:22 2024 INFO: Fill completed in 6min 18secs

Jordan confirmed a good fill curbside.

TITLE: 07/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 5mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:

• The IFO is still down, pending more investigations

There was a pressure glitch on PT100B HAM1 cold-cathode at 07:25:15 PDT this morning. There was no corresponding glitch on any other corner station gauge as far as I can tell.

Pressure increased from 3.3e-08 to 4.0e-08 Torr, recovered in 4 minutes.

I've gathered several trends comparing the before and after of the alignment shift the IFO saw in the past couple of days, likely on Friday the 12th.

On all of these plots, I use y-cursors to indicate the levels each signal was at the appropriate time leading up to the last lock H1 had fully up to Nominal Low Noise, which lasted from 10:14 to 16:45 UTC on July 12th, in order to compare to what was likely the last time H1 was behaving "well."

DRMI signals with OM1/2 OSEMs:

The last DRMI acquisition before NLN, POP18 was around 66 counts while POP90 was around 11 counts.

The most recent DRMI acquisition, POP18 has dropped to 42 counts while POP90 has risen to 16 counts. The only significant alignment difference I see is with OM1 Y being about 70 counts different.

Optic alignments during 10W single-bounce: (times all taken from when ALIGN_IFO was in the state SR2_ALIGN [58])

The last SR2 alignment before NLN (P/SRC optics; large optics), AS_C_NSUM signal was at 0.0227 counts.

The next time in this configuration (P/SRC optics; large optics), AS_C_NSUM signal had dropped to 0.018 counts. The most obvious alignment changes are with PR2Y, PRM, some SR2, and SRM.

The most recent time in this configuration (P/SRC optics; large optics), AS_C_NSUM signal had dropped again to 0.014 counts. The same optics are off in alignment as before, but PRM has now flipped to being off in the opposite directions for both pitch and yaw.

OFI and KAPPA_C:

I recreated trends like Camilla did in alog78399 to check the behavior of the OFI. See attached for multi-day trend, zoomed in on shakes during high-state locklosses, and zoom in on one of these shakes.

HAM6 vacuum pressure:

Out of an abundance of caution, I trended the pressure gauge on HAM6 since we've seen pressure spikes somewhat recently, and I don't see anything that hasn't been previously noted on a 21-day timescale (the pressure rise ~20 days ago was noted to be because of the OM2 thermistor in alog78829).

To help characterize the recent issues we've seen with IFO alignment that are familiar to the output change in late April, I moved SR3 in 10W single-bounce configuration with the SRC2 ASC loop on so that SR2 would follow (ALIGN_IFO's SR2_ALIGN state), much like in alog77694. I moved SR3 far enough in each direction (+/- pitch and yaw) to bring the ASC-AS_C_NSUM signal back up to our target value of 0.022, which I was successful doing in every direction except -Y, where I could only bring AS_C up to 0.018. In the three directions where I was successful, the beam spot on the AS AIR camera looked much more like the clear circle we're used to seeing and less like the upside-down apostrophe we have now.

It seems that our old output spot (starting place in TJ's alog) is still bad (-Y direction with SR3 from current place) since that was the only direction where I couldn't get the AS_C power back up.

Slider values of SR3 after each move:

Attachment 1 is the starting AS AIR camera image (and our current spot), attachment 2 is after the +P move, attachment 3 is after the -P move, and attachment 4 is after the +Y move.

Yesterday, the fast shutter test failed due to dark offsets in the AS WFS DC NSUM channels.

The guardian started running the TwinCAT testing code, which works as intended: it sends a close command to the trigger logic, which in turn fires the fast shutter. The fast shutter works fine as can be seen on the HAM6 geophones. The slow controls readbacks also indicate that both fast shutter and PZT shutter are closed no later than 200ms after the trigger. However 0.5sec after the guardians started the test, it checks the AS WFS DC NSUM outputs and compares them against dark offset limits of ±15. Since the dark offset on WFS B was over 30, the guardian then sent an abort command to the TwinCAT code and reported a failure.

16:44 UTC lockloss. PRCL was oscillating at about ~3.6 Hz.

Ibrahim, Rahul

BOSEM counts have been visibly drifting over the last few days since I centered them last week. Attached are two screenshots:

1. Screenshot 1 shows the 48hr shift of the BOSEM counts as the temperature is varying
2. Screenshot 2 shows the full 8 day drift since I centered the OSEMs.

I think this can easily be explained by Temperature Driven Suspension Sag (TDSS - new acronym?) due to the blades. (Initially, Rahul suggested maybe the P-adjuster was loose and moving but I think the cyclic nature of the 8-day trend disproves this)

I tried to find a way to get the temp in the staging building but Richard said there's no active data being taken so I'll take one of the thermometer/temp sensors available and place it in the cleanroom when I'm in there next, just to have the available data.

On average, the OSEM counts for RT and LF, the vertical facing OSEMs have sagged by about 25 microns. F1, which is above the center of mass, is also seeing a long-term drift. Why?

• Is this because there's an ever-so-slight P instability, so the sag "reveals" it?
• Is it because the blades are differentially sagging
• Is it because (assuming temp. driven DC sag offset), the flag is slowly moving up or down and allowing less light in?

More importantly, how does this validate/invalidate our OSEM results given that some were taken hours after others and that they were centered days before the TFs were taken?

LHO control room crew-

We suspect that something about the output Faraday has changed. 

• One observation is that the powers in HAM6 dropped 5% in Monday night's lock (see first attachment). 
• We spent a lot of time in SRY and single bounce, and comparing today's single bounce times to times Jennie Wright found from the last few weeks ago 77204.  Jenne Driggers was able to restore the same throughput from input power to HAM6 powers in single bounce by moving SR3 and SR2 by 800 urads in yaw, 77388.  We checked many witness sensors for large alignment shifts, it doesn't seem that any suspension shifted enough to explain this.
• We then tried to reproduce a SQZ single bounce from 2 weeks ago OMC scans, by reseting all PSAMs and osems on ZM4/5/6 SRM, and OM1/2.  This resulted in much lower power on AS_A and AS_B than two weeks ago, indicating that something is also shifting this beam.  We were able to restore the power by moving in alignment into the OFI by ~200 urad in yaw or pitch of ZM5.

The only thing that seems common to these two is the OFI, a baffle in it's vicinity or something like a wire from the fast shutter impacting the beam on the way to OM1.  We'd actuated the fast shutter and see no change in the AS camera, so we don't think that the fast shutter is the issue, which points us to looking at the OFI.

Editing to add that Keita points out something like a bad spot on OM1 is also a possibility.