Elenna noticed that pimon that we run on nuc25 hadn't been consistently saving data. There were large periods of time that didn't have anything recorded. I first verified that it would still write log and npz files, then I redirected them to /ligo/data/pimon/locklosses/ where it will now put all of the npz files. Elenna also asked that the script saves the time series and not the PSDs, so I made that change as well. I'm not sure the file size difference, the PSDs were about 3.6Mb, so we should keep an eye on that and get rid of old ones.

Operators, please verify that pimon is running on nuc25, and restart it if necessary. The normal launch script will start it up.

The calibration lines still appear in DHARD P and Y to varying degrees. Here is a higher resolution spectrum of DHARD P and Y, with OMC DCPD sum for reference. DHARD P shows all four calibration lines (L1 15.6 Hz, L2 16.4 Hz, L3 17.6 Hz and PCAL 17.1 Hz) between 15-18 Hz, while DHARD Y only seems to show the 16.4 Hz line in L2.

Today, Sheila and I also looked at the phasing of AS A/B 45 WFS. Our thinking was that a DARM line could be phased to be mainly in Q for each segment. The segments all show the calibration lines prominently, so they can be used as a phasing reference. It was immediately apparently that for all segments of both WFS, the DARM line is at least a factor of 2 higher in I than in Q. We decided not to make any phasing changes at this time, since rephasing these lines into Q would change the sign dramatically and may impact locking when we put DARM on RF.

Instead of redesigned the MICH ASC loops, I just updated the lowpasses to be at 12 Hz instead of 15 Hz, which should reduce the gain by 15 dB between 10-20 Hz for both pitch and yaw. I tested them today with no issues, so I adjusted the MICH ASC engagement in ISC_DRMI guardian, and updated SDF (accidentally overwrote the screenshot with my screenshot of the filter).

Elenna and I checked the FC detuning by taking it slowly from nominal -26 to -50, where we could clearly see low freq DARM was worse, and then stepping to -15 (any lower unlocked the FC in 85886) using ezcastep H1:IOP-LSC0_RLF_FREQ_OFS -s 30 '+1,35', see attached plot. The best de-tuning is close to where we already are, leaving it at -27.

Mon Jul 21 10:08:47 2025 INFO: Fill completed in 8min 44secs

First I tried taking H1:IOP-LSC0_RLF_FREQ_OFS down towards zero but we lost FC lock at  H1:IOP-LSC0_RLF_FREQ_OFS = -8. FC would not re-lock until I brought it back to -15.

Wanted to scan FC de-tuning from -15 to -60 but the command didn't seem to work with negative steps.

Then 5s after moving FC detuning from -15 to -60, the IFO lost lock, 1437146099. The lockloss appears unrelated to the FC detuning change as the FC lost lock after we see  ETMX_L3 start to be unstable, plot. There was slightly increased noise at low frequency but not enough scatter to cause a lockloss.

Ideally would have ran ezcastep H1:IOP-LSC0_RLF_FREQ_OFS -s 30 '+1,45' to get 30sec steps from -60 to -15. We've been at -44 before with no issues, 83526, but wanted to make the low frequency noise worse to be able to fit to the best place.

I'm followed the pydarm deployment instructions here, to update the LHO pydarm install.

This is the 20250721.0 tag for pydarm, which includes a bug fix for high frequency roam line handling, letting us indicate the correct reference model in the pydarm_H1.ini file.

This is not the default cds conda environment, but the default you get when typing pydarm at a command line, or specifically invoking by running "conda activate /ligo/groups/cal/conda/pydarm".

TITLE: 07/21 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 13mph Gusts, 9mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY:

Arriving to a Dust Alarm in the PSL over the last hr (there have been winds around the corner station the last 2hrs but only about 15mph).

For H1 over the last 12 hrs we have had 4 locks (current one is 90min) where 2 (of3) locklosses were ETMx Glitches, and it looks like all reacquisition was automatic overnight.

H1 is scheduled for Commissioning from 15-19utc (8-noonPDT).

TITLE: 07/21 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:
Started the Shift unlocked due to earthquake.

Got back to observing for 1 minute before a lockloss.
Then relocked again for an hour and 17 minutes before another Unknown Lockloss from NLN.

Relocking now,  currently at LOW_NOISE_COIL_DRIVERS we should be locked by soon .

 

TITLE: 07/21 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 12mph Gusts, 9mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

Went up to the Roof to check for fires or smoke. Took some pictures in all directions.

Inherited an unlocked H1 after a large earthquake from Alaska.
Did an Initial Alignment which had an SRM WD trip towards the end. So I reran it with out any issues.
We reached NOMINAL_LOW_NOISE at 01:25 UTC
Observing at 1:27 UTC.

And Lost lock to an Unknown Lockloss at 1: 29 UTC.



 

TITLE: 07/20 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

Flurries of earthquakes from Japan.  Had another HAM6 High Voltage power supply failure.  Able to finally finish the change for removing the SR3 Dither Pitch Offset.  Shift ended with a M6.2 Alaska earthquake which took H1 down.
LOG:

• 1600 Verbal:  "nearby earthquake from Japan"  (was a M5.2).  Received several of these notifications as flurry of EQs in Japan rolled through.
• 1732 LOCKLOSS seismic motion is elevated, but doesn't look like it was EQ-caused
  • 1745 Removed SR3 Dither Pit Offset and updated safe.snap (alog)....and...
  • 1800-1822 Ran Initial Alignment with new SR3 pointing due to offset being zeroed
• 1736-1746 EQ Mode activated
  • 1813-1823 EQ Mode activated
• 1845 Noticed H1 was STUCK at PREP_DC_READOUT_TRANSITION.  This was due to the OMC not locking. 
  • 1913-2004 Phoned Elenna for help.  She eventually found out we had another HAM6 High Voltage Power Supply issue for the OMC 100V PZTs.
• 2030 OBSERVING
• 2119-2150 PCal Measurement in the PCal Lab (tony)
• 2233 LOCKLOSS due to M6.1 Alaska earthquake.

(Corey, Elenna [remote], Tony)

Recent Backstory Alogs:  

• JUN 1st:  (Corey, Craig, Georgia) HAM6 HV PZT Kepco Power Supply was stuck in the Constant Current ("CC") state (alog84698)
• JUN 13th:  (Keita, RyanC) Output Voltage was 0V and it was also in the Constant Current ("CC") state (alog85030)

Today, while locking H1 noticed that we were stuck in PREP_DC_READOUT_TRANSITION.  DIAG_MAIN was mentioning an OMC issue.  OMC_LOCK guardian was in a loop of trying to sweep the PZT to FIND_CARRIER, but Elenna noticed that the PZT was not sweeping.  Then she remmebered the previous issues!  

(I was on the wrong trail, because I had assumed the SR3 change I made somehow was the cause of the issue---it was NOT.)

• Went out to the CER and went upstairs to the rack which has the HAM6 HV OMC PZT power supply....
• It was OFF!  (see 1st photo) The rocker switch was in the OFF state.  (This is different from both cases noted above from last month.)
• Read up on Craig's and Keita's alogs
• Turned the Power Supply ON
• But had 0.0V and 0.0mA + it also was stuck in the CC state (didn't take a photo, but basically it looked like THIS from Keita's alog)
• Checked the "V SET" and saw that it was 0 V, so took this to 110V (which is also noted on the sticker on the front of the Power Supply).  Went to "I SET" and saw that it was 0.0 mA, and did not change this value.
• Still had the Constant Current .
• Eventually did another Power Cycle via the front rocker switch and this time the Power Supply came up in the Constant Voltage ("CV")--which we wanted.  (Is there an easier way to transition to this state?  I went through the MENU and did not find anything.
• WE BACK!:  At this point, we had:  CV, 110V, and a non-zero Current.  (see 2nd photo for photo of the power supply display)
• Elenna took the OMC_LOCK guardian from MANAGED  --to--> AUTO, and started moving the PZT2 in 1.0 count steps.  She saw this working on her medm and I saw the Current changing on the front display of the Power Supply.
• At this point, we were back in business and returned to the Control Room.
• ISC_LOCK was continuing on (because the OMC had locked).

We still had the OMC_LOCK in a non-ISC_LOCK-managed AUTO state and we were not totally sure of how to have ISC_LOCK remanage OMC_LOCK.  Tony found a 2015 alog from Patrick which we used to get OMC_LOCK managed again.  H1 continued on and has been in OBSERVING.

Finally had the opportunity (due to H1 Lockloss during my shift) to address the SR3 Dither Pitch Offset Oli noted last week (alog 85830), and I first looked at during a lockloss at the end of my shift Friday evening (alog 85855)---but my changes did not hold for the Dither Pitch Offset due to the H1SUSSR3 SDF needing its safe.snap changed/accepted.

It's been ages since I've updated an safe.snap, so pardon the less elegant steps I took to update the SR3 SDF here---Basically I 

1. Turned the OFFSET button OFF (then Accepted the SDF diff for the safe.snap) and then
2. Took the OFFSET from 32.3 to 0 (then Accepted the SDF diff, once again, for the safe.snap). 

Both updates are screenshot-ed separately! Ha.  The new SR3 OPTICALIGN_P_OFFSET was already at its new offset (of 457.9...which used to be 445.8) since Friday (and aligned).  So now that the SR3 pointing changed with the Dither Offset going to zero, I ran a new alignment. 

Currently, H1 has locked DRMI and I'm sure the final step here would be updated thing observe.snap with the changes noted above.

ADDENDUM:  Currently stuck at PREP_DC_READOUT_TRANSITION, where the OMC can't lock.  I'm wondering if this is due to the SR3 change....Where the SR3 Top Mass was at one spot since Fri evening, and now with the Dither Offset being zeroed, we now have SR3 Top Mass back to it's pointing we had for the last few weeks up until Friday night.  (see attached ndscope trend over the weekend)

Sun Jul 20 10:08:56 2025 INFO: Fill completed in 8min 52secs

Prompted by me noticing on-off behaviors in the daily strain spectrogram for today at around 20.2 Hz, I've done some additional investigations into the source and behavior of this line: 

The 20.2 Hz line, which is currently prominent in DARM, first appeared in accelerometer and microphone data from the corner station on June 9. The first appearance of this line that I found was in the PSL mics, as shown in this spectrogram. This line then appeared in DARM in the first post-vent locks a few days later. The summary of work from June 9 does not show anything obvious to me that would be the source of this new noise.

This feature also turns off and on multiple times during the day. An example from today can be seen in this spectrogram. Most corner station microphones and accelerometers exhibit this feature, but it is most pronounced visually in the PSL microphone spectrograms. I was unable to identify any other non-PEM channels that showed the same on-off behavior, but this does reveal many change points that should aid in tracking down the source. Almost every day, this line exhibits abrupt on-off features at different times of the day and for varying durations. Based on my initial review, these change points appear to be more likely during the local daytime (although not at any specific time).  When the line first appeared, it was usually in the "off" state and then turned on for short periods. However, this has slowly changed, so that now the line is generally in the "on" state and turns off for brief periods. 

Sheila, Camilla

We ran a couple of squeezing angle scans to check the settings of the ADF servo. 

One thing that we realized is that the ADF Q demod signal is divided by H1:SQZ-ADF_OMC_TRANS_Q_NORM rather than mulitplied which is what we had thought.  We changed the coefficent from 0.18 to 5.8. The first png attachment shows that this transforms the blue ellipse into the orange one.  It would be a bit better if we first adjusted the demod phase to maximize the Q signal, so that the ellipse would be aligned along the axis, and the rescaled version would be more like a circle.  However you can see in the right side plot that this gives us a reasonably linear readback of sqz angle as we change the RF6 demod angle (which is actually cabled up to RF3 phase) about 150 degrees where our best squeezing is.

Camilla turned the servo back on in sqzparams. 

For future reference, a slightly better way to do this would be to move the demod phase to maximize Q, do a scan and set H1:SQZ-ADF_OMC_TRANS_Q_NORM to the ratio (max of Q)/ (max of I).  Then you can do a smaller scan around the point with the best squeezing, and in sqzparams set sqz_ang_adjust_ang to the readback angle that you think is best.

I previously noted a glitch about 30 seconds before lockloss in LOWNOISE ASC, 85685. However, we had two more locklosses from this state last night and I do not see such a glitch so that is a random coincidence. One of those locklosses appears to be caused by an earthquake. However, since 6/11, we have had 9 locklosses in this state that occurred exactly 47 seconds into the state, which seems suspicious, one of those occurred last night, and the lockloss with the glitch was the same.

This seems to be coincident with the engagement of a few DHARD P filters:

2025-07-14_14:26:54.641531Z ISC_LOCK executing state: LOWNOISE_ASC (522)
2025-07-14_14:26:54.642230Z ISC_LOCK [LOWNOISE_ASC.enter]
2025-07-14_14:26:54.655894Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_PIT3_OSC_CLKGAIN => 300
2025-07-14_14:26:54.656325Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_PIT4_OSC_CLKGAIN => 300
2025-07-14_14:26:54.656732Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_PIT5_OSC_CLKGAIN => 300
2025-07-14_14:26:54.657043Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_YAW3_OSC_CLKGAIN => 300
2025-07-14_14:26:54.657438Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_YAW4_OSC_CLKGAIN => 300
2025-07-14_14:26:54.657892Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_YAW5_OSC_CLKGAIN => 300
2025-07-14_14:26:54.658134Z ISC_LOCK [LOWNOISE_ASC.main] timer['LoopShapeRamp'] = 5
2025-07-14_14:26:54.658367Z ISC_LOCK [LOWNOISE_ASC.main] timer['pwr'] = 0.125
2025-07-14_14:26:54.783581Z ISC_LOCK [LOWNOISE_ASC.run] timer['pwr'] done
2025-07-14_14:26:59.658298Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:26:59.719537Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_Y_GAIN => 200
2025-07-14_14:26:59.720456Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_Y_SW1 => 256
2025-07-14_14:26:59.846249Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_Y_SW2 => 20
2025-07-14_14:26:59.971686Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_Y => ON: FM3, FM8, FM9
2025-07-14_14:26:59.972384Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_Y_SW1 => 5392
2025-07-14_14:27:00.098073Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_Y_SW2 => 4
2025-07-14_14:27:00.223528Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_Y => ON: FM1, FM3, FM4, FM5, FM8
2025-07-14_14:27:00.224135Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CSOFT_P_SMOOTH_ENABLE => 0
2025-07-14_14:27:00.224497Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CSOFT_Y_SMOOTH_ENABLE => 0
2025-07-14_14:27:00.224868Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DSOFT_P_SMOOTH_ENABLE => 0
2025-07-14_14:27:00.225188Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DSOFT_Y_SMOOTH_ENABLE => 0
2025-07-14_14:27:00.225433Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 10
2025-07-14_14:27:10.225728Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:27:10.281803Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P_TRAMP => 5
2025-07-14_14:27:10.408120Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P_SW2 => 16
2025-07-14_14:27:10.533563Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P => OFF: FM9
2025-07-14_14:27:10.534285Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P_SW1 => 256
2025-07-14_14:27:10.660088Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P_SW2 => 4
2025-07-14_14:27:10.785453Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P => ON: FM3, FM8
2025-07-14_14:27:10.786315Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P_GAIN => 208
2025-07-14_14:27:10.786535Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 5
2025-07-14_14:27:15.786858Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:27:15.847152Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DSOFT_Y_TRAMP => 5
2025-07-14_14:27:15.847580Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DSOFT_Y_GAIN => 5
2025-07-14_14:27:15.848666Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DSOFT_P_GAIN => 5
2025-07-14_14:27:15.848917Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 5
2025-07-14_14:27:20.849050Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:27:20.906577Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ITMX_M0_DAMP_Y_TRAMP => 10
2025-07-14_14:27:20.907206Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ETMX_M0_DAMP_Y_TRAMP => 10
2025-07-14_14:27:20.907700Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ITMY_M0_DAMP_Y_TRAMP => 10
2025-07-14_14:27:20.908422Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ITMX_M0_DAMP_Y_GAIN => -0.5
2025-07-14_14:27:20.908830Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ETMX_M0_DAMP_Y_GAIN => -0.5
2025-07-14_14:27:20.909148Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ITMY_M0_DAMP_Y_GAIN => -0.5
2025-07-14_14:27:20.909562Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ETMY_M0_DAMP_Y_GAIN => -0.5
2025-07-14_14:27:20.909789Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 10
2025-07-14_14:27:30.910055Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:27:30.968166Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_P_GAIN => -0.2
2025-07-14_14:27:30.968527Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_Y_GAIN => -0.2
2025-07-14_14:27:30.968806Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_L_GAIN => -0.2
2025-07-14_14:27:30.969073Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_R_GAIN => -0.2
2025-07-14_14:27:30.969343Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_T_GAIN => -0.2
2025-07-14_14:27:30.969606Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_V_GAIN => -0.2
2025-07-14_14:27:30.969838Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 10
2025-07-14_14:27:40.970003Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:27:40.972085Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_P_SW1 => 1024
2025-07-14_14:27:41.097962Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_P_SW2 => 4
2025-07-14_14:27:41.223313Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_P => ON: FM4, FM8
2025-07-14_14:27:41.223637Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 10
2025-07-14_14:27:41.593743Z ISC_LOCK [LOWNOISE_ASC.run] Unstalling IMC_LOCK
2025-07-14_14:27:41.765955Z ISC_LOCK JUMP target: LOCKLOSS

I will take a look and see if there is anything unstable about these filters. Whatever is occurring seems to be too fast to be seen in the ASC signals themselves, and at first glance I don't see anything strange in the suspension channels either.

DHARD FM4 is engaged with a 10 second ramp- this is a change I made on 6/11: 84973 because we had lost lock on that day twice in the same spot. Two of the locklosses at 47 seconds occurred before that change. Then, later that day on 6/11 I reengaged a boost in DHARD P, which only has a 5 second ramp, 84980. Engaging that boost shouldn't be unstable, but maybe something bas occurrs when they ramp at different times. I'm lengthing the ramp to 10 seconds.