Naoki, Sheila, Elenna

We tested high NLG. We increased the OPO trans power from 85uW to 105uW, which corresponds to NLG of 22.7 and generated squeezing of 18.3dB according to the NLG calculator. We measured sqz/asqz with high NLG as shown in the attached figure. The squeezing above 1 kHz slightly improved, but the squeezing below 500 Hz got worse. 

At 1kHz, squeezing is 3.6dB, mean squeezing is 13.5dB, and anti squeezing is 16.7dB.

We suspect that the degradation below 500 Hz is due to the SRCL detuning so we tried to change the SRCL offset. When we changed the SRCL offset from -175 to -150, squeezing below 500 Hz slightly improved. So the degradation below 500 Hz is likely due to the SRCL detuning. We also tried the SRCL offset of -125 and -100, but they are almost the same as -150. Everytime we changed the SRCL offset, we adjusted the sqz angle to maximize the squeezing at high frequency.

How to increase the pump power

• original SHG launch power is 23.1 and SHG rejected power is 4.9 with ISS
• request OPO guardian to LOCKED CLF DUAL NO ISS
• rotate HWP to minimize the rejected SHG power
• change the ISS drivepoint from 5 to 3 to keep the ISS control monitor around 3. The SHG launch power reached around 28.
• change the ISS setpoint in sqzparams to 105, load the OPO guardian, and request OPO guardian to LOCKED CLF DUAL

Noise between 15 and 50 Hz seems to be highly non-stationary, as shown in the median-whitened spectrogram attached, computed over 9 hours of data.

TITLE: 07/05 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 6mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.08 μm/s 
QUICK SUMMARY:

Inherited from Tony in Nominal_Low_Noise, detector has been locked for 17:43 and just came out of commissioning. Now in Observing.

Leftover from last Wednesday's update to the LSC iterative feedforward was another update to the MICH FF. I was able to test it today. Attached is a screenshot of the injections. The blue reference trace is the injection with no MICH feedforward on (SRCL FF on). The green trace was our previous MICH feedforward and the red live trace is the new MICH feedforward. There is a small improvement from 20-30 Hz, so we are going with the new feedforward. Old FF filter was in FM5 ("6-22b-23"), new feedforward filter is in FM3 ("6-28b-23"). The update is SDFed and guardianized.

Elenna, Sheila, Brina

When we engaged the OM2 heater on June 27th, we saw a reduction in input jitter coupling to DARM (70864) and a shift in OMC alignment (70886).  Today we attempted to recreate the changes we saw when turning on the TSAMS by shifting the OMC alignement, by watching IMC PZT lines, OM1 + OM3 dither lines, and the optical gain. We see that the alignment shifts are the right size to explain what happened when the TSAMs came on, but the optical gain shifts from alignment are smaller. We also see that the aligments that reduce input jitter aren't the same as those that reduce output jitter.

The first screenshot shows the alignment shift that happened when we first turned on OM2 heater.  Today we added offsets into OMC QPD A and B, we found that adding an offset of about 0.03 to OMC B made a shift somewhat simliar to what happened when OM2 was heating up (second attachment shows a step which was roughly undoing the alignment shift from June 27th).  We saw that there was a 13.8% decrease in IMC PZT P coupling to DARM when we undid that step.  On the 27th there was a 26% decrease in jitter coupling to DARM looking at the known peak around 120Hz.  So the change that we see in jitter coupling from this alignment shift has the right order of magnitude to explain the change in jitter coupling on the 27th, although confusingly we improved the jitter by undoing the move.  This move also made the coupling of the OM1 pitch dither line to DARM worse, while OM3 got slightly better. (3rd screenshot attached)

We quickly tried to move this DOF to see if we could further reduce the jitter coupling, we weren't able to get much more of an improvement. 

During all of our moves today we saw small changes in optical gain, for OMC alignment changes that were comparable in size to what happened on the 27th the optical gain changes were smaller than the 2% drop seen with TSAMs.  

TITLE: 07/05 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 8mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

Inherited 9.75 hour lock.
19:00 Commissioning Time!!
We are in Comissioning State for some Calibration Measurements, Squeezing updates, and a filter change for LSC Mitch.
19:00 UTC Calibration Measurements.
20:00 UTC OMC changes
22:27 SEI Measurements on HAM3.
22:30 SQUEEZE Guardian changes and reload.  
22:43 UTC LSC Mitch changes.

Current IFO Status: Locked in NOMINAL_LOW_NOISE for 17.5 Hours and Just got back to OBSERVING,

Tony, Jonathan, Erik, Keith T, Mike T, Dave:

FRS-28,463

Executive Summary:

psinject spontaneously restarted itself due to an out-of-memory problem on h1hwinj1 yesterday morning.

This was a result of the upgrade of the system on Tue 27 June 2023. It takes 7 days to run out of memory.

In the short term (over next few days) we will monitor the memory usage, and restart the psinject process during a lock_loss event to make it through to next week.

Next Tuesday we will downgrade to the original version of the LAL pulsar binary.

Details:

On Tuesday 4th July 2023 at 10:32 the psinject process on h1hwinj1 cleanly shutdown and was then restarted by monit. The shutdown and restart of this process involves ramping the output of the INJ_CW filtermodule on h1calinj, which takes H1 out of observation mode.

Looking through the logs we found that h1hwinj1 had slowly ran out of memory over the 7 days since the upgrade of the code on Tuesday 27th June 2023.

We did a quick estimate of the memory leak rate between 10am and 2pm today and came up with 1MB/min. Starting with 10GB of free memory, at this rate the memory is exhausted in about 7 days, which is what we saw.

Last Tuesday several things were upgraded on h1hwinj1:

1. psinject code was changed to use gpstime instead of tconvert (makes LHO = LLO)

2. python3 version increased from 3.4 to 3.6 (makes LHO = LLO)

3. lalapps was upgraded from 6.25 to 9.2 (not done at LLO)

We think it is most probably the lalapps upgrade which is causing the memory leak. LLO did the first two upgrades two weeks ago on l1hwinj1 and are not seeing any memory issues.

In lalapps 6.25 /usr/bin/lalapps_Makefakedata_v4 is a 70K binary. In lalapps 9.2.1 is is a launcher script, spawning /usr/bin/lalpulsar_Makefakedata_v4 which is a 53K binary. It also issues the warning

"WARNING: 'lalapps_Makefakedata_v4' has been renamed to 'lalpulsar_Makefakedata_v4'"

Actions:

We will restart psinject before it stops itself next Tuesday during an appropriate lock loss time.

Next Tuesday we will downgrade LAL from 9.2.1 to 6.25.1 so LHO and LLO are identical.

Over the weekend we ran into a few times (alog71043, alog71026, alog71008) that we tried to get data via cdsutils getdata function in an ISC_LOCK guardian state, and it returned nothing. This caused an error in ISC_LOCK, fixed by simply reloading the node since the function just had to try again to get the data. This is not a new thing, but it's definitely another reminder that we have to be prepared for different outcomes anytime we request data.

Some months ago I made with Jonathan's help, a function wrapper that can be used to handle hung data grabs. While not the issue we saw over the weekend, it's still a good idea to use this whenever we try getting data in a Guardian node. The file is (userapps)/sys/h1/guardian/timeout_utils.py and there is either a decorator (@timeout) or a wrapper function (call_with_timeout) than can be used.

For the specific issue we saw over the weekend, a solution is to just do a simple check that the data is actually there before trying to do anything with it (ie. if data:). Using this situation as a good example:

# This wrapper should handle hung nds data grabs

popdata_prmi = call_with_timeout(cdu.getdata, 'LSC-POPAIR_B_RF90_I_ERR_DQ', -60)

# This conditional handles None data returned

if popdata_prmi.data:

    if popdata_prmi.data.max() < 20:

        log('no POPAIR RF90 flashes above 20, going to CHECK MICH FRINGES')

        return 'CHECK_MICH_FRINGES'

    else:

        self.timer['PRMI_POPAIR_check'] = 60

Erik, TJ, Dave:

TJ found that gpstime on h1guardian1 is warning of an expired leapseconds file. The file in question is /usr/share/zoneinfo/leapseconds which has an expiration time of 1687910400 in UNIX seconds which is Tue 27 Jun 2023 05:00:00 PM PDT.

Currently H1 is locked for commissioning, so we don't want to do any updates on h1guardian1 at this time. As a temporary measure I increased the expiration time to 1787910400, Fri 28 Aug 2026 02:46:40 AM PDT by manually editing the file.

Current IFO Status: LOCKED in NLN_CAL_MEAS & Not OBSERVING
We are in Comissioning State for some Calibration Measurements, Squeezing updates, and a changes for LSC Mitch and OMC.

Wed Jul 05 10:05:49 2023 INFO: Fill completed in 5min 49secs

Gerardo confirmed a good fill from curbside.

TITLE: 07/05 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 144Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 6mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:
Inherited an IFO that hasbeen locked for 9.75 hours.

TITLE: 07/05 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
SHIFT SUMMARY: Very quiet shift mostly just monitoring violin mode damping. H1 has now been locked and observing for 9.5 hours.

• Engaged damping for ETMX violin mode 19 according to full power settings found in the spreadsheet
  • This setting is working, but very slowly; likely needs further testing

Handing off to Tony for the day.

LOG:

No log for this shift.

FAMIS 19665, last checked in alog 70845

ITMX_ST2_CPSINF_V1 high freq noise is high!

State of H1: Observing at 144Mpc

A couple of small earthquakes rolling through, otherwise a very quiet night. Violin damping has been going slowly, but surely. The new settings for ETMX mode 19 (FM1, FM2, FM10 w/ gain of -30) have been working, albeit very slowly.

TITLE: 07/04 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 13mph Gusts, 9mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

16:00 Acknowledge Test T415139 SNEWS.  
17:31 H1:CAL-INJ_MASTER_SW was changed somehow, and took us into commissioning. A moment later it was undone and I took H1 back to observing.

17:54 Lockloss
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1372528500

Relocking without Initial Alignment.
18:45 OMC WHITENING
20:23 NOMINAL_LOW_NOISE & Observing

22:22 Lockloss
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1372544545
 

Current IFO Status: Relocking.

J. Kissel

To help out studies of the end station VEA temperatures and HVAC system, I've created templates of the relevant channels similar to those described in LHO:70284, but for the X and Y-end VEAs.

You can find them in 
    /ligo/home/jeffrey.kissel/Templates/NDScope/
        xvea_temp_study.yaml
        yvea_temp_study.yaml