The TRANSITION_BACK_TO_ETMX state of the SUS_CHARGE  guardian has been causing the majority of our locklosses prior to Tuesday Maintenance, e.g. 69437, 70861.

I looked into this again and found that the ITMX LOCK_L gain is being turned off before the ITMX to ETMX transition has completed. The ramp time of the swap is 20 seconds but the code only waits 15 seconds before moving on, see attached plot and code.

We are using a ezca.get_LIGOFilter and the wait=True parameter isn't waiting the full ramp_time specified, created Issue 15. I've changed the wait to be False and added a 20s sleep timer. The SUS_CHARGE guardian will need to be reloaded when we are out of observing. 

I just turned off the EX19 damping Ryan and I turned on last night (FM1,2,10 Gain -30 71066) as the mode seemed to be very slowly turning around and the damping was so slow anyway, see attached. Tagging SUS.

Following up on the DARM bicoherence observation and the non-stationarity of low frequency noise: the noise in DARM between 20 and 40 Hz is correlated with the amplitude of the 2.6 Hz peak

The attached plot is an histogram of the DARM RMS in the 20-37 Hz region (computed by summing bins in a whitenend spectrogram) and the DARM RMS around the 2.6 Hz peak (computed with a band-pass filter between 2.4 and 2.7 Hz).

There is a clear correlation between the DARM noise in the 20-40 Hz region and the RMS in the 2.6 Hz region.

Summary of the report:

• The summary page is not updated after 10:00 UTC for 26th June 2023. The range plot is also not available after 06:00 UTC for 2nd July 2023.
• The average range throughout the week around 140 Mpc.
• Some of the locklossed during this week were due to ground motion, power glitches or maintenance. Some locklosses did not have any apparent reason.
• There is a constant high noise (~500 Hz) strain line present in the h(t) spectrogram on several days, except Wednesday and Thursday. No apparent cause found.
• The glitchgram showed an increased number of glitches on Saturday and Sunday. Mostly ground and wind motion were responsible for this.
• The noises in the BSCs got better during the end of the week.
• The EY accoustic showed low noises (~0-100 Hz) throughout the week, at different times of the day. Some of them can be attributed to ground motion and wind. Also, they might be the reason behind the glitches and the strain line noises.
• Hveto: H1:PEM-EX_VMON_ETMX_ESDPOWER48_DQ channel was the first round winner for everyday. The new channels are H1:LSC-REFL_A_LF_OUT_DQ, H1:SUS-ETMX_L3_OPLEV_PIT_OUT_DQ, H1:PEM-EX_VMON_ETMX_ESDPOWER18_DQ. The third channel had appeared in the previous week.
• Fscan: All the plots have appeared. Nothing significantly new.

Link to the full report: Report

V. Bossilkov, L. Dartez

Summary
We gave the simulines calibration injection scheme another go at LHO during this week's commissioning window. In short, the commissioning endeavor was partially successful: we did not lose lock and we were able to gather useful data for the actuation stages and the DARM OLG. However, additional commissioning time is required before we can fully adopt simulines for the regular calibration measurements at LHO.

Objective
The goal of today's session included the following:
1.) Determine whether we can run simulines at LHO reliably without losing lock,
2.) assess the data quality of the recorded simulines hdf5 files and confirm that they can be read and post-processed
3.) evaluate the spacing of the common frequency vector that is used for all measurements
4.) tweak the injection amplitudes and integration length to get a good enough SNR while keeping the amplitudes and integration lengths as low as possible

Code Execution
We ran two trials with injections at different amplitudes: 0.5*A and 0.1*A, where A is the corresponding amplitude typically used for the same injections in DTT. Simulines uses a configuration file in .INI format to inform the various injections of their frequency vectors, amplitudes, and other settings. The two files we used, both of which are located in /ligo/groups/cal/src/simulines/h1_test_settings, are named as follows:

Half the DTT amplitude: settings_h1_0.5.ini
One tenth the DTT amplitude: settings_h1_0.1.ini

These files are contained within the simulines repo as of commit bf25b1b4.

The simulines code was run using the following command from within the repo's simulines directory:
python simulines.py -i ../h1_test_settings/settings_h1_0.5.ini

The terminal output for both runs is attached here. All measurements were placed in /ligo/groups/cal/H1/measurements as they would during a normal pydarm measure execution.

Once recorded, we ran a quick script that Vlad wrote to take a look at each measurement's uncertainty as it compares to those taken with DTT. The script used lives at /ligo/home/louis.dartez/projects/20230705/simulines_commissioning/simulinesUncert_H1.py. This script compares each simulines injection with a corresponding DTT injection. The exact DTT injections used for this initial test run are listed in the script source code (uploaded to this alog for redundancy).

Results
We were able to successfully run simulines for all of the normal calibration measurement sweeps (we did not take any broadband measurements). The table below contains plots comparing the uncertainty from the simulines (SL) injections vs the same using diaggui (DTT). The top plot in each figure shows an overlay of the uncertainty from the SL injection and the DTT injection. The bottom plot contains the residual (SL_unc / DTT_unc). The left and right columns correspond to the 0.5*"DTT Amplitude" and 0.1*"DTT Amplitude", respectively. Each actuation stage is displayed on its own row.



0.5*Ampl
0.1*Ampl
ETMX L1


ETMX L2


ETMX L3






For a first shot at taking a serious look at demo-ing and commissioning the simulines tool at LHO, we have been quite successful. The uncertainty comparisons below tell us that we're able to achieve comparable uncertainty in several parts of the band with a significantly lower amplitude than what we've been using in DTT (this isn't a statement about the tools, rather that it's about time we take a new look at the injection amplitudes). However, we will need to zoom in a bit more and study the data we've already recorded to determine where and by how much to adjust our injection amplitudes to maintain an acceptable coherence for all measurements. 

Unfortunately, we weren't able to plot up any of the usual transfer functions today because of an issue (bug?) that prevented us from processing any of the PCal measurements. Luckily, the data is thought to be processable once a few kinks get ironed out with the simulines processing code. The current theory is that a bug in the recording script caused nonsensical coherence values to be recorded in the hdf5 output files. 

I'm going to continue working with Vlad and the calibration team to get this up and running.

== Relevant Logs ==
simulines was first tried out at LHO (LHO:66284) 

Vlad lays out initial plan for testing simulines at LHO (LHO:70998)

Naoki, Sheila, Vicky.  B/c of recent issues with SQZ_ASC not engaging (eg 71057, 70890), range can hover ~120 instead of >145 (!!). So, we added a state to SQZ_MANAGER that operators can use to clear asc, "RESET_SQZ_ASC". See new guardian graph (vs old graph).

Tagging OpsInfo: If range hovers low ~120 and SQZ_MANAGER has the yellow notification like "SQZ ASC AS42 not on?? Please RESET_SQZ_ASC", can do the following (~10 seconds):

• From the drop-down menu on SQZ_MANAGER,
• Select "RESET_SQZ_ASC" (right under FDS),
• Wait for it to complete (~5 seconds), then
• Re-select "FREQ_DEP_SQZ"

If this doesn't work, the following is a longer solution (~5 min) which includes resets the squeezer ASC offsets, in case that's the problem:

• Take SQZ_MANAGER to "NO_SQUEEZING"
• From sitemap > SQZ > SQZ ASC IFO > see screenshot circling buttons to click:
  • First, purple button that says "! reset AS42 offsets". Wait for script to run (it averages for ~30 sec). Log/post a screenshot of the output if you can. Close pop-up terminal when done.
  • Next, purple button for "! graceful clear history". Close terminal when finished.
• Re-select "FREQ_DEP_SQZ" to re-inject squeezing.

Usually our ASC offsets don't change much, unless IFO output alignment or beam changes. SVN code diffs attached, and committed to version 25949.

Naoki, Sheila, Elenna

We tested high NLG. We increased the OPO trans power from 85uW to 105uW, which corresponds to NLG of 22.7 and generated squeezing of 18.3dB according to the NLG calculator. We measured sqz/asqz with high NLG as shown in the attached figure. The squeezing above 1 kHz slightly improved, but the squeezing below 500 Hz got worse. 

At 1kHz, squeezing is 3.6dB, mean squeezing is 13.5dB, and anti squeezing is 16.7dB.

We suspect that the degradation below 500 Hz is due to the SRCL detuning so we tried to change the SRCL offset. When we changed the SRCL offset from -175 to -150, squeezing below 500 Hz slightly improved. So the degradation below 500 Hz is likely due to the SRCL detuning. We also tried the SRCL offset of -125 and -100, but they are almost the same as -150. Everytime we changed the SRCL offset, we adjusted the sqz angle to maximize the squeezing at high frequency.

How to increase the pump power

• original SHG launch power is 23.1 and SHG rejected power is 4.9 with ISS
• request OPO guardian to LOCKED CLF DUAL NO ISS
• rotate HWP to minimize the rejected SHG power
• change the ISS drivepoint from 5 to 3 to keep the ISS control monitor around 3. The SHG launch power reached around 28.
• change the ISS setpoint in sqzparams to 105, load the OPO guardian, and request OPO guardian to LOCKED CLF DUAL

Noise between 15 and 50 Hz seems to be highly non-stationary, as shown in the median-whitened spectrogram attached, computed over 9 hours of data.

TITLE: 07/05 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 6mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.08 μm/s 
QUICK SUMMARY:

Inherited from Tony in Nominal_Low_Noise, detector has been locked for 17:43 and just came out of commissioning. Now in Observing.

Leftover from last Wednesday's update to the LSC iterative feedforward was another update to the MICH FF. I was able to test it today. Attached is a screenshot of the injections. The blue reference trace is the injection with no MICH feedforward on (SRCL FF on). The green trace was our previous MICH feedforward and the red live trace is the new MICH feedforward. There is a small improvement from 20-30 Hz, so we are going with the new feedforward. Old FF filter was in FM5 ("6-22b-23"), new feedforward filter is in FM3 ("6-28b-23"). The update is SDFed and guardianized.

Elenna, Sheila, Brina

When we engaged the OM2 heater on June 27th, we saw a reduction in input jitter coupling to DARM (70864) and a shift in OMC alignment (70886).  Today we attempted to recreate the changes we saw when turning on the TSAMS by shifting the OMC alignement, by watching IMC PZT lines, OM1 + OM3 dither lines, and the optical gain. We see that the alignment shifts are the right size to explain what happened when the TSAMs came on, but the optical gain shifts from alignment are smaller. We also see that the aligments that reduce input jitter aren't the same as those that reduce output jitter.

The first screenshot shows the alignment shift that happened when we first turned on OM2 heater.  Today we added offsets into OMC QPD A and B, we found that adding an offset of about 0.03 to OMC B made a shift somewhat simliar to what happened when OM2 was heating up (second attachment shows a step which was roughly undoing the alignment shift from June 27th).  We saw that there was a 13.8% decrease in IMC PZT P coupling to DARM when we undid that step.  On the 27th there was a 26% decrease in jitter coupling to DARM looking at the known peak around 120Hz.  So the change that we see in jitter coupling from this alignment shift has the right order of magnitude to explain the change in jitter coupling on the 27th, although confusingly we improved the jitter by undoing the move.  This move also made the coupling of the OM1 pitch dither line to DARM worse, while OM3 got slightly better. (3rd screenshot attached)

We quickly tried to move this DOF to see if we could further reduce the jitter coupling, we weren't able to get much more of an improvement. 

During all of our moves today we saw small changes in optical gain, for OMC alignment changes that were comparable in size to what happened on the 27th the optical gain changes were smaller than the 2% drop seen with TSAMs.  

TITLE: 07/05 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 8mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

Inherited 9.75 hour lock.
19:00 Commissioning Time!!
We are in Comissioning State for some Calibration Measurements, Squeezing updates, and a filter change for LSC Mitch.
19:00 UTC Calibration Measurements.
20:00 UTC OMC changes
22:27 SEI Measurements on HAM3.
22:30 SQUEEZE Guardian changes and reload.  
22:43 UTC LSC Mitch changes.

Current IFO Status: Locked in NOMINAL_LOW_NOISE for 17.5 Hours and Just got back to OBSERVING,

Tony, Jonathan, Erik, Keith T, Mike T, Dave:

FRS-28,463

Executive Summary:

psinject spontaneously restarted itself due to an out-of-memory problem on h1hwinj1 yesterday morning.

This was a result of the upgrade of the system on Tue 27 June 2023. It takes 7 days to run out of memory.

In the short term (over next few days) we will monitor the memory usage, and restart the psinject process during a lock_loss event to make it through to next week.

Next Tuesday we will downgrade to the original version of the LAL pulsar binary.

Details:

On Tuesday 4th July 2023 at 10:32 the psinject process on h1hwinj1 cleanly shutdown and was then restarted by monit. The shutdown and restart of this process involves ramping the output of the INJ_CW filtermodule on h1calinj, which takes H1 out of observation mode.

Looking through the logs we found that h1hwinj1 had slowly ran out of memory over the 7 days since the upgrade of the code on Tuesday 27th June 2023.

We did a quick estimate of the memory leak rate between 10am and 2pm today and came up with 1MB/min. Starting with 10GB of free memory, at this rate the memory is exhausted in about 7 days, which is what we saw.

Last Tuesday several things were upgraded on h1hwinj1:

1. psinject code was changed to use gpstime instead of tconvert (makes LHO = LLO)

2. python3 version increased from 3.4 to 3.6 (makes LHO = LLO)

3. lalapps was upgraded from 6.25 to 9.2 (not done at LLO)

We think it is most probably the lalapps upgrade which is causing the memory leak. LLO did the first two upgrades two weeks ago on l1hwinj1 and are not seeing any memory issues.

In lalapps 6.25 /usr/bin/lalapps_Makefakedata_v4 is a 70K binary. In lalapps 9.2.1 is is a launcher script, spawning /usr/bin/lalpulsar_Makefakedata_v4 which is a 53K binary. It also issues the warning

"WARNING: 'lalapps_Makefakedata_v4' has been renamed to 'lalpulsar_Makefakedata_v4'"

Actions:

We will restart psinject before it stops itself next Tuesday during an appropriate lock loss time.

Next Tuesday we will downgrade LAL from 9.2.1 to 6.25.1 so LHO and LLO are identical.

Over the weekend we ran into a few times (alog71043, alog71026, alog71008) that we tried to get data via cdsutils getdata function in an ISC_LOCK guardian state, and it returned nothing. This caused an error in ISC_LOCK, fixed by simply reloading the node since the function just had to try again to get the data. This is not a new thing, but it's definitely another reminder that we have to be prepared for different outcomes anytime we request data.

Some months ago I made with Jonathan's help, a function wrapper that can be used to handle hung data grabs. While not the issue we saw over the weekend, it's still a good idea to use this whenever we try getting data in a Guardian node. The file is (userapps)/sys/h1/guardian/timeout_utils.py and there is either a decorator (@timeout) or a wrapper function (call_with_timeout) than can be used.

For the specific issue we saw over the weekend, a solution is to just do a simple check that the data is actually there before trying to do anything with it (ie. if data:). Using this situation as a good example:

# This wrapper should handle hung nds data grabs

popdata_prmi = call_with_timeout(cdu.getdata, 'LSC-POPAIR_B_RF90_I_ERR_DQ', -60)

# This conditional handles None data returned

if popdata_prmi.data:

    if popdata_prmi.data.max() < 20:

        log('no POPAIR RF90 flashes above 20, going to CHECK MICH FRINGES')

        return 'CHECK_MICH_FRINGES'

    else:

        self.timer['PRMI_POPAIR_check'] = 60

Erik, TJ, Dave:

TJ found that gpstime on h1guardian1 is warning of an expired leapseconds file. The file in question is /usr/share/zoneinfo/leapseconds which has an expiration time of 1687910400 in UNIX seconds which is Tue 27 Jun 2023 05:00:00 PM PDT.

Currently H1 is locked for commissioning, so we don't want to do any updates on h1guardian1 at this time. As a temporary measure I increased the expiration time to 1787910400, Fri 28 Aug 2026 02:46:40 AM PDT by manually editing the file.

J. Kissel

To help out studies of the end station VEA temperatures and HVAC system, I've created templates of the relevant channels similar to those described in LHO:70284, but for the X and Y-end VEAs.

You can find them in 
    /ligo/home/jeffrey.kissel/Templates/NDScope/
        xvea_temp_study.yaml
        yvea_temp_study.yaml