Camilla, Vicky

Screenshot of how we found the SQZ_FC and SQZ_OPO this AM. Two separate issues.

Only to-do might be fixing the SQZ_FC issue in FIND_IR. I hope the SQZ_OPO_LR issue is solved, but we will watch out for it.

More description of the issues from this morning/last night and how to fix it:

• SQZ_FC -  same stuck in FIND_IR issue as yesterday 76862. See the execution time over-run circled in today's guardian screenshot. 
   
  • To fix: brought SQZ_FC to "DOWN". That was it, SQZ_MANAGER took over the rest.
     
  • To-do: Unsure why we have gotten multiple timeouts in the past 2 days, try to resolve this. 
    • Maybe in SQZ_FC, on Line 112, we need to wrap the cdu.getdata() function call in TJ's timeout checker function. 
    • In the meantime, I have added a log statement to Line 111 just before cdu.getata() is called, and reloaded SQZ_FC. So if FIND_IR gets stuck again, we can confirm whether it is really this function call that times out the guardian.
       
• SQZ_OPO_LR - did not scan to high enough PZT voltage to find co-resonance. Hopefully we have now increased the scan range enough to fix this for O4b. Steps taken today:

  1. From it being STALLED and MANAGED, I took SQZ_OPO_LR to AUTO. This worked to un-stall it, and let it continue locking. If the scan is fixed, this should be enough to solve the problem.
      --> Not sure why OPO guardian was "STALLED" when it jumped to "IDLE" state, but I think this is related to the guardian "jump" to IDLE. I'm not sure if we even want to fix this -- leaving the SQZ_OPO_LR guardian in STALLED was probably the correct move, otherwise it would have spent hours scanning the PZT without finding co-resonance. Better to fix issues with not finding co-resonance, maybe not much to do here.

  2. OPO couldn't lock on co-resonance b/c that was at 104V, but PZT1 was only scanning up until 100V -- I changed the scan offset.

     • OPO did not find co-resonance within its scan range, then it timed out, jumped into IDLE, and the guardian STALLED. See sqz locking screenshot -- increasing the scan range let it find co-resonance at 104V, and then SQZ_MANAGER easily went back up to FDS.

     • I increased H1:SQZ-OPO_PZT_1_OFFSET from 0V to 15V, and saved this 15V offset in SDF. We cannot increase H1:SQZ-OPO_PZT_1_SCAN_STOP above 100V in slowcontrols, so we have to increase the scan offset to get the PZT there.

       • Note, back when OPO_PZT2 was at 0V (before 76688), I think the OPO would often find co-resonance between 100-110 V during O4a. 

     • Based on a previous opo co-resonance issue, 76642 seemed to want to the scan starting around ~50V. Today's issue shows we want to scan beyond 105V, probably up to 110V or even 115V (the max). Based on OPO cavity scans (e.g. screenshot from LHO:75285), I think OPO PZT1 is not super effective (in terms of um/V scanning) below 50V or so.

     • Let's try scanning from 55V - 115V to guarentee we find co-resonance. This is how it's currently set. This corresponds to the first screenshot, where H1:SQZ-OPO_PZT_1_OFFSET = 15V (saved in SDF), H1:SQZ-OPO_PZT_1_SCAN_START = 40, H1:SQZ-OPO_PZT_1_SCAN_START = 100V.

TITLE: 04/02 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.25 μm/s
QUICK SUMMARY:

H1 lost lock 1458 (Camilla thinks the in-lock charge measurement might have run...but caused the lockloss due to New-DARM.).

MAINTENANCE has begun!

Paused TCS_ITM_CO2_PWR guardians so that when the IFO unlocks the CO2s will stay on so we can take absorption measurements of the ITMs using HWS (need ITMs and SR2 to stay aligned for 1 hour after lockloss). 

Workstations were updated and rebooted.  This was an OS package update.  No Conda packages were changed.

TITLE: 04/02 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
INCOMING OPERATOR: None
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as of 23:00 UTC (8hr 36 min lock)

Nothing else of note

LOG:

Here are the highlights of my data quality shift report for the week of 3/25-3/31.

• Duty cycle of 49.7% for the week between ~140-160 Mpc
• Roughly hour long periods of very low frequency (<30 Hz) noise in the strain which could not be reliably seen in seismic BLRMS or PEM overviews (see figure 1) 
  • Associated with range drops
  • Seen only on Tuesday
  • LASSO should be rerun on this day to try to identify correlations
• The SNR distribution of glitches was unexpected on Friday and Saturday - there is an overabundance of SNR ~10 Omicron triggers (see figure 2)
  • Hveto was not successful in vetoing this population
• There is a population of moderate (order 10) SNR glitches at 1100 Hz that appeared in the second half of the week (see figure 3)
  • The central frequency is apparently constant over time, so I suspect there is a common mechanism for this glitch
  • They start at roughly midnight UTC on Friday (3/29)
  • No clear morphology in Qscans
• There were several days where the number of spectral artifacts was over the alert threshold. It's unclear to me whether that is due to a lack of observing data or poor data quality

The full report can be found here: https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20240325

IFO is in NLN and OBSERVING as of 23:00 UTC  (4hr 45 min lock)

Secondary microseism is steadily decreasing.

With apologies that this is so delayed, I have prepared updated jitter cleaning.  Since we're already in Observing, and the IFO isn't yet all the way thermalized, I'll push these and see how they do sometime tomorrow.

To do this, I:

• Made sure everything in /ligo/gitcommon/NoiseCleaning_O4/Frontend_NonSENS/lho-online-cleaning/Jitter/ was up-to-date and committed in the git repository.
• Found a nice looking "glitch free" section of low noise time from many hours into our long lock last night.  By "glitch free", I just mean there are no major dips down in range during the time I pick +30 mins.
• Set my new start time in /ligo/gitcommon/NoiseCleaning_O4/Frontend_NonSENS/lho-online-cleaning/Jitter/Jitter_TrainSubtraction.py
  • Line 44-ish, I added: params['gps'] = 1395993819 # ER16. OM2 cold. Many hours into a lock.
• Enabled the nonsens conda environment in my terminal:
  • conda activate nonsens
• In my terminal, in the Jitter folder, ran the training:
  • python3 Jitter_TrainSubtraction.py
• Checked the resulting cleaning
  • In a web browser, go to https://lhocds.ligo-wa.caltech.edu/exports/jenne.driggers/lho-online-cleaning/Jitter/plots/?C=M;O=D and select the most recent "Jitter_subtracted_H_...... .html" file.
  • This time, that's at: https://lhocds.ligo-wa.caltech.edu/exports/jenne.driggers/lho-online-cleaning/Jitter/plots/Jitter_subtracted_H_1395993819_1800_2024_04_01_16h30m24s.html
  • I was happy with this cleaning.
• In the sub-folder ..../Jitter/trained_models/ I copied the most recent .pickle file to Jitter_best.pickle:
  • cp Jitter_results_H_1395993819_1800_2024_04_01_16h30m24s.pickle Jitter_best.pickle
• Back in the ..../Jitter/ folder, I ran the _extractCoeffs.py script:
  • python3 Jitter_extractCoeffs.py
  • Note that this just uses whatever is in the ...../Jitter/trained_models/Jitter_best.pickle file, so make sure to have copied forward the pickle file that you liked the cleaning from.
• Check that there is a new coefficients file in ..../Jitter/CoeffFilesToWriteToEPICS/
  • Jitter_writeEPICS.py
• This is where I stopped, since we are in Observing.  However, tomorrow, the next step is to (after we've been locked and are thermalized) take a reference of H1:GDS-CALIB_STRAIN_CLEAN, and then run that _writeEPICS.py file, and then see how the cleaning is improved.  If it is improved, I will accept the SDF diffs in both safe.snap and Observe.snap.
  • To push the coefficients, run the following: python3 Jitter_writeEPICS.py

TITLE: 04/01 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 5mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.19 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 23:00 UTC (45 min lock)

Other:

H1:PEM-CS_DUST_PSL101                    Error: data set contains 'not-a-number' (NaN) entries

TITLE: 04/01 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:

H1 Is currently Locked for 44 Min and Observing for 8 Min.  
The Myster H0:CDS-VID_CAM_REQ Button was pushed by a button pusher.

In unrelate news:
Unknown Lockloss 21:22 https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1396041753

Relocked with out an Initial Alignment
The SQZ_FC Gaurdian got stuck for 10 minutes, Vicky jumped on TeamSpeak and helped us out. 
Nominal_Low_Noise 22:24 UTC

LOG:

SUS SDF changes saved to Safe.snap.
IM
IM1,2,3,4 Pitch
IM1,2,3,4 Yaw

OMC, OFI, OM1, 2, 3

J. Kissel
ECR E1700387
IIET 9392
WP 11743

Here, this week we tackle the last of the suspension types that need suspension watchdog upgrades after having finished the End Station, Corner BSC, and HAM Triple suspensions this past month (LHO:76269, LHO:76545, and LHO:76712, respectively).

WP 11797
While doing so, I'm also going to address the OMC ASC routing bugs found in the h1susifoout top-level model and its OMCS_MASTER library part (which used to live in h1susomc.mdl), and fix those too.

As such, I'm doing a linear combination of the following changes:
    (1) In library part, removed sending watchdog out to top level
    (2) In library part, removed ODC parts from DAMP and LOCK banks
    (3) In library part, removed never-usered software shutter system
    (4) In library part, upgraded watchdog trigger generator to actual functional RMS
    (5) At top level, removed USER DACKILL
    (6) At top level, removed wires to ground (i.e. things never used) at top-level for WD, SHUTTER, and ODC outputs and SHUTTER input
    (7) At top level, cleaned up receiving of ISC ASC signals (see discussion in LHO:76842)

Below is the list of models, their respective library parts, and the optics in that model that this impacts, followed by a call out of which of the above changes happened
h1susim       HSSS_MASTER    IM1, IM2, IM3, IM4      (1), (2), (4), (5), (6)
h1sushtts     HSSS_FF_MASTER RM1, RM2                (1), (2), (3), (4), (6)
h1susifoout   OMCS_MASTER    OMC                     (1), (2), (4), (5), (6), (7)
              OFIS_MASTER    OFI                     (1), (2), (4), (6)
              HSSS_MASTER    OM1, OM2, OM3           (1), (2), (4), (5), (6)
h1sussqzout   HSSS_MASTER    ZM4, ZM5, ZM6           (1), (2), (4), (5), (6)
h1sussqzin    HDDS_MASTER    ZM1, ZM3                (1), (4)
              OPOS_MASTER    OPO                     (1), (2), (4), (6)
              HSSS_MASTER    ZM2                     (1), (2), (4), (5), (6)

I've test-compiled all of the above listed models, so they're ready for re-compile, install, and restart of the front-end process tomorrow morning.
All model changes have been committed to the userapps repo, either under
    /opt/rtcds/userapps/release/sus/h1/models/
    /opt/rtcds/userapps/release/sus/common/models/
where the above mentioned models live.

Screenshots describing the changes will be posted in the comments.

J. Kissel, [Corroborated by J. Wright and J. Driggers]
WP 11797

While upgrading the OMCS_MASTER.mdl library part for the OMC Suspension (which now lives in the h1susifoout.mdl top-level model), I found two distressing things:
    (1) The OMC ASC signals that are sent from the h1omc.mdl model -- running at 2kHz -- are not anti-imaged at all before being sent to the OMC SUS DAC -- running at 16 kHz. So, we're sending unnecessary imaging noise to the SUS DAC.
    (2) The OMC ASC L, P, amd Y signals are being sent through a filter bank that does NOT past through the DRIVEALIN matrix, rendering 2019's attempt at diagonalization (LHO:47488) unused.

These bugs have been long standing.
I see it now as I touch the h1susifoout.mdl and the OMCS_MASTER library part for watchdog upgrades, but I'd previously identified the issue when 
 - I was upgrading the front-end models for O4 (i.e. when the OMC SUS transitioned from the h1susomc.mdl to the h1susifoout.mdl) in 2021 (LHO:59652), and
 - When we were identifying which SUS models used the broken "TrueRMS" bloick -- there's a preceding note in the simulink model that suggests this has been around since 2015 or earlier.

We can fix this tomorrow, with *only* a change to the h1susifoout.mdl and OMCS_MASTER.mdl models, and we're already doing that anyways for the SUS watchdog upgrade.


Attached are several series of screenshots showing:
    (1) The signal path from the h1omc.mdl model where the OMC ASC control signals begin, to where they land in the OMC SUS driving in the EULER basis (again WITHOUT going through the DRIVEALIGN MATRIX)
    (2) MEDM Screens showing that in practice we only drive L, P, and Y (as one should for an ASC signal) and we don't use T, V, or R drive and likely never will.
    (3) The existing filters that will need to be copied over, and the DRIVEALIGN gain values that will likely need to be re-addressed -- since effectively we've NOT been using them.

Note -- I checked on how this is done at LLO, and they *do* drive through the LOCK banks and DRIVALIGN banks (BUT they don't have any anti-imaging filters in pace either, even though their ISCINF FM1 banks are ON... whoops)...
So, when we change the OMCS_MASTER library part, this won't affect them.

TITLE: 03/31 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:
LOG:
H1 now Locked for 24 hours , and Observing as of 23:00 UTC.
It's been really quiet day for Robert to do PEM injections all day.

SQZers requested a few minutes of SQZ_MANAGER taken to SCAN_SQZANG  which happened at 21:23 UTC                                                                                                                                                                        

Sheila, Jennie W

At around 21:49 UTC we used the step_45MHz.py in userapps/isc/h1/scripts to step the modulation depth in steps of 1dB down from 21dBm to 18dBm. This script adjusts the loop gains to compensate for a drop in power on the RF45 PDs.

Its also important to open the POP beam diverter to monitor what is happening to the RF 45 PD signals.

This is to check that the noise does not decrease with decreasing modulation depth which would imply that the modulator was imposing noise on the carrier through the ISS loop.

At around 22:18 UTC we put the modulation depth down by 3 dB we saw a 4% increase in the DARM broadband noise (measured the level at 2kHz with cursors) which is the green trace in the second image. The level of KAPPA C (shown in first image on bottom plot) only decreased by around 1% over this time, so we suspect some other cause other than a change in optical gain being responsible for this.

When we stepped the modulation depth up by 3dB (third image) we saw no change in the noise from the nominal state (purple trace on second image). Kappa C (bottom plot) looks almost the same as the nominal also.

In the fourth image it can be seen that f_C does not change much either.