Betsy, TJ, Rahul

I have accepted or unmonitored a bunch of settings on SUS SDF for the OBSERVE STATE. I trended all the settings difference and accepted everything which didn't change in the last few months. I have also set the offsets (for COILOUTF) to zero for all the QUADs and R0 chain after going through all of them. There were a few things which were changed by the GRD, hence I have either accepted them or unmonitored (things like TRAMP) them.

Attached below are the screenshot of all the SDF which original existed when I started yesterday morning. Hence if changes were accepted by mistake then please compare it with the screenshot and make the necessary changes.

Still there are a few SUS SDF remaining and we are planning to tackle this today (currently ongoing).

My first attempt at Calibration Sweeps created a report here : 
/ligo/groups/cal/data/H1/reports/20230426T224835Z
This measurement was interupted by a lockloss .

Jeff and I spoke about what measurments it was able to get done before the lock loss and were surprised that it didn't make any BroadBand measurements. We had also edited line 28 of /ligo/groups/cal/ifo/H1/pydarm_cmd_H1.yaml by adding: - pcal before - bb. I will have to look more into what this script does to understand why it performed like this.
 

Diaggui is updated to 3.1.5~dev11.  This version fixes a crash when an NDS connection is dropped.  When the '--fom' or '--autostart' option is used, diaggui will try to restart failed tests, e.g. from a dropped connection.  These options are used on wall displays to ease startup and to keep the diaggui screens running.

- all calibartion lines off (NLN_CAL_MEAS) with frequency dependent squeezing
    from  PDT: 2023-04-27 07:56:44 PDT
          UTC: 2023-04-27 14:56:44 UTC
          GPS: 1366642622

    to    PDT: 2023-04-27 08:06:56 PDT
          UTC: 2023-04-27 15:06:56 UTC
          GPS: 1366643234
- no squeezing
    from  PDT: 2023-04-27 08:07:28 PDT
          UTC: 2023-04-27 15:07:28 UTC
          GPS: 1366643266
    to    PDT: 2023-04-27 08:17:29 PDT
          UTC: 2023-04-27 15:17:29 UTC
          GPS: 1366643867

TITLE: 04/27 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 15mph Gusts, 12mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:
IFO Status: NLN_CAL_MEAS, Comissioning.

We are sitting in some quiet time.
Big Red has to be moved from the Water tank to the staging building before a large delivery arrives in a connex container today.
Picket fence seems like it's not working properly, I'll reboot it.
 

This morning Kim Stewart heard a beeping from the Diode room.  We investigated and the Low Level alarm was coming in and out.  I added 152 ml of water to the chiller and the alarm stopped.

TITLE: 04/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Calibration
SHIFT SUMMARY:

- IFO was locked until 23:34 UTC when there was a DCPD saturation and a subsequent lockloss - Reason unknown

- Lock #1-2:

- GRD needed to do an increase flashes on X but it was able to catch ALS after

- Unfortunately it could not find IR on ALS COMM and flashes were not getting better after ~5 minutes so I tried to rerequest FAST_SEARCH so GRD could cycle through the offsets again, but had a subsequent lockloss ~3 minutes after this request was made

- Jenne noticed that the H1:LSC-TR_X_NORM_OUTPUT channel was hovering below 0 (~ -0.6, which shouldn't be the case since this channel looks at the DC ouput of a PD and you cannot have negative photons), and recommended that we zero the COMM offset values.

• In order to do this, we took the value in the H1:LSC-TR_X_QPD_B_SUM_INMON channel (found on LSC --> LOCKING --> TR CARM (bottom right ish) --> TRX QPD B SUM Overview), and then zeroed out that value by taking the inverse value of that channel, and inputting it into H1:LSC-TR_X_QPD_B_SUM_OFFSET (the offset channel directly below)
• After this was reset, we tried to let GRD find ALS COMM flashes to no avail, then tried to manually cycle through the entire offset range, but could not get any good flashes (capped out at 0.1)
• We are cutting our losses and doing an initial alignment (read Notes portion before continuing to Lock #3)

- Lock #3:

- After finishing an initial alignment after the troubleshooting saga - looks like we are stuck at the same spot as we were before despite all our changes :))

- AHA, something interesting - Ran through INIT one more time but this time ran through CHECK_SDF and FINALLY was able to catch CHECK_IR

• Coincidence? Possibly. But definitely worth looking into the SDF_REVERT state tomorrow. Something to chew on though, if this is the culprit to the issue, how come there was no issue locking today/yesterday - both of which were post SDF_REVERT changes?
• Tried looking in SDF to see if any of the SDF_REVERT models monitor slider values, but couldn't find anything like that

- Acquired NLN @ 06:35 UTC!

- Other Notes:

- Having issues with initial alignment, seems to be stuck in ACQUIRE_XARM_IR, Gabriele and I are looking into it

Issue Troubleshooting Log:

•  After some trouble shooting, we saw that there was no light on the TR_X QPD, so we trended PR2, PR3, and IM4 sliders to see if anything moved but all looked ok
• The WD trip shortly after so we decided to restart the initial alignment to check green alignment
• The second initial alignment worked fine, but again we ran into the issue at ACQUIRE_XARM_IR. This time we tried walking the 3 aforementioned suspensions to try and increase flashes on TR_X, with not much luck - we then trended the M1 IM4 and M3 PR2 witness sensors and saw that they did move about ~2 hours ago, scope attached below - we tried reverting the witness sensors back to the old spot, but flashes did not improve (pain)
• After this we contacted Jenne for help with locking and she recommended lowering the LSC TRIG thresh value (H1:LSC-XARM_TRIG_THRESH_ON, found on the LSC Trigger Matrix) to 0.1 to see if we can even catch it. After painstakingly moving IM4, PR2, and PR3 both P and Y to try and catch to this lower threshold (with the intent of then trying to get flashes up to 0.5), we gave up and tried instead running a manual initial alignment...stay tuned.
• Jenne came on TS to help troubleshoot, suggested we tried to align IR on Y arm instead of X - no luck :(
• Next we tried, running the dark offsets scripts found in: ./isc/h1/scripts/dark_offsets/dark_offsets_exe.py (Attached accepted SDF changes for ALSX/Y)  - hit INIT on ISC LOCK - then trying another manual initial alignment
• Same issue - trying to retouch PR2/IM4
• :'(²
• Jenne tried to change the PSL POWER SCALE OFFSET value to 1/1.5 and then walking the suspensions again - negative
• Ok hail mary time - we reverted all suspensions back to original time when the last initial alignment was completed - ~1366491294
• Worth noting that after this revert, we DID see good flashes in LSC-TR_X briefly but quickly went away, interestingly enough I trended back the guardian states on ISC LOCK and flashes improved at READY and went back down to 0 once it went to SDF_REVERT. Screenshot attached
• Post revert - we decided to skip SDF REVERT - touch up green arms and try another manual alignment...SUCCESS! Finally able to catch XARM IR
• INITIAL ALIGNMENT IS DONE #dontletyourdreamsbedreams - on to ISC LOCK

Jenne's comments regarding the ISS loops during this whole debacle:

I checked that the number that Daniel said in a comment that he had turned to zero from ten, went back to 10 via guardian or some SDF thing, and tried disabling the whole ISS second loop output, no change

And, I put that offset 'thirdloop' (even though it's not actually for the 3rd loop anymore) back to the 500-ish val. All no success

- 1:05 UTC - ITMX ST/2 WD Tripped - This is weird...no one is in the LVEA, plot attached

- Leaving the IFO locked at NLN and in UNDISTURBED, twas an eventful night to kick off ER15 🥲, but we are back in business!
LOG:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     

We attempted to move to a new router tonight from roughly 6pm-7pm localtime.  We ran into some issues and had to roll back.

Some items we need to check on.

 * We had significantly less light on the fiber when using the new router, close to non-viable levels.
 * While we did get light we where not able to exchange bgp routes.

The attached screenshot shows a filter module and its 2 hour old counterpart. Comparing the two it looks like the only difference in the settings is that that the decimate button was turned off in the past. However, the actually difference is that the hold button was on!

J. Kissel, E. Goetz

As Evan and I continue to debug / populate the new demodulation of the PCALX calibration lines in order to get a front-end made, real-time estimate of the systematic error in CAL-DELTAL_EXTERNAL (see e.g. LHO:69061), we found and fixed some bugs in yesterday's update of the MEDM user interface for the time-dependent correction factor and systematic error monitoring infrastructure (LHO:68999). 

Some minor typos in the CAL_CS_TDEP_OVERVIEW.adl, but really overhauled the PCAL_LINE.adl DEMOD OVERVIEW screen with the following changes:
    - Increased the precision on the displayed frequency of the oscillator
    - Moved the newish uncommissioned line-subtraction stuff (in the Yellow Box) out of the way, improving the representation of what's happening along the way
    - Flipped (and therefore corrected) the visual representation of how the I and Q inputs to the DARM/PCAL transfer function COMPLEX_RATIO block are shown, 
    - Replaced (and therefore corrected) SUS tags with PCAL tags in the visual representation of I and Q inputs of the denominator of the DELTAL/PCAL transfer function COMPLEX_RATIO block,
    - Updated the graphical representation of the COMPLEX_RATIO between (DARM/PCAL) and (DELTAL/PCAL) with their respective (DARM_CORR/PCAL_CORR) and (DELTAL_CORR/PCAL CORR) corrective transfer function values
    - Showed the rotation between Real / Imaginary to Mag / Phase for the corrected (DELTAL / PCAL) transfer function.
    - Added a bunch of notation defining the physical components of what each of the corrective transfer functions should be.

See attached before vs. after screenshots.

The screens
    /opt/rtcds/userapps/release/cal/common/medm
        CAL_CS_TDEP_PCAL_LINE.adl
        CAL_CS_TDEP_OVERVIEW.adl
have been committed to the repo.

J. Kissel, E. Goetz

Now that some front end infrastructure is in-place for monitoring the systematic error of CAL-DELTAL_EXTERNAL against PCAL, we have to include corrections to DELTAL_EXTERNAL and PCAL to account for the impacts of anti-aliasing, delays, and imperfect estimation of the inverse sensing and actuation in the CALCS model. To that end, I have included these corrections as transfer function values stored in EPICS records (note that these are higher precision than what is currently written to the EPICS records; in the next days we will have a better solution to push these EPICS records to the front end)
    'CAL-CS_TDEP_PCAL_LINE1_DELTAL_PCAL_CORR_REAL': -0.006073503332940157,
    'CAL-CS_TDEP_PCAL_LINE1_DELTAL_PCAL_CORR_IMAG': -0.0013606856967624704,
    'CAL-CS_TDEP_PCAL_LINE2_DELTAL_PCAL_CORR_REAL': 5.899769524086836e-06,
    'CAL-CS_TDEP_PCAL_LINE2_DELTAL_PCAL_CORR_IMAG': -3.699645821510619e-07,
    'CAL-CS_TDEP_PCAL_LINE3_DELTAL_PCAL_CORR_REAL': 8.386397287449958e-07,
    'CAL-CS_TDEP_PCAL_LINE3_DELTAL_PCAL_CORR_IMAG': -1.5081314667072642e-07,
    'CAL-CS_TDEP_PCAL_X_COMPARE_DELTAL_PCAL_CORR_REAL': -5.9041552773359905e-06,
    'CAL-CS_TDEP_PCAL_X_COMPARE_DELTAL_PCAL_CORR_IMAG': 3.7352291714501767e-07,
    'CAL-CS_TDEP_PCAL_Y_COMPARE_DELTAL_PCAL_CORR_REAL': 5.899769524086836e-06,
    'CAL-CS_TDEP_PCAL_Y_COMPARE_DELTAL_PCAL_CORR_IMAG': -3.699645821510619e-07,
    'CAL-CS_TDEP_PCAL_LINE5_CORRECTION_REAL': -0.000891357719598607,
    'CAL-CS_TDEP_PCAL_LINE5_CORRECTION_IMAG': -6.855875512501219e-05,
    'CAL-CS_TDEP_PCAL_LINE5_DELTAL_PCAL_CORR_REAL': 0.0019506922912537398,
    'CAL-CS_TDEP_PCAL_LINE5_DELTAL_PCAL_CORR_IMAG': -0.0012367241766306207,
    'CAL-CS_TDEP_PCAL_LINE6_CORRECTION_REAL': -0.0003463121477292491,
    'CAL-CS_TDEP_PCAL_LINE6_CORRECTION_IMAG': -2.2361738471929133e-05,
    'CAL-CS_TDEP_PCAL_LINE6_DELTAL_PCAL_CORR_REAL': 0.00031722752183511923,
    'CAL-CS_TDEP_PCAL_LINE6_DELTAL_PCAL_CORR_IMAG': -0.0011010520099375882,
    'CAL-CS_TDEP_PCAL_LINE7_CORRECTION_REAL': -0.00016511918561500546,
    'CAL-CS_TDEP_PCAL_LINE7_CORRECTION_IMAG': -1.0773445503552214e-05,
    'CAL-CS_TDEP_PCAL_LINE7_DELTAL_PCAL_CORR_REAL': -0.0002494269255759141,
    'CAL-CS_TDEP_PCAL_LINE7_DELTAL_PCAL_CORR_IMAG': -0.0005356018866498055,
    'CAL-CS_TDEP_PCAL_LINE8_CORRECTION_REAL': -9.56073427066754e-05,
    'CAL-CS_TDEP_PCAL_LINE8_CORRECTION_IMAG': -6.836645060102044e-06,
    'CAL-CS_TDEP_PCAL_LINE8_DELTAL_PCAL_CORR_REAL': -0.0002616172956290841,
    'CAL-CS_TDEP_PCAL_LINE8_DELTAL_PCAL_CORR_IMAG': -0.00018845087631648119,
    'CAL-CS_TDEP_PCAL_LINE9_CORRECTION_REAL': -1.2253118668315672e-05,
    'CAL-CS_TDEP_PCAL_LINE9_CORRECTION_IMAG': -1.866404318229895e-06,
    'CAL-CS_TDEP_PCAL_LINE9_DELTAL_PCAL_CORR_REAL': -9.67618312303927e-06,
    'CAL-CS_TDEP_PCAL_LINE9_DELTAL_PCAL_CORR_IMAG': 5.78098087799133e-06,
    'CAL-CS_TDEP_PCAL_LINE10_CORRECTION_REAL': -6.138898212894542e-07,
    'CAL-CS_TDEP_PCAL_LINE10_CORRECTION_IMAG': -4.1525414776953105e-07,
    'CAL-CS_TDEP_PCAL_LINE10_DELTAL_PCAL_CORR_REAL': -7.42023357685704e-07,
    'CAL-CS_TDEP_PCAL_LINE10_DELTAL_PCAL_CORR_IMAG': 1.4076461554037913e-07


These were computed by using the method pydarm.calcs.CALCSModel.deltal_ext_pcal_correction() for the CAL-CS_TDEP_PCAL_LINE*_DELTAL_PCAL_CORR_* channels. Note that this calculates DELTAL/PCAL correction and what we need to install (because the front end does a division instead of multiplication) is the inverse of what this method returns. We have also specified include_whitening=False, endstation=False because the DELTAL_EXTERNAL signal is not whitened and the PCAL channels are corner station channels.

Note that CAL-CS_TDEP_PCAL_LINE*_CORRECTION_* channels are computed the usual way using pydarm.pcal.PcalModel.pcal.compute_pcal_correction() using endstation=False inside of pydarm.calcs.CALCSModel.compute_epics_records().

We also had to be a little careful because the LINE1-3 PCAL channels are Y-arm channels while 5-10 are X-arm channels, so we had to make sure to set ref_pcal_2_darm_act_sign=1 for X and ref_pcal_2_darm_act_sign=-1 for Y EPICS records. This should be improved in a future merge request.

When the gain of the ISS second loop was changed, the input offset was not adjusted. This results in the output of the 2nd loop ISS to saturate after a lock loss, until the offset slowly gets compensated by the offset loop. But, this now typically takes like 5 minutes. I adjusted the offset (H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET) from 587.3 to 1272. We will have to check after the next lock loss that this actually works.

J. Kissel, E. Goetz, T. Sanchez

While teaching Tony how to use the command line infrastructure to gather a full set of calibration sweeps, we noticed that the default list of measurements was missing a companion PCAL measurement to the last TST stage actuation strength measurement.

Before we changed anything we ran the check of "what will the command do if I just run it," and got
    $ pydarm measure
    available measurements:
      pcal: PCal response, swept-sine (/ligo/groups/cal/ifo/H1/templates/PCALY2DARM_SS__template_.xml)
      bb  : PCal response, broad-band (/ligo/groups/cal/ifo/H1/templates/PCALY2DARM_BB__template_.xml)
      sens: sensing function (/ligo/groups/cal/ifo/H1/templates/DARMOLG_SS__template_.xml)
      act1x: actuation X L1 (UIM) stage response (/ligo/groups/cal/ifo/H1/templates/SUSETMX_L1_SS__template_.xml)
      act2x: actuation X L2 (PUM) stage response (/ligo/groups/cal/ifo/H1/templates/SUSETMX_L2_SS__template_.xml)
      act3x: actuation X L3 (TST) stage response (/ligo/groups/cal/ifo/H1/templates/SUSETMX_L3_SS__template_.xml)
    measurement sequence:
      ['bb', 'sens', 'pcal', 'act1x', 'pcal', 'act2x', 'pcal', 'act3x', 'bb']
    Use --run-* option to actually execute measurement
You'll see that the list is missing a companion 'pcal' to 'act3x'.

So, we edited the default list by changing 
     /ligo/groups/cal/ifo/H1/
         pydarm_cmd_H1.yaml
Specifically, the lines under the parameter "measurement_sequence" such that it now reads
 measurement_sequence:
  - bb
  - sens
  - pcal
  - act1x
  - pcal
  - act2x
  - pcal
  - act3x
  - pcal
  - bb


Now, the dry-run of pydarm measure reads
    $ pydarm measure
    available measurements:
      pcal: PCal response, swept-sine (/ligo/groups/cal/ifo/H1/templates/PCALY2DARM_SS__template_.xml)
      bb  : PCal response, broad-band (/ligo/groups/cal/ifo/H1/templates/PCALY2DARM_BB__template_.xml)
      sens: sensing function (/ligo/groups/cal/ifo/H1/templates/DARMOLG_SS__template_.xml)
      act1x: actuation X L1 (UIM) stage response (/ligo/groups/cal/ifo/H1/templates/SUSETMX_L1_SS__template_.xml)
      act2x: actuation X L2 (PUM) stage response (/ligo/groups/cal/ifo/H1/templates/SUSETMX_L2_SS__template_.xml)
      act3x: actuation X L3 (TST) stage response (/ligo/groups/cal/ifo/H1/templates/SUSETMX_L3_SS__template_.xml)
    measurement sequence:
      ['bb', 'sens', 'pcal', 'act1x', 'pcal', 'act2x', 'pcal', 'act3x', 'pcal', 'bb']
    Use --run-* option to actually execute measurement

I've committed this new version of the yaml to the Calibration/ifo/H1/ git repo, as of 092dc147.

Adrian, Camilla, Gabriele, Fil, Richard

While trying to track down the source of the 4 Hz bumps in DARM, Richard noticed the cosmic ray photomultiplier tube (PMT) power supply voltage display numbers were jumping around. When he reset the dial, we noticed the 4 Hz oscillations in H1:PEM-ADC_5_29_2K_OUT_DQ disappeared for a few minutes. We unplugged the power supply to the PMTs at 19:05:00 UTC and have not seen 4 Hz features in the ADC channel or DARM since. Plots of the change in ADC channel timeseries behavior when Richard first touched the power supply knobs as well as the ADC channel behavior after the power supply was unplugged are attached. A spectrum comparing the ADC power supply plugged in (blue) and unplugged (red) is attached. We had about 40 minutes to watch DARM before some commissioning tasks started and we didn't see any more bumps. Another plot is forthcoming for this...

(edit for spelling...)

Sidd, Peter, Craig, Jeff

This is a copy/paste of an email thread to the alog for the purpose of outside citations.

Hey Sidd, 

Thanks for putting this study together, it was very interesting at the Heavy Sus meeting.

I'm looking at the following two plots together:
https://ldas-jobs.ligo-wa.caltech.edu/~siddharth.soni/ESD_study/GRD_states/H1_1354310689/zoomed_force_count_H1-ETMX-1354310689.pdf
https://ldas-jobs.ligo-wa.caltech.edu/~siddharth.soni/ESD_study/GRD_states/H1_1354310689
/zoomed_grd_state_H1-1354310689.pdf

Here are the names of the ISC_LOCK state numbers at Hanford



Looks like the high noise starts in state 304 (DARM_TO_RF) at Hanford and slowly goes away during 308 (CARM_OFFSET_REDUCTION).
I took a quick look at the DARM_TO_RF run state.  



Here's the code from the state.  It seems that the high noise begins after line 2182, which is when we turn off FM9 in the LSC-DARM2 filter bank "LP55", which is just a modest 55 Hz low pass filter with a two second ramp time:



So the excess noise we see is a direct result of us turning off some LP55 filter in LSC-DARM2.
This filter never gets turned on again, and appears to go away in CARM_OFFSET_REDUCTION when we bring CARM closer to RESONANCE.  
This part of the guardian is some pre-conditioning for the CARM_OFFSET_REDUCTION state.  The noise we witness is terrible sensing noise from locking on the fringe, having the transmitted power be in loop (from the power normalization), and not filtering it away because it's not really necessary.

The conclusion from the meeting still stands in my opinion:  if we had needed to we would have commissioned this excess noise away.  Because we weren't hitting any saturations or locklosses here we've left it like this.  
We could even try locking CARM and DARM directly from ALS_COMM and DIFF at this point (as the powers that be intended), but that is extremely low priority.

The real question of "how much maximum force is required to lock" is may be during state 15: LOCKING_ALS,
which the ALS_DIFF guardian actually controls.  I'm seeing a 300k count peak-to-peak initial swing in the EX UL MASTER OUT during the latest example lock there, afterwards it calms down significantly.  Probably also possible to acquire ALS_DIFF more gently.  
Would be interesting if you compiled a bunch of these ALS_DIFF acquisitions similar to what you've already done for state 304, and maybe we can do a deeper dive into what ALS_DIFF acquisition is actually doing similar to this study.

Craig

P.S. We sometimes have trouble locking ALS_DIFF for ~hour in bad conditions.  If we looked deeply at this, we may be able to improve IFO uptime.