At some point either last friday or over the weekend we started to see some very strange behavior in the H1:GDS-CALIB_STRAIN_NOLINES channel in the low-latency frames.  It manifested as what appeared to be a large DC offset in the channel, and polution of the spectrum at higher frequencies.  The problem was not seen in the data on disk, and only in the data in the low-latency frames.  Due to a quirk in the DMT systems, the low-latency hoft frames and the to-disk hoft frames are created by different DMT machines, running their own gstlal-calibration processes. It appears that the calibration process on h1dmt1 which writes out the low-latency hoft frame somehow got into a bad state.  The issue was resolved by restarting the gstlal-calibration processes on the two DMT machiens (h1dmt1 and h1dmt2).  It's unclear why this happened, the calibration group is looking into it.

We narrowly avoided disaster today when CSOFT P again rung up at 0.45 Hz. I was able to drop the gain and save the lock. This was in the midst of many SRCL offset changes and measurements. Once we settle on a SRCL offset, I will take another sensing matrix measurement and check if the input matrix needs updating again. I have updated the soft loop gains in the guardian again.

When we get back to NLN, there will be SDFs that need to be accepted in the ASC model.

TITLE: 05/08 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 12mph Gusts, 10mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:
IFO Has been locked for 25:05.
ISC LOCK STATE : NLN_CAL_MEAS
Currently some SQZ and SRCL measurements are being done and the IFO is in commissioning mode.
CDS Overview has some red bits on it denoting SDF Diffs. Pic of CDS overview included.

TITLE: 05/08 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 136Mpc
SHIFT SUMMARY:

Main Notes:

- IFO has been locked for ~25 hours! Range is 112 Mpc

- SQZ and SCRL measurements are currently ongoing

Lock #1:

• Arrived with IFO locked for ~17 hrs 
• Camilla took us out of OBSERVING @ 16:01 to get 30 mins data of no squeezing, then we started a calibration suite @ 16:40-18:23
• Jennie ran a SCRL detuning measurement @ 18:25
• Switched to COMMISSIONING @ 20:02 for SQZ measurements
• EX saturation @ 20:10/22:09

LOG:

Adrian, Marc

Last week we looked for evidence of crosstalk between PEM accelerometer channels in the PEM rack at the CS CER. We compared the crosstalk performance of channels which use an inductor versus a resistor by shaking a temporary accelerometer plugged into a permanent PEM channel at 25 Hz with a damped speaker. We find that there is no difference between the two configurations - the ratio of counts in the unexcited channels (i.e. possibly from crosstalk) to counts in the excited channel at 25, 50, 75 and 100 Hz range from O(1E-5) to O(1E-2). Attached are plots of ASDs of the CS accelerometers we tested during the tests as well as a file with the list of ASD ratios. The test were 5 minutes apiece, but the times given below used in our analysis are a little shorter - we want to build in some buffer time for us moving around after setting a test up.

Channel 1 Test Details:

Excited Channel: H1:PEM-CS_ACC_HAM6VAC_SEPTUM_Y_DQ

Start: 1367347574 (May 5, 2023 18:45:56 UTC)

End: 1367347744 (May 5, 2023 18:49:56 UTC)

Channel 16 Test Details:

Excited Channel: H1:PEM-CS_ACC_PSL_TABLE2_Z_DQ

Start: 1367347851 (May 5, 2023 18:50:33 UTC)

End: 1367348041 (May 5, 2023 18:50:56 UTC)

J. Kissel

In hopes that we would have a few thermalization periods measured over the weekend in our new SRCL offset at -265 (LHO:69359) and our new DARM offset set such that there's 40 mA total of photocurrent on the DCPDs (H1:OMC-READOUT_X0_OFFSET ~= 9.7, see LHO:69358), I went to run Evan's thermalization script (LHO:69301) ... but came up with noise only for the PCALY 8.925, 11.575, 15.275, 24.5 Hz lines driven by CAL_AWG_LINES. 

Sadly, it looks like the PCAL model's test points (FEC DCUID 125) were cleared on 2023-05-06 15:57 UTC, or (Saturday morning at 08:57 PDT), and they've not been restored. I think this might have resulted in some confusion regarding instructions on how to run calibration measurements headlessly, in that the test points were cleared even though the headless measurement succeeded normally without the IFO losing lock. (LHO:69396).
 
I've edited the wiki page instructions to be more clear.

I'll get lines restarted ASAP so we can catch the next lock acquisition.

Accepted attached JITTER_NONSENS_COEFF sdfs on behalf of Jenne.

Ryan C, Camilla.  As suggested by Ryan, we've edited the SUS_CHARGE guardian to take the In-lock Charge Measurements at 5.45am if we are locked tomorrow. This is 2 hours earlier than the nominal, during observing run, time of 7:45am to maximize our changes of getting data as we have lost lock before they started/finished the last 2 weeks. 

Referencing Camilla's alog 69333, I ran a calib suite.

• Took us out of OBSERVING & took ISC_LOCK to NLN_CAL_MEAS
• Opened terminal and entered/ran:  "pydarm measure --run-headless" at 16:40 UTC.
• Script finished at 18:23 UTC
• Moving on to SCRL detuning measurement

Attached is a screenshot of the sitemap > CAL CS > Calibration Monitor screen shortly after starting the measurement.

• Measurement at:  /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20230508T164115Z.xml

'pydarm report' command gave attached report: 

• /ligo/groups/cal/H1/reports/20230508T180014Z/H1_calibration_report_20230508T180014Z.pdf

FAMIS 19975

Looks like the anteroom relative humidity sensor went dead ~4 days ago, at the same time as the temperature in there dropped by 2 degrees F (environment trends, tagging PEM).

Incursion work 6 days ago and calibration for AMP2 power update shown (see alog 69259 for full summary of work last week).

Austin and I took SQZ_MANAGER from FREQ_DEP_SQZ to SQZ_READY_IFO for 30 minutes of no-squeezing time 16:02UTC to 16:34UTC. 

The IFO had been locked for 18 hours. Range dropped from 133MPc down to 113MPc. Both of these range values seem lower than earlier in the week but there was a lot of changes on Friday (69365). 

TITLE: 05/06 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning at 130 Mpc
SHIFT SUMMARY:

Main Notes:

- Second ER15 candidate! @ 17:09:35

- ACQUIRE_XARM_IR during IA is still failing, and therefore I cannot run a full initial alignment

- ACQUIRE DRMI 1F has been quite cumbersome recently trying to acquire

- IFO is at OBSERVING @ NLN @23:02
Lock #1:

- Came in with IFO locked, switched to OBSERVING @ 14:56 UTC, out of OBSERVING @ 17:00 UTC due to difference from VAC SDF (was only temporary, but maybe this channel should be unmonitored), screenshot attached - Tagging Vac

- PI 24 ring ups at 15:26/15:37 UTC, but the SUS_PI guardian was able to damp it - no intervention needed

- Starting a calibration suite @ 17:01 UTC in NLN CAL MEAS

- EX saturation @ 18:02 UTC and again @ 20:20, huge jump in ADS signals and OMC DCPD output went off the rails - glitch?

- Back to NLN and OBSERVING @ 18:53 UTC

- LOCKLOSS @ 2026 UTC - this looks like an ASC ringup, potentially from SRC1/2 and/or MICH? - however ground motion looks a bit elevated so that could be a culprit as well

Lock #2:

- IMC had issues acquiring, but resetting ISC_LOCK to INIT and having it tried again worked fine

- DRMI issues 3x times trying to lock, gave up and trying an initial alignment - just like yesterday I again cannot ACQUIRE_XARM_IR. Skipping IA and trying again.

• Ended up having to move BS P from 89.7 --> 90.36, PRM P from -1510.6 --> 1525.2 and PRM Y from 344.3 --> 354.3

- Acquired NLN @ 22:38

- Some OBSERVE snaps are preventing me from going to OBSERVING, SUSITMY - Screenshots attached, accepting and went into OBSERVING, Tagging SUS

LOG:

No log for this shift.

Tito Dal Canton from PyCBC reported that there are periods with many ADC overflows of the DCPD in the last few days. This is probably due to the increased DARM offset alog 69358. See attached plots of the DQ vector. There's no obvious effect on the spectrum or the glitchiness, so it must not be a severe overflow. But at least some of the searches are using this as a veto and aren't analysing the data.

• Whitening gain on POP45 reduced by 6dB (alog 69350)
• DARM offset increased from 20mA to 40mA (alog 69358)
• SCRL offset changed to -265 (alog 69373)
• MICH and SRCL ASC changes (alog 69372)

LOCK#1

After the IFO was locked 5h13, we lost lock at 01:06UTC from ASC changes.

Elenna loaded a new MICH filter after the lockloss.

LOCK #2

Had a tough time relocking (later we noticed it seemed like CHECK_MICH_FRINGES moved the BS the wrong way at around 2:00UTC). This BS move made the DIFF beat note too low (-30) for LOCKING_ALS. I attempted an initial alignment. Couldn't get XAMR IR locked eso exited from initial alignment. Reverted BS pointing to before CHECK_MICH_FRINGES and touched PRM during ACQUIRE PRMI. Finally locked again. Inputting 76W at 03:45 UTC and at NLN at 04:20UTC. Delay due to OMC_WHITENING limited needed to be changed, alog69371.

Observing at 04:35UTC. SDFs accepted:

• SQZ sdfs screenshot for FC TRAMP, ZM3 offset, FC detuning and PSAMS;
• LSC and ASC sdf screenshot for alog 69373 and 69372;
• OMC and CAL sdf screenshot for 69375 The OMC TRAMP sdf may need to be re-accepted on next lock as it may been from Elenna manually changing Whitening.

At 05:59UTC we had the 10.4Hz PI ring up (PI # 24), I was notified via verbalalarms and it and was successfully damped by the PI_DAMPING guardian, no intervention needed, Vicky's alog 69318 changes were good. See attached plot.

Naoki, Vicky

We successfully implemented the beam spot control with green QPD at FC trans by feeding back to ZM3 (Fig.1). The middle two are the green QPD P/Y signals and the bottom two are the error signals of the beam spot control P/Y. The cursors show the optimal green QPD offset for FC2 center which is decided in alog69338. After engaging the control, the green QPD signals settle to the optimal offset and the error signals become 0. The time scale of this control is ~1 min.

Fig. 2 shows the medm of FC ASC and Fig. 3-6 show the input/output matrix of FC ASC.

For stable FC green and IR lock with the beam spot control, we needed to clear history of filter bank of the beam spot control when the FC loses lock. Vicky added the clear history in down state of SQZ_FC guardian. The FDS with the beam spot control can be operated well by guardian.

Several special calibration lines have been running for the past couple of weeks so that the calibration group can assess how IFO thermalization impacts the response function (see LHO aLOG 69284). We observe the coupled-cavity pole frequency and optical gain evolving over the first ~2 h of lock when the IFO is "cold" and still thermalizing to normal high-power operations as measured by the calibration lines.

These additional lines, together with the standard set of calibration lines, have been processed offline to measure the transfer function of PCAL / DARM_IN1 = ( 1 + G) / C = R, and DARM_EXC / DARM_IN1 = (1 + G) so that we can actively track the changing response function (R) and the sensing function, derived from the combination of the measurements to isolate C [(DARM_EXC/DARM_IN1) x (DARM_IN1/PCAL)]. The DARM_EXC lines are placed close in frequency to additional PCALY lines so that is is straightforward to remove the effect of the closed servo loop.

At present the script to do this is still in "draft" form (see merge request), but we expect further improvements to analysis, plotting, and general robustness for future use cases.

I downloaded this script to my home directory (~evan.goetz/lscrepos), activated Louis' environment (conda activate cds-pydarm-test), and ran:
$ python process_sensing_darm_comb.py -i H1 -s 'may 2 2023 11:00:00' -e 'may 2 2023 11:30:00' --pydarm_ini=/ligo/groups/cal/H1/ifo/pydarm_H1.ini --pydarm_unc_ini=/ligo/groups/cal/H1/ifo/pydarm_uncertainty_H1.ini

This queries for the data and saves it so that re-running the script does not require re-querying for the data. It processes the data to determine R(t,f) and C(t,f). Attached is an initial series of plots showing the evolution from the start of the analysis period (data points in blue) to the end of the analysis period (data points in yellow). The pyDARM modeled values are shown in blue lines for comparison. This initial measurement shows the sensing function evolving from a pro-spring response to nearly no spring over the course of 30 minutes.

Future work should be to go back and run this program on all of the lock stretches for at least the last few days. Going back to the initial locks may not prove as useful because the front-end TDCF values were not correct, owing to incorrect "reference transfer function" EPICS records. Nevertheless, we can collect a few more of these "thermalization" measurements over the next several lock stretches will help inform the additional uncertainty needed for start of lock stretches.

Unfotunately turning off the cosmic ray power supply did not remove the intermittent 4.05 Hz bumps in DARM, even though all the CS signals that showed a large peak at 4.06 Hz are now clean.

Looking again at all the signals that contain a peak around 4.05 Hz, we now have

          H1:PEM-CS_MAG_LVEA_VERTEX_Z_DQ 4.050 Hz
               H1:PEM-EX_ADC_0_08_OUT_DQ 4.040 Hz
               H1:PEM-EX_ADC_0_09_OUT_DQ 4.040 Hz
          H1:PEM-EX_LOWFMIC_VEA_FLOOR_DQ 4.040 Hz
         H1:PEM-EX_MAG_EBAY_SUSRACK_Y_DQ 4.040 Hz
         H1:PEM-EX_MAG_EBAY_SUSRACK_Z_DQ 4.040 Hz
             H1:PEM-EX_MIC_EBAY_RACKS_DQ 4.040 Hz
      H1:PEM-EX_TEMPERATURE_BSC9_ETMX_DQ 4.040 Hz
       H1:PEM-EX_VMON_ETMX_ESDPOWER18_DQ 4.040 Hz
               H1:PEM-EY_ADC_0_14_OUT_DQ 4.040 Hz
         H1:PEM-EY_MAG_EBAY_SEIRACK_X_DQ 4.040 Hz
         H1:PEM-EY_MAG_EBAY_SEIRACK_Y_DQ 4.040 Hz
         H1:PEM-EY_MAG_EBAY_SEIRACK_Z_DQ 4.040 Hz
         H1:PEM-EY_MAG_EBAY_SUSRACK_X_DQ 4.040 Hz
         H1:PEM-EY_MAG_EBAY_SUSRACK_Y_DQ 4.040 Hz

Attached plots shows spectra and time series. Although the frequency of all peaks is very close, EX signals are only coherent among themselves, and not with EY signals, and the other way around too. So it looks like at both end stations there is a source of 4.04 Hz that is local. Maybe a clue? Still, we don't have evidence that those sources are related to the DARM noise.