Looks like the lock loss was caused by an ADC error on h1iopseib3. Erik is currently working on the issue.

Gabriele, Sheila, Artem, Louis


Since Gabriele found low frequency noise (~4 Hz) non-linearly coupling into DARM at higher frequencies (~15-25 Hz) (see LHO:73937), Sheila & I have been working to prop up a dependable model of the DARM loop that can be used to estimate stability margins and offload actuation from ETMX L3 to L2 and L1. N.B. we're not 100% certain that the noise we're chasing is due to the ETMX ESD.


1. The DARM Model
To estimate DARM loop stability, we've been primarily looking at modeling and directly measuring IN1/IN2 at the input to each stage's LOCK bank filter. IN1/IN2 at the input to the L3 LOCK bank filter (often measured as (IN1/EXC)/(IN2/EXC) so as to not be biased towards 1) is the open loop gain, G, of the DARM loop. We've been calling IN1/IN2 measured at the input to the L2 and L3 LOCK banks Gp and Gu, respectively. This is because they each behave somewhat like an effective open loop gain as measured at those points in the offloaded DARM loop. Similar to G, Gp, and Gu (the "OLGs"), we calculate the loop suppression, 1/(1-G), and the closed loop gain, G/(1-G), for each our three test points (G, Gp, and Gu) at the input of each ETMX LOCK bank filter.






Figure 1 above is a Bode plot of G, Gp, and Gu for the LHO DARM loop overlaid with measurements taken for G and Gu. The OLG, G, measurement was taken from a recent calibration swept sine and the measurement of Gu is courtesy of Sheila from LHO:74226. The two measurements shown for G and Gu match the model pretty well. There is some noise in the Gu measurement but the structure is in pretty good agreement between the model and the data. The measurement of G also matches pretty well but begins to deviate below 10 Hz. This is likely in large part due to error in fitting the sensing function, C, at low frequencies. It's worth noting that the UIM stage has multiple UGFs just below 2 Hz and 4 Hz; this is okay but not ideal.

We tried taking a measurement of Gp but we've been unable to get good agreement between the current DARM loop model and these measurements (this is mentioned in LHO:74264). We've only tried relatively low coherence measurements so far in this current sprint so we plan to get some better measurements of this soon to compare against the model. However, it's worth noting that the PUM has been a bear to model properly for a while (see for example the PUM discussion in LHO:63450). A few years ago Jenne successfully modeled length-to-pitch-to-length cross-coupling that matched data taken at the PUM stage (LHO:48738). In light of the recent modeling push, Sheila and I are have begun working to follow up on this work to include yaw in the hopes of modeling and suppressing this effect from coupling to DARM. More on that later as our current major goal is to reduce ETMX actuation as introduced at the top.

The Gu points in Figure 1 were calculated from a measurement of IN1/EXC at the input of the L2 stage LOCK bank filter. Direct measurement of Gu = IN1/IN2 was difficult due to low coherence. The relationship between Gu and IN1/IN2 is Gu = (IN1/EXC)/(1+(IN1/EXC)). See Section 4 of DCC:LIGO-T2300436 for a discussion on that relationship.

2. Comparison of LHO and LLO DARM Models

Here is a quick comparison of recent DARM loop models for LHO and LLO. 

For searching purposes I'm writing their modeled OLG UGFs:
LHO UGFs:
  1: 70.45 Hz, 26.97 degrees
LLO UGFs:
  1: 78.75 Hz, 35.89 degrees











The DARM OLG magnitudes between LLO and LHO differ at low frequencies by roughly 3 orders of magnitude near 1Hz. Here's an even higher zoom near 1Hz: LHO_LLO_comparison_OLG_super_zoom_low_freq.png. As an independent check into whether that makes sense, Artem pulled up the external disturbance sent to each actuation stage at both sites using CAL-DELTAL_CTRL_{UIM,PUM,TST}_DBL_DQ and saw that they're similar between LLO and LHO (direct link to this plot). Then he looked at the LHO and LLO error signals and found a discrepancy of 2 orders of magnitude in the expected direction (direct link to this plot). So that all seems to roughly track, which is great news. See LHO:74744 for Artem's discussion.







3. Instability of 4.5Hz boost filter not predicted by DARM model
In LHO:74226, Sheila tried engaging a L2 LOCKL FM2, "boost4.5", which caused a lockloss. She successfully engaged the boost without a lockloss the day before just before the Tuesday maintenance period. 

Here is a comparison of the LHO model with and without this boost engaged.






The hope was that we would look at the model with the boost included and be able to confirm that the resultant loop configuration is unstable. However, we aren't able to point to anything that obviously indicates that that's the case. The corresponding loop suppression plot shows some peaking near 10Hz but nothing that would have us consider it a smoking gun.


=====

So the DARM modeling efforts are progressing but there is still much to do..

TITLE: 12/13 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
00:27 UTC HWS was restarted and took us out of Observing by making SF changes for a minute. The HWS Engineer making the changes was still in the control room and managed to handle the SDF changes which allowed us to get back to Observing.  See Alog : https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=74770

00:29 UTC OBSERVING was reached again.

After this a small earthquake rolled through which we survived, but otherwise the night was quite.
Everthing went smooth like butter.

LOG:
No log

TITLE: 12/13 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.33 μm/s
QUICK SUMMARY:
H1 Is locked at NOMINAL_LOW_NOISE and OBSERVING for the past hour.
Violins lookin great!

TITLE: 12/12 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Relatively light maintenance day followed by some trouble getting locked, but after adjusting thresholds in ALS Guardians, was able to lock without much issue.

• 16:07 Lockloss from commissioning test, ISC_LOCK to IDLE
• 20:27 Maintenance finished, starting initial alignment
  • Troubleshooting lower ALS power
  • Manual-ed ALS Guardians to find IR; see alog74766
  • Rest of locking was fully automated
• 22:51 Reached NLN
• 22:55 Started observing

LOG:

Ryan S, Jenne D, Sheila

The ALS_COMM and DIFF nodes will check on the green arm transmission when finding IR to ensure that the arms are aligned well enough to move on. This threshold has been set to 0.75 but I lowered it to 0.65 since we've ran into this threshold as few times in the last week and have been unable to improve the arm transmission.

Last week this happened on Thursday (alog74655) and I just manualed around the state to continue locking. It worked for that lock to get us back up, and then it wasn't a problem with future locks until today. Today, the arm transmission was low for both arms so we went to confirm that there was no PR3 movement, and then we restored optics back to the start of the previous lock. This also didn't help. Looking at the arm power over the last few weeks (attachment 1), after the PR3 and table work from last Tuesday (alog74618) it has been lower than before then but as low as we were seeing today. The current thinking is that the PR3 move last week brought our powers lower, and while the table work improved the COMM beatnote, it did nothing for the arm powers. Once the IFO gets to a better full lock alignment, this alignment tends to give us better ALS power next lock acquisition, so lowering this threshold should help us get to that point and hopefully help sequential locks.

The long term solution would be to go back on table to realign the rest of the COMM path to camera, then renormalize the arm powers so that 1 is actually our max power, then bring this threshold back to the 0.75 that it was*. Going on that table always carries rick of making things worse, so we definitely don't want to be doing this right before the holidays. We will reconvene in 2024 on this issue if it's still an issue.

*There's a comment in the guardian code that it was 0.85 from Aug 12, 2020.

Jordan, Janos

We ran the functionality test on the main turbopumps in MX and EX during Tuesday Maintenance (12/12/23). The scroll pump is started to take pressure down to low 10^-02 Torr, at which time the turbo pump is started, the system reaches low 10^-08 Torr after a few minutes, then the turbo pump system is left ON for about 1 hour, after the hour the system goes through a shut down sequence.
 

MX Turbo:

Bearing Life:100%

Turbo Hours: 120

Scroll Pump Hours: 211

EX Turbo:

Bearing Life:100%

Turbo Hours: 277

Scroll Pump Hours: 6317

Closing WP 11573 and FAMIS 24846 FAMIS 24870

TJ, Camilla. WP11574

We borrowed the Ophir PH00235 USB MSP-NS-Pyro 9/5 bean scanner from CIT. It has a PH00092 removable 7.5" FL lens on the front and is on a 500mm rail. WE checked the scanner and rail worked in the lab using our Nanoscan laptop and the Nanoscan v1 software. 

On CO2X (layout T1200007) we added three Gold coated steering mirrors just after the power control waveplate. Added a beam block panel on the back of the table. While aligning we kept ther CO2 power requested at 0.1W.  Started aligning the beam to the beamscan but didn't finish before the end of maintenance. 

We left the beanscan and the three mirrors on the table, we moved the mirror that was in the path ("A" on photo) out of the path. Plan to finish aligning and take data next Tuesday. The cables and driver boxes are in a tote labeled "nanoscan" in the TCS cabinet and the laptop is back in the optics lab cabinet.  Photos of table and beampath (nominal lighter red) attached. 

Genie lift found plugged into outlet near TCSY table, unplugged.

Robert's shaker is still connected to near HAM2, it is plugged in but powered off.

All else looked good, followed T1500386

All but a few maintenance activities have wrapped up to the point where I've turned sensor correction back on and H1 has started lock acquisition.

As the first part of the TW0 raw minute trend file offload, tw0 is now writing to a new area freeing up the old files for transfer. 

nds0 was restarted at 10:44 PST to serve the past 6 months of data from their temporary location as the files are being transferred to h1daqframes-0. The file copy takes about 30 hours.

This morning at 8:15am (16:15UTC) I turned off the external power supply that powered both ITM HWS CCD cameras (Dalsa 1M60), located on the floor under the HWS table.  We'll plan to check the magnetometer data to see if the 74738 comb is still present.

After we stopped HWS code and remotely turned off camera's in 74738, the DARM comb still remained. Dan, Daniel, Nutsinee expect this could be the framegrabber in the HWS computer still sending a 7Hz signal. We can troubleshoot this and how the camera's are grounded to the optics table later. 

First observed as a persistent mis-calibration in systematic error monitoring Pcal lines which measure PCAL / GDS-CALIB_STRAIN affecting both LLO and LHO, [LLO Link] [LHO Link], characterised by these measurements consistently disagreeing with the uncertainty envelope.
It us presently understood that this arises from bugs in the code producing the GDS FIR filters there exists a sizeable discrepancy, which Joseph Betzwieser is spear-heading a thorough investigation to correct,

I make a direct measurement of this systematic error by dividing CAL-DARM_ERR_DBL_DQ / GDS-CALIB_STRAIN , where the numerator is further corrected for kappa values of the sensing, cavity pole, and the 3 actuation stages (GDS does the same corrections internally). This gives a transfer function of the difference induced from errors in the GDS filters.

Attached in this aLog, and its sibling aLog in LLO, is this measurement in blue, the PCAL / GDS-CALIB_STRAIN measurement in orange, and the smoothed uncertainty correction vector in red. Attached also is a text file of this uncertainty correction for application in pyDARM to produce the final uncertainty, in the format of [Frequency, Real, Imaginary].