pi31 slow ringup causing range drop

We noticed that the range was dropping even lower than our already low ~145Mpc. There was a lot of low frequency noise that was creeping up. Sheila suggested that we look at the PIs and sure enough, PI31 had started to slowly creep up the same time as the range degredation (see attached). The SUS_PI guardian node will turn on the damping at 3 according to the H1:SUS-PI_PROC_COMPUTE_MODE31_RMSMON channel and turn it back off when it gets below 3. For now we just tried changing that threshold to 0 to all for continuous damping. This let the mode damp down completely and it brought our range back up.

We should think more about damping this down lower and having two thresholds, an on an off.

range trends with sqz angle servo on

The attached screenshot shows that turning on the SQZ angle servo 83270, which is shown by the time when CLF_RF6 phase starts to move around, the 350Hz squeezing (yellow BLRMS3) has been better, which is what this servo is intended to do. 

You can also see that the higher frequency squeezing has been worse, and now seems correlated with the range dropping.  Early in each lock there is a time when the high freq squeezing reaches 4.7dB, which corresponds with the times when the range is highest around 156 Mpc.  This is during the thermalization, you can also see "f_s" which is the estimate of the SRC spring frequency based on the calibration lines, and kappa_c are also changing rapidly in this time, so the change in range could be explained by these changes more than the impact on squeezing.

TJ has posted a sensitivity comparison with coherence here:  83327 which shows that we have some A2L coherence to fix. 

Wed CP1 Fill

Wed Mar 12 10:08:11 2025 INFO: Fill completed in 8min 8secs

Gerardo confirmed a good fill curbside.

VACSTAT restart for BSC3 sensor glitch

Late entry from last night. VACSTAT went into single-alarm-mode with a BSC3 sensor glitch 20:29:44 Tue 11mar2025 PDT. I reset the vacstat_ioc.service on cdsioc0 at 20:36 to clear this.

Ops Day Shift Start

TITLE: 03/12 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 141Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 15mph Gusts, 11mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.31 μm/s
QUICK SUMMARY: Locked for just over and hour, looks like Ryan had to accept some SDFs overnight, and there were some ALS issues yesterday, but we had 3 relocks overnight. No alarms this morning.

 

OPS OWL assistance

Another ALS_Y sdf diff

13:33 UTC Observing

OPS OWL assistance

I had to accept two sets of SDF diffs to get us into observing, for HAM1 and ALS_Y.

09:16 UTC Observing

Ops Eve Shift End

TITLE: 03/12 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently relocking and in POWER_25W. We've been down for a while trying to figure out a weird issue with the ALSY WFS and we at least have a plan for the night, but it might end up really annoying for Ryan C unfortunately.

For Ryan C (and maybe TJ tomorrow morning): tagging OPSINFO

• If we are in initial alignment and ALSY keeps drifting down and then unlocking, have the initial alignment align ALSY using ETM_TMS_WFS_OFFLOADED instead of INITIAL_ALIGNMENT_OFFLOADED (You'll then need to manual to green arms offloaded in INIT_ALIGN and then continue the alignment)
• Once we get to ARMS_OFF_RESONANCE and you'll see ALSY transmission start dropping and DOF2 WFS diverging, turn off the ALSY DOF2 WFS for both pitch and yaw

I started an FRS ticket FRS#33562. Wasn't sure what category to put it under though so I moved it to be under operators, but that's definitely not right.

 

What we saw/what we tried to do to fix it:

After the lockloss earlier, I wanted to run an initial alignment since we had not run one since yesterday. I noticed that once the WFS would turn on for ALSY, the transmission would slowly drift down until it lost lock(ndscope1, ndscope2). I thought the issue might be DOF2 or 3, since both of those seemed to be diverging and DOF3 WFS were much larger than they usually are(ndscope3). I tried turning DOF2, DOF3, and both sets of WFS off, but we would just lose lock. I left IA and tried locking green arms the normal relocking way, which only uses DOF1 and DOF2 aka ETM and TMS, and that worked well, so I knew it had to be an issue with DOF3 specifically. I ended up needing an initial alignment, and either way we didn't want it to need an initial alignment in the middle of the night. Jenne suggested I take the YARM to LOCKED_SLOW_ETM_TMS_WFS and then slowly align ITMY to lower the ITMY camera error. I tried that, but I kept getting to a point where the DOF2 WFS stopped being able to handle it and would just diverge until we lost lock, even if I tried stepping back the other way, all while the camera errors didn't get much better. After trying this multiple times, Jenne came onto TS and we tried this method again with no success. We also tried adjusting ITMY (usually in Pitch) with YARM in LOCKED_SLOW_NO_WFS, again in LOCKED_SLOW_ETM_TMS_WFS but with DOF3 YAW WFS engaged, we tried resetting the slider values for ITMY, ETMY, and TMSY to the end of the last initial alignment, and we tried aligning MICH to get the beamsplitter in a better spot. None of these methods worked.
Eventually we decided that the only thing we could do for tonight was to have an initial alignment run with ALSY aligning only using ETMY and TMSY. Hopefully during the night there isn't a need for another initial alignment, because if there is Ryan C will need to change the states ALSY will need to go through to be ETM_TMS_WFS_OFFLOADED instead of INITIAL_ALIGNMENT_OFFLOADED.

Now, after finishing an initial alignment where we only used the ETMY and TMSY WFS, I've been trying to relock, but it looks like DOF2 diverging is still happening, ALSY goes down, and then ALSY loses lock(ndscope4). This wasn't happening during the relocks earlier in the day right after maintenance. To get around this, once we got to ARMS_OFF_RESONANCE and ALSY transmission started dropping, I turned off the ALSY DOF2 WFS in both pitch and yaw. We were able to get through the rest of the ALS states like this.

LOG:

22:30 Observing

23:45 We dropped out of Observing for a second and then went back in because I accidently clicked something monitored by SDF

01:02 Lockloss                                                                                                                                                                                                                                                                                                                                                                                                                         

Start Time	System	Name	Location	Lazer_Haz	Task	Time End
22:19	ISC	Mayank, Siva	Opt Lab	Local	ISS PD array	00:59
23:42	ISC	Keita	OpticsLab	n	WFS work	01:09

Lockloss

Lockloss @ 03/12 01:02 UTC

Ops Day Shift End

TITLE: 03/11 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: List of maintenance activities below, the most impactful to locking being the camera server change that we ended up having to revert to resume locking (alog83308). We will try to implement these cameras that are used in locking a different day when we can dedicate time to setting the IFO to their new offsets.

• Vacuum compressor replacement continues at EY, and cleanup from EX - 83066
• CP2 LN2 delivery
• UPS batteries replaced in all builds for VAC racks - 83313
• HAM1 ISI electronics were installed - 83309 83305
• FARO in LVEA
• Replaced, then reverted some, cameras on old server - 83308
• isiham1 model additions and DAQ restart - 83289
• PM1 infrastructure added, PR3 cable moved, other collateral models affected and changed - 83293 83288
• BSC1 temperature sensor troubleshooting - 83294
• susb123 power supply fan needed replacement, taking down those models - 83311
• EY chiller pump seal fixed - 83307
• EY well pump testing - 83291
• More I missed

Locking notes:

• 2230 Observing
  • Locking was automated the first go around, but then after we lost lock from the camera offsets, the BS in particular had been pulled far off and had to be brought back to get ALSY to lock. Everything else was automated though.

LOG:

Start Time	System	Name	Location	Lazer_Haz	Task	Time End
14:36	FAC	Nelly	Opt Lab	n	Tech clean	14:48
15:06	SUS	Jeff	CER	n	Move PR3 OpLev cable	16:01
15:06	FAC	Kim	EY	n	Tech clean	16:53
15:06	FAC	Nelly	EX	n	Tech clean	16:05
15:14	CDS	Dave, Erik, Jeff	CR	n	Model restarts (PR3, HTTS, IMs, SUSAUX) also SEIH16	15:35
15:17	CDS	Fil, Erik, Marc	CER	n	SEI rack work (HAM1)	18:21
15:18	FAC	Tyler	LVEA, Ends, Mids	n	3IFO checks	17:34
15:20	VAC	Ken	EY MR	n	Compressor electrics	18:45
15:25	FAC	Contractor	EY	n	Well testing - Large van, water truck, SUV	16:22
15:29	IAS	Jason, Ryan C	LVEA	n	FARO	18:59
15:35	OPS	Richard	EY	n	Check on panels and Ken	16:51
15:35	VAC	Janos, Travis, Jordan	EY, EX, mids	n	Compressor cleanup	18:18
15:51	PCAL	Rick, Francisco, Sivananda	EX	not yet	PCAL meas	17:35
16:17	VAC	Gerardo	LVEA	n	Vac rack UPSs	17:07
16:18	ISC	Camilla, job shadow	LVEA	n	Mark table location	17:17
16:29	FAC	Nelly	FCES	n	Tech clean	17:14
16:32	VAC	Norco	CP2 (CS)	n	LN2 fill	18:19
16:49	CDS	Jonathan, Tony	MSR	n	Pulling digivideo0&1	18:03
16:54	FAC	Kim, Nelly	LVEA	n	Tech clean	18:16
16:55	SEI	Jim, Mitchell	Ends	n	Dropping off wind fence materials	17:58
17:10	VAC	Gerardo, Jackie	Ends, Mids	n	Vac rack UPSs	18:36
17:11	SYS	Betsy	LVEA	n	Check on HAM1 vent prep	17:11
17:30	VAC	Janos, job shadow	LVEA	n	Turbo test	18:45
17:31	SYS	Betsy	Garbing room	n	Check on covers and other	18:03
17:35	FAC	Tyler, Eric	EY	n	Replacing pump	18:23
17:41	ISC	Mayank, Sivananda, Matt	Opt Lab	Local	ISS PD array	20:12
17:46	VAC	Travis, Jordan	LVEA	n	HAM1 feedthrough check	18:18
18:25	-	Daniel, Fil, Betsy	LVEA	n	HAM1 vent prep	19:04
18:34	-	Richard	LVEA	n	Look at cleanrooms	18:42
18:38	VAC	Gerardo, Jackie	LVEA	n	Join on turbo testing	18:55
18:41	FAC	Eric	EY, MY	n	Glycol pumping	19:59
18:45	OPS	Tony	LVEA	n	Sweep	19:09
18:47	VAC	Marc, Fil	LVEA	n	BSC1 temperature sensor	18:55
18:58	VAC	Janos	LVEA	n	Turbo test end	19:04
19:47	PCAL	Dripta	PCAL Lab	Yes	Setting PS5//PS12 Measurement	20:00
20:31	CDS	Fil, Jackie, Marc	CER	n	Replacing power supply for susb123	20:33
20:34	FAC	Jim, Mitch	EndY	N	Wind fence investigation	21:36
22:19	ISC	Mayank, Siva	Opt Lab	Local	ISS PD array	ongoing

Ops Eve Shift Start

TITLE: 03/11 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 2mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY:

Observing at 152Mpc and have been Locked for 1 hour.

CDS Maintenance Summary: Tuesday 11th March 2025

WP12374 WP12370 ASC and SUS model changes.

Daniel, Jeff, Jonathan, Erik, TJ:

New models were installed for h1asc, h1susim, h1sushtts and h1suspr3. A DAQ restart was required.

WP12372 Add ISI HAM1 cards to h1seih16 IO Chassis

Fil, Marc, Erik, Dave, Jim, TJ:

h1seih16 was powered down. In its IO Chassis the bad A2 Adnaco backplane was replaced and the cards which had been moved to A3 were returned. Two additional ADCs and one additional BIO6464 were installed as per drawing (attached) which will be used by the new h1isiham1 model. Currently these cards have no external cables attached.

A new h1iopseih16 model was installed with these cards added. Note an additional 16bitDAC was not required because seismic ham models allocate a DAC per chamber, with the HEPI model using the lower 8 channels and the ISI model using the upper 8 channels. DAC-1 was already installed for h1hpiham1.

Note that this was one of two changes to the h1iopseih16 model (see SWWD section).

DAQ restart was required.

WP12373 Install new h1isiham1 Model

Jonathan, Erik, Jim, Dave:

The first version of the ISI HAM1 model was added to h1seih16. It is currently running with nothing connected to its ADC, DAC and BIO cards.

This is not something we do regularly, so we took the opportunity to document the steps needed. The list we have so far has been included as a spreadsheet.

The model was added to the DAQ. A related change to h1hpiham1 was also made. A DAQ restart was required for both.

Attachment shows the CDS Overvew MEDM h1seih16 section, now with a full complement of models.

WP12382 HAM1 SWWD

Jeff, Dave:

In preparation for HAM1 ISI install, we created a HAM1 Software Watchdog (SWWD) on h1iopsush2b and h1iopseih16. Note this is the second change to h1iopseih16 today.

There are two suspensions on HAM1; RM1 and RM2.

Despite its name, h1iopsush2b also controls ham1 suspensions. A SWWD part on h1iopsush2b reads the UL, LL, UR, LR raw ADC channels and calculates their integrated RMS values. If any value exceeds the trip level, the Dolphin IPC channel sent to h1iopseih16 is set to zero.

The IPC receiver on h1iopseih16, on receipt of a trigger signal, starts a DACKILL count-down timer. If the timer expires, it executes a DACKILL for the second DAC card, which drives HAM1 HEPI+ISI.

New h1iopsush2b and h1iopseih16 models were installed. The h1iopsush2b was done at the same time as the h1susim and h1sushtts models since these are the only models running on this front end. h1iopseih16 restart was done when this front end was powered back up after the IO Chassis work.

The SWWD MEDMs were updated with the new HAM1 system (attached)

Note: one complication is that up until now SUS SWWDs have been for larger suspensions with 6 OSEM channels (F1, F2, F3, LF, RT, SD). RM1,2 only have 4 (UL, LL, UR, LR) and so the two unused OSEM inputs have been grounded in the models. On the MEDM I just show ADC0_30, the channel prior to the duotone which is zero.

WP12358 Move Digital Video Cameras from h1digivideo2 to h1digivideo5

Patrick, Jonathan, Erik, TJ, Sheila, Dave:

We moved the six cameras which were on the last of the old servers (h1digivideo2) onto the new AMD/deb12 h1digivideo5 and powered down the old machine.

All was good until locking started, which is was discovered that for cameras which are copied to front end models the X,Y centroid EPICS channels now had offsets which the models could not handle.

For locking expediency it was decided to move the cameras back to the old server for now until the models could be configured to use to the new centroid values. Of the six models, five were moved back to h1digivideo2 (BS, ITMX_RED, ITMX_GREEN, ITMY_RED andITMY_GREEN). POP-AIR is not copied to FE models and so could stay on h1digivideo5.

At time of writing H1 has locked and been in OBSERVE for 50 minutes.

To "green up" the EDC I am running a dummy IOC on opslogin0 to serve the new channels for these cameras which were lost when we reverted them to the old software.

The camera MEDM shows which ones are still running on VID2 (attached).

DAQ Restarts:

Jonathan, Erik, Dave:

Due to the large number of changes we did two DAQ restarts today. The first was primarily for the ASC and SUS model changes and SWWD. The second was for the addition of the new h1isiham1 model and the camera move.

The EDC restart was due to new:

H1EPICS_DIGVIDEO.ini (at the time, moved 6 cameras to new server)

H1EPICS_DAQ.ini (add h1isiham1)

H1EPICS_FEC.ini (add h1isiham1)

H1EPICS_SDF.ini (add h1isiham1)

The camera server code on both h1digivideo4 and h1digivideo5 was updated from 0.1.16 to 0.1.17 to fix a bug in the periodic archiving of camera images (https://git.ligo.org/cds/software/cameras/pylon-camera-server/-/issues/6).

In addition Tony and Jonathan removed h1digivideo0 and h1digivideo1 from the racks as a cleanup job as part of the camera update WP.

Replaced batteries on the Vacuum UPS systems

(Jackie F., Gerardo M.)

Replaced the old batteries on the UPS units at both mid stations, both end stations, and two at the corner station.
The UPS units are dedicated supplies for the vacuum racks, the following racks were updated today:

• Corner station, h0vaclx (located by the X-beam manifold) and h0vacmr (located in the mechanical room).
• X-Mid, h0vacmx
• Y-Mid, h0vacmy
• X-End, h0vacex
• Y-End, h0vacey

No issues were encountered.  One set of batteries had been already installed previously, see aLOG 83151.

Work done under WP #12377.