Ops Day Shift Start

TITLE: 03/12 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 141Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 15mph Gusts, 11mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.31 μm/s
QUICK SUMMARY: Locked for just over and hour, looks like Ryan had to accept some SDFs overnight, and there were some ALS issues yesterday, but we had 3 relocks overnight. No alarms this morning.

 

OPS OWL assistance

Another ALS_Y sdf diff

13:33 UTC Observing

OPS OWL assistance

I had to accept two sets of SDF diffs to get us into observing, for HAM1 and ALS_Y.

09:16 UTC Observing

Ops Eve Shift End

TITLE: 03/12 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently relocking and in POWER_25W. We've been down for a while trying to figure out a weird issue with the ALSY WFS and we at least have a plan for the night, but it might end up really annoying for Ryan C unfortunately.

For Ryan C (and maybe TJ tomorrow morning): tagging OPSINFO

• If we are in initial alignment and ALSY keeps drifting down and then unlocking, have the initial alignment align ALSY using ETM_TMS_WFS_OFFLOADED instead of INITIAL_ALIGNMENT_OFFLOADED (You'll then need to manual to green arms offloaded in INIT_ALIGN and then continue the alignment)
• Once we get to ARMS_OFF_RESONANCE and you'll see ALSY transmission start dropping and DOF2 WFS diverging, turn off the ALSY DOF2 WFS for both pitch and yaw

I started an FRS ticket FRS#33562. Wasn't sure what category to put it under though so I moved it to be under operators, but that's definitely not right.

 

What we saw/what we tried to do to fix it:

After the lockloss earlier, I wanted to run an initial alignment since we had not run one since yesterday. I noticed that once the WFS would turn on for ALSY, the transmission would slowly drift down until it lost lock(ndscope1, ndscope2). I thought the issue might be DOF2 or 3, since both of those seemed to be diverging and DOF3 WFS were much larger than they usually are(ndscope3). I tried turning DOF2, DOF3, and both sets of WFS off, but we would just lose lock. I left IA and tried locking green arms the normal relocking way, which only uses DOF1 and DOF2 aka ETM and TMS, and that worked well, so I knew it had to be an issue with DOF3 specifically. I ended up needing an initial alignment, and either way we didn't want it to need an initial alignment in the middle of the night. Jenne suggested I take the YARM to LOCKED_SLOW_ETM_TMS_WFS and then slowly align ITMY to lower the ITMY camera error. I tried that, but I kept getting to a point where the DOF2 WFS stopped being able to handle it and would just diverge until we lost lock, even if I tried stepping back the other way, all while the camera errors didn't get much better. After trying this multiple times, Jenne came onto TS and we tried this method again with no success. We also tried adjusting ITMY (usually in Pitch) with YARM in LOCKED_SLOW_NO_WFS, again in LOCKED_SLOW_ETM_TMS_WFS but with DOF3 YAW WFS engaged, we tried resetting the slider values for ITMY, ETMY, and TMSY to the end of the last initial alignment, and we tried aligning MICH to get the beamsplitter in a better spot. None of these methods worked.
Eventually we decided that the only thing we could do for tonight was to have an initial alignment run with ALSY aligning only using ETMY and TMSY. Hopefully during the night there isn't a need for another initial alignment, because if there is Ryan C will need to change the states ALSY will need to go through to be ETM_TMS_WFS_OFFLOADED instead of INITIAL_ALIGNMENT_OFFLOADED.

Now, after finishing an initial alignment where we only used the ETMY and TMSY WFS, I've been trying to relock, but it looks like DOF2 diverging is still happening, ALSY goes down, and then ALSY loses lock(ndscope4). This wasn't happening during the relocks earlier in the day right after maintenance. To get around this, once we got to ARMS_OFF_RESONANCE and ALSY transmission started dropping, I turned off the ALSY DOF2 WFS in both pitch and yaw. We were able to get through the rest of the ALS states like this.

LOG:

22:30 Observing

23:45 We dropped out of Observing for a second and then went back in because I accidently clicked something monitored by SDF

01:02 Lockloss                                                                                                                                                                                                                                                                                                                                                                                                                         

Start Time	System	Name	Location	Lazer_Haz	Task	Time End
22:19	ISC	Mayank, Siva	Opt Lab	Local	ISS PD array	00:59
23:42	ISC	Keita	OpticsLab	n	WFS work	01:09

Lockloss

Lockloss @ 03/12 01:02 UTC

Ops Day Shift End

TITLE: 03/11 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: List of maintenance activities below, the most impactful to locking being the camera server change that we ended up having to revert to resume locking (alog83308). We will try to implement these cameras that are used in locking a different day when we can dedicate time to setting the IFO to their new offsets.

• Vacuum compressor replacement continues at EY, and cleanup from EX - 83066
• CP2 LN2 delivery
• UPS batteries replaced in all builds for VAC racks - 83313
• HAM1 ISI electronics were installed - 83309 83305
• FARO in LVEA
• Replaced, then reverted some, cameras on old server - 83308
• isiham1 model additions and DAQ restart - 83289
• PM1 infrastructure added, PR3 cable moved, other collateral models affected and changed - 83293 83288
• BSC1 temperature sensor troubleshooting - 83294
• susb123 power supply fan needed replacement, taking down those models - 83311
• EY chiller pump seal fixed - 83307
• EY well pump testing - 83291
• More I missed

Locking notes:

• 2230 Observing
  • Locking was automated the first go around, but then after we lost lock from the camera offsets, the BS in particular had been pulled far off and had to be brought back to get ALSY to lock. Everything else was automated though.

LOG:

Start Time	System	Name	Location	Lazer_Haz	Task	Time End
14:36	FAC	Nelly	Opt Lab	n	Tech clean	14:48
15:06	SUS	Jeff	CER	n	Move PR3 OpLev cable	16:01
15:06	FAC	Kim	EY	n	Tech clean	16:53
15:06	FAC	Nelly	EX	n	Tech clean	16:05
15:14	CDS	Dave, Erik, Jeff	CR	n	Model restarts (PR3, HTTS, IMs, SUSAUX) also SEIH16	15:35
15:17	CDS	Fil, Erik, Marc	CER	n	SEI rack work (HAM1)	18:21
15:18	FAC	Tyler	LVEA, Ends, Mids	n	3IFO checks	17:34
15:20	VAC	Ken	EY MR	n	Compressor electrics	18:45
15:25	FAC	Contractor	EY	n	Well testing - Large van, water truck, SUV	16:22
15:29	IAS	Jason, Ryan C	LVEA	n	FARO	18:59
15:35	OPS	Richard	EY	n	Check on panels and Ken	16:51
15:35	VAC	Janos, Travis, Jordan	EY, EX, mids	n	Compressor cleanup	18:18
15:51	PCAL	Rick, Francisco, Sivananda	EX	not yet	PCAL meas	17:35
16:17	VAC	Gerardo	LVEA	n	Vac rack UPSs	17:07
16:18	ISC	Camilla, job shadow	LVEA	n	Mark table location	17:17
16:29	FAC	Nelly	FCES	n	Tech clean	17:14
16:32	VAC	Norco	CP2 (CS)	n	LN2 fill	18:19
16:49	CDS	Jonathan, Tony	MSR	n	Pulling digivideo0&1	18:03
16:54	FAC	Kim, Nelly	LVEA	n	Tech clean	18:16
16:55	SEI	Jim, Mitchell	Ends	n	Dropping off wind fence materials	17:58
17:10	VAC	Gerardo, Jackie	Ends, Mids	n	Vac rack UPSs	18:36
17:11	SYS	Betsy	LVEA	n	Check on HAM1 vent prep	17:11
17:30	VAC	Janos, job shadow	LVEA	n	Turbo test	18:45
17:31	SYS	Betsy	Garbing room	n	Check on covers and other	18:03
17:35	FAC	Tyler, Eric	EY	n	Replacing pump	18:23
17:41	ISC	Mayank, Sivananda, Matt	Opt Lab	Local	ISS PD array	20:12
17:46	VAC	Travis, Jordan	LVEA	n	HAM1 feedthrough check	18:18
18:25	-	Daniel, Fil, Betsy	LVEA	n	HAM1 vent prep	19:04
18:34	-	Richard	LVEA	n	Look at cleanrooms	18:42
18:38	VAC	Gerardo, Jackie	LVEA	n	Join on turbo testing	18:55
18:41	FAC	Eric	EY, MY	n	Glycol pumping	19:59
18:45	OPS	Tony	LVEA	n	Sweep	19:09
18:47	VAC	Marc, Fil	LVEA	n	BSC1 temperature sensor	18:55
18:58	VAC	Janos	LVEA	n	Turbo test end	19:04
19:47	PCAL	Dripta	PCAL Lab	Yes	Setting PS5//PS12 Measurement	20:00
20:31	CDS	Fil, Jackie, Marc	CER	n	Replacing power supply for susb123	20:33
20:34	FAC	Jim, Mitch	EndY	N	Wind fence investigation	21:36
22:19	ISC	Mayank, Siva	Opt Lab	Local	ISS PD array	ongoing

Ops Eve Shift Start

TITLE: 03/11 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 2mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY:

Observing at 152Mpc and have been Locked for 1 hour.

CDS Maintenance Summary: Tuesday 11th March 2025

WP12374 WP12370 ASC and SUS model changes.

Daniel, Jeff, Jonathan, Erik, TJ:

New models were installed for h1asc, h1susim, h1sushtts and h1suspr3. A DAQ restart was required.

WP12372 Add ISI HAM1 cards to h1seih16 IO Chassis

Fil, Marc, Erik, Dave, Jim, TJ:

h1seih16 was powered down. In its IO Chassis the bad A2 Adnaco backplane was replaced and the cards which had been moved to A3 were returned. Two additional ADCs and one additional BIO6464 were installed as per drawing (attached) which will be used by the new h1isiham1 model. Currently these cards have no external cables attached.

A new h1iopseih16 model was installed with these cards added. Note an additional 16bitDAC was not required because seismic ham models allocate a DAC per chamber, with the HEPI model using the lower 8 channels and the ISI model using the upper 8 channels. DAC-1 was already installed for h1hpiham1.

Note that this was one of two changes to the h1iopseih16 model (see SWWD section).

DAQ restart was required.

WP12373 Install new h1isiham1 Model

Jonathan, Erik, Jim, Dave:

The first version of the ISI HAM1 model was added to h1seih16. It is currently running with nothing connected to its ADC, DAC and BIO cards.

This is not something we do regularly, so we took the opportunity to document the steps needed. The list we have so far has been included as a spreadsheet.

The model was added to the DAQ. A related change to h1hpiham1 was also made. A DAQ restart was required for both.

Attachment shows the CDS Overvew MEDM h1seih16 section, now with a full complement of models.

WP12382 HAM1 SWWD

Jeff, Dave:

In preparation for HAM1 ISI install, we created a HAM1 Software Watchdog (SWWD) on h1iopsush2b and h1iopseih16. Note this is the second change to h1iopseih16 today.

There are two suspensions on HAM1; RM1 and RM2.

Despite its name, h1iopsush2b also controls ham1 suspensions. A SWWD part on h1iopsush2b reads the UL, LL, UR, LR raw ADC channels and calculates their integrated RMS values. If any value exceeds the trip level, the Dolphin IPC channel sent to h1iopseih16 is set to zero.

The IPC receiver on h1iopseih16, on receipt of a trigger signal, starts a DACKILL count-down timer. If the timer expires, it executes a DACKILL for the second DAC card, which drives HAM1 HEPI+ISI.

New h1iopsush2b and h1iopseih16 models were installed. The h1iopsush2b was done at the same time as the h1susim and h1sushtts models since these are the only models running on this front end. h1iopseih16 restart was done when this front end was powered back up after the IO Chassis work.

The SWWD MEDMs were updated with the new HAM1 system (attached)

Note: one complication is that up until now SUS SWWDs have been for larger suspensions with 6 OSEM channels (F1, F2, F3, LF, RT, SD). RM1,2 only have 4 (UL, LL, UR, LR) and so the two unused OSEM inputs have been grounded in the models. On the MEDM I just show ADC0_30, the channel prior to the duotone which is zero.

WP12358 Move Digital Video Cameras from h1digivideo2 to h1digivideo5

Patrick, Jonathan, Erik, TJ, Sheila, Dave:

We moved the six cameras which were on the last of the old servers (h1digivideo2) onto the new AMD/deb12 h1digivideo5 and powered down the old machine.

All was good until locking started, which is was discovered that for cameras which are copied to front end models the X,Y centroid EPICS channels now had offsets which the models could not handle.

For locking expediency it was decided to move the cameras back to the old server for now until the models could be configured to use to the new centroid values. Of the six models, five were moved back to h1digivideo2 (BS, ITMX_RED, ITMX_GREEN, ITMY_RED andITMY_GREEN). POP-AIR is not copied to FE models and so could stay on h1digivideo5.

At time of writing H1 has locked and been in OBSERVE for 50 minutes.

To "green up" the EDC I am running a dummy IOC on opslogin0 to serve the new channels for these cameras which were lost when we reverted them to the old software.

The camera MEDM shows which ones are still running on VID2 (attached).

DAQ Restarts:

Jonathan, Erik, Dave:

Due to the large number of changes we did two DAQ restarts today. The first was primarily for the ASC and SUS model changes and SWWD. The second was for the addition of the new h1isiham1 model and the camera move.

The EDC restart was due to new:

H1EPICS_DIGVIDEO.ini (at the time, moved 6 cameras to new server)

H1EPICS_DAQ.ini (add h1isiham1)

H1EPICS_FEC.ini (add h1isiham1)

H1EPICS_SDF.ini (add h1isiham1)

The camera server code on both h1digivideo4 and h1digivideo5 was updated from 0.1.16 to 0.1.17 to fix a bug in the periodic archiving of camera images (https://git.ligo.org/cds/software/cameras/pylon-camera-server/-/issues/6).

In addition Tony and Jonathan removed h1digivideo0 and h1digivideo1 from the racks as a cleanup job as part of the camera update WP.

Replaced batteries on the Vacuum UPS systems

(Jackie F., Gerardo M.)

Replaced the old batteries on the UPS units at both mid stations, both end stations, and two at the corner station.
The UPS units are dedicated supplies for the vacuum racks, the following racks were updated today:

• Corner station, h0vaclx (located by the X-beam manifold) and h0vacmr (located in the mechanical room).
• X-Mid, h0vacmx
• Y-Mid, h0vacmy
• X-End, h0vacex
• Y-End, h0vacey

No issues were encountered.  One set of batteries had been already installed previously, see aLOG 83151.

Work done under WP #12377.

Kepco Power Supply For ITMY/ITMX/BS Replaced

WP 12384

Power supply that provides ±18V to SUS Rack SUS-C6 (ITMY, ITMX, BS) was replaced. Fan bearings was starting to fail, causing loud grinding sound. Coordinated work with control room as this required powering off the the SUS Watchdog chassis.

F. Clara, J. Figueroa

IO Chassis H1seih16 - HAM1 ISI Upgrade

WP 12372

The h1seih16 IO chassis was upgraded with new hardware in preparation for the HAM1 ISI installation:

1. Adnaco Card (Backplane slot A2)
2. ADC cards - QTY 2
3. ADC adapter cards and ribbon cables - QTY 2 each
4. Binary 6464 card

The Adnaco in slot A2 was replaced. Timing issues reported in May 2023, alog 70010. The existing ADCs and DAC were moved to the new Adnaco card slot A2. The two new ADC’s were installed in Adnaco card slot A3.

D. Barker, F. Clara, E. Von Reis

New camera server introduced offsets into calculated X & Y values

Patrick T, Dave B, Jonathan H, Sheila D, TJ S

Summary: Today during maintenance more of the cameras were swapped over to the new camera code on a new server. This caused a lock loss since there was an offset with the new camera code, so the code and servers were swapped back for most cameras.

Some of these swapped cameras are ones we use in locking. At the end of maintenance, I skipped over initial alignment and started to lock the IFO. When we got to the ISC_LOCK state ADS_TO_CAMERAS, we could see the PRG get pulled away over a minute, as well as other signals on the FOMS, before we lost lock. Looking into it, the calulated P & Y, or X & Y depending on which channel you're looking at, channels were different from before the camera swap. The offset that this created moved us away from our alignment setpoints to cause the lock loss. Had I ran an initial alignment we might have run into this soon as we use the green ITM cameras, but we might have lost our alignment references there as well.

While it is possible to find and try to adjust for all of the offsets that the new code introduced, it would be much safer to do this after the IFO achieved full lock and we could transition the IFO off of the cameras, switch to the new code/servers, and then see what the offset is and adjust accordingly. Doing it while not locked risks losing our saved offsets, so we decided to revert the code and server changes for today and plan for another day to implement this.

 

End Y chiller pump repair

The leaking pump seal in the chiller 1 pump has been repaired at End Y. 

HAM1 ISI Electronics

WP 12366
ECR E2200171

The following electronics were installed in the SEI-C1 rack:

U42: BSC-ISI Interface S1203890
U41: BSC-ISI Interface S1203891
U40: BSC-ISI Interface S1203983

U39: HAM ISI AA Interface S1202074
U38: HEPI AI S1202068
U37: HAM ISI AA Interface S1202075

U36: ISI Coil Driver S1103327
U34: ISI Coil Driver S1103350

U32: Binary In Interface S1203076

Part of the HAM1 ISI installation. Units left powered off. Field cabling arrived onsite today, will be installed next Tuesday.

F. Clara, J. Figueroa, M. Pirello

LVEA Sweep.

The LVEA has been swept.
 
 

EY compressor installation progress

At the EY station the compressor is being replaced - after the one at EX is done. In this aLog, in the comments, the progress of this operation is tracked continuously, until the 1st startup by the supplier, Rogers Machinery.
Another important consideration here, is that the purge line at EY needs to be replaced (based on an FTIR test - see DCC LIGO-E2300222-v2. As it can be seen, the level of contamination reaches even the 10 ug/cm2 value). This operation will be done after the April-May vent, so the EY station will be ready to be vented after O4.

02-25 (maintenance Tuesday): the old compressor was pulled out (it is temporarily stored in the EY receiving area). The beginning of the purge and TMDS lines with the associated brackets and unistruts were taken off. The new compressor unit and dryer skid were anchored in the mechanical room. Here it is important to mention that the orientation of the inlet was brought closer to the purge line inlet into the VEA, so the overall length of the associated circulation lines will be much shorter.
Next is the electrical and pneumatic installation, which will be completed in the next 1-2 weeks.

The 1st startup of the compressor was carried out by Rogers Machinery on March 18th, during maintenance hours.

OPS OWL report SDF diffs

To get in Observing I had to accept some SDF diffs for SQZ, and PCALY. There was also still a PEM CS excitation point open as well. There was a notification about PCALY OFS servo malfunction so I looked at it and it was railed at -7.83, so I toggled it off and back on and it brought it back to a good value. I also did not receive a call, a voicemail just appeared.

09:21 UTC observing