OMCS was touching.
Rahul made OMCS transfer functions and found that something was touching. He, Koji and Betsy went to the floor to center the OSEMs. T1 was touching, which was fixed. Depth of OSEMs were adjusted, too.

Something is funny about OM3 damping.

Rahul also made TFs for OMs and thought that they were fine. However, OM3 PIT damping was oscillating at 1.75Hz. I flipped the sign of H1:SUS-OM3_M1_DAMP_P_GAIN (nominally -1, now +1) and it damped fine. I hate this.

1st attachment red shows the OLTF of OM3 P damping loop at around the offending resonance. This was of course measured with the flipped damping gain, and the OLTF looks absolutely strange. The 1st UGF is ~1.73Hz with -42deg phase, and the 2nd UGF is ~1.77Hz with -180deg phase.

If you draw a Nyquist diagram, you will find that this is a stable TF. OTOH, with the "correct" damping gain, this will become unstable. So we know why this works and why the correct sign doesn't, but we don't know what changed and why. 

Alignment converges.

With the OMC suspended without any biases, we adjusted OM1/2/3 biases to align the beam into the OMC (i.e. centering both of the OMC QPDs).

At some point we found that the ASC-AS_C was totally off-centered, used SRM to recenter, and continued the alignment.

2nd attachment shows the status as of now. Both of the OMC QPDs are reasonably centered, ASC-AS_C too. BOSEM DAC output for OM1 is ~11k max. OM2 and OM3 are at ~7k max  and ~9k max, respectively. These are OK though OM1 is not super.

Current status of the LVEA etc.

It's laser hazard. SQZ manager was set to DOWN. Koji closed small HAM5 GV and HAM7 -X curtain.

Remaining work (Monday).

• Fix the balance mass for OMCS.
• Check the beam positions on OM1, OM2 and OM3 (again).
• Adjust the WFS / OMCR paths.
• OMC trans video path?
• Restore the shroud panels.
• Restore the OMC trans beamdump on the -Y side.
• Check the centering on the irises on HAM6.

Rick and I cleaned the optics out here at End X.
Noted a drop in MAX Diffracted Power between M5 and the Wedged Beam Splitter from 1.1W to 1.08 W.
This may be due to the laser being turned off and not thermalized.

We would like to come back to End X and adjust the AOM alignment and we are leaving PCAL in a LASER SAFE status, LASER OFF and Key out.

Today's activities:
- HAM3 was prepared for door removal: the AIP was valved out, and the Annulus system was vented with N2
- EX was fully prepared for pumpdown, which happens next week
- RGA at EX was finished, it's totally leak tight - although its Ion-pump needs some troubleshooting
- There was a solution found for the filters' installation to the Hepta headers: the ISO-fitting will be done with a unique part (https://ancorp.com/product/4500060/), as the more standard claw clamps cannot be fit there
- The 2nd bellows for the relay tube was located - it needs class-A C+B
- Dew points were measured at the corner and at EX, all is good: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=75590

Marc, Erik, Corey, Betsy, Jim, Dave:

h1seih16 and h1oaf0 were power cycled to recover them from this afternoon's crashes. Sequence is (* applies to h1seih16 only)

1. stop all models
2. fence from Dolphin fabric
3. power down fe computer
4. * power down AI chassis
5. power down IO Chassis
6. power up IO Chassis
7. power up fe computer
8. * power up AI chassis

Everything restarted with no issues.

Jim has requested that no watchdogs be untripped on the h1seih16 system. The first DAC (HAM6 ISI+HPI) is still in its DACKILL state because the SWWD is untripped. The second DAC (HAM1 HPI) has zero drive because the h1hpiham1 model is untripped.

Latched IPC and DAQ-CRC errors have been cleared.

Replaced the Endevco Accelerometer Power Conditioners with LIGO Accelerometer Power Conditioners. WP11653

Order in the rack:

U19 - Accelerometer Power Conditioner S2300062

U16-U17 AA Chassis S1300101

U14 - Accelerometer Power Conditioner S2300063

U13 - Accelerometer Power Conditioner S2300065

U12 - Accelerometer Power Conditioner S2300069

U11 - Accelerometer Power Conditioner S2300064

Still have EX, EY, MX, MY, FCES to complete.

M. Pirello, J. Jimerson. R. Schofield

TITLE: 01/26 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:

Afternoon shift's main item of note  is the CRASH of the OAF front end as well as the SEIH16 model; recovery of this would be made after the shift (after CER accelerometer work).

OMC aligning (from the Control Room) has also been ongoing through the afternoon by Keita/Koji/Rahul.
LOG:

• 2102 Tagging in for Oli's shift (corey)
• 2113-2151 MY part search (gerardo)
• 2123-2144 Hartman table checks (camilla)
• 2153 HAM3 baffle bolt assy prep in HAM3 cleanroom (mitch, tony)
• 2145-0000 Accelerometer work in CER (marc, jabari)
• 2207-2252 HAM3 vent prep (jordan, janos)
• 2223 OAF model crash (On CDS Overview, we have Front End & ADC faults/red, but also have issues with HAM SEI1/6 model [which is a surprise]).  Suspect this is due to the Accelerometer work in the CER.  Plan is to let Marc finish his work before starting front end recovery work.
• 2249 External Alert tests (tj)
• 2306-2329 Locking HAM3 HEPI (jim)
• 2307-2347 EX pump down prep (jordan)
• 2318 Grabbing ladder at EY (ken)

Measured the purge air dew point at the corner station during "lunch break", no people in chamber, measured -42.0 degrees C.

Travis measured the purge air dew point at EX, all ports closed, purge valve closed, no usage from vacuum volume.  Measured at purge valve area -41.3 degrees C.

all the models on h1oaf0 have stopped running. At time of writing it looks like a possible timing glitch caused by in-rack work in the CER. We will hold off on restarts until the rack wiring work is completed because there are ongoing alignment measurements.

Fri Jan 26 12:33:45 2024 INFO: Fill completed in 3min 42secs

Second time was a charm, Gerardo confirmed a good fill curbside.

Replaced the 6 port +24V  DC power strip with a 24 port DC power strip in the TCS / PEM racks per WP11649

Marc, Oli, Dave:

For Marc's change over to a bigger +24V power strip in the OAF/TCS CER rack, I powered down the h1oaf0 frontend to permit its IO Chassis to be power cycled.

Sequence was

• h1oaf0 model SDF diffs captured (see attachments)
• h1oaf0 models stopped
• h1oaf0 fenced from Dolphin fabric
• h1oaf0 computer powered down 11:58 PST
• Marc in CER powered off IO Chassis and Beckhoff chassis, rerouted power cords to new strip, powered chassis back on.
• h1oaf0 computer powered up, models started.
• I did a CDS overview cleanup, reset IPC errors and DAQ CRC errors.

All is looking good from a CDS perspective.

FRS30184

Patrick, Jonathan, Erik, Dave:

A summary and follow up on the recent FMCS EPICS IOC freeze ups.

Timeline (all times local)

Sat 13 Jan 15:57 FMCS IOC flatlining started. After restart IOC would run anywhere from 1 to 14 hours before flatlining again

Tue 16 Jan 23:12 FMCS IOC running under systemd control, auto-restart code running which restarts IOC if flatlined for 10 mins

Wed 17 Jan 10:48 fmcs-epics-cds machine power cycled

Thu 18 Jan 11:25 After power cycle, IOC ran 25 hours with no flatline, previous longest run 14 hours [Power cycle of computer fixed it]

Thu 18 Jan 11:25 Patrick installed new version of the FMCS IOC code. This added new diagnostic EPICS channels.

Thu 18 Jan 14:58 systemd control of FMCS IOC under puppet configuration management

Tue 23 Jan 10:33 DAQ+EDC restarted to trend new FMCS diagnostic channels

Summary:

After running for many years error free, and 90 days after the last computer reboot, the FMCS IOC code became unstable. At random times it would stop updating its EPICS values, flatlining them at their last value.

The code at this point was started manually and ran in a screen environment.

To facilitate the auto restart of the code while the problem was being investigated, the IOC code was moved to a procServ environment and put under systemd control. A script running on cdsmanager monitored the FMCS channel H0:FMC-EX_CY_H2O_SUP_DEGF every minute. If its value did not change for 10 minutes, the systemd fmcs_ioc.service was restarted on fmcs-epics-cds.

A soft reboot of fmcs-epics-cds did not fix the problem. We then tried a hard power down, wait 30 seconds, then power back on. This appeared to fix the immediate problem.

We decided to upgrade the software to see if this would prevent future occurances, in 90+ days time.

At time of writing, +8 days after upgrade, there have been zero flatline instances.

So far, the O4 break vent & commissioning work is proceeding as planned, with only minor hiccups.  Yesterday's TCS water leak caused a pause to the HAM6 in-chamber OMC alignment work, but we were able to resume this morning pretty quickly with only a short amount of time down.

The attached snapshot shows the planned schedule outline with green checkmarks next to the items completed so far on the left hand side.  On the right is a running log of the activities which have taken place daily (for ease of viewing all in one place).

Teams currently on:

• HAM6 new OMC in-chamber alignment
• EX prepping to pump down post cryo baffle damper install, ETMX SUS appears healthy with few TFs spot checked
• Prep for HAM3 baffle work next week, chamber cleaned
• Starting EY wind fence repair
• TCS system maintenance
• FARO setup in LVEA for monument surveying

Today's activities:
- 3 viewports have been taken off at EX for baffle works. These have been inspected, and found intact in every aspect - the inspection report is in the comment section
- These viewports have been already taken back, as the works were already finished
- The EX RGA is being rebuilt after many leaks have been found - it is in progress, will be finished tomorrow
- The new Hepta header elbow pieces have been staged at all stations

Today's activities:
- The RGA tree at EX was leak-checked, and multiple flanges were leaky, possibly because of some overheating during a bakeout in the past. The gaskets will be redone, and the all-metal angle valve will be also replaced
- EX was vented
- The VPW vacuum chamber was opened, after the 2 stuck screw was removed
- Both 2 spare roughing stations are entirely finished
- All the Hepta header filter works were finished in the LVEA

Dana, Jenne

We looked at how the microseism levels have impacted the detector duty factor since the start of aLIGO to see if our ability to maintain lock during microseisms has gotten better or worse. The attached figure shows the fraction of time the detector maintained lock over the course of a week as a function of the average microseism level over that same week for each week in O4a, as well as each week in (1) O1, (2) O2, (3) O3a, and (4) O3b.

According to the data, it seems we may be slightly worse off in terms of our ability to maintain lock during high levels of microseisms in O4 than we were in O3, but it seems we are better off now than we were in O1 and O2. However, it is difficult to directly compare the data between observing runs given that during certain runs the average level of microseisms seemed consistently higher than in others. There were also certainly other sources of lockloss besisdes microseisms. But it can at least be said that maintaining lock during microseisms does not appear to be a bigger issue now than it used to be.

Note: The two large data gaps present in O2 were removed.