all the models on h1oaf0 have stopped running. At time of writing it looks like a possible timing glitch caused by in-rack work in the CER. We will hold off on restarts until the rack wiring work is completed because there are ongoing alignment measurements.

Fri Jan 26 12:33:45 2024 INFO: Fill completed in 3min 42secs

Second time was a charm, Gerardo confirmed a good fill curbside.

Replaced the 6 port +24V  DC power strip with a 24 port DC power strip in the TCS / PEM racks per WP11649

Marc, Oli, Dave:

For Marc's change over to a bigger +24V power strip in the OAF/TCS CER rack, I powered down the h1oaf0 frontend to permit its IO Chassis to be power cycled.

Sequence was

• h1oaf0 model SDF diffs captured (see attachments)
• h1oaf0 models stopped
• h1oaf0 fenced from Dolphin fabric
• h1oaf0 computer powered down 11:58 PST
• Marc in CER powered off IO Chassis and Beckhoff chassis, rerouted power cords to new strip, powered chassis back on.
• h1oaf0 computer powered up, models started.
• I did a CDS overview cleanup, reset IPC errors and DAQ CRC errors.

All is looking good from a CDS perspective.

Dana, Louis

We analyzed every measurement of the sensing function taken between the start of O4 and October 27th to see if they were reliable and came up with the following, summarized in the table below:

Ideally, the detector should be in lock state at least three hours before making a sensing function measurement to make sure the thermalization process is complete. However, there were a couple measurements that were made when the detector had only been locked for about two hours (06/21, 08/02), and there was one particularly problematic measurement that was made when the detector had only been locked for about 10 minutes (05/05). This last measurement should certainly not be included in the GPR calculation.

The code used to obtain the detector lock state and history given a report ID is attached below. Note: To run this code, you will need access to pydarm, so run the following command in the terminal before executing the file: source /ligo/groups/cal/local/bin/activate

FRS30184

Patrick, Jonathan, Erik, Dave:

A summary and follow up on the recent FMCS EPICS IOC freeze ups.

Timeline (all times local)

Sat 13 Jan 15:57 FMCS IOC flatlining started. After restart IOC would run anywhere from 1 to 14 hours before flatlining again

Tue 16 Jan 23:12 FMCS IOC running under systemd control, auto-restart code running which restarts IOC if flatlined for 10 mins

Wed 17 Jan 10:48 fmcs-epics-cds machine power cycled

Thu 18 Jan 11:25 After power cycle, IOC ran 25 hours with no flatline, previous longest run 14 hours [Power cycle of computer fixed it]

Thu 18 Jan 11:25 Patrick installed new version of the FMCS IOC code. This added new diagnostic EPICS channels.

Thu 18 Jan 14:58 systemd control of FMCS IOC under puppet configuration management

Tue 23 Jan 10:33 DAQ+EDC restarted to trend new FMCS diagnostic channels

Summary:

After running for many years error free, and 90 days after the last computer reboot, the FMCS IOC code became unstable. At random times it would stop updating its EPICS values, flatlining them at their last value.

The code at this point was started manually and ran in a screen environment.

To facilitate the auto restart of the code while the problem was being investigated, the IOC code was moved to a procServ environment and put under systemd control. A script running on cdsmanager monitored the FMCS channel H0:FMC-EX_CY_H2O_SUP_DEGF every minute. If its value did not change for 10 minutes, the systemd fmcs_ioc.service was restarted on fmcs-epics-cds.

A soft reboot of fmcs-epics-cds did not fix the problem. We then tried a hard power down, wait 30 seconds, then power back on. This appeared to fix the immediate problem.

We decided to upgrade the software to see if this would prevent future occurances, in 90+ days time.

At time of writing, +8 days after upgrade, there have been zero flatline instances.

Closes FAMIS#26439, last checked 74987

BRS Driftmon (attachment1)
All within range, however in the last day, BRS-X has increased steeply.

Aux BRS Channels (attachment2)
ETMX BRS Temp has declined in temperature over the last day, lining up with the increase in BRS-X Drift.

Fri Jan 26 10:03:24 2024 INFO: Fill completed in 3min 22secs

Dubious fill. From the discharge line pressure it looks like a reasonable fill. From the TC perspective it was not optimal, TC-B min was only -72C which barely tripped the fill. There is no indication of a steady LN2 flow after the trip.

I've zoom in on the plot to show the TC details.

Closes FAMIS#26278, last checked 75471

Corner Station Fans (attachment1)
MR_FAN5_170_2 became noisier after being turned off and back on last Thursday (attachment2), but is still well below being of concern and has been getting quieter over the course of the week.
All other fans are looking normal and within range.

Outbuilding Fans (attachment3)
All fans are looking normal and within range.

Closes FAMIS#26228, last checked 75219

Laser Status:
    NPRO output power is 1.819W (nominal ~2W)
    AMP1 output power is 68.31W (nominal ~70W)
    AMP2 output power is 138.9W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN

PMC:
    It has been locked 5 days, 19 hr 23 minutes
    Reflected power = 18.55W
    Transmitted power = 108.4W
    PowerSum = 127.0W

FSS:
    It has been locked for 10 days 1 hr and 27 min
    TPD[V] = 0.5292V

ISS:
    The diffracted power is around 1.9%
    Last saturation event was 10 days 1 hours and 59 minutes ago

Possible Issues:
    PMC reflected power is high
    FSS TPD is low
    ISS diffracted power is low

So far, the O4 break vent & commissioning work is proceeding as planned, with only minor hiccups.  Yesterday's TCS water leak caused a pause to the HAM6 in-chamber OMC alignment work, but we were able to resume this morning pretty quickly with only a short amount of time down.

The attached snapshot shows the planned schedule outline with green checkmarks next to the items completed so far on the left hand side.  On the right is a running log of the activities which have taken place daily (for ease of viewing all in one place).

Teams currently on:

• HAM6 new OMC in-chamber alignment
• EX prepping to pump down post cryo baffle damper install, ETMX SUS appears healthy with few TFs spot checked
• Prep for HAM3 baffle work next week, chamber cleaned
• Starting EY wind fence repair
• TCS system maintenance
• FARO setup in LVEA for monument surveying

TJ, Jason, Camilla, Fil.

WP11612, FRS25384, swapping because current CO2X laser was slowly deteriorating, details in 75249.

Removed: Access 50LT 20706.21015D. Installed 20306-20419D, which was refurbished/re-gassed in March 2020. Table layout: T1200007.

• On Tuesday 23rd:
  • Checked polarizers on both CO2X and CO2Y (pictures) and spares were fine after issues at LLO (FRS30220). Will update FRS with details. 
  • Added labels to RF cables and water tubes on CO2 laser and RF driver.
  • Scribed current position of CO2 laser and photographed.
• On Wednesday 24^th AM:
  • Placed beam dump in front of periscope.
  • Remembered L1 has some damage on lens, caused by the plastic lens ring, photo. In future we should use metal lens rings.   
  • Turned on CO2 laser.
  • Check position on FLIR, with no mask, central and annual masks (needed to turn on FLIR – power supply inside enclosure).
  • Checked beam is going though all irises with beam card kit and FLIR one camera. Pictures
    • On masks: beam slightly high/masks low
    • On irises – beam a little low, but well centered with masks in.
    • Will not adjust irises 
  • Turned off: chiller, power supply, RF and power supply in mechanical room mezzanine,  AA chassis
  • Disconnected water lines (outside CO2 table). Drained table lines with gravity and wet-vac.
    • The small on table water lines have a lot of old contamination in them, photos of RF driver and CO2 lines, comment added to FRS10612 so we can think about replacing these lines post O4. 
  • Disconnected all cables and water pipes to CO2X/RF driver and removed laser and RF driver. 
  • Installed new CO2 laser (needed to move back foot back one set of bolt holes and turn water connections around 180deg). 
  • Installed new RF driver after Fil reattached it's electrical plug
  • Reconnected all cables and water pipes (expect table to main line connection). Photos of installed laser and RF driver.
  • Powered on CO2X chiller (had been turn off in medm so usually wouldn't turn on but it did)
  • The CO2X water pipe coming from the mezzanine leaked under the high pressure (as not yet connected to table), spraying water onto the cable trays above SQZ0/SQZT7 and SQZT0. 
  • Betsy initiated stop work M2400022, we transitioned to laser safe and Fil disconnected electrical connections. 
  • Richard, Fil, Jason and TJ dried the area as best they could. We see no sign of water inside SQZT0. 
  • Will reassess next steps on Thursday 25th. 

Still to do after the CO2X cooling line issue is solved:

• Refill table with water and reconnect to chiller.
• Turn on laser in PWM mode for low power realigning.
• Realign laser to irises and FLIR camera using M1 and PM1 mirrors
• Measure power out of laser head using S322C
• Check alignment to power meter and masks
• During commissioning time: spend ~2 hours realigning in-vac using HWS and periscope picos. 

Today's activities:
- 3 viewports have been taken off at EX for baffle works. These have been inspected, and found intact in every aspect - the inspection report is in the comment section
- These viewports have been already taken back, as the works were already finished
- The EX RGA is being rebuilt after many leaks have been found - it is in progress, will be finished tomorrow
- The new Hepta header elbow pieces have been staged at all stations

Dana, Jenne

We looked at how the microseism levels have impacted the detector duty factor since the start of aLIGO to see if our ability to maintain lock during microseisms has gotten better or worse. The attached figure shows the fraction of time the detector maintained lock over the course of a week as a function of the average microseism level over that same week for each week in O4a, as well as each week in (1) O1, (2) O2, (3) O3a, and (4) O3b.

According to the data, it seems we may be slightly worse off in terms of our ability to maintain lock during high levels of microseisms in O4 than we were in O3, but it seems we are better off now than we were in O1 and O2. However, it is difficult to directly compare the data between observing runs given that during certain runs the average level of microseisms seemed consistently higher than in others. There were also certainly other sources of lockloss besisdes microseisms. But it can at least be said that maintaining lock during microseisms does not appear to be a bigger issue now than it used to be.

Note: The two large data gaps present in O2 were removed.