Fri Jan 26 12:33:45 2024 INFO: Fill completed in 3min 42secs

Second time was a charm, Gerardo confirmed a good fill curbside.

Replaced the 6 port +24V  DC power strip with a 24 port DC power strip in the TCS / PEM racks per WP11649

Marc, Oli, Dave:

For Marc's change over to a bigger +24V power strip in the OAF/TCS CER rack, I powered down the h1oaf0 frontend to permit its IO Chassis to be power cycled.

Sequence was

• h1oaf0 model SDF diffs captured (see attachments)
• h1oaf0 models stopped
• h1oaf0 fenced from Dolphin fabric
• h1oaf0 computer powered down 11:58 PST
• Marc in CER powered off IO Chassis and Beckhoff chassis, rerouted power cords to new strip, powered chassis back on.
• h1oaf0 computer powered up, models started.
• I did a CDS overview cleanup, reset IPC errors and DAQ CRC errors.

All is looking good from a CDS perspective.

Dana, Louis

We analyzed every measurement of the sensing function taken between the start of O4 and October 27th to see if they were reliable and came up with the following, summarized in the table below:

Ideally, the detector should be in lock state at least three hours before making a sensing function measurement to make sure the thermalization process is complete. However, there were a couple measurements that were made when the detector had only been locked for about two hours (06/21, 08/02), and there was one particularly problematic measurement that was made when the detector had only been locked for about 10 minutes (05/05). This last measurement should certainly not be included in the GPR calculation.

The code used to obtain the detector lock state and history given a report ID is attached below. Note: To run this code, you will need access to pydarm, so run the following command in the terminal before executing the file: source /ligo/groups/cal/local/bin/activate

FRS30184

Patrick, Jonathan, Erik, Dave:

A summary and follow up on the recent FMCS EPICS IOC freeze ups.

Timeline (all times local)

Sat 13 Jan 15:57 FMCS IOC flatlining started. After restart IOC would run anywhere from 1 to 14 hours before flatlining again

Tue 16 Jan 23:12 FMCS IOC running under systemd control, auto-restart code running which restarts IOC if flatlined for 10 mins

Wed 17 Jan 10:48 fmcs-epics-cds machine power cycled

Thu 18 Jan 11:25 After power cycle, IOC ran 25 hours with no flatline, previous longest run 14 hours [Power cycle of computer fixed it]

Thu 18 Jan 11:25 Patrick installed new version of the FMCS IOC code. This added new diagnostic EPICS channels.

Thu 18 Jan 14:58 systemd control of FMCS IOC under puppet configuration management

Tue 23 Jan 10:33 DAQ+EDC restarted to trend new FMCS diagnostic channels

Summary:

After running for many years error free, and 90 days after the last computer reboot, the FMCS IOC code became unstable. At random times it would stop updating its EPICS values, flatlining them at their last value.

The code at this point was started manually and ran in a screen environment.

To facilitate the auto restart of the code while the problem was being investigated, the IOC code was moved to a procServ environment and put under systemd control. A script running on cdsmanager monitored the FMCS channel H0:FMC-EX_CY_H2O_SUP_DEGF every minute. If its value did not change for 10 minutes, the systemd fmcs_ioc.service was restarted on fmcs-epics-cds.

A soft reboot of fmcs-epics-cds did not fix the problem. We then tried a hard power down, wait 30 seconds, then power back on. This appeared to fix the immediate problem.

We decided to upgrade the software to see if this would prevent future occurances, in 90+ days time.

At time of writing, +8 days after upgrade, there have been zero flatline instances.

Closes FAMIS#26439, last checked 74987

BRS Driftmon (attachment1)
All within range, however in the last day, BRS-X has increased steeply.

Aux BRS Channels (attachment2)
ETMX BRS Temp has declined in temperature over the last day, lining up with the increase in BRS-X Drift.

Fri Jan 26 10:03:24 2024 INFO: Fill completed in 3min 22secs

Dubious fill. From the discharge line pressure it looks like a reasonable fill. From the TC perspective it was not optimal, TC-B min was only -72C which barely tripped the fill. There is no indication of a steady LN2 flow after the trip.

I've zoom in on the plot to show the TC details.

Closes FAMIS#26278, last checked 75471

Corner Station Fans (attachment1)
MR_FAN5_170_2 became noisier after being turned off and back on last Thursday (attachment2), but is still well below being of concern and has been getting quieter over the course of the week.
All other fans are looking normal and within range.

Outbuilding Fans (attachment3)
All fans are looking normal and within range.

Closes FAMIS#26228, last checked 75219

Laser Status:
    NPRO output power is 1.819W (nominal ~2W)
    AMP1 output power is 68.31W (nominal ~70W)
    AMP2 output power is 138.9W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN

PMC:
    It has been locked 5 days, 19 hr 23 minutes
    Reflected power = 18.55W
    Transmitted power = 108.4W
    PowerSum = 127.0W

FSS:
    It has been locked for 10 days 1 hr and 27 min
    TPD[V] = 0.5292V

ISS:
    The diffracted power is around 1.9%
    Last saturation event was 10 days 1 hours and 59 minutes ago

Possible Issues:
    PMC reflected power is high
    FSS TPD is low
    ISS diffracted power is low

Late entry.
Activities at 1-25:
- The RGA was completely rebuilt at EX, and the RGA tree was already leak checked. Currently it is pumped, and it is already at the E-9 Torr region
- The 12" 3IFO feedthrough was replaced with a blank on BSC5 at EX
- The spare roughing stations were completely finished
So far, everything went as scheduled.

Mechanical

• Alignment check: Confirm if the current mechanical Pitch/Yaw of OMC3 and OMCS are sufficient. -> (If not, touch OMC3 and OMCS more)
• Check the bias values of OMC1/2/3 and OMCS ->(Do we need any debias?)
• Check if the OSEM DC values shows the light close to half shadowed.
• Check if the TFs of the OMC1/2/3 and OMCS are still acceptable.
• Adjust the WFS / OMCR paths
• Restore the shroud panels
• Restore the OMC trans beamdump

Electrical tests:

• PZT1 test
• Fast Shutter functionality/delay time test

So far, the O4 break vent & commissioning work is proceeding as planned, with only minor hiccups.  Yesterday's TCS water leak caused a pause to the HAM6 in-chamber OMC alignment work, but we were able to resume this morning pretty quickly with only a short amount of time down.

The attached snapshot shows the planned schedule outline with green checkmarks next to the items completed so far on the left hand side.  On the right is a running log of the activities which have taken place daily (for ease of viewing all in one place).

Teams currently on:

• HAM6 new OMC in-chamber alignment
• EX prepping to pump down post cryo baffle damper install, ETMX SUS appears healthy with few TFs spot checked
• Prep for HAM3 baffle work next week, chamber cleaned
• Starting EY wind fence repair
• TCS system maintenance
• FARO setup in LVEA for monument surveying

Summary:

We restarted the alignment work. The angle and/or the position of the new OMC relative to the cage is significantly different from the old one, and we had to mechanicaly rebalance not just the OMC, but OM3 and OM2 too, and rotate the cage of OM3. We'll see if these were good enough tomorrow.

1. Rebalancing the OMC

Yesterday, we rebalanced the OMC, the beam was on both of OMC QPDs but we didn't have time to touch up OM1 and OM2 to center the beam on OMC QPDs.  This morning we centered the QPDs and relieved the PIT offset of the OM3 as much as I could by using OM1 and OM2. Then we did another push of the balance mass, recentered the QPDs, and relieved PIT offsets of OM1/2/3 such that everybody kind of equally suffer. We ended up doing four iterations of this in total.

Following is the PIT/YAW offset slider values (ASC output to OM1/OM2 due to earlier WFS centering was already folded into these numbers).

Note1: OMn (n=1/2/3) sliders are not calibrated. Due to different EUL2OSEM matrix, OM2 apparent "range" for the sliders is about a factor of 1.7 larger than OM1 and OM3 (i.e. if you compare OM1 and OM2 coil output, the latter has a factor of 1.7 smaller output than the former given the same slider offset).

Note2: By "centering", we really mean centering w/o offsets. Since the position of the QPDs relative to the cavity mode differs from one assembly to another, there's no reason to assume that the good QPD offsets for the new OMC will be the same as the old one.

2. Beam height measurement, beam centering on OMs

At this point Koji measured the beam height at OM1, OM2 and OM3.

The beam was about 3/64" or about 1mm too high on OM1 (it was 5mm higher than 4" in the past) but it was too low on OM2 by ~16/64" or about 6mm. It quickly went up from OM2 to OM3 where the beam height was about 2/64" or about 0.8mm too low. The beam was clearing the input hole of the OMC shroud cleanly, and the OMCR beam was level.

These meant that OM2 was excessively tilted up and OM3 excessively down, both mechanically. Koji and I agreed that this is most likely due to the difference in angle for the old and the new OMC.

1/4" mis-centering on OM2 sounded too much to ignore to me, so we decided to bring the beam up at OM2 by using OM1 bias, and mechanically relieve the PIT of OM2/OM3 and YAW of OM3.

We're done with the 1st round of mechanical relieving. We adjusted OM2 PIT. We adjusted OM3 PIT and rotated OM3 cage for YAW while making sure that the beam is at least partially visible on both of the QPDs. After centering QPDA, the following shows the numbers as of now.

Overall, this looks to be a good improvement, but the beam is still hitting QPDB at the edge. We still have to see if it's possible at all to center both QPDs without further mechanical relief.

3. PZT and DCPD electronics are working

Koji scanned the OMC PZT and saw flashing on the DCPD A as well as B. PZT as well as the DCPD preamp are working.

Remaining tasks:

Center the OMC QPDs using mostly OM2 and OM3. If further mechanical relief is required, do.

After the OMC QPDs are centered, confirm that OMCR path is still reasonably level. Center the beam on the OMCR DCPD.

Center the AS_A and AS_B.

I don't know how to confirm that the OMC transmission will still hit the viewport for the camera. We'll try to use the card and the viewer combined.

Check the grounding.

Fil, Jonathan, Erik, Dave:

At EY Fil found h1susauxey has its front panel FAIL LED lit. He tracked the issue down to a failed power supply. This is a redundant dual power supply system.

Strangely this system is not beeping, which is the default behaviour if a power supply unit is inserted but not operational.

The IPMI shell confirms that the system has correctly identified the failure:

ipmi sensor output:

  OK     | (2751) PS1 Status        |           Presence detected           |
  Fail   | (2818) PS2 Status        |     Power Supply Failure detected     |
 

TJ, Jason, Camilla, Fil.

WP11612, FRS25384, swapping because current CO2X laser was slowly deteriorating, details in 75249.

Removed: Access 50LT 20706.21015D. Installed 20306-20419D, which was refurbished/re-gassed in March 2020. Table layout: T1200007.

• On Tuesday 23rd:
  • Checked polarizers on both CO2X and CO2Y (pictures) and spares were fine after issues at LLO (FRS30220). Will update FRS with details. 
  • Added labels to RF cables and water tubes on CO2 laser and RF driver.
  • Scribed current position of CO2 laser and photographed.
• On Wednesday 24^th AM:
  • Placed beam dump in front of periscope.
  • Remembered L1 has some damage on lens, caused by the plastic lens ring, photo. In future we should use metal lens rings.   
  • Turned on CO2 laser.
  • Check position on FLIR, with no mask, central and annual masks (needed to turn on FLIR – power supply inside enclosure).
  • Checked beam is going though all irises with beam card kit and FLIR one camera. Pictures
    • On masks: beam slightly high/masks low
    • On irises – beam a little low, but well centered with masks in.
    • Will not adjust irises 
  • Turned off: chiller, power supply, RF and power supply in mechanical room mezzanine,  AA chassis
  • Disconnected water lines (outside CO2 table). Drained table lines with gravity and wet-vac.
    • The small on table water lines have a lot of old contamination in them, photos of RF driver and CO2 lines, comment added to FRS10612 so we can think about replacing these lines post O4. 
  • Disconnected all cables and water pipes to CO2X/RF driver and removed laser and RF driver. 
  • Installed new CO2 laser (needed to move back foot back one set of bolt holes and turn water connections around 180deg). 
  • Installed new RF driver after Fil reattached it's electrical plug
  • Reconnected all cables and water pipes (expect table to main line connection). Photos of installed laser and RF driver.
  • Powered on CO2X chiller (had been turn off in medm so usually wouldn't turn on but it did)
  • The CO2X water pipe coming from the mezzanine leaked under the high pressure (as not yet connected to table), spraying water onto the cable trays above SQZ0/SQZT7 and SQZT0. 
  • Betsy initiated stop work M2400022, we transitioned to laser safe and Fil disconnected electrical connections. 
  • Richard, Fil, Jason and TJ dried the area as best they could. We see no sign of water inside SQZT0. 
  • Will reassess next steps on Thursday 25th. 

Still to do after the CO2X cooling line issue is solved:

• Refill table with water and reconnect to chiller.
• Turn on laser in PWM mode for low power realigning.
• Realign laser to irises and FLIR camera using M1 and PM1 mirrors
• Measure power out of laser head using S322C
• Check alignment to power meter and masks
• During commissioning time: spend ~2 hours realigning in-vac using HWS and periscope picos. 

Dana, Jenne

We looked at how the microseism levels have impacted the detector duty factor since the start of aLIGO to see if our ability to maintain lock during microseisms has gotten better or worse. The attached figure shows the fraction of time the detector maintained lock over the course of a week as a function of the average microseism level over that same week for each week in O4a, as well as each week in (1) O1, (2) O2, (3) O3a, and (4) O3b.

According to the data, it seems we may be slightly worse off in terms of our ability to maintain lock during high levels of microseisms in O4 than we were in O3, but it seems we are better off now than we were in O1 and O2. However, it is difficult to directly compare the data between observing runs given that during certain runs the average level of microseisms seemed consistently higher than in others. There were also certainly other sources of lockloss besisdes microseisms. But it can at least be said that maintaining lock during microseisms does not appear to be a bigger issue now than it used to be.

Note: The two large data gaps present in O2 were removed.