FAMIS 28455, last checked in alog81106

Only things of note on these trends are that the ITMX spherical power has been dropping for the past week and a half, which Camilla agrees looks strange, and the ITMY SLED power has reached the lower limit of 1, so it will need replacing soon. Everything else looks normal compared to last check.

TITLE: 12/04 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.17 μm/s
QUICK SUMMARY: H1 is running smoothly, has been observing for 20+ hours.

• Wind and secondary microseism low
• All other systems nominal

TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Locked for 20 hours. Very quiet shift with nothing to report.
LOG:                                                                                        

FAMIS 21691

pH of PSL chiller water was measured to be just above 10.0 according to the color of the test strip.

Wed Dec 04 10:07:33 2024 INFO: Fill completed in 7min 30secs

Gerardo confirmed a good fill.

FAMIS 26019

I somehow missed this email last week so it's a week late. The last time this was ran on Nov 10, the main differences are:

• I noticed more narrow band peaks in some of the plots, particularly the HAMs, seemed higher than in the past.
• HAM8 looks a touch quieter than before
• ITMY looks much quieter

TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 5mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.16 μm/s
QUICK SUMMARY: Locked for 12 hours, range looks good.

The slab has been poured and finished, and the steel erection is well underway. Civil inspections took place today and I discussed with Jake a peripheral slab adjoined to the walking path to the man door for an air compressor. Insulation is beginning to get shaken out while siding and roofing goes up. Progress against the initial DGR schedule looks good. 

TITLE: 12/04 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Mostly quiet shift with just one lockloss with an ETMX glitch; relocking afterwards was simple. H1 has now been observing for 2.5 hours.

• 02:07 - Lockloss - alog81610
  • BS and PRM needed adustment to lock DRMI
• 03:37 - Observing

Lockloss @ 02:07 UTC - link to lockloss tool

Looks like an ETMX glitch about 200ms before the lockloss.

FAMIS 31062

For some reason, the NPRO current has been very slightly rising, correlating with a rise in power from both NPRO diodes, but overall output NPRO power is largely unchanged.

Jason's alignment tweaks of the PMC and RefCav this morning (alog81600) can be clearly seen on the stabilization trends. In addition to alignment changes improving PMC and RefCav transmission, the signal on PMC REFL and the average ISS diffracted power are both significantly less noisy.

WP12221

Kepco supplies with failed fans noted last week were replaced with with refurbished Kepco power supplies.  These supplies have the updated sealed ball bearing fan motor.  Alog of from last week 81498.

The following supplies were replaced.
EX VDD-1 U22-U24 +/- 24V this powers ISC-R1, we replaced both supplies.
**  one of these supplies has a bad voltmeter, we will replace this at the next opportunity
CER C5 U9-U11 +/- 24V this powers ISC-R2&R4, we replaced both supplies.

Did the wind fence inspection today. The EY fence looks fine, no further damage to the section that was found damaged previously. The EX fence has at least 8 broken connections, scattered over the length of the entire fence. All of the breaks are the original wires go through the split clamps that attach the wires to the uprights. I think it might be possible to patch many of the breaks to get by, but I don't know when we will have time to do the work.

EX is shown in the first two images, EY in the last two.

TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Network outage and maintenance day today. The network outage did not have a direct affect on the relocking of the interferometer, which was straight forward aside from my testing of ALS automation code (alog81597). At this point I believe all systems have been recovered and H(t) looks to be going offsite. The LVEA was swept.
LOG:                                                                                                 

As a follow-up to https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=81595 with some more information.

Today the many of the site general computing services where down.

While preparing for WP 12215, Nyath was migrating systems on the GC hypervisor cluster.  This was to make a configuration change on the nodes when they were not running anything.  At the end of moving one of the VMs the network switches connecting the hypervisors and storage went into a bad state.  We saw large packet loss on the systems connected to the switches.  This manifested itself as the systems seeing disk/io errors (due to timeouts when trying to write/read data).  This had wide ranging impacts on GC.  It also caused issues on the GC to CDS switch, setting a key link to a blocking state so no traffic flowed between GC and CDS (which points to the issues being related to a spanning-tree problem).  I will note that the migration of systems is part of the designed feature set of the system and part of the normal procedure for doing maintenance on hypervisor nodes.

The first steps of work were to get access to the hypervisors and storage, trying to make sure the those items where in a good state.  Later after working through restarts on various components and consulting with Dan and Erik the main switch stack for the VM system was rebooted and that seems to have cleared up the issues.

Work in the control room continued, using the local controls account.  Though we did have to make a change to the system config.  This needs to be looked at.  We have several KDCs configured so that authentication can go to multiple locations and does not need to rely on DNS, but the setup caused us issues.  To get things working we commented out the KDC lines in the krb5.conf file.  This essentially stopped the krb5 authentication (LIGO.ORG), but allowed local auth to go forward (which is what we had designed it for, so we will re-check the configs).

TITLE: 12/03 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 1mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.21 μm/s
QUICK SUMMARY: Despite the network outage and maintenance day activities today, H1 seems to have recovered easily and has been observing for 2 hours.

• Wind is low, secondary microseism descending to the 50th percentile
• All other systems nominal

Tue Dec 03 10:11:42 2024 Fill completed in 11min 39secs

Gerardo confirmed a good fill curbside. Late entry due to morning network issues.

The LHO offsite network has been down since 06:30 PST this morning (Tue 03 Dec 2024). Alog has just become operational, but network access to CDS is still down.

Camilla C., TJ

Recently, the CO2Y laser that we replaced on Oct 22 has been struggling to stay locked for long periods of time (alog81271 and trend from today). We've found some loose or bad cables in the past that have caused us issues, so we went out on table today to double check they are all ok.

The RF cables that lead into the side of the laser can slightly impact the output power when wiggled, in particular the ones with a BNC connector, but not to the point that we think it would be causing issues. The only cable that we found loose was for the PZT that goes to the head of the laser. The male portion of the SMA that comes out of the laser head was loose, and cannot be tightened from outside of the laser. We verified that the connection from this to the cable were good, but wiggling it did still introduce glitched in the PZT channel. I don't think that we've conviced ourselves that this is a problem though, because the PZT doesn't seem to glitch when the laser loses lock and instead it will run away.

An unfortunate consequence of the cable wiggling was that one of the Beckhoff plugs at the feedthrough must have been unseated slightly and caused our mask flipper read backs to read incorrectly. The screws for this plug were not working so we just pushed the plug back in to fully seat it and all seemed to work again.

We still are not sure why we've been having these lock losses lately, the 2nd and 3rd attachments show a few of them from the last day or so. They remind me of back in 2019 when we saw this - example1, example2. The fix ultimately a chiller swap (alog54980), but the flow and water temp seem more stable this time around. Not completely ruling it out yet though.