TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 5mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.16 μm/s
QUICK SUMMARY: Locked for 12 hours, range looks good.

The slab has been poured and finished, and the steel erection is well underway. Civil inspections took place today and I discussed with Jake a peripheral slab adjoined to the walking path to the man door for an air compressor. Insulation is beginning to get shaken out while siding and roofing goes up. Progress against the initial DGR schedule looks good. 

TITLE: 12/04 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Mostly quiet shift with just one lockloss with an ETMX glitch; relocking afterwards was simple. H1 has now been observing for 2.5 hours.

• 02:07 - Lockloss - alog81610
  • BS and PRM needed adustment to lock DRMI
• 03:37 - Observing

Lockloss @ 02:07 UTC - link to lockloss tool

Looks like an ETMX glitch about 200ms before the lockloss.

FAMIS 31062

For some reason, the NPRO current has been very slightly rising, correlating with a rise in power from both NPRO diodes, but overall output NPRO power is largely unchanged.

Jason's alignment tweaks of the PMC and RefCav this morning (alog81600) can be clearly seen on the stabilization trends. In addition to alignment changes improving PMC and RefCav transmission, the signal on PMC REFL and the average ISS diffracted power are both significantly less noisy.

WP12221

Kepco supplies with failed fans noted last week were replaced with with refurbished Kepco power supplies.  These supplies have the updated sealed ball bearing fan motor.  Alog of from last week 81498.

The following supplies were replaced.
EX VDD-1 U22-U24 +/- 24V this powers ISC-R1, we replaced both supplies.
**  one of these supplies has a bad voltmeter, we will replace this at the next opportunity
CER C5 U9-U11 +/- 24V this powers ISC-R2&R4, we replaced both supplies.

Did the wind fence inspection today. The EY fence looks fine, no further damage to the section that was found damaged previously. The EX fence has at least 8 broken connections, scattered over the length of the entire fence. All of the breaks are the original wires go through the split clamps that attach the wires to the uprights. I think it might be possible to patch many of the breaks to get by, but I don't know when we will have time to do the work.

EX is shown in the first two images, EY in the last two.

TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Network outage and maintenance day today. The network outage did not have a direct affect on the relocking of the interferometer, which was straight forward aside from my testing of ALS automation code (alog81597). At this point I believe all systems have been recovered and H(t) looks to be going offsite. The LVEA was swept.
LOG:                                                                                                 

As a follow-up to https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=81595 with some more information.

Today the many of the site general computing services where down.

While preparing for WP 12215, Nyath was migrating systems on the GC hypervisor cluster.  This was to make a configuration change on the nodes when they were not running anything.  At the end of moving one of the VMs the network switches connecting the hypervisors and storage went into a bad state.  We saw large packet loss on the systems connected to the switches.  This manifested itself as the systems seeing disk/io errors (due to timeouts when trying to write/read data).  This had wide ranging impacts on GC.  It also caused issues on the GC to CDS switch, setting a key link to a blocking state so no traffic flowed between GC and CDS (which points to the issues being related to a spanning-tree problem).  I will note that the migration of systems is part of the designed feature set of the system and part of the normal procedure for doing maintenance on hypervisor nodes.

The first steps of work were to get access to the hypervisors and storage, trying to make sure the those items where in a good state.  Later after working through restarts on various components and consulting with Dan and Erik the main switch stack for the VM system was rebooted and that seems to have cleared up the issues.

Work in the control room continued, using the local controls account.  Though we did have to make a change to the system config.  This needs to be looked at.  We have several KDCs configured so that authentication can go to multiple locations and does not need to rely on DNS, but the setup caused us issues.  To get things working we commented out the KDC lines in the krb5.conf file.  This essentially stopped the krb5 authentication (LIGO.ORG), but allowed local auth to go forward (which is what we had designed it for, so we will re-check the configs).

TITLE: 12/03 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 1mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.21 μm/s
QUICK SUMMARY: Despite the network outage and maintenance day activities today, H1 seems to have recovered easily and has been observing for 2 hours.

• Wind is low, secondary microseism descending to the 50th percentile
• All other systems nominal

Tue Dec 03 10:11:42 2024 Fill completed in 11min 39secs

Gerardo confirmed a good fill curbside. Late entry due to morning network issues.

After seeing the reflected spots on the PSL Quad display this morning and the "ISS diffracted power low" notification, since no one was using the beam today I performed a remote beam alignment tweak for both the PMC and RefCav.  I started shortly after 10am, after Masayuki and I finished his PMC measurements and sufficient time had passed for the PMC to recover.  The PMC Refl spot looked brighter on the left side vs. the right, indicating a potential alignment shift in yaw, while the RefCav Refl spot looked like it needed a pitch adjustment.

Starting with the PMC, with the ISS OFF there was ~103.3 W in transmission and ~25.0 W in reflection.  Tweaking beam angle (entirely in yaw as expected based on the Refl spot, pitch adjustment acheived nothing) resulted in ~105.2 W transmitted and ~22.7 W reflected; walking the beam got things a little better, with ~105.5 W transmitted and ~22.3 W reflected.  Seems this drop in PMC transmission was due to a slow alignment drift as the enclosure recovered from our NPRO swap ~10 days ago.  With the ISS ON and diffracting ~3.8% (I had to adjust the RefSignal to -1.97 V from -1.95 V due to the increase in PMC transmission) the PMC is now transmitting ~105.7 W and reflecting ~22.4 W.

Moving on to the RefCav, with the IMC locked the RefCav TPD was ~0.79 V.  Adjusting the beam angle got things a little better, with a TPD of ~0.82 V; as expected this adjustment was almost entirely in pitch.  While walking the beam alignment the IMC unlocked and was having a hard time relocking, so I asked TJ to take it to OFFLINE while I finished the alignment; I note this as with the IMC unlocked the RefCav TPD is generally a little bit higher than with the IMC locked.  With the IMC now unlocked I was able to get the RefCav TPD to ~0.86 V by walking the beam alignment.  While I was not able to get the TPD higher than this, the Refl spot on the PSL Quad display still looks like there's some alignment work to do; the central spot is usually centered in the ring that surrounds it, but now it looks a little low and left.  I don't have an explanation for this currently, we'll keep an eye on this in the coming days and see if/how things change.  TJ relocked the IMC and it did not have a problem this time.

Following from last week (alog81493) I've now replaced the Increase_Flashes (IF) state calls with SCAN_ALIGNMENT (SA). The former still exists, I've created parallel paths in the ALS arm nodes, but it's not called by ISC_LOCK or INIT_ALIGN. If there are any troubles, then reverting ISC_LOCK and INIT_ALIGN should be all that's needed (no need to touch ALS nodes).

Last week I wanted to return the functionality to immediately stop if it sees a flash above threshold. I added this back in, but since it has to watch slow channels, it rarely happens. Still though, SA seems to be slightly faster than IF, accuracy seems similar.

I've also added some more fault checking and a way to "gracefully" escape the state. I would still like to go through and clean up the code a bit since it's currently a Frankenstein of comments from me integrating this. This includes some log and notifications that might still reference IF. For other time.

Maintenance wrapped up, the most notable item was an internet outage. We are back to Observing.

The LHO offsite network has been down since 06:30 PST this morning (Tue 03 Dec 2024). Alog has just become operational, but network access to CDS is still down.

We've now had two lock losses from the Transition from ETMX state or immediately after while trying to reacquire. Unlike back on Nov 22 (alog81430), SRCL seems fine. For the first one, the locklost tool 1417133960 shows the IMC-SERVO_SPLITMON channel is saturating ~5sec before lock loss, then there is some odd LSC signals 40msecs before the tool tagged the lock loss (attachment 1). This might just be the lock loss itself though. The second one (locklost tool 1417136668) hasn't tagged anything yet, but ETMY has a glitch 2.5 sec before lock loss and ETMX seems to move more from that point (attachment 2).

Oli, Camilla WP12203. Repeat of some of the work done in 2019: EX: 52608, EY: 52636, older: part 1, part 2, part 3.

We misaligned ITMY and turned off the ALS-Y QPD servo with H1:ALS-Y_PZT_SWITCH and placed the Ophir Si scanning slit beam profiler to measure both the 532nm ALSY outgoing beam and the ALSY return beam in the HWS path.

The outgoing beam was a little oblong in the measurements but looked pretty clean and round by eye, the return beam did not! Photos of outgoing and return beam attached.  Outgoing beam was 30mW, return beam 0.75mW. 

Attached is the 13.5% and D4sigma measurements, I also have photos of the 50% measurements if needed. Distances are measured from the optic where HWS and ALS beams combine, ALS-M11 in D1400241. 

We had previously removed HWS-M1B and HWS-M1C and translated HWS-M1A from whats shown in D1400241-v8 to remove clipping. 

Ansel reported that a peak in DARM that interfered with the sensitivity of the Crab pulsar followed a similar time frequency path as a peak in the beam splitter microphone signal. I found that this was also the case on a shorter time scale and took advantage of the long down times last weekend to use  a movable microphone to find the source of the peak. Microphone signals don’t usually show coherence with DARM even when they are causing noise, probably because the coherence length of the sound is smaller than the spacing between the coupling sites and the microphones, hence the importance of precise time-frequency paths.

Figure 1 shows DARM and the problematic peak in microphone signals. The second page of Figure 1 shows the portable microphone signal at a location by the staging building and a location near the TCS chillers. I used accelerometers to confirm the microphone identification of the TCS chillers, and to distinguish between the two chillers (Figure 2).

I was surprised that the acoustic signal was so strong that I could see it at the staging building - when I found the signal outside, I assumed it was coming from some external HVAC component and spent quite a bit of time searching outside. I think that this may be because the suspended mezzanine (see photos on second page of Figure 2) acts as a sort of soundboard, helping couple the chiller vibrations to the air. 

Any direct vibrational coupling can be solved by vibrationally isolating the chillers. This may even help with acoustic coupling if the soundboard theory is correct. We might try this first. However, the safest solution is to either try to change the load to move the peaks to a different frequency, or put the chillers on vibration isolation in the hallway of the cinder-block HVAC housing so that the stiff room blocks the low-frequency sound. 

Reducing the coupling is another mitigation route. Vibrational coupling has apparently increased, so I think we should check jitter coupling at the DCPDs in case recent damage has made them more sensitive to beam spot position.

For next generation detectors, it might be a good idea to make the mechanical room of cinder blocks or equivalent to reduce acoustic coupling of the low frequency sources.