For about 50 minutes starting at 01:45 PST this morning we had another jump of the EY CNS-II reference GPS 1PPS compared with the LIGO timing. Previous jump was Mon 18nov2024 05:55. Prior to that no jumps for 3 weeks since the power supply was replaced.

We had a slight mains power glitch at 07:00:26 PST this morning, the GC UPS reported being on battery power for 4 seconds.

Attached plot shows all three phases in the corner station CER.

Richard turned back on the high voltage and plugged back in the DB37 cable that was unplugged yesterday afternoon.

TITLE: 11/20 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Microseism
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM_EARTHQUAKE
    Wind: 17mph Gusts, 11mph 3min avg
    Primary useism: 0.40 μm/s
    Secondary useism: 1.56 μm/s
QUICK SUMMARY:

• Wind is moderate, 2ndary microseism is quite high. We're also being shook by a 4.9 from the Mariana islands
• The IMC, FSS, PMC, and ISS are all unlocked

Since the secondary microseism has been very high this evening and preventing H1 from locking, we decided to leave just the PMC locked (no FSS, ISS, or IMC) for an extended time and watch for any glitches. At around 23:45 UTC, we unlocked the FSS, Richard turned off the high voltage supply for the FSS, and Jason and I unplugged the DB37 cable from the FSS fieldbox in the PSL-R1 rack in order to ensure no feedback from the FSS made it to the NPRO. Pictures of the DB37 cable's location are attached.

The first attachment shows the changes seen when the FSS was unlocked. Since then, I've seen several instances of groups of glitches come through, such as those shown in the second and third attachments. These glitches in the PMC_MIXER channel are smaller than ones seen previously that have unlocked the IMC (like in alog81228). There have also been times where the PMC_MIXER channel gets "fuzzier" for a bit and then calms down, shown in the fourth attachment; it's possible this is due to the NPRO frequency not being controlled so the PMC sees some frequency changes? Finally, I only recall one instance of the NPRO jumping in power like in the final attachment; the PMC doesn't seem to care much about this, only having one very small glitch at this time.

I'll leave the PSL in this configuration to collect more data overnight as the secondary microseism is still much too high for H1 to successfully lock.

TITLE: 11/20 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Microseism
INCOMING OPERATOR: None
SHIFT SUMMARY: H1 remained unlocked the whole evening due to very high secondary microseismic motion, and there is no plan to attempt locking overnight. In the meantime, we took the opportunity to leave just the PMC locked to look for glitches (more details in alog81371). Very quiet evening otherwise with nothing else to report.

At 17:52:10 PST Tue 19 Nov 2024 we had another sharp positive glitch in the BSC3  vacuum gauge signal (PT132_MOD2). VACSTAT did not alarm because no other gauge tripped around this time. The glitch was a 2 second wide square wave, and as before it was the delta-P which tripped.

vacstat was reset at 18:39 to restore it to its monitoring state.

I startedtesting the PCALX_STAT Guardian node today.

/opt/rtcds/userapps/release/cal/h1/guardian/PCALX_STAT.py

It created a new ini file, But the DAQ was not restarted after this new ini file creation.
As it currently stands this is a draft of the final product that will be tested for a week and further refined.
This Guardian node, does not make any changes to the IFO, it's only job is to determine if PCALX arm is broken or not. TJ has already added it to the Guardian Ignore list. 
 

TITLE: 11/19 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: 2ndary microseism has been consistently high (above the 90th percentile) for the whole day and has risen even more in the last few hours of the shift and has reached the top of the plot. It will likely will get even worse tomorrow as the storm off the pacific coast closes in. Wind has been pretty low today. ITMY5/6 is still kind of high.

As of 23:30 UTC we're back to sitting in idle, we unlocked the FSS, ISS, and IMC just leaving the PMC locked.
LOG:

IFO/Locking:

• 16:35 UTC While we were down Sheila set the IMC to offline and turned off the ISS first loop off (PMC and FSS locked).
• 16:53 UTC HAM7 ISI tripped, from a CPS glitch, possibly from the fire crew checking out extinguishers. This is a sensitive area, people walking around can cause CPS glitches.
• 17:22 UTC LVEA back to LASER SAFE (Transition)
• 21:23 UTC I started relocking, ALS_X had to be recovered due to the restart. I just trended top mass OSEMs and set the sliders accordingly
• 21:55 - 22:02 UTC I ran a manual IA to align the SRC as DRMI didn't look good but PRMI did. DRMI looked great afterwords
• 22:00 UTC Oli swept the LVEA alog81363
• 22:15 UTC "FAST_SHUTTER test failed", Keita's AS_B FM4 gain filters were still on when they shouldn't have been
• 23:12 UTC Lockloss at MOVE_SPOTS, lockloss tool. ASC loops were working pretty hard
• 23:30 UTC ISC_LOCK to IDLE

Some of the maintenance work completed includes:

• Glycol piping checks
• Fire alarm testing & inspection around site (confirmed PSL encl strobes are working)
• CO2Y table loose cables investigated and beam mask adjustment alog81362
• New VAC scroll pumps at LVEA and Ends
• Replaced bad pressure gauge at PT154 alog81361
• EPO pictures and videos of site
• ALS beam profiling alog81358
• ETMX DAC swap, back to the old DAC alog81350
• LVEA oscilloscope installed alog81359
• IMC testing alog81356
  • IMC offline ISS off, PMC and FSS locked
  • Unplugging and plugging in ISS FF cable
  • IMC offline, ISS off, FSS off, only PMC

We did not do a DAQ restart

Oli, Camilla WP12203. Repeat of some of the work done in 2019: EX: 52608, EY: 52636, older: part 1, part 2, part 3.

We misaligned ITMY and turned off the ALS-Y QPD servo with H1:ALS-Y_PZT_SWITCH and placed the Ophir Si scanning slit beam profiler to measure both the 532nm ALSY outgoing beam and the ALSY return beam in the HWS path.

The outgoing beam was a little oblong in the measurements but looked pretty clean and round by eye, the return beam did not! Photos of outgoing and return beam attached.  Outgoing beam was 30mW, return beam 0.75mW. 

Attached is the 13.5% and D4sigma measurements, I also have photos of the 50% measurements if needed. Distances are measured from the optic where HWS and ALS beams combine, ALS-M11 in D1400241. 

We had previously removed HWS-M1B and HWS-M1C and translated HWS-M1A from whats shown in D1400241-v8 to remove clipping. 

TITLE: 11/19 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 10mph Gusts, 8mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 1.47 μm/s
QUICK SUMMARY: Due to very rapidly rising microseism, H1 will not be locking this evening or overnight, which allows us to run a long test for PSL glitch hunting by leaving just the PMC locked (no FSS, ISS, IMC). The DB37 cable was unlplugged from the FSS fieldbox at 23:45 UTC.

Camilla C., TJ

Recently, the CO2Y laser that we replaced on Oct 22 has been struggling to stay locked for long periods of time (alog81271 and trend from today). We've found some loose or bad cables in the past that have caused us issues, so we went out on table today to double check they are all ok.

The RF cables that lead into the side of the laser can slightly impact the output power when wiggled, in particular the ones with a BNC connector, but not to the point that we think it would be causing issues. The only cable that we found loose was for the PZT that goes to the head of the laser. The male portion of the SMA that comes out of the laser head was loose, and cannot be tightened from outside of the laser. We verified that the connection from this to the cable were good, but wiggling it did still introduce glitched in the PZT channel. I don't think that we've conviced ourselves that this is a problem though, because the PZT doesn't seem to glitch when the laser loses lock and instead it will run away.

An unfortunate consequence of the cable wiggling was that one of the Beckhoff plugs at the feedthrough must have been unseated slightly and caused our mask flipper read backs to read incorrectly. The screws for this plug were not working so we just pushed the plug back in to fully seat it and all seemed to work again.

We still are not sure why we've been having these lock losses lately, the 2nd and 3rd attachments show a few of them from the last day or so. They remind me of back in 2019 when we saw this - example1, example2. The fix ultimately a chiller swap (alog54980), but the flow and water temp seem more stable this time around. Not completely ruling it out yet though.

LVEA has been swept. Just had to unplug a(n already off) PEM function generator and turn off the lights for the HAM5/6 giant cleanroom.

Today we swapped the PT154 gauge on the FC-A section that was reported faulty in alog 81078 with a brand new gauge, same make/model.

FCV1 & 2 were closed to isolate the FCA section, and the angle valve on the A2 cross closed to isolate the gauge. Volume was vented with dry nitrogen and gauges swapped. CF connection was helium leak checked, no He signal above the HLD background ~2e-10 Torr-l/s.

Closing WP 12200

Related alog: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=81320

Sometime ago (July/09/2024) the analog DC gain of H1:ASC-AS_B_DC was set from "High" to "Low" when I was looking to extract more information from WFSs about the power coming past the Fast Shutter when the shutter bounces down after high power lock losses. Since this is not necessary any more (see plots in alog 81130) and since keeping this means that we have to either adjust dark offset once in a while and/or making the "shutter closed" threshold for Fast Shutter test (alog 81320), I set the gain switch back to "HIGH", and disabled the +20dB digital gain in H1:ASC-AS_B_DC_SEG[1234] FM4.

Interestingly, when I flipped the gain switch, SEG3 output didn't change (1st attachment). It could be that the specific DC channel is stuck to HIGH or LOW, but it could be that the analog offset happens to be really small for that channel. I cannot test it right now as we're leaving IMC unlocked. As soon as the light comes back I will go to the floor and test.

The second attachment is the relevant MEDM screen (that shows the state it should be in now) and the third one is the picture of the interface chassis (the switch I flipped is circled in red, and it's supposed to be HIGH now).

It looks like usually the PSL PMC + FSS can stay locked for long periods of time, for example >1 month over the emergency vent in August. See the attached trends of the # of locked days and hours in the last few months (ndscope saved as /ligo/home/victoriaa.xu/ndscope/PSL/pmc_fss_can_stay_locked_over_emergency_o4b_vent.yaml).

The last time the PMC was locked for almost a month ended on Sept 23, 2024. Since then the PSL PMC has not stayed locked for over 5 days, but this is most likely due to commissioning tests and debugging which started around then.

2024 Nov 12

Neil, Fil, and Jim installed an HS-1 geophone in the biergarten (image attached). HS-1 is threaded to plate and plate is double-sided taped to the floor. Signal given was non-existent. Must install pre-amplifier to boost signal.

2024 Nov 13

Neil and Jim installed an amplifier (SR560) to boost HS-1 signal (images attached). Circuitry checked to ensure signal makes it to the racks. However, when left alone there is no signal coming through (image attached, see blue line labelled ADC_5_29). We suspect the HS-1 is dead. HS-1 and amplifier are now out of LVEA, HS-1's baseplate is still installed. We can check one or two more things, or wait for more HS-1s to compare.

After discussions with control room team: Jason, Ryan S, Sheila, Tony, Vicky, Elenna, Camilla 

• Prior to September 13th:
  • No IMC locklosses 80951
  • NPRO/FSS glitches present but not bothering us (June 80520, last year 67718).
• September 12th: IMC gain redistribution 79998
• September 13th: First IMC lockloss
• September to October:
  • We start noticing FSS glitches and do of tests to understand these.
  • They were present in NPRO only with everything else unlocked (part of reason for swapping NPRO)
  • Found that these glitches are not new, present in June 80520 and even last year 67718. Maybe they got worse, or maybe we just stopped surviving them as well.
  • IMC has periods of being unstable (have to wait in DOWN)
• October 9th: IMC gain redistribution reverted 80566
  • We have 6 days with only one IMC lockloss 
• October 15th: NPRO current increased 80687
  • IMC lockloses started again 
• Ocotber 18th: NPRO current reverted 80746
  • IMC lockloses continued
• October 23rd to 29th: NPRO swapped 80929
  • Swapped to new faraday and EOM, both found to be dodgy
• Ocotber 30th: Back to NLN
  • Had issues with NPRO 81073: found to be NPRO mode hopping 
  • IMC locklosses continued 
• October 30th to November 1st: Tests.
  • CARM OLG 80956
  • IMC OLG 80979
  • FSS OLG 80993 
  • If the PSL ISS is unlocked the IMC can stay locked: 81056
  • Attempt to swap TTFSS (reverted) 80983
  • Hardboot of PSL chassis h1psl0: 81048 
• November 6th: PSL mode hopping was fixed by changing NPRO temperature, 81107. 
  • PSL now stable
  • IMC has periods of being unstable
  • IMC locklosses continue 

Conclusions: The NPRO glitches aren't new. Something changed to make us not be able to survive them as well in lock. The NRPO was swapped so isn't the issue. Looking the timing of these "IMC" locklosses, they are caused by something in the IMC or upstream 81155.

Tagging OpsInfo: Premade templates for looking at locklosses are in /opt/rtcds/userapps/release/sys/h1/templates/locklossplots/PSL_lockloss_search_fast_channels.yaml and will come up with command 'lockloss select' or 'lockloss show 1415370858'.