TITLE: 11/16 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 11mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY:

Walked in to a locking h1, which then has moved on to INITIAL ALIGNMENT.  Ibrahim shared how H1 has had a better time--so am cautiously optimistic for the night---with the goal of OBSERVING!

Environmentally we are becoming quieter with µseism dropping below the 95th percentile and winds not too breezy.

Ibrahim, Ryan S

Seems like an IMC lockloss. Interestingly, there was no obviously glitchy behavior (as previously observed) before or after the lockloss. Attached screenshot of ASC and IMC losing lock.

Sheila, Vicky. 

We changed the low pass filtering on the ADF SQZ angle servo to be lower, going from 1 Hz low pass to 0.1 Hz low pass on the ADF I/Q demod filter banks. I think this reduced the sqz angle readback noise a bit (trends before/after). It didn't make a huge impact on the control signal that goes to the CLF_RF6 demod phase (bottom purple trace), and it might be worth considering low pass filtering that control signal too. I accepted the SDFs for this filter bank LPF change.

Ibrahim, Sheila, Ryan S

Ibrahim noticed that the guardian was in a loop of locking PRMI something like 20 times overnight.

What happened was that PRMI was poorly aligned (POP18 and 90 I ERR  both below 20 counts), but PRMI was able to grab lock for a couple of seconds, but would not hold lock as the guardian engaged the top mass offloading, adjsuted gains and filters.  Eventually this hit the timer in H1 manager to do an initial alignmemt, after just over an hour. 

There is also a timer in ISC that checks if PRMI hasn't locked in 10 minutes it should move onto MICH fringes.  This didn't happen in this case because of the very short locks.  Ryan and I read through the ISC_LOCK AQUIRE_PRMI state, and realized that it had a check (line 1371) for DRMI arrived (in the state PRMI locked), which didn't include a check for the state being done, which would return true in ISC LOCK before ISC_DRMI finished the final steps in PRMI locked  .  In the second, zoomed screenshot, you can see that as soon as the DRMI guardian enters state 35 (PRMI locked), ISC lock moves on immediately from state 50 (ACQUIRE PRMI) to 51 and 52 (PRMI ASC), which resets the timer so that we never went to MICH fringes. 

We added a check for the ISC_DRMI state to be both arrived and done in PRMI locked, so that PRMI will have to survive the offloading and boost engagement for the timer to be reset.  Ibrahim has now reloaded this.  We think that if this situation came up again, now we would only spend 10 minutes relocking PRMI, before going to MICH fringes.  If H1 is under the manager control, it would run initial alignment if running MICH fringes didn't help. 

I ran the range_compare script using the 10-12 minute low range stretch from the most recent lock and some time before it of the same lock where the range was better, the construction crew by staging was also seen moving large equipment around this time on the cameras.

To produce this figure I ran "python3 /ligo/gitcommon/ops_tools/rangeComparison/range_compare.py --span 600 1415725641 1415728983", I only did 600 seconds of data. The range drop looks to be from low frequency noise, mostly below ~300Hz, the most obvious peak difference looks to be at 46/47 (mostly on sensmon), 36 (mostly on sensmon), and 60Hz (both sensmon and DARM) for the low range stretch. The lines are hard to see, but if you open the pdf in LibreOffice (linux default pdf viewer) and right-click -> Arrange -> Behind Object, then you can hover your cursor over the curves and it will be much easier to see the differences.

Ryan S, Ibrahim

Unexpected lockloss being investigated but confirmed not the IMC nor the environment. Ryan S is seeing that it is looking like the ETM Glitch.

FAMIS 31059

Late entry; these are trends taken on Monday as usual but I neglected to post them until now.

Since troubleshooting of laser glitches is still ongoing, several things in the PSL have been changing more than usual, including enclosure incursions, temperature changes, and ISS diffracted power increases. The only unexpected thing of note is that PMC reflected power seems to have been rising slowly over the past few days, but I think it's too soon to tell if this is a similar increase to what we were seeing here pre-NPRO swap or if it's related to laser troubleshooting. Will certainly be keeping an eye on this.

Fri Nov 15 10:12:59 2024 INFO: Fill completed in 12min 55secs

TITLE: 11/15 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 1mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.35 μm/s
QUICK SUMMARY:

IFO is LOCKING at PRMI_ASC

I arrived and guardian has recently finished an initial alignment and was at CHECK_IR. Microseism is much lower than last few days.

TITLE: 11/15 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: TJ
SHIFT SUMMARY:

Started shift with beginning locking for H1.  Had a couple of locklosses (as posted earlier), but made it to NLN on the 3rd attempt.  had a lockloss a few min ago due to M6.7 EQ from south pacific.  Have been touching base with TJ (who will be on OWL), and he wanted to receive Owl alerts and if locking is rough overnight, he'll switch to the PMC/FSS tests overnight.
LOG:

• 0003-0043 INITIAL ALIGNMENT (w/ ALSy needing touch up by hand)
• 0044 Locking
  • 0044 LOCK#1
    • 0122:  LOCKLOSS at LOW_NOISE_ESD_ETMX [558]....before the new T-Ramp change  --->  In the Control Room we saw ASC ring up for a few seconds before the lockloss ("it was DHARD PIT ringing up at 1Hz; probably a not great alignme," according to Elenna).
      • For next lock, will pause at LOWNOISE COIL DRIVERS [552] to make sure ASC looks ok.
  • LOCK #2:  DRMI looked its ugly self.  Needed to run CHECK MICH FRINGES and the BS was definitely the culprit for the nasty alignment and was fixed.  PRMI & DRMI both locked immediately.
    • 0205 LOCKLOSS while holding at TRANSITION FROM ETMX [557] to look for the ASC ring-up....but while here, there was NO ring up and it looks like an FSS (or IMC?) glitch lockloss.
  • LOCK #3:  0303 at NLN!
    • 0307:  Phoned Louis so he could do Calibration checks
      • Also took Observatory Mode to CALIBRATION (from PREVENTATIVE MAINTENANCE)
      • Louis and JoeB discussed the H1 Calibration and then gave the OK to Go for Observing.
• 0332 OBSERVING
• 0535 Alerts for M6.7 Solomon Islands earthquake (in the Yellow band ["large eq"] of Seismon)
• 0542 begin seeing earthquake on picket fence and seismic FOMs
• 0546 Lockloss due to Earthquake after a 2.75hr lock

After tonight's Initial Alignment, H1's been fairly good with getting through the bulk of ISC_LOCK, but have had 2 consecutive locklosses a couple of states after MAX POWER: 

• 1st lockloss was ASC-related
• 2nd lockloss was possibly an FSS/IMC glitch

Locking Notes:

0003-0043 INITIAL ALIGNMENT (w/ ALSy needing touch up by hand)

0044 LOCK#1

• 0122:  LOCKLOSS at LOW_NOISE_ESD_ETMX [558] due to ASC ring up (happened before the new T-Ramp change)  --->  In the Control Room we saw ASC ring up for a few seconds before the lockloss ("it was DHARD PIT ringing up at 1Hz; probably a not great alignme," according to Elenna).
• For next lock, will pause at LOWNOISE COIL DRIVERS [552] to make sure ASC looks ok.

LOCK #2:  DRMI looked its ugly self.  Needed to run CHECK MICH FRINGES and the BS was definitely the culprit for the nasty alignment and was fixed.  PRMI & DRMI both locked immediately.

• 0205 LOCKLOSS while holding at TRANSITION FROM ETMX [557] to look for the ASC ring-up....but while here, there was NO ring up and it looks like an FSS (or IMC?) glitch lockloss (see attached plot by Elenna from Sheila's template), BUT....
• There was NOT an FSS or IMC tag on the Lockloss page

Will continue locking for the next 2.5-3hrs, and then take H1 to IDLE and leave FSS & PMC -ON- for the night.

But if H1 makes it to NLN, will contact Louis or Joe B for a ~15min calibration check.

Ansel reported that a peak in DARM that interfered with the sensitivity of the Crab pulsar followed a similar time frequency path as a peak in the beam splitter microphone signal. I found that this was also the case on a shorter time scale and took advantage of the long down times last weekend to use  a movable microphone to find the source of the peak. Microphone signals don’t usually show coherence with DARM even when they are causing noise, probably because the coherence length of the sound is smaller than the spacing between the coupling sites and the microphones, hence the importance of precise time-frequency paths.

Figure 1 shows DARM and the problematic peak in microphone signals. The second page of Figure 1 shows the portable microphone signal at a location by the staging building and a location near the TCS chillers. I used accelerometers to confirm the microphone identification of the TCS chillers, and to distinguish between the two chillers (Figure 2).

I was surprised that the acoustic signal was so strong that I could see it at the staging building - when I found the signal outside, I assumed it was coming from some external HVAC component and spent quite a bit of time searching outside. I think that this may be because the suspended mezzanine (see photos on second page of Figure 2) acts as a sort of soundboard, helping couple the chiller vibrations to the air. 

Any direct vibrational coupling can be solved by vibrationally isolating the chillers. This may even help with acoustic coupling if the soundboard theory is correct. We might try this first. However, the safest solution is to either try to change the load to move the peaks to a different frequency, or put the chillers on vibration isolation in the hallway of the cinder-block HVAC housing so that the stiff room blocks the low-frequency sound. 

Reducing the coupling is another mitigation route. Vibrational coupling has apparently increased, so I think we should check jitter coupling at the DCPDs in case recent damage has made them more sensitive to beam spot position.

For next generation detectors, it might be a good idea to make the mechanical room of cinder blocks or equivalent to reduce acoustic coupling of the low frequency sources.

TITLE: 11/15 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 15mph Gusts, 10mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.68 μm/s 
QUICK SUMMARY:

H1 was down for troubleshooting all day shift and at the end of TJ's shift he started an Initial Alignment which we are running now. 

The goal for today is similar to last night's shift:  Will try locking from 430-9pm.

• If locking is GOOD, get some Observing time,
  • But before that, when H1 gets to NLN, contact Louis (or JoeB) for Calibration Pipeline stuff.
• If locking is BAD, OWL canceled, AND set H1 for
  • -only- FSS & PMC -ON- for the night
  • Take the following down:  IMC to OFFLINE, ISS autolocker OFF, ISC_LOCK to IDLE, IMC to AUTO

Operator NOTE from TJ:  When restoring for locking--order is PMC -> FSS -> ISS  (as they are listed on the Ops Overview)

Environmental Notes:  µseism is worse than last night (is clearly higher than the 95th percentile & touching "1 count" on FOM).  Had been windy most of the day, but has become calmer in the last hour.

Initial Alignment Note (since it has just completed while I've been trying to write this alog for the last 45min!):  ALSy wasn't great after INCREASE FLASHES, Elenna touched up the ETMy by hand and this immediately helped-----EVEN WITH HIGH MICROSEISM!

We restarted the calibration pipeline with a new configuration ini such that it no longer subtracts the 60 Hz (and harmonics up to 300 Hz).  The configuration change is recorded in commit here: https://git.ligo.org/Calibration/ifo/H1/-/commit/53f2e892a38cfb18815912c33b1f1b8385cfff62

I restarted the pipeline at around 9:35 am PST. Around the same time as this was done at LLO (LLO:74051).

The IFO was down at the time, so I left a request with the H1 operators to contact both me and Joe B. when H1 is back at NLN but before going to Observing mode so that we (whoever is first) can confirm that the GDS restart is behaving as expected. Initial checks at LLO indicate that things are working properly, which is promising.

Ryan, Jason, Patrick, Filiberto

As part of troubleshooting the PSL we hardware power cycled the PSL Beckhoff computer in the diode room this morning, along with all of the associated diode power supplies and a chassis in the LVEA.

I had guessed that everything would autostart, but I was wrong, so I took the opportunity to set it up to do so. This required putting a shortcut to the EPICS IOC startup script in the C:\TwinCAT\3.1\Target\StartUp directory (see attached screenshots), and selecting an option in the TwinCAT Visual Studio project to autostart the TwinCAT runtime. We software restarted the computer again to test this, and after logging in, the Beckhoff runtime and PLC code started, along with the EPICS IOC, but the visualization did not. I found documentation that pointed to the location of the executable that starts the visualization, and added a shortcut to that to the startup directory as well. We didn't have time to restart the computer again to see if that would autostart correctly.

For some reason there seemed to be issues with processes reconnecting to the EPICS IOC channels. I tested running caget on the Beckhoff computer itself and got a message about connecting to two different instances of the channel, and a couple of pop up windows related I think to allowing network access, which I said to allow. caget worked, although it gave a blank space for the value, so I tried it again with an invalid channel name, which it correctly gave an error for. On the Linux workstation we were using, the MEDM screens were not reconnecting, even after closing and reopening them, but again caget worked. We had to restart the entire medm process for it to reconnect.

The EDCU and SDF also had issues reconnecting, and they had to be restarted too.