TITLE: 11/17 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 15mph Gusts, 11mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.24 μm/s
QUICK SUMMARY:

H1 Has been locked for 1 Hour and 45 minutes.
DCPD's are diverging, Which I usually attribute to the Violins growing.
Turns out it's one of the usual suspects..... ITMY Mode 5 increasing.

I have turned the Gain off of ITMY m5 to see if it stays at that level or decreases.

The IFO stays locked until 1:25 UTC then I can do a Calibration sweep are the calibration time early today was missed by a lockloss.

TITLE: 11/16 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as of 22:35 UTC

There was one IMC-caused Lockloss which ended our 13hr and 36 min lock acquisition. (alog 81310)

Stayed locked for the majority of this shift with one somewhat troublesome lock acquisiton for a number of reasons:

• ALSX was stuck at 0.15 cts, with a max of 0.3 cts after intervension. I went into manual initial alignment to continue troubleshooting. Jenne advised to touch ITMX, which worked. I was able to get ALSX to the normal ~0.8cts and let guardian successfully take it from there. Initial alignment happened fully auto.
• We were able to lock up until OFFLOAD_DRMI_ASC until a lockloss occured due to the IMC glitch. The IMC was still coming in and out, locking and unlocking for 4 more minutes.
• We were able to get all the way up to LOWNOISE_ESD_ETMX, before losing lock again.
• IFO was able to lock to NLN on the third and final try.

I was not able to take a calibration measurement due to the Lockloss, and we're not thermalized yet. I've informed Tony, the EVE operator, and he will take a calibration sweep once thermalized (01:24 UTC).

LOG:

None

Lockloss seemingly caused due to the IMC with MC2 losing lock within 1ms of ASC (attached).

Sat Nov 16 10:10:51 2024 INFO: Fill completed in 10min 47secs

TITLE: 11/16 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 164Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 12mph Gusts, 9mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.20 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 05:34 UTC (10 hr lock!)

TITLE: 11/16 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: TJ
SHIFT SUMMARY:

Most of the meat of the shift has already been posted earlier.

Had a return of the FSS/IMC glitches this afternoon/evening.  Have made it back to Observing at the end of the shift though.  

For the night we are set for locking for the OWL.  But if locking becomes an issue that lasts over 2hrs, plan is to take ISC LOCK to IDLE for the night.
LOG:

• 0007-0028 initial alignment
• 0111 OBSERVING
  • 0122-0131 PI 31 begins increasing pretty steadily over atleast 5min and started causing DCPD glitches, but then guardian finally was able to bring it down 
  • 0208 LOCKLOSS (looks like an FSS/IMC glitch, @0211 IMC also caused on the order of min of downtime before relocking)
  • Several FSS and/or IMC glitches for a few locking attempts

• 0534 OBSERVING

Had about 20hrs with no IMC/FSS glitches from 9pm Thurs night until about 340pmPT Fri afternoon.  But have now had several for the last 5hrs.

Also for one ~60min stretch, H1 almost made it to NLN, but had rung up violins which kept us at OMC Whitening for 20min until the next IMC glitch.

Here are some running notes for the rough 5hrs filled with glitches (currently have H1 in IDLE).

Notes During Return of IMC/FSS Glitch Locklosses

2340utc (340pmPT) LOCKLOSS with IMC tag was the beginning of IMC/FSS glitch locklosses (where we went almost 20hrs since the last glitch around 5utc (9pmPT).

0111-0208:  OBSERVING

0208utc (608pmPT):  Attached screenshot shows a coincident glitch lockloss seen on FSS & IMC.  Here is the LOCKLOSS page also tagged with IMC glitch.

Starting an FRS Ticket to mark the DOWNTIME (FRS-32644) for 3.5hrs of DOWNTIME so far.  Where there was notable part of downtime due to IMC not locking easily.

• 0241 Lockloss at DARM to DC ReadOut looked like a FSS/IMC glitch lockloss---> IMC + FSS lockloss tags.
• 0255 Lockloss at RESONANCE also looked like a FSS/IMC glitch lockloss---> IMC lockloss tag
• 0256 Lockloss at READY ---> FSS lockloss tag
• 0334 lock made it almost to NLN, Yay Yay Yay!!!!   BUT it was a little delayed due to....

VIOLINS Rung Up A Bit: (haven't had this since my shift on Tues night!)   So were stuck at OMC WHITENING for over 20min before H1 had....

• 0357 Lockloss at OMC WHITENING (with an IMC LOCKLOSS tag)

So for the last 3hrs:

• 0208-0256:  50min with a flurry of FSS/IMC glitches causing locklosses at various states also causing IMC locking grief.
• 0256-0357:  Hour of NO glitch locklosses, and almost made it to NLN (but rung up violins prevented that).  Then another IMC glitch.
• 0420 Went to IDLE due to more locklosses and for a pause

Not sure what is more useful.  Keep having ISC LOCK continue to locking, or to leave it in IDLE for IMC/PMC/FSS monitoring....

For FAMIS #26340:  All looks well for the last week for all site HVAC fans (see attached trends).

TITLE: 11/16 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

IFO is LOCKING in ENGAGE_ASC_FOR_FULL_IFO

Just had a lockloss and trying to get back to observing. IFO has been quite well behaved today. Both lock acquisitions were fully automatic with H1 needing no intervention.

That being said, we did have two locklosses.

Lockloss 1: ETM glitch caused.

Lockloss 2: IMC caused.

LOG:

None

TITLE: 11/16 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 11mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY:

Walked in to a locking h1, which then has moved on to INITIAL ALIGNMENT.  Ibrahim shared how H1 has had a better time--so am cautiously optimistic for the night---with the goal of OBSERVING!

Environmentally we are becoming quieter with µseism dropping below the 95th percentile and winds not too breezy.

Ibrahim, Ryan S

Seems like an IMC lockloss. Interestingly, there was no obviously glitchy behavior (as previously observed) before or after the lockloss. Attached screenshot of ASC and IMC losing lock.

Sheila, Vicky. 

We changed the low pass filtering on the ADF SQZ angle servo to be lower, going from 1 Hz low pass to 0.1 Hz low pass on the ADF I/Q demod filter banks. I think this reduced the sqz angle readback noise a bit (trends before/after). It didn't make a huge impact on the control signal that goes to the CLF_RF6 demod phase (bottom purple trace), and it might be worth considering low pass filtering that control signal too. I accepted the SDFs for this filter bank LPF change.

Ibrahim, Sheila, Ryan S

Ibrahim noticed that the guardian was in a loop of locking PRMI something like 20 times overnight.

What happened was that PRMI was poorly aligned (POP18 and 90 I ERR  both below 20 counts), but PRMI was able to grab lock for a couple of seconds, but would not hold lock as the guardian engaged the top mass offloading, adjsuted gains and filters.  Eventually this hit the timer in H1 manager to do an initial alignmemt, after just over an hour. 

There is also a timer in ISC that checks if PRMI hasn't locked in 10 minutes it should move onto MICH fringes.  This didn't happen in this case because of the very short locks.  Ryan and I read through the ISC_LOCK AQUIRE_PRMI state, and realized that it had a check (line 1371) for DRMI arrived (in the state PRMI locked), which didn't include a check for the state being done, which would return true in ISC LOCK before ISC_DRMI finished the final steps in PRMI locked  .  In the second, zoomed screenshot, you can see that as soon as the DRMI guardian enters state 35 (PRMI locked), ISC lock moves on immediately from state 50 (ACQUIRE PRMI) to 51 and 52 (PRMI ASC), which resets the timer so that we never went to MICH fringes. 

We added a check for the ISC_DRMI state to be both arrived and done in PRMI locked, so that PRMI will have to survive the offloading and boost engagement for the timer to be reset.  Ibrahim has now reloaded this.  We think that if this situation came up again, now we would only spend 10 minutes relocking PRMI, before going to MICH fringes.  If H1 is under the manager control, it would run initial alignment if running MICH fringes didn't help. 

I ran the range_compare script using the 10-12 minute low range stretch from the most recent lock and some time before it of the same lock where the range was better, the construction crew by staging was also seen moving large equipment around this time on the cameras.

To produce this figure I ran "python3 /ligo/gitcommon/ops_tools/rangeComparison/range_compare.py --span 600 1415725641 1415728983", I only did 600 seconds of data. The range drop looks to be from low frequency noise, mostly below ~300Hz, the most obvious peak difference looks to be at 46/47 (mostly on sensmon), 36 (mostly on sensmon), and 60Hz (both sensmon and DARM) for the low range stretch. The lines are hard to see, but if you open the pdf in LibreOffice (linux default pdf viewer) and right-click -> Arrange -> Behind Object, then you can hover your cursor over the curves and it will be much easier to see the differences.

Ryan S, Ibrahim

Unexpected lockloss being investigated but confirmed not the IMC nor the environment. Ryan S is seeing that it is looking like the ETM Glitch.

FAMIS 31059

Late entry; these are trends taken on Monday as usual but I neglected to post them until now.

Since troubleshooting of laser glitches is still ongoing, several things in the PSL have been changing more than usual, including enclosure incursions, temperature changes, and ISS diffracted power increases. The only unexpected thing of note is that PMC reflected power seems to have been rising slowly over the past few days, but I think it's too soon to tell if this is a similar increase to what we were seeing here pre-NPRO swap or if it's related to laser troubleshooting. Will certainly be keeping an eye on this.

Ryan, Jason, Patrick, Filiberto

As part of troubleshooting the PSL we hardware power cycled the PSL Beckhoff computer in the diode room this morning, along with all of the associated diode power supplies and a chassis in the LVEA.

I had guessed that everything would autostart, but I was wrong, so I took the opportunity to set it up to do so. This required putting a shortcut to the EPICS IOC startup script in the C:\TwinCAT\3.1\Target\StartUp directory (see attached screenshots), and selecting an option in the TwinCAT Visual Studio project to autostart the TwinCAT runtime. We software restarted the computer again to test this, and after logging in, the Beckhoff runtime and PLC code started, along with the EPICS IOC, but the visualization did not. I found documentation that pointed to the location of the executable that starts the visualization, and added a shortcut to that to the startup directory as well. We didn't have time to restart the computer again to see if that would autostart correctly.

For some reason there seemed to be issues with processes reconnecting to the EPICS IOC channels. I tested running caget on the Beckhoff computer itself and got a message about connecting to two different instances of the channel, and a couple of pop up windows related I think to allowing network access, which I said to allow. caget worked, although it gave a blank space for the value, so I tried it again with an invalid channel name, which it correctly gave an error for. On the Linux workstation we were using, the MEDM screens were not reconnecting, even after closing and reopening them, but again caget worked. We had to restart the entire medm process for it to reconnect.

The EDCU and SDF also had issues reconnecting, and they had to be restarted too.