TITLE: 11/15 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Corey
SHIFT SUMMARY: The entire shift was dedicated to troubleshooting the laser glitching issue. We also had high winds and high useism, so it was good timing. The wind has died down and the troubleshooting has ended for the day so we are starting initial alignment.
LOG:

• 1629 IMC offline, ISS off
• 1644 IMC offline, ISS off, FSS off
• 1756 Turning off PSL and power cycling the Beckhoff computer
• 1933 Laser back up
• 2350 Started relocked                                                                                     

Vicky and Jason measured the PMC olg, and I grabbed the data from the SR785 for them. The plots are attached. The second measurement is a zoomed in version of the first.

Looks like the feature above 5 kHz is around the same frequency as the peak we are seeing in the intensity and frequency noise (alogs 80603, 81230)

These are the steps I took to get the data:

> cd /ligo/gitcommon/psl_measurements

> conda activate psl

> python code/SRmeasure.py -i 10.22.10.30 -a 10 --getdata -f data/name

This will save your data in the data folder as "name_[datetime string].txt"

To confirm connection before running, try

> ping 10.22.10.30

you should get something like

PING 10.22.10.30 (10.22.10.30) 56(84) bytes of data.
64 bytes from 10.22.10.30: icmp_seq=1 ttl=64 time=1.26 ms
64 bytes from 10.22.10.30: icmp_seq=1 ttl=64 time=1.54 ms (DUP!)
64 bytes from 10.22.10.30: icmp_seq=2 ttl=64 time=0.748 ms
64 bytes from 10.22.10.30: icmp_seq=2 ttl=64 time=1.03 ms (DUP!)
64 bytes from 10.22.10.30: icmp_seq=3 ttl=64 time=0.730 ms
64 bytes from 10.22.10.30: icmp_seq=3 ttl=64 time=1.02 ms (DUP!)
^C
--- 10.22.10.30 ping statistics ---
3 packets transmitted, 3 received, +3 duplicates, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.730/1.054/1.538/0.282 ms
----

run Ctrl-C to exit. If you don't get these messages, you are probably not plugged in properly.

To plot the data (assuming that you are measuring transfer functions), use

> python code/quick_tf_plot.py data/name_[datetime str].txt

Craig has lots of great options in this code to make nice labels, save the plot in certain places, etc. if you want to get fancy. Also, he has other scripts that will plot spectra, multiple spectra or multiple tfs.

If you want to measure from the control room, there are yaml templates that run different types of measurements, such as carm or imc olgs.

We restarted the calibration pipeline with a new configuration ini such that it no longer subtracts the 60 Hz (and harmonics up to 300 Hz).  The configuration change is recorded in commit here: https://git.ligo.org/Calibration/ifo/H1/-/commit/53f2e892a38cfb18815912c33b1f1b8385cfff62

I restarted the pipeline at around 9:35 am PST. Around the same time as this was done at LLO (LLO:74051).

The IFO was down at the time, so I left a request with the H1 operators to contact both me and Joe B. when H1 is back at NLN but before going to Observing mode so that we (whoever is first) can confirm that the GDS restart is behaving as expected. Initial checks at LLO indicate that things are working properly, which is promising.

Ryan, Jason, Patrick, Filiberto

As part of troubleshooting the PSL we hardware power cycled the PSL Beckhoff computer in the diode room this morning, along with all of the associated diode power supplies and a chassis in the LVEA.

I had guessed that everything would autostart, but I was wrong, so I took the opportunity to set it up to do so. This required putting a shortcut to the EPICS IOC startup script in the C:\TwinCAT\3.1\Target\StartUp directory (see attached screenshots), and selecting an option in the TwinCAT Visual Studio project to autostart the TwinCAT runtime. We software restarted the computer again to test this, and after logging in, the Beckhoff runtime and PLC code started, along with the EPICS IOC, but the visualization did not. I found documentation that pointed to the location of the executable that starts the visualization, and added a shortcut to that to the startup directory as well. We didn't have time to restart the computer again to see if that would autostart correctly.

For some reason there seemed to be issues with processes reconnecting to the EPICS IOC channels. I tested running caget on the Beckhoff computer itself and got a message about connecting to two different instances of the channel, and a couple of pop up windows related I think to allowing network access, which I said to allow. caget worked, although it gave a blank space for the value, so I tried it again with an invalid channel name, which it correctly gave an error for. On the Linux workstation we were using, the MEDM screens were not reconnecting, even after closing and reopening them, but again caget worked. We had to restart the entire medm process for it to reconnect.

The EDCU and SDF also had issues reconnecting, and they had to be restarted too.

TJ, Vicky, Camilla 

Vicky set up an oscilloscope on a cable simular to the PMC mixer and we watched the second trend of Sheila's PSL ndscope. The largest cause of repeatable glitches by touching cables: ISS AOM 80MHz that we found loose and tightened, it sounds like the PSL team found the PSL side of this cable loose and tightened on Tuesday too. Times noted below if we want to go back and look at the raw data.

• 12:12UTC Started in PSL R1.
  • TJ tightened cable in ISS AOM 80MHz that he found loose. The power changed a lot on PMC MIXER, this was repeatable, at 12:25PT loosened, repeated, saw glitches, tightened again. 
  • 12:20 injection pmc locking box in R1
• 12:21 PSL-R1 row 25
• 12:22 top of R1, saw repeatable glitch in LO powermon from in2 of delay line phase shift to 35MHz PMC
• 12:26:40 moved to PSL-R2 EOM driver by bottom of rack
• 12:28:30 moved to ISC-R1
  • Glitches when touching 24Mhz in row42. More noise in PMC mixer afterwards. Not repeatable
  • RF amp for 75MHz/25MHz see wiggle in ISS PDs  and PMC got noisier
• 12:36 Glitches without touching anything
• 12:37 moved to ISC-R1 below 33
  • Found loose out2 in ALS COMM VCO now tightened
  • Row14 common mode servo for IMC? In2 feels loose, could pull out. Pushed in but still loose (Push/pull connector)
• 12:43:45 moved to ISC-R2
  • Lots of glitches while touching wavefront sensor demod patch panel around row 19. Stopped and No big glitches when repeated at 12:48:00
• 12:49:30 moved to Last rack/ isc-r4. Slow controls
• 12:51 moved to back of PSL R1
• 12:53 moved to back of PSL-R2
  • Mixer saw glitches when touching 35.5HMz PMC but continues once we stopped touching
• 12:56PT Done, Vicky left oscilloscope plugged in. 

Thu Nov 14 10:15:06 2024 INFO: Fill completed in 15min 3secs

Jordan confirmed a good fill curbside.

New Vacuum section on CDS Overview shows CP1 LLCV percentage open as a red bar. Bar limits are between 40% and 100% open so normally you won't see any red in this widget.

Sheila, Vicky, Elenna, TJ, Marc, Filiberto, Richard, Daniel, Jason, Ryan Short

The IMC did not stay locked overnight, after Corey left it locked at 2W with the ISS on (screen shot from TJ). Vicky noticed that the SQZ 35MHz LO monitor sees something going on sometimes before the IMC lost lock, screenshot from Elenna.  A few days ago Nutsinee flagged this squeezer 35MHz LO monitor channel, it does show increased noise when the FSS is unlocked which doesn't make sense but it seems like this is at least partially due to cross talk (when we intentionally unlock the FSS, there is extra noise in this channel).

A bit before 8:29 I unplugged the cable from the IMC servo to the PSL VCO, and we left the FSS and PMC locked with the ISS off.  At 8:31 pacific time the FSS came unlocked, and glitches were visible in the PMC mixer and HV (screenshot) .  The new channel plugged in on TUesday H1:PSL-PWR_HPL_DC_OUT_DQ might show some extra noise, more visible in the zoomed in screenshot.  There are some small glitches seen in the SQZ 35MHz LO monitor at the time of the reference cavity glitches, the squeezer 35MHz LO is shared by the SQZ + PSL PMCs. 

8:44 unlocked the FSS and sitting with the PMC only locked, a few seconds later we had a few glitches in the mixer and HV. PMC alone screenshot. 

The PSL was powered down to restart the beckhoff, a few minutes before 11 until 11:30 or so.

A few minutes before 11 pacific time, Marc Filiberto and Richard went to the CER, measured the power out of the 35MHz source (11.288dBm) and adjusted the Marconi setting to match the power measured on the RF meter to 11.313dBm.  There is a 10MHz signal which is locked to gps through the timing system plugged into the back of the Marconi, Daniel says that he thinks the Marconi is locked to that source if it is plugged in.

At 19:33 (11:33 UTC) the PMC is relocked after the beckhoff reboot with the 35MHz source changed.

TITLE: 11/14 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 33mph Gusts, 23mph 3min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.66 μm/s
QUICK SUMMARY: The IMC dropped lock many times overnight. I imagine that this will shape our plans for the day once we have time to dicuss next steps.

TITLE: 11/14 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

Had a nice 5hr lock for H1.  Running an IMC-only lock overnight to see if there are any FSS/IMC glitches.
LOG:

• 0002 OBSERVING
• 0101 Beckhoff work in MSR done for the day (Fernando)
• 0118-0120 TCSy Laser unlocks:  Dropped out of OBSERVING due to TCS ITMy CO2 laser unlocking and then coming back on its own.
  • 0142-0144 Another TCSy laser unlock
• 0247 Superevent S241114y
• 0459 Lockloss (5hr2min lock)
• 0540 IMC Lock test overnight started
• 0558 Observatory mode taken to PREVENTATIVE MAINTENANCE for overnight.

The plan for tonight's shift was to see how locking would go for H1.  Well....TJ handed over a locked H1, and then it stayed locked for 5hrs of the 6hr Eve shift!  

• So did not determine if locking is "tough" since it was locked most of the shift
• First action was to see if H1 would relock on its own.  Made it to DRMI, but flashes were ugly, so ran CHECK MICH FRINGES, but there was a lockloss  during CHECK MICH FRINGES (at 5:22:27utc).
• Then it was iffy on what to do next.  Would love to run an alignment, to see how H1 would look overnight for locking, but it's minutes from the end of the shift.
• SO---will opt for:  IMC-ONLY Lock Overnight
  • ISC_LOCK taken to IDLE at 05:40:49 <----This would be the time for the start of the "clean" period for checking Overnight IMC/FSS/PSL glitches (if any)
  • IMC_LOCK at LOCKED state and taken to AUTO state (from MANAGED).
  • At 0558 took OBSERVATORY MODE to CORRECTIVE MAINTENANCE

Vicky mentioned that the Lockloss at 0459utc looked like possible FSS glitch, but it looked "different".  She is going to post her plot about that.

H1 was dropped out of OBSERVING due to the TCS ITMy CO2 laser unlocking at 0118utc.  The TCS_ITMY_CO2 guardian relocked the 2min.

It was hard to see the reason why at first (there were no SDF Diffs, but eventually saw a User Message via GRD IFO (on Ops Overview) pointing to something wrong with TCS_ITMY_CO2.  Oli was also here and they mentioned seeing this, along with Camilla, on Oct9th (alog)---this was the known issue of the TCSy laser nearing the end of its life.  It was replaced a few weeks later on  Oct22nd (alog).

Here are some of the lines from the LOG:

2024-11-13_19:43:31.583249Z TCS_ITMY_CO2 executing state: LASER_UP (10)
2024-11-14_01:18:56.880404Z TCS_ITMY_CO2 [LASER_UP.run] laser unlocked. jumping to find new locking point

.

.
2024-11-14_01:20:12.130794Z TCS_ITMY_CO2 [RESET_PZT_VOLTAGE.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 35.109375
2024-11-14_01:20:12.196990Z TCS_ITMY_CO2 [RESET_PZT_VOLTAGE.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 35.0
2024-11-14_01:20:12.297890Z TCS_ITMY_CO2 [RESET_PZT_VOLTAGE.run] timer['wait'] done
2024-11-14_01:20:12.379861Z TCS_ITMY_CO2 EDGE: RESET_PZT_VOLTAGE->ENGAGE_CHILLER_SERVO
2024-11-14_01:20:12.379861Z TCS_ITMY_CO2 calculating path: ENGAGE_CHILLER_SERVO->LASER_UP
2024-11-14_01:20:12.379861Z TCS_ITMY_CO2 new target: LASER_UP

TITLE: 11/14 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 13mph Gusts, 10mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.50 μm/s
QUICK SUMMARY:

TJ was taking H1 to OBSERVING as I arrived!  Plan for tonight is to keep H1 NLN as much as possible---if H1 goes down and locking is tough toward the end of the shift, will redirect and focus on only the IMC and keeping it locked overnight to log time to monitor for any glitches.  If this happens, will set ISC_LOCK to IDLE and take IMC_LOCK to from MANAGED to AUTO.

Environmentally it is less breezy than 24hrs ago.  µseism is about the same as 24hrs ago (it rose a little but came back down).  

TITLE: 11/14 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: We took 4 or so hours out today to troubleshoot the laser glitches that we've been having. After that we were able to relock quickly and test out some longer ramp times for te ETMX transition state.
LOG:

• 1658 Observing
• 1718 Lock loss
• 1845 Lost lock from ETMX transition, planning to stay in IDLE with the IMC locked to test out the stability of the ISS, FSS, IMC
• 2253 Started relocking
• 0002 Observing

We paused locking for just over 4 hours today to try and troubleshoot the glitching that we've been having. After that was wrapped up we relocked without an initial alignment, but I did help DRMI lock by touching SRM and a little BS. Once at low noise there were 3 SDF diffs related to the new TRAMPS for the transition from ETMX state.

Today TJ had the IMC locked without the IFO (with the ISS first loop) from 18:45- 22:45, without the IMC losing lock.  This might suggest that the change of the FSS EOM path op amp yesterday 81247 did indeed solve the issue that was making the IMC unstable even without the interferometer locking, but more time would make this a more definitive test. 

During the 4 hours Jason and Marc also went to the floor and measured the FSS OLG, 81254, which showed some places where there are mulitple places where it came close to unity gain.  They lowered the gain by 0.75 dB.  Later, Marc and I went and measured the IMC OLG, and we tried lowering the gain by 3dB (common gain of 12dB), to give the FSS OLG more clearance from unity gain, we measured the IMC OLG before and after this change 81259. 

We decided to try locking tonight with the FSS gain at 12dB.  If the interferometer is not stable through the evening shift, Corey will set the IFO to down and leave the IMC locked overnight so that we can get more time for the test of IMC stability on it's own. 

Unfortunately, changing the ramp time in the IX L3 to EX L3 transition to 2 seconds (from 10 seconds), seems to overall have made things worse. The first lock attempt through this state was successful, but there were four locklosses last night in state 558, each with 1 second state duration and none tagged as IMC.

Sheila and I changed the ramp time to 20 seconds, to test if ramping slower is better. Besides that, it seems like we should measure the loops before and after the transition on the way up to check the stability of the DARM loop.

TITLE: 11/13 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 16mph Gusts, 11mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.65 μm/s
QUICK SUMMARY: Looks like H1 made it up to low noise two times in the last 3 hours, but lost lock immediately. useism is still quite high, and some wind overnight didn't help. The IMC is now struggling to stay locked after a few failed DRMI attempts, but a successful PRMI. Starting initial alignment now.