This morning I remotely tweaked the beam alignment into the PMC.  With the ISS OFF (note, the diffracted power % was ~3.1%, a little high, so I adjusted this later), I started with PMC Trans of ~108.8 W and PMC Refl of ~16.1 W.  Using the picomotor-controlled mounts I tweaked the alignment to max the PMC Trans and minimize PMC Refl.  Once done, still with the ISS OFF, we now have a PMC Trans of ~109.0 W and a PMC Refl of ~15.7 W.  I turned the ISS back ON and, seeing the higher than normal diffracted power % noted earlier, I adjusted the ISS RefSignal to get the diffracted power % back to ~2.5%; this necessitated a RefSignal change from -1.97 V to -1.99 V.  With the ISS ON we now have a PMC Trans of ~109.4 W and a PMC Refl of ~15.7 W (moving between 15.7 W and 15.8 W).

While I was hoping for move improvement in PMC Trans and Refl, I was only expecting the small change I was able to achieve.  To my mind this means it's getting close to time to adjust the operating currents of the amplifier pump diodes.  Will continue to monitor.

Following instructions from Sheila, I stepped the CARM gains (H1:LSC-REFL_SERVO_IN{1, 2}GAIN) by 1 about every minute starting at 15:11:30 UTC. I started at a gain of 8 and we lost lock as soon as I got to 20. We lost lock at 15:24 UTC.

TITLE: 10/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 7mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.20 μm/s
QUICK SUMMARY:

Have been Locked for  28.5 hours. SUS Charge measurements finished and Camilla is blasting the CO2s. Long maintenance day today.

Workstations updated and rebooted.  This was an update to OS packages.  Coinda packages were not changed.

TITLE: 10/23 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 165Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY:

H1 continues with a lock approaching 20.5hrs.  Winds have died down since "lunch".
LOG:

• ~0045-0024 PCal Lab work (Tony)
• (2) EQ triggered EARTHQUAKE Mode, but H1 rode through.

H1's running on a 16.5hr lock and range around 165Mpc.  Rode through a M4.6 Tongan earthquake and now the chilly breezes are picking up (with gusts peaking above 30mph recently).

TITLE: 10/23 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 7mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.33 μm/s
QUICK SUMMARY:

Receiving a locked H1 from RyanC w a lock running 12.5+hrs.  Slight step up of useism in the last 24hrs (but still low).

On the CDS Overview, there is a RED CFC block for H1SUSETMYPI_GDS_TP which happened just a few minutes ago (2308utc).

TITLE: 10/23 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

PSL 101 yellow alarm at 19:55UTC, came down after about 15 minutes.

We dropped out of observing from 22:01 to 22:03UTC for Camilla and I to adjust the OPO temperature, but unfortunately it made the range worse. Seeing LLO have a lockloss, we went out again to adjust it from 22:34 to 22:40UTC.

LOG:                                                                                                                                                                                                  

In addition to the 1.23Hz line seen in DARM from ETMX ISI (alog 73625), there is another line closeby at 1.39Hz (seen in plots attached to 73622) which is coming from ITMY ISI (as reported by Jim here). It is also visible on the BS ISI sensors, but it is probably being fed from the ITM to the BS via the CPS diff loop. The attached plot on p1 shows the stage 2 ISI dofs in the beam direction where we see the 1.23Hz line in ETMX (green) and 1.39Hz line in the BS and ITMY (blue and cyan). p2 shows DARM IN1, p3 shows the coherence between DARM and the ISI and p4 is the zoomed in coherence version between 1-2Hz, where we can see coherence between BS/ITMY and DARM at 1.39Hz. ETMY is also coherent, which I presume is because the ITMY motion is fed to ETMY also via CPS diff. 

While we look at ETMX tomorrow during maintenance, I would also be interested to take some data with the ITMY ISI offline.

FAMIS26159
 

Added 100mL to TCSX and 150mL to TCSY chillers. The filters were looking good and there was no evidence of coolant leaks on either. The LDU (leak detection unit) Dixie cup was dry.

FAMIS 26459
There was a large spike 20 days ago (October 3rd) which would have been on a Tuesday & I imagine this is just a daq restart or something.
So breaking this up into before the spike and after the spike.

Before the October 3rd Spike:
~35 days ago (Septemper 18th)  H1:HPI-PUMP_LO_CONTROL_VOUT had an increase and it has persisted ever since, even after the daq restart.

After the October 3rd Spike:
Every thing looks relatively normal, Aside from the persistent inscrease seen on september 18th and another much smaller spike about 6 days ago which was likely another tuesday maintenance reboot.
 
 

Mon Oct 23 10:11:20 2023 INFO: Fill completed in 11min 16secs

https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1382089144

This lockloss had a bit of a wiggle in LSC darm, but it looked a little different from the other times its noted. DHARD_Y might have rung up a bit?

Closes  FAMIS 26214

Laser Status:
    NPRO output power is 1.814W (nominal ~2W)
    AMP1 output power is 67.79W (nominal ~70W)
    AMP2 output power is 135.3W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN

PMC:
    It has been locked 5 days, 22 hr 39 minutes
    Reflected power = 16.23W
    Transmitted power = 109.1W
    PowerSum = 125.3W

FSS:
    It has been locked for 0 days 5 hr and 45 min
    TPD[V] = 0.9668V

ISS:
    The diffracted power is around 2.5%
    Last saturation event was 0 days 5 hours and 45 minutes ago

Possible Issues: None

TITLE: 10/23 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 163Mpc
OUTGOING OPERATOR: Camilla (OWL)
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 7mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.33 μm/s
QUICK SUMMARY:

• We've been locked for 4:26, we survived a 6.0 from New Zealand ~4.5 hours ago
• All systems look good

Below are the steps for a (failed) attempt at running a broadband + simmulines calibration mesaurement w/ H1 was locked for 3+hrs and with L1 still in recovery from a power outtage:

• Following instructions (link)
• Screenshot of Cailbratoin Monitor medm taken (see attached)
• 2332 Out of OBSERVING and ISC LOCK to NLN_CAL_MEAS
• 2333 in a terminal started broadband calibration meas: pydarm measure --run-headless bb
  • 2339 this ended with:  ICE default IO error handler doing an exit(), pid = 99711, errno = 32
  • (Listing last few lines from this measurement below)
• 2340 Assuming this was the "end" of the broadband measurement, so I next ran simmulines measurement (in same terminal, per instructions):  gpstime;python /ligo/groups/cal/src/simulines/simulines/simuLines.py -i /ligo/groups/cal/src/simulines/simulines/settings_h1.ini;gpstime 
  • This had a similar error after 2-min.  Here are the last few lines from the terminal:

2023-10-22 23:42:28,941 | INFO | Drive, on L3_SUSETMX_iEXC2DARMTF, at frequency: 544.99, is now running for 23 seconds.
2023-10-22 23:42:35,388 | INFO | Drive, on PCALY2DARMTF, at frequency: 8.69, is finished. GPS start and end time stamps: 1382053346, 1382053370
2023-10-22 23:42:35,389 | INFO | Scanning frequency 8.99 in Scan : PCALY2DARMTF on PID: 106437
2023-10-22 23:42:35,389 | INFO | Drive, on PCALY2DARMTF, at frequency: 8.99, is now running for 31 seconds.
^C2023-10-22 23:42:40,347 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L1_CAL_EXC
2023-10-22 23:42:40,347 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L3_CAL_EXC
2023-10-22 23:42:40,347 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L2_CAL_EXC
2023-10-22 23:42:40,347 | INFO | Ending lockloss monitor. This is either due to having completed the measurement, and this functionality being terminated; or because the whole process was aborted.
2023-10-22 23:42:40,348 | ERROR | Ramping Down Excitation on channel H1:LSC-DARM1_EXC
2023-10-22 23:42:40,348 | ERROR | Aborting main thread and Data recording, if any. Cleaning up temporary file structure.
2023-10-22 23:42:40,348 | ERROR | Ramping Down Excitation on channel H1:CAL-PCALY_SWEPT_SINE_EXC
ICE default IO error handler doing an exit(), pid = 106373, errno = 32
PDT: 2023-10-22 16:42:44.751376 PDT
UTC: 2023-10-22 23:42:44.751376 UTC
GPS: 1382053382.751376

• 2351 After ligo.chat with Jenne, decided to stop Calibration Measurements (she mentioned possibly the dolphin crash from Fri night could be a possible culprit?)
• 2351 OBSERVING (and I restored H1 Manager to MANAGED)

Here were the last few lines at the end of the broadband measurement:

notification: new test result
notification: new test result
notification: new test result
notification: end of measurement
notification: end of test
diag> save /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20231022T233325Z.xml
/ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20231022T233325Z.xml saved
diag> quit
EXIT KERNEL

INFO | bb measurement complete.
INFO | bb output: /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20231022T233325Z.xml
INFO | all measurements complete.
ICE default IO error handler doing an exit(), pid = 99711, errno = 32

TITLE: 10/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Spent most of the shift continuing to try and recover alignments from the dolphin glitch at 8:05UTC. Currently waiting for ADS to converge to go into observing.

I had lots of issues to start in the morning, ALS_X would lose lock for no apparent reason or its WFS would oscilate too hard and kill its lock, then Diff_IR kept being "not really found" then PRMI buildups were very unstable and we couldn't get good enough flashes to get DRMI. I tried an IA, and then moving SR3 and PR3 based on their OPLEVS to where they were before the dolphin glitch, adjusting SR3 made PRMI better and PR3 made it worse.

I ended up trying to a full slider restore to a time when we were locking for the lock that was killed by the computer crashes (10/19 18:09UTC), still no luck getting PRMI or DRMI. I was considering doing another IA or restoring the sliders to after Camilla's IA this morning, after calling and talking with Jenne I did an initial alignment and not another restore. After this IA we went back up to NLN without any interventions, I stepped slowly through the states just in case. The violins had rung up a bit, but they damped down pretty quickly and weren't much of a hinderance.

Once I got to NLN, the SQZ manager had the notification "SQZ ASC AS42 not on?" So I went to RESET_SQZ_ASC then back to FREQ_DEP_SQZ following Vickys instructions in alog71083. While waiting for the ADS to converge I went through the large amout of SDF diffs for BOS and OAF from 8:50UTC which looked to be when Dave restarted their models, I reverted almost all of them after talking to Jenne (they were all COEFF) except for 1 in BOS. Naoki also hopped on to optimize the squeezing before we went back into observing, I optimized the OPO temperature when he was done and accepted it in SDF then went into observing at 20:49UTC.

Lockloss at 21:22UTC

Lost lock during PRMI_ASC from Yarm losing lock. PRMI keeps locking then losing it seconds later, flashes were super low. Tried CHECK_MICH after a few PRMIs. I trended PRM, PR3, SR3, and the BS to see if anything had moved since the LL, I ended up adjusting PR3 in yaw by .2 (155.5 ->155.3) since it looked like it drifted a little bit and then the flashes looked much better. Not sure why it moved? I also had to tap PRM a little but in pitch (2 microradians, -1502.4 -> -1504.4) and it caught PRMI then DRMI. Guardian brought us up the rest of the way no problem.

LOG:

No log

• 8:05UTC We had a 8:05UTC lockloss where HAM1 HEPI WD tripped (ATC type trip)
• 8:18UTC I saw no obvious issue with ground motion or HAM1 so Untripped HAM1
  • Then noticed that CDS screen was red with timing card, adc, dac, dk errors on machines sus123, sush2a, sus34, sus56, iopoaf, and ioplsc0. 
• 08:24UTC Left voice message with Dave Barker. 08:32UTC Voicemail left with Erik. 
• 08:35UTC Dave Barker online and looking at front end crashes. alog 73629
• 08:45UTC Dave restarting computers as all DAC cards look okay. I requested ISC_LOCK to IDLE and all corner station (not HAM7) SUS to SAFE (Safe State for IFO wiki)  and Dave bypassed SEI. 
• 09:10UTC CDS Okay
  • SUS WD untripped and back to nominal configuration. 
  • SEI ISI WD tripped (payload type) when Dave bypassed them reset those WDs too. 
• 09:35UTC All SUS and SEI nominal. Resumed locking with ISC_LOCK via DOWN, INIT, NLN. 
• 09:45UTC No light seen on green arms so plan to run baffle align scripts (62024).

J. Kissel, J. Driggers

I was brainstorming why LOWNOISE_LENGTH_CONTROL would be ringing up a Transmon M1 to M2 wire violin mode (modeled to be at 104.2 Hz for a "production" TMTS; see table 3.11 of T1300876) for the first time on Aug 4 2023 (see current investigation recapped in LHO:72214), and I remembered "TMS tracking..."

In short: we found that ETMX M0 L OSEM damping error signal has been fed directly to TMSX M1 L path global control path, without filtering, since Sep 28 2021. Yuck!

On Aug 30 2021, I resolved the discrepancies between L1 and H1 end-station SUS front-end models -- see LHO:59772. Included in that work, I cleaned up the Tidal path, cleaned up the "R0 tracking" path (where QUAD L2 gets fed to QUAD R0), and installed the "TMS tracking" path as per ECR E2000186 / LLO:53224. In short, "TMS tracking" couples the ETM M0 longitudinal OSEM error signal to the TMS M1 longitudinal "input to the drivealign bank" global control path, with the intent of matching the velocity of the two top masses to reduce scattered light.

On Aug 31 2021, the model changes were installed during an upgrade to the RCG -- see LHO:59797, and we've confirmed that I turned both TMSX and TMSY paths OFF, "to be commissioned later, when we have an IFO, if we need it" at
    Tuesday -- Aug 31 2021 21:22 UTC (14:22 PDT) 

However, 28 days later,
    Tuesday -- Sept 28 2021 22:16 UTC (15:16 PDT)
the TMSX filter bank got turned back on, and must have been blindly SDF saved as such -- with no filter in place -- after an EX IO chassis upgrade -- see LHO:60058. At the time, that RCG 4.2.0 still had the infamous "turn on a new filter with its input ON, output ON, and a gain of 1.0" feature, that has been since resolved with RCG 5.1.1. So ... maybe, somehow, even though the filter was already installed on Aug 31 2021, the IO chassis upgrade rebuild, reinstall, and restart of the h1sustmsx.mdl front end model re-registered the filter as new? Unclear. Regardless this direct ETMX M0 L to TMSX M1 L path has been on, without filtering, since Sep 28 2021. Yuck!

Jenne confirms the early 2021 timeline in the first attachment here.
She also confirms via a ~2 year trend of the H1:SUS-TMSY_M1_FF_L filter bank's SWSTAT, that no filter module has *ever* been turned on, confirmed that there's *never* been filtering.

Whether this *is* the source of 102.1288 Hz problems and that that frequency is the TMSX transmon violin mode is still unclear. Brief investigations thus far include
    - Jenne briefly gathered ASDs of ETMX M0 L (H1:SUS-ETMX_M0_DAMP_L_IN_DQ) and TMSX M1 L OSEMs' error signal (H1:SUS-TMSX_M1_DAMP_L_IN1_DQ) around the time of Oli's LOWNOISE_LENGTH_CONTROL time, but found that at 100 Hz, the OSEMs are limited by their own sensor noise and don't see anything.
    - She also looked through the MASTER_OUT DAC requests (), in hopes that the requested control signal would show something more or different, but found nothing suspicious around 100 Hz there either.
    - We HAVE NOT, but could look at H1:SUS-TMSX_M1_DRIVEALIGN_L_OUT_DQ since this FF control filter should be the only control signal going through that path. I'll post a comment with this.

Regardless, having this path on with no filter is clearly wrong, so we've turned off the input, output, and gain accepted the filter as OFF, OFF, and OFF in the SDF system (for TMSX, the safe.snap is the same as the observe.snap).