Naoki noticed the pump fiber rejected PD (in ham7, rejects pump fiber light that comes out in the wrong polarization) was saturated, so today I re-aligned the pump fiber polarization using sqzt0 pico's (described recently also in 71761).

I'm not sure why pump fiber needs to have its input polarization readjusted so often; I checked the CLF fiber and FCGS fiber, and they both seemed relatively well-aligned despite having not adjusted those in a while. Especially this time, it seems the fiber polarization got misaligned more quickly than before.

Tue Aug 22 10:08:15 2023 INFO: Fill completed in 8min 11secs

Note TCs did not saturate at -200C, the outside temperatures are lower today.

TITLE: 08/22 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventative Maintenance
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 12mph Gusts, 7mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.10 μm/s
QUICK SUMMARY: We're going to try and stay locked for a while this maintenance period, the BRS at EndX has been turned off for the PCAL team working down there.

• CDS/VAC/SUS looks good
• A few dust alarms, from the smoke? EX, PSL anteroom
• The inlock charge measurements did not run this morning, nothing in the SUS_CHARGE log and the state never changed on that guardian -Tagging SUS

TITLE: 08/21 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: H1 was able to relock and start observing an hour into the shift, then remained observing for the rest of the evening. H1 has now been locked for 7 hours.

• 22:59 Lockloss - alog
• 00:00 Reached NLN
  • Fully automatic lock acquisition
• 00:10 Started observing

The dust monitors for PSL101, EX VEA, and Optics Lab have been alarming on and off throughout the evening. Trends of counts attached.

LOG:

State of H1: Observing at 152Mpc

H1 was able to relock automatically following the lockloss at the start of the shift. H1 has now been locked for 3.5 hours.

FAMIS 19990

The PMC and FSS alignment from last Tuesday (alog 72222) caused the expected drop in PMC_REFL power and increase in PMC_TRANS. The FSS_TPD signal didn't improve much with that tweak, but over the past 3 days it's risen by ~50mV. The ISS diffracted power is continuting to jump around as we've seen it do recently.

Lockloss @ 22:59 UTC

Immediately prior to the lockloss (at 22:59:42.89 to be precise), the SQZ_MANAGER guardian reported the notifications "SQZ ASC AS42 not on?? Please RESET_SQZ_ASC" and "FC-IR UNLOCKED!"

ISC_LOCK jumped to LOCKLOSS at 22:59:43.58, so I suspect the filter cavity unlocking had something to do with this lockloss. However, I'm not sure if we've seen a lockloss from just the FC unlocking in the past, so I hesitate to say this was the definitive cause or just a case of guardian loop timing showing a red herring.

TITLE: 08/21 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 9mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY: H1 just lost lock, cause still being determined.

• All systems look good. Ground motion and wind are low.

TITLE: 08/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

- Couple of yellow dust monitor alerts from the Optics Lab

- Lockloss @ 22:59 (15 seconds before the end of my shift :(). Leaving a locking IFO to Ryan S.
LOG:                                                                                                                                                                                                                                                                                                                                                                                                                                                                  

J. Kissel

As I continue game-planning the integration of O5 suspensions into the existing system, (see progress thus far in G2301306) one of the questions is the percentage of computer usage needed on a given computer core running different numbers and amounts of suspensions. Given the sheer amount of new SUS coming in coupled with the limited number of computers and IO chassis, we'll need to pull a similar computer core assignment re-arrangement game that we've done before previous observing runs -- (for O4 see G2101636, and implementation aLOGs LHO:59651 for HAM56 and LHO:59669 for HAM7. For O3 see LHO:38827).

Summary of findings shown in the attached trends of the CPU_METER_MAX values for SUSB123 and SUSH34 and SUSH56 and SUSH7 since May 24 2023 16:00 UTC: 

    :: All 65 kHz IOP Models, be it SUSH34, SUSB123, SUSH56, or SUSH7 are consistently using 8 usec / 15 usec ~ 53% of capacity.

    On SUSB123, 
    :: The ITM 16 kHz user models consume the most clock cycle time, running with maximums of ~26 usec / 61 usec ~42%. (I presume the abnormally large computation comes from violin mode damping)
    :: The BS 16 kHz user model only consumes 14 / 61 ~ 23% of its core.

    On SUSH34,
    :: All three HSTSs, MC2, PR2, and SR2 run between 13 / 61 usec ~ 21% of core capacity. MC2 had "run the longest" at 14 and I presume that's from the fast longitudinal global control filtering (where PR2 and SR2 only receive slow angular drift control), but the restart after reboot from the dolphin crash on July 30 2023 (see LHO:71829) seems to have brought it back in symmetry with the other two at 13.

    On SUSH56 and SUSH7,
    :: all SUSH56 and SUSH7 16 kHz user models are using at most only 15 / 61 usec ~ 25% (it's SUSSRM running one globally controlled HSTS, and SUSSQZIN running three ZMs [two HPDS and one HDDS] and the OPO). Some, like SUSSQZOUT which are only running damping loops on ZM4, ZM5, and ZM6 top masses (two HPDSs and one HSDS, respectively) are using as little as 8 / 61 ~13%
Their 65 kHz IOPs are running at ~50% capacity.
    :: The SUSAUXH7 model is running at 4 kHz, and consumes only 4 / 244 ~ 6% of its core on SUSH7. I'll note that no one has populated the coil driver monitor filter banks with any filters, but I'm sure even if it was chock full I doubt it would even come close to "breaking the bank."


This bodes well for re-organizing models for O5; we're not computationally limited in doing so for any suspensions on the move, even if we keep running things at 16 kHz. Further, we can consider combining even more suspensions' controls into single models if we expect the SUS's filter computation to be light [e.g. only damping like in SUSSQZOUT].

H1 has been locked for 20 hours, all systems appear stable and seismic motion is low.

Mon Aug 21 10:08:31 2023 INFO: Fill completed in 8min 27secs

Travis confirmed a good fill curbside.

Genevieve, Lance, Robert

To further understand the roughly 10Mpc lost to the HVAC (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72308), we made several focussed shutdowns today. These manipulations were made during observing (with times recorded) because such HVAC changes happen automatically during observing and also, we were reducing noise rather than increasing it. The times of these manipulations are given below.

One early outcome is that the peak at 52 Hz in DARM is produced by the chilled water pump at EX (see figure). We went out and looked to see if the vibration isolation was shorted, it was not, though there are design flaws (the water pipes arent isolated). We switched from CHWP-2 to CHWP-1 to see if the particular pump was extra noisy. CHWP-1 produced a similar peak in DARM at its own frequency. The peak in accelerometers is also similar in amplitude to the one from the water pump at EY. One possibility is that the coupling at EX is greater because of the undamped cryobaffle at EX.

Friday HVAC shutdowns; all times Aug. 18 UTC

15:26 CS SF1, 2, 3, 4 off

15:30:30 CS SF5 and 6 off

15:36 CS SF5 and 6 on

15:40 CS SF1, 2, 3, 4 back on

16:02 EY AH2 (only fan on) shut down

16:10 EY AH2 on

16:20 EY AH2 off

16:28 EY AH2 on

16:45 EY AH2 and chiller off

16:56:30 EY AH2 and chiller on

17:19:30 EX chiller only off, pump stays on

17:27 EXwater pump CHWP-2 goes off

17:32: EX CHWP-2 back on chiller back on right after

19:34:38 EX chiller off, CHWP-2 pump stays on for a while

19:45 EX chiller back on

20:20 EX started switch from chiller 2 to chiller 1 - slow going

21:00 EX Finally switched

21:03 EX Switched back to original, chiller 1 to chiller 2

Robert did an HVAC off test. Here is a comparison of GDS CALIB STRAIN NOLINES from earlier on in this lock and during the test. I picked both times off the range plot from a time with no glitches.

Improvement from removal of 120 Hz jitter peak, apparent reduction of 52 Hz peak, and broadband noise reduction at low frequency (scatter noise?).

I have attached a second plot showing the low frequency (1-10 Hz) spectrum of OMC DCPD SUM, showing no appreciable change in the low frequency portion of DARM from this test.

J. Kissel, J. Driggers

I was brainstorming why LOWNOISE_LENGTH_CONTROL would be ringing up a Transmon M1 to M2 wire violin mode (modeled to be at 104.2 Hz for a "production" TMTS; see table 3.11 of T1300876) for the first time on Aug 4 2023 (see current investigation recapped in LHO:72214), and I remembered "TMS tracking..."

In short: we found that ETMX M0 L OSEM damping error signal has been fed directly to TMSX M1 L path global control path, without filtering, since Sep 28 2021. Yuck!

On Aug 30 2021, I resolved the discrepancies between L1 and H1 end-station SUS front-end models -- see LHO:59772. Included in that work, I cleaned up the Tidal path, cleaned up the "R0 tracking" path (where QUAD L2 gets fed to QUAD R0), and installed the "TMS tracking" path as per ECR E2000186 / LLO:53224. In short, "TMS tracking" couples the ETM M0 longitudinal OSEM error signal to the TMS M1 longitudinal "input to the drivealign bank" global control path, with the intent of matching the velocity of the two top masses to reduce scattered light.

On Aug 31 2021, the model changes were installed during an upgrade to the RCG -- see LHO:59797, and we've confirmed that I turned both TMSX and TMSY paths OFF, "to be commissioned later, when we have an IFO, if we need it" at
    Tuesday -- Aug 31 2021 21:22 UTC (14:22 PDT) 

However, 28 days later,
    Tuesday -- Sept 28 2021 22:16 UTC (15:16 PDT)
the TMSX filter bank got turned back on, and must have been blindly SDF saved as such -- with no filter in place -- after an EX IO chassis upgrade -- see LHO:60058. At the time, that RCG 4.2.0 still had the infamous "turn on a new filter with its input ON, output ON, and a gain of 1.0" feature, that has been since resolved with RCG 5.1.1. So ... maybe, somehow, even though the filter was already installed on Aug 31 2021, the IO chassis upgrade rebuild, reinstall, and restart of the h1sustmsx.mdl front end model re-registered the filter as new? Unclear. Regardless this direct ETMX M0 L to TMSX M1 L path has been on, without filtering, since Sep 28 2021. Yuck!

Jenne confirms the early 2021 timeline in the first attachment here.
She also confirms via a ~2 year trend of the H1:SUS-TMSY_M1_FF_L filter bank's SWSTAT, that no filter module has *ever* been turned on, confirmed that there's *never* been filtering.

Whether this *is* the source of 102.1288 Hz problems and that that frequency is the TMSX transmon violin mode is still unclear. Brief investigations thus far include
    - Jenne briefly gathered ASDs of ETMX M0 L (H1:SUS-ETMX_M0_DAMP_L_IN_DQ) and TMSX M1 L OSEMs' error signal (H1:SUS-TMSX_M1_DAMP_L_IN1_DQ) around the time of Oli's LOWNOISE_LENGTH_CONTROL time, but found that at 100 Hz, the OSEMs are limited by their own sensor noise and don't see anything.
    - She also looked through the MASTER_OUT DAC requests (), in hopes that the requested control signal would show something more or different, but found nothing suspicious around 100 Hz there either.
    - We HAVE NOT, but could look at H1:SUS-TMSX_M1_DRIVEALIGN_L_OUT_DQ since this FF control filter should be the only control signal going through that path. I'll post a comment with this.

Regardless, having this path on with no filter is clearly wrong, so we've turned off the input, output, and gain accepted the filter as OFF, OFF, and OFF in the SDF system (for TMSX, the safe.snap is the same as the observe.snap).

J. Kissel, D. Barker

As of today Dave helped me install the new front-end, EPICs controlled oscillators discussed in LHO:71746. Then, after crafting a few new MEDM screens (see comments below), I've turned ON some of those oscillators in order to replace the unstable function of the CAL_AWG_LINES guardian.

So, there're no "new" calibration lines (not since we turned CAL_AWG_LINES back ON last week at 2023-07-25 22:21:15 UTC -- see LHO:71706) -- but they're now driven by front-end, EPICs controlled oscillators rather than by guardian using the python bindings for awg (which was unstable across computer crashes, and other connection interruptions).

This is true as of the first observation segment today: 2023-08-01 22:02 UTC
However, due to a mishap with me misunderstanding the state of the PCALY SDF system (see LHO:71879), I accidentally overwrote the PCALXY comparison line at 284.01 Hz, and we went into observe. Thus, 
The short observation segment between 22:02 - 22:11 UTC is out of nominal configration, because there's no PCALY line contributing to the PCALXY comparison.
 
The was rectified by the second observation segment starting on 2023-08-01 22:16 UTC.

Also, because of these changes the subtraction team should switch their witness channel for the DARM_EXC frequencies to H1:LSC-CAL_LINE_SUM_DQ.
The PCALY witness channel remains the same, H1:CAL-PCALY_EXC_SUM_DQ, as the newly used oscillators sum in to the same channel.
Below, I define which oscillator number is assigned to which frequency.

Here's the latest list of calibration lines:
Freq (Hz)   Actuator                   Purpose                      Channel that defines Freq             Changes Since Last Update (LHO:69736)     
8.825       DARM (via ETMX L1,L2,L3)   Live DARM OLGTFs             H1:LSC-DARMOSC1_OSC_FREQ              Former CAL_AWG_LINE, now driven by FE OSC; THIS aLOG
8.925       PCALY                      Live Sensing Function        H1:CAL-PCALY_PCALOSC5_OSC_FREQ        Former CAL_AWG_LINE, now driven by FE OSC; THIS aLOG
11.475      DARM (via ETMX L1,L2,L3)   Live DARM OLGTFs             H1:LSC-DARMOSC2_OSC_FREQ              Former CAL_AWG_LINE, now driven by FE OSC; THIS aLOG
11.575      PCALY                      Live Sensing Function        H1:CAL-PCALY_PCALOSC6_OSC_FREQ        Former CAL_AWG_LINE, now driven by FE OSC; THIS aLOG
15.175      DARM (via ETMX L1,L2,L3)   Live DARM OLGTFs             H1:LSC-DARMOSC3_OSC_FREQ              Former CAL_AWG_LINE, now driven by FE OSC; THIS aLOG
15.275      PCALY                      Live Sensing Function        H1:CAL-PCALY_PCALOSC7_OSC_FREQ        Former CAL_AWG_LINE, now driven by FE OSC; THIS aLOG
24.400      DARM (via ETMX L1,L2,L3)   Live DARM OLGTFs             H1:LSC-DARMOSC4_OSC_FREQ              Former CAL_AWG_LINE, now driven by FE OSC; THIS aLOG
24.500      PCALY                      Live Sensing Function        H1:CAL-PCALY_PCALOSC8_OSC_FREQ        Former CAL_AWG_LINE, now driven by FE OSC; THIS aLOG
15.6        ETMX UIM (L1) SUS          \kappa_UIM excitation        H1:SUS-ETMY_L1_CAL_LINE_FREQ          No change
16.4        ETMX PUM (L2) SUS          \kappa_PUM excitation        H1:SUS-ETMY_L2_CAL_LINE_FREQ          No change
17.1        PCALY                      actuator kappa reference     H1:CAL-PCALY_PCALOSC1_OSC_FREQ        No change
17.6        ETMX TST (L3) SUS          \kappa_TST excitation        H1:SUS-ETMY_L3_CAL_LINE_FREQ          No change
33.43       PCALX                      Systematic error lines       H1:CAL-PCALX_PCALOSC4_OSC_FREQ        No change
53.67         |                            |                        H1:CAL-PCALX_PCALOSC5_OSC_FREQ        No change
77.73         |                            |                        H1:CAL-PCALX_PCALOSC6_OSC_FREQ        No change
102.13        |                            |                        H1:CAL-PCALX_PCALOSC7_OSC_FREQ        No change
283.91        V                            V                        H1:CAL-PCALX_PCALOSC8_OSC_FREQ        No change
284.01      PCALY                      PCALXY comparison            H1:CAL-PCALY_PCALOSC4_OSC_FREQ        Off briefly between 2023-08-01 22:02 - 22:11 UTC, back on as of 22:16 UTC
410.3       PCALY                      f_cc and kappa_C             H1:CAL-PCALY_PCALOSC2_OSC_FREQ        No Change
1083.7      PCALY                      f_cc and kappa_C monitor     H1:CAL-PCALY_PCALOSC3_OSC_FREQ        No Change
n*500+1.3   PCALX                      Systematic error lines       H1:CAL-PCALX_PCALOSC1_OSC_FREQ        No Change (n=[2,3,4,5,6,7,8])

I un-commented the lines in ISC_LOCK so that both CO2_PWR guardians will again go to NO_OUTPUT in DOWN, initial alignment and manual initial alignment states. This is undoing the change we made in alog 66535. C)2s were on all weekend. I turned them off around 8:15am. We can re-think this change later in the week.

Ryan S locked the IFO with no issues on on Friday while the CO2s remained on but we had only been unlocked for 2 hours. Comparing Ryan's lock to alog 66520, the circulating arm powers still increase by 40kW in the forst 2 hours of lock with the CO2s left on when locking. I predited this would reduce but it didn't.

Vent progress today:

HAM2 (Hugh, Kissel, Jim, Rick, Sheila)

HAM2 ISI dampers are installed, minus the 3 TMDs which are currently being tuned by Jeff K and will be reinstalled in ~ the morning.  Thanks to Sheila The Bolt Breaker for crumpling and contorting her skeleton in order to reach those difficult to reach ISI guts in that one corner inside the ISI.

Rick assisted with unplugging and moving the ISS array out of the way for the damper install.  He also hunted more appropriate cable relief hardware which he'd like to install in the next few days when the ISS goes back into place.

BSC3 (Betsy, Travis)

ITMX was thoroughly inspected to check for any mechanics* which may have cause the IFO to need it to be maxed in pitch after the May cleaning vent.  All looked fine with zero pitch offset and with maxed pitch offset.  We have zeroed the biases on the main and reaction chains and will reset both chain mechanical pointing to the optical lever when we reinstall the new ITM optic. 

While looking at it, we also mapped the optical lever beams heading out of the viewport down the manifold (Travis hoofed it down there with a head lamp, target and a pen while I steered the suspension around with a CDS laptop).  I'll post results of that separately - at first glance the CP-HR beam and ITM-AR beam look much closer together than we found months ago on the ITMy, good.

Started work to remove the sleeve, vibration absorber blocks, wedges, structure cross braces, and flooring panels (to get the sleeve off).  Face shields were added to the exposed QUAD optic surfaces.  (The suspension is still suspended so violin measurements can be done in the next day or so after we get the LSAT on the structure for measurement equipment support.)

HAM3

Bubba, Mark, and Tyler removed the doors from HAM3 today.

Jim locked the HAM3 ISI.

* All 4 main chain pitch adjuster rods were found tightly locked in place.  All chain cabling was as we left it - nominal, not touching any place it shouldn't.  The ACB assembly was still nominal at only ~1mm away from the QUAD structure, but not touching. All magnet flags looked reasonably centered at all stages.  All EQ stops looked well positioned and not touching.  Nothing was found that possibly fell from higher up and landed on a stage in order to cause a pitch change.  (Recall the suspension had a clean bill of health via TFs a few times over the last month, so we didn't expect to find anything really.)