TITLE: 09/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: One lock loss with a full auto relock. Quiet shift otherwise.
LOG:

• 2137 Lock loss
• 2253 Observing

1378762650

Looks like the same one as yesterday and similar to others recently were LSC-DARM_IN1 shows the first signs of movement. Need to look into this further. Ibrahim has dug a good amount here 72876.

I received an email from the control room (thank you TJ) about the Mid Y AHU Fan 1 showing some excessive vibration. Tyler went down to investigate and had me turn the fan off to listen to the sound as it was ramping down. It was terrible, one or both bearings are damaged or destroyed. We switched to Fan 2 which should be much smoother. We will investigate Fan 1 and replace the bearings. 

An alarm was noted this morning on the Alerton System at End X chiller 2. The alarm information is not accessible remotely. At the control a "Low Refrigerant Temperature" alarm was present. Because this is a non-latched alarm, I took note and cleared it before returning to the corner station. 

Shortly after clearing, the alarm returned. I investigated the setup of the chiller via compass and noticed that the chiller was set to "manually enable". It is my understanding that a manual enable of the chiller will hold the chiller on regardless of the demand for cooling. I suspect that the cooler temperatures combined with the manual enable kept the chiller running well past the buildings demand for chiller water and possible driving refrigerant temperatures lower than the chiller would like.

When returning this afternoon to clear the second alarm, I observed, for the first time in recent memory, EX chiller 2 ramping down indicating that the requirement for cooling had been satisfied. The control boards diagnostics agreed.

Im hopeful the chiller refrigerant will return to normal following this change but I will continue to monitor it in the coming days.

B. Gateley T. Guidry 

R. Short, T. Shaffer

Our automation has called for assistance when earthquakes roll through and make locking H1 difficult, which typically just has an operator request H1 to 'DOWN' and wait until ground motion is low, then try locking again. In an attempt to further improve our automation and lower the need for intervention, I've added a 'WAITING' state to H1_MANAGER that holds ISC_LOCK in 'READY' and waits for the SEI_ENV guardian to leave its 'EARTHQUAKE' state before moving back to 'RELOCKING.' H1_MANAGER will jump from 'RELOCKING' to 'WAITING' if the SEI_ENV node is in 'EARTHQUAKE' or 'LARGE_EQ' and ISC_LOCK is not past 'READY' (the motivation for this being that if H1 is making progress in locking when an earthquake hits, we don't want it to stop if the earthquake is harmless enough).

These changes are committed to svn and H1_MANAGER has been loaded.

I have put together an MEDM for the FCES (HAM8 shack) FMCS. It can be opened from the FMCS Overview MEDM.

On a related note, I have added a button on the Overview to run a script which restores the FMCS EPICS alarm settings.

We are now in day 113 of O4 and we have not had any spontaneous IPC receive errors on any model throughout this time.

During Tuesday maintenance this week I forgot to issue a DIAG_RESET on h1oaf after the pem models were restarted, and therefore it is showing latched IPC errors from this time which I just noticed today.

To elevate the visibility of latched transient IPC errors, I have added a new block on the CDS overview which will turn yellow if the model has a latched IPC error. This block does not differentiate between IPC type (shared-memory, local-dolphin, x-arm, y-arm). The new block is labeled lower case "i". Clicking on this block opens the model's IPC channel table.

The upper case "I" block remains as before which turns red if there are any ongoing IPC errors (reported as a bit in the model's STATE_WORD)

To make space for this new block (located at the end by the CFC) I have reduced the width of the DAQ-STAT and DAQ-CFC triangles to the same width as the blocks (10 pixels).

Thu Sep 14 10:07:22 2023 INFO: Fill completed in 7min 18secs

Gerardo confirmed a good fill curbside.

TITLE: 09/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 145Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: SEISMON_ALERT
    Wind: 1mph Gusts, 0mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY: Locked for 9 hours. A 5.2M earthquake from Columbia is starting to roll through and elevate 30-100mHz.

TITLE: 09/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 142Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:
IFO is in NLN and OBSERVING as of 06:04 UTC

Lock acquisition was fully automatic, but had to go through PRMI and MICH_FRINGES.

3 IY SDF diffs accepted as per Jenne's instruction (screenshotted)

Lockloss Investigation (alog 72875)

“EX” Saturation one second before the lockloss.

The lockloss happened at 04:45:48 UTC. In order to find the so-called “first cause”, I realized that there was an EX saturation at 04:45:47. I went and checked the L3 actuators since those were the highest (blinking yellow) on the saturation monitor.

Upon trending this lockloss, I found that there was indeed a very big actuation at 04:45:47 at around the same time for all 4 ESDs. The fact that they were also in the same millisecond tells me that they were all caused by something else. (Screenshot "EXSaturation")

More curious however is that there was a preliminary negative “kick” about half a second before (1.5 seconds pre-lockloss at 04:45:46.5 UTC). This wasn’t a saturation but contributes to instability in EX (I’m assuming). This “kick” did not happen regularly before the saturation so I think may be relevant to the whole mystery. All of these also happened at the same time (to the nearest ms) and all had the same magnitude of ~-330,000 cts. I think that this half a second kick was caused by something else since it’s the same magnitude and time for all 4.

It’s worth noting that this preliminary kick was orders of magnitude higher (lower) than the EX activity leading up to it (Screenshot "EXSaturation2") and the saturation was many orders of magnitude higher than that one, which went up to 1*10¹¹ counts.

It is equally worth noting that EX has saturated 5 separate times since the beginning of the shift:

1. 00:43:44 UTC
2. 01:52:23 UTC
3. 02:04:00 UTC
4. 04:33:33 UTC (12 mins pre-lockloss)
5. 04:45:47 UTC (1 second pre-lockloss)

Now the question becomes: was it another stage of “EX” or was it something else?

First, we can trend the other stages and see if there is any before-before-before lockloss behavior. (This becomes relevant later so I’ll leave this investigation for now).

Second, we can look at other fun and suspicious factors:

• One such “other” factor is the glitch that started it all - a big peak showing up on all the bands of the BLRMs. Going into all of them, we can actually see that the same type of pre-peak “kick” is there. Though this happens pretty much exactly when the lockloss occurs, at 4:45:48 UTC, implying that this was not the cause (Screenshot "BLRMSringup" 3). A note is that the kick that coincided with the lockloss is just a vertical line in the picture, but that is because it was much higher in magnitude compared to the preliminary kick, which again, was much higher in magnitude to all of the motion happening leading up to it. Potentially relevant here is that the 38-60Hz band sees this first (4:45:48 UTC) and the 10-20Hz band sees this last, about a second later (4:45:49 UTC). 
• Another “other” factor is that upon opening up this issue briefly with TJ today, we found that the OMC DCPD saw yesterday’s lockloss (alog 72852) kick issue first, so this is worth investigating aside from the discussion above.

This leads me to believe that whatever caused the glitch behavior may be related to the EX saturation, which begs the question: Have the recent EX saturations all prompted these BLRM all-bands glitches? Let’s find out.

Matching the EX saturation timestamps above:

Looking at the saturation 12 mins before, there was indeed a ring up, this time with 20-34 Hz band being first and highest in magnitude (Screenshot "EXSaturation4"). It’s worth saying that this rang up much less (I had to zoom in by many orders of magnitude to see the glitch behavior), which makes sense because it didn’t cause a lockloss but also tells us that these locklosses are particularly aggressive if:

1. When we do lose lock, we have to go all the way to MICH_FRINGES in order to align properly (and yesterday even that didn’t work).
2. Even the EX saturations that cause these glitches that don’t cause locklosses are orders of magnitudes lower than the ones that do.

Anyway, all EX saturations during this shift caused this behavior in the BLRMs screen. All of the non-lockloss causing ones were about 1*10⁸ times lower in magnitude than this one.

This all isn’t saying much other than that an EX ring-up in DARM will show up in DARM.

But, now we have confirmed that these glitches seem to be happening due to EX saturations so let’s find out if this is actually the case. So far, we know that a very bad EX saturation happens, the BLRMs screen lights up, and then we lose lock. This splits up our question yet again:

Does something entirely different (OMC perhaps) cause EX to saturate or is the saturation happening and caused within another EX stage? (Our “first” from before) We can use our scope archeology to find out more.

But sadly, not today because it’s close to 1AM now - though the investigation would continue as such:

1. Trend EX actuators and see when the “first kick” happens. Remember our first evidence of anything is 04:45:46.5 UTC in L3 but all at the same time.
   1. If one actuator is clearly first, this supports the claim that that specific actuator is the problem
   2. If none are first, then it is likely something else
2. Trend OMC DCPD channels in a similar manner to EX
3. Do “all of the above” for locklosses of a similar breed (yesterday’s would be a good one).
4. Continue exploring “first causes” among those locklosses if nothing else yields conclusive answers.

I will continue investigating this in tomorrow’s shift.

P.S. If my reasoning is circular or I’m missing something big, then burst my bubble as slowly as possible so I could maximize the learning (and minimize the embarrassment)

LOG:

None

Lockloss due to another of the same type of Glitch as yesterday's Lockloss Alog 72852 (and less than 20 minutes apart too). Attempting to re-lock automatically now while investigating the cause of the glitch.

IFO is in NLN as of 7:24 UTC (19:37 hr lock) and OBSERVING as of 23:12 UTC

Other:

IY violin modes 5 (and 6) have been going down steadily for the last 4 hrs since comissioning rang them up and a gain was applied to mode 5 (Screenshot). All violins are now under 10^-17 m/Hz^1/2 as of about 2:00 UTC.

Based on TJ's log 72857 about recent FC alignment drifts, I wanted to check out the situation with possible scattered light from the filter cavity since alignments can drift around (worth investigating more), and I wanted to check the CLF fiber polarization (this was basically fine).

Summary screenshot shows some measurements from today. I'm wondering if we might be closer than I realized to filter cavity backscatter. With an excitation that is ~5-10x greater than the ambient lsc control to FC2, it created scatter about 2x above darm at 100 Hz. Based on previous (higher-freq) measurements (LHO:68022) and estimates (LHO:67586), I had thought ambient LSC control was about 10-fold below DARM; this suggests we are within a factor of ~5? Though, there is a lot of uncertainty (on my end) of how hard we are driving FC2 given the suspension/loop roll-offs/etc, so I need to think more about the scaling between measured scatter and the excitation.

Strange line in FC-LSC error signal -- it seems to wander; I've seen it (or things like it) sometimes in other SQZ-related signals; but I haven't figured out where it comes from yet. It can be easily seen on SQZ summary pages (sqz > subsystems tab) since Derek and Iara helped us it up (thank you Detchar!!). I don't see it in CLF_ISS but sometimes in other error signals. Not clear to me that this is an issue if the peak is >100 Hz, but if it drifts to lower frequencies and this line is real/physical (not some random artifact), it could be problematic. The peak amplitude seems large enough that if it were in-band of the FC2 suspension and controls, it could plausibly get injected as real FC length noise and drive some measurable backscatter.

For the excitation -- I used the template from LHO:68022 but ran it to lower frequencies, in-band of FC-LSC. Compared to the FC error signal (specifcally H1:SQZ-FC_WFS_A_I_SUM_OUTPUT which is the input sensor to FC_LSC_DOF2_IN1), this DTT injection of 30,000 counts increased the integrated RMS of the in-loop error signal at 10 Hz by about 9-fold (= 3375 (w/excitation) / 387 (ambient), measured from dtt rms cursors). I injected into the fc-lsc loop at H1:SQZ-FC_LSC_DOF2_EXC, with various amplitudes (like 30k), and filter = butter("BandPass",4,10,300); this should then go to FC2_L for suspension feedback. I'm not sure that I'm using the best witness sensors for actual length noise driven in this excitation, but wasn't able to totally figure it out in time.

With this excitation going, I tried to walk the alignment to see if there was an alignment that minimizes backscatter, but I didn't figure this out in time. I tried to walk ZM2 with beam spot control off, and then set the QPD offsets where it landed. This was probably the wrong approach, since I wasn't able to then set the QPD offsets in time; maybe I should have walked the FC-QPD offsets with full ASC running at higher gain, since this loop is so slow. Might be worth trying this again for a bit; with the injection running, I wasn't sure if I was able to minimize scatter by walking ZM2 (p/y/ maybe psams?), but there were a couple directions in both pitch and yaw that looked promising. 

In the end, SDF diffs for ZM2 were accepted (ZM2 is not under asc-control), and I accepted the beam spot position change in the FC QPD Pitch offset (from 0.07 --> 0.08) while reverting the yaw change that I didn't figure out in time. I don't anticipate much overall change in the squeezer after these tests.

[Jenne, Gabriele, with thoughts and ideas from Elenna and Sheila]

The last few days, our sensitivity has been degrading a small amount, and Gabriele noted from a bruco that we're seeing increased MICH and SRCL coherence.  It hasn't even been a full 2 weeks since Gabriele and Elenna last tuned the MICH FF, so this is disappointing. Elenna has made the point in alog 72598 that the effectiveness of the MICH FF seems to be related to the actuation strength of the ETMX ESD.  We certainly see that the Veff of ETMX has been marching in a monatonic line for the last few months in Ibrahim's alog 72849. After roughly confirming that this makes sense, Gabriele and I took measurements in preparation for soon switching the LSC FF to use the ITMY PUM, just like LLO does, in hopes that makes us more immune to these gain changes.

Today, at Sheila's suggestion, I tried modifying the ETMX L3 DriveAlign L gain to counteract this actuation strength change.  (Fear not, I reverted the gain before commissioning ended, so our Observe segments do not have any change to any calibration.)  To check the effect of changing that drivealign vlaue, I looked both at the DARM open loop gain, as well as the coupling between MICH and DARM. 

• As-found, DARM OLG at 60 Hz was gain of 1.14 and 30.1 deg, when EX L3 drivealign L was at the nominal value of 184.65.  Corresponds to blue trace in first attachment.
• I adjusted the EX L3 drivealign L gain to get the DARM OLG back closer to its nominal value of 60 Hz.  After adjustment, the DARM OLG at 60 Hz is gain of 1.02, 29.8 deg, with EX drivealign 166.8. The MICHFF is quite bad though.  Ran a bb calib meas to check the calibration, although this is before Jeff reminded me that it would be useful to have the CAL-CDS_DARM_FE mimic of that drivealign gain match the actual gain: /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20230913T195421Z.xml, but H1:CAL-CS_DARM_FE_ETMX_L3_DRIVEALIGN_L2L_GAIN had not been fixed to match my updated drivealign gain.  Corresponds to green trace in first attachment.
• I modified the calibration mimic of the ETMX L3 drivealign gain, ran another broadband measurement, and indeed saw that this made the calibration better.  This indicated that much of the loss in sensitivity was due to the poor LSC FF.  The bb measurement for this is at /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20230913T200136Z.xml . 
• I realized that probably we didn't want the DARM OLG to be the 60 Hz value it was when we started O4, but rather we wanted it to match what it was on the day the LSC FF was last tuned (which is closer to 65 Hz or so, from this plot from alog 72422 from Louis).  So, I switched to using drivealign to minimize MICH-> DARM coupling, and got a drivealign gain of 182.5.  This puts DARM 60 Hz values at mag 1.14 and phase of 30.3 deg.  (So, basically the same as earlier today....Hmmm. Likely some measurement uncertainty there.)  Had a small amount of quiet time when H1:CAL-CS_DARM_FE_ETMX_L3_DRIVEALIGN_L2L_GAIN (the Calib matching value) was also 182.5, so that's the channel to check for a quiet time.  Corresponds to red in first attachment. 
• Reverted ETMX L3 drivealign L gain back to 184.65 (original value, so we haven't altered our calibration).

This all seemed to jive with the MICH FF effectiveness being related to ETMX ESD actuation strength.  So, rather than try to track that, we decided to work on changing over to use the ITMY PUM like LLO does.  I note that our Transition_from_ETMX guardian state uses ITMX, not ITMY, so it should be safe to have made changes to the ITMY settings.

• Used -1 on ITMY in LSCFF output matrix, for both MICH and SRCL.  I'm not sure why LLO uses a -2 there.  Made sure signal goes through L3 filter banks but not to the ITMY ESD, and that signal does not go up to L1 or M0 (so, we'll only be actuating on L2).
• Gabriele and I copied over the f^2-ish filter that is in the LLO ITMY L2 LOCK filter bank, which is used to increase the drive at high frequencies since we're going to be driving up one stage.
• Our first try at measuring coupling functions rang up some violin modes, so Gabriele copied over a violin bandstop from the ITMY angular L2 dofs.  Our nasty doublet of 508.35 Hz violin modes is on ITMY, so perhaps that's why they got rung up, even though LLO does not use a bandstop and we didn't have one for ETMX.
• Gabriele took measurements for updating both MICHFF and SRCLFF.
• After this, we reverted everything back to using our nominal ETMX LSC FF actuation, since we didn't have time for fitting and testing the new filters today.
• Gabriele is working on fitting the new filters for the ITM, and we will come back soon to trying them.  I am hopeful that this will alleviate much of our need to retune these regularly!
• We did save the new ITMY settings in the Observe SDF (There were 2 LOCK_L gains, and then two filters engaged in ITMY L2 LOCK L).  I did not think to save these in the safe.snap file, so they will likely get reverted, and cause an SDF diff for our next lock :/ .

Between 09/04 00:48 and 00:50UTC the TCSX CO2 laser lost lock and pushed us out of Observing(attachment1). There were two SDF diffs for TCS-ITMX_CO2_CHILLER_SERVO_GAIN and TCS-ITMX_CO2_PZT_SERVO_GAIN(attachment2), both of which resolved themselves as the laser locked back up. Currently these unlocks are happening every ~2.8ish days. Jeff and Camilla noted(72627) that the ITMX TCS laser head power has been declining over the course of the past 1.5 months, and the drops in the power output line up with every time that the TCSX laser loses lock(attachment3).

08/15 22:00UTC - TCSX Chiller Swap to spare -> ..sn813 (72220)

08/16 08:30UTC TCSX unlock

08/18 13:30UTC ''

08/23 08:42UTC ''

08/26 09:46UTC ''

08/29 08:27UTC ''

09/01 01:44UTC ''

09/04 00:48UTC ''

Naoki and I unmonitored  H1:SQZ-FIBR_SERVO_COMGAIN and H1:SQZ-FIBR_SERVO_FASTGAIN from syscssqz observe.snap. They have been regularly  taking us out of observing (72171) by changing when the TTFSS isn't really unlocking, see 71652. If the TTFSS really unlocks there will be other sdf diffs and the sqz guardians will unlock. 

We still plan to investigate this further tomorrow. We can monitor if it keeps happening using the channels.

First observed as a persistent mis-calibration in systematic error monitoring Pcal lines which measure PCAL / GDS-CALIB_STRAIN affecting both LLO and LHO, [LLO Link] [LHO Link], characterised by these measurements consistently disagreeing with the uncertainty envelope.
It us presently understood that this arises from bugs in the code producing the GDS FIR filters there exists a sizeable discrepancy, which Joseph Betzwieser is spear-heading a thorough investigation to correct,

I make a direct measurement of this systematic error by dividing CAL-DARM_ERR_DBL_DQ / GDS-CALIB_STRAIN , where the numerator is further corrected for kappa values of the sensing, cavity pole, and the 3 actuation stages (GDS does the same corrections internally). This gives a transfer function of the difference induced from errors in the GDS filters.

Attached in this aLog, and its sibling aLog in LLO, is this measurement in blue, the PCAL / GDS-CALIB_STRAIN measurement in orange, and the smoothed uncertainty correction vector in red. Attached also is a text file of this uncertainty correction for application in pyDARM to produce the final uncertainty, in the format of [Frequency, Real, Imaginary].