TITLE: 07/13 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY:  H1 has been down since the lockloss at 16:44 UTC earlier today.

Since midshift, Sheila was working with me on diagnosing locking issues. After locking DRMI, we turned on the DRMI ASC loops one-by-one until we found that turning on the SRC1_P&Y loops caused the buildups and camera image on AS AIR to get worse, so we've commented out using those loops in the ISC_DRMI Guardian. Locking DRMI after that and running DRMI ASC all at once made DRMI look much better and we didn't see the oscillations and saturations that I was seeing earlier. We also commented out the use of SRC1 in ENGAGE_AS_FOR_FULL_IFO since Sheila and Keita saw earlier in the day that it was not minimizing POP90 well at that point.

Once we made it past DRMI, we then encountered our next issue at CHECK_AS_SHUTTERS. The FAST_SHUTTER Guardian ran through its shutter test, then jumped to SHUTTER_FAILURE where it reported "Fast shutter failed tests! Do not power up!" and the AS port protection screen shows the error messages "Protection in fault" and "Power interlock is on". Unsure of what these errors meant, and having been unsuccessful in running the fast shutter test again manually, I called Fil for support. He had me power cycle the fast shutter driver chassis in the ISC rack by HAM6, which showed gibberish on the LCD screen (a previously seen issue), and the screen looked better and showed 255V DC charge. This didn't seem to fix the issue of the test failing, so I also power cycled the shutter logic chassis at the bottom of the rack, but this also didn't solve it.

Later, Keita discovered that the shutter test was failing because when the shutter closes, the signal on ASC-AS_B_DC_NSUM was not low enough, so Guardian didn't think it was closed even though it was. By just pressing the open and close buttons manually on the fast shutter screen, the shutter was able to run through its test and the Beckhoff errors cleared. Nothing was fundamentally changed here to fix this, aside from adding some log statements to the FAST_SHUTTER Guardian, but at least it sounds less like an electronics issue at this point. Keita commented that this perhaps isn't an alignment issue, unless some alignment in HAM6 changed recently.

Even though the issue with the fast shutter seems to be dealt with (but untested during a lock acquisition), there are still IFO locking issues that Sheila and I were unable to fully address this evening due to the fast shutter issues, including the 0.5Hz oscillation and lockloss when going to full power and the strange image on the AS AIR camera, which will need to be addressed in the morning. I'm leaving H1 DOWN for the night until more thorough investigations can be done.

Ryan S and Filiberto remotely are troubleshooting why the fast shutter isn't working now.  I just had a look at our last high power lockloss, from MAX power at around 6:45 local time.  In that lockloss the fast shutter did function and protect the AS port. 

We've had a strange looking AS air camera all afternoon, screenshot of the camera with DRMI locked is attached.  Hopefully the shutter issue is just the controller needing to be reset.  In the past when the shutter wires were in the beam path towards OM1, the AS camera image did look strange.

H1 has been unable to relock so far this shift. After running an initial alignment at the start of the evening, we were able to reach MAX_POWER fairly consistently before a 0.5Hz oscillation started in ASC MICH_P and also smaller in DHARD_P and some SRC_P which would cause a lockloss. After one of these locklosses, when relocking DRMI, there would be many SRM and SR2 saturation callouts after DRMI ASC turned on and DRMI would lose lock. I dropped back down to run another initial alignment, paying closer attention this time, and everything went smoothly aside from SRC alignment, where no matter what SRM and SR2 moves I made, we could not acquire SRY. Trending OSEMs, optics in the SRC are in generally the same place as they are when locking normally, so I'm unsure at this point where the misalignment is (and I'm hesitant to move SR3 at this stage). The AS AIR camera has looked quite bad the whole time as well.

Starting the locking sequence to see what things look like now.

TITLE: 07/12 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Currently finishing up with a IA to align SRM.

Lock1:

• 15:32 to 16:01 UTC dropped Observing to do some range checks alog79069
• 16:44 UTC lockloss alog79073

Lock2:

• 18:07 UTC lost it at INJECT_SQUEEZING, we were getting shaken by an earthquake at the time, it looks like MICH rang up?
• We kept losing it at PREP_DC_READOUT, while the OMC was trying to lock, at different points of the OMC locking sequence. The ASC signals go crazy, maybe the SRM alignment isn't good enough?
  • Tried turning off the DARM offset once we got to PREP_DC
  • Sitting in SOFT_LOOPS we turned off ASC and adjusted SRM, lowered the DARM OFFSET, then thesholds were lowered like OMC-DCPD_SUM_OUTPUT from 10 to 7 which got it to lock alog79082
  • 22:49 UTC lockloss at TRANSITION_FROM_ETMX
  • AS AIR looked much worse now, after losing it at DRMI we dropped to start an initial alignment at 23:12 UTC, skipping green arms
  • We had to align SRM by hand as it was too badly misaligned for the GRD
• 20:15 UTC erroneous signal from the HWUT seismometer on picket-fence

LOG:                                                                                                           

TITLE: 07/12 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 8mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY: Sounds like H1 has been having some locking troubles today, which I'll be aware of as we relock. Currently H1 is in the process of locking PRMI.

• Wind and microseism low
• Just dipped below 100degF on-site
• All other systems nominal

Ibrahim

Attached are the new set of BBSS TF measurement screenshots with slightly adjusted offsets applied. These offsets account for the Pitch drift we're seeing in F1 and for any other non-centered OSEM behavior due to temperature sagging. These are also the first TFs since moving the EQ stops back and re-confirming that nothing is rubbing or touching. See alog 79079 for the summary of what was done right before these were taken. Screenshot 1 shows the new offsets that are currently in. Screenshot 2 shows the old offsets that were from the OLV script - if the temperature stops varying to this degree, it may be worth reverting these.

The red trace are results as of today (07-12-2024)

The green results are results from earlier this week 07-10-2024, which were presented in alog 79032 had the issues mentioned in alog 79042 and alog 79036.

The blue results are the results from 01-05-2024, which we were using as a reference for expected coherence.

The new measurements are saved under a new file in the Data folder: /ligo/svncommon/SusSVN/sus/trunk/BBSS/X1/BS/SAGM1/Data/2024-07-12_1400_tfs. I've svn commited this Data file so it should be visible both in the X1 and normal workstations.

Interpretations:

The new transfer functions look much better than the ones from two days ago, meaning that moving the EQ stops back and adjusting for the Temp Driven sag was extremely fruitful. We can clearly see cleaner results, with almost eliminated noise in the previously problematic 1.5-6hz region and the results are markedly more coherent, comprable to the pre-rebuild state from 01-05. Now we can begin interpreting these results seriously with respect to the dynamic model, which I will leave for next week.

Next Steps (Next Week):

• Produce the BBSS Model-Result overlay comparison and look for cross coupling, unexpected resonances and other oddities.
• Make decisions about how to change the mechanical build or the parametric models for the BBSS
• Fix the M3 roll
• Discuss, then fix the M1 Pitch issue if it is still there an apparent
• If there's time, recenter OSEMs mechanically and set offsets back to their OLV values (since weather is going to be reliably around 100F all next week at the same times)
• Place a thermometer/temperature sensor in the staging building to track temp drift more accurately

Keita, Sheila, Ryan C, Ibrahim

Two previous alogs about this problem: 78332 78258

We tried to open the SRC1 loop and align SRM manually, we were able to decrease POP90 that way but when we again turned on the DARM offset we saw the same drop in powers that preceeds locklosses. (see screenshot, when things recover we manually turned off the DARM offset.)

We were able to move past this for today by lowering the DARM offset used from 9e-5 to 6e-5 in ISC_LOCK DARM_OFFSET.  Then to get the OMC guardian to lock we had to lower the threshold on the peak height in OMC_LOCK FIND_CARRIER line 328 to 7.  Then we had to lower the check in OMC_LOCKED from 10 to 7, OMC_LOCK line 512.

We manually reengaged the SRC1 ASC loops, and are now powering up.

J. Kissel, L. Dartez, F. Llamas

As Louis and I were wracking our brains for all the changes to the IFO that might have impacted the IFO's calibration in 2024 in order to update / maintain T2300297, we realized that Francisco's work -- changing the PCAL "inner beam" spot position of PCALX to explore and validate models of systematic errors the PCAL system -- would potentially impact the measurements that are constantly running on PCALX -- the so-called "roaming lines" which is a really long duration, repeating sweep of the sensing function between 1 and 5 kHz. Because this long-duration sweep relies on nominal low noise data, the speed at which the sweep frequency advances is dependent on the IFO's duty cycle, and is thus irregular; sometimes complete in 6 days, other times taking 10 days or more.

To facilitate this research, I plot the trend of the PCALX Oscillator Frequency that's used to define the sweep frequency over time and overlay Francisco, et. al's PCALX spot moves.

And just so you can join us in our worry -- while Francisco's work (see latest in LHO:78964) is showing 0.03%-level changes in the comparison between PCALX and PCALY at 283.91 and 284.01 Hz respectively (via plots like this), the roaming lines measurement is only PCALX and the sweep is from 1-5 kHz, where it's been known for a long time that PCAL spot position changes of ~5 [mm] at RX, or ~2.5 [mm] scale at the Test Mass, can cause 1-to-10%-level errors in reported displacement values at 3, 4, 5 kHz (largest at highest frequencies) -- see, for example, figures 2.30 through 2.35 in T2300381.

Stay tuned!

This aLOG does not suggest we've done the leg work to quantify that we've seen the a problem, but the start of doing such a study.

Ibrahim

Today, I went into the staging building to see if there were any visible or otherwise apparent reasons for our L, P, Y M1 Transfer Function incoherence.

Here's what I did:

• Backed off all EQ stops, none were too close to touching, but the temp drif sag was higher yesterday than today so I just made sure they were all the way back.
• Re-did the foil, double triple checking nothing was touching, rubbing or in contact
• Checked every BOSEM Flag

Here's what I found:

• M1 is visibly pitched - this is corroborated by the 8-day drift of the F1 OSEM counts (as opposed to the daily sag and rebound of the LF-RT OSEMs. I've copied over the F1 section of the drift screenshot from yesterday's alog 79042 on this topic, since it seems the pitch instability suggestion from that alog was the right hypotehesis (F1 is used in P and is above the M1 center of mass). Image 1 shoes this clearly.
  • This could either be differential sagging of the Top Stage blades causing it to pitch in one direction over another.
  • It could also be a pitch hysteresis issue we had with the Top Mass during assembly - this was fixed by setting the heights of the blades to a different nominal position.
  • It could also be that the the Top Mass blades are differentially sagging too, causing more pull on one side than the other.
  • Or a combo, or something else.
• The M1 flags are surprisingly not visibly sagging that much - I guess this makes sense since 25 microns (our measured offset from centered position last week) is only 0.025mm.
• The M2 flags are much more visibly sagging, which means that the Top Mass blades are more affected by the temperature drift. This sort of corroborates the idea that the Top Mass blade sag is affecting the pitch moreso than the top stage blades or a previous ptich hysteresis blade height problem. Image 2 shoes one of the 4 M2 BOSEMs substantially lower than their centered positon.

Here's what I'm going to to:

I'm going to set an offset in order to "fake" center the BOSEMs and then take data but I expect that with a Pitch instability the results won't be too coherent either. We'll see. We're at least now ensured that there won't be any rubbing or contact.

What I'm going to do later:

We will fix the percieveable Roll issue on Monday when there are more hands on deck. I'll also consider re-centering the OSEMs at noon and measuring the temp at that time so we have some sort of informed reference.

Laser Status:
    NPRO output power is 1.821W (nominal ~2W)
    AMP1 output power is 65.27W (nominal ~70W)
    AMP2 output power is 138.5W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 2 days, 23 hr 37 minutes
    Reflected power = 17.94W
    Transmitted power = 107.8W
    PowerSum = 125.8W

FSS:
    It has been locked for 0 days 0 hr and 49 min
    TPD[V] = 0.7949V

ISS:
    The diffracted power is around 3.2%
    Last saturation event was 0 days 0 hours and 49 minutes ago

Possible Issues:
    ISS diffracted power is high, seems like its been high since early morning this past maintenance.

16:44 UTC lockloss. PRCL was oscillating at about ~3.6 Hz.

Ryan C, Sheila D

15:32 to 16:01 UTC we dropped Observing to do some range checks/investigations.

Sheila and I did some range checks following the wiki. Running the coherence check showed high coherence with CHARD and the SQZer BLRMs did not look as good as previous locks, particularly in the 10-20, 20-34, and 60-100 HZ bandwidths. So we decided to drop out of observing since LLO was down to run the SQZ alignment and angle scans and I concurrently ran the A2L_min script (TJs alog78552). After these finished we also took 5 minutes of NO_SQZ time starting at 15:49 UTC which showed that the extra noise is not coming from the SQZer. We gained a few Mpcs from the SQZ scan and A2L_min script.

Coherence comparison before and after SQZ and A2L checks

We were only about 2 and half hours into lock when I did this test due to our earthquake lockloss this morning.

I ran the

python auto_darm_offset_step.py

in /ligo/gitcommon/labutils/darm_offset_step

Starting at GPS 1404768828

See attached image.

Analysis to follow.

Returned DARM offset H1:OMC-READOUT_X0_OFFSET to 10.941038 (nominal) at 2024 Jul 11 21:47:58 UTC (GPS 1404769696)

DARM offset moves recorded to 
data/darm_offset_steps_2024_Jul_11_21_33_30_UTC.txt

We're running a patched version of simuLines during the next calibration measurement run. The patch (attached) was provided by Erik to try and get around what we think are awg issues introduce (or exacerbated) by the recent awg server updates (mentioned in LHO:78757).

Operators: there's is nothing special to do. just follow the normal routine as I applied the patch changes in place. Depending on the results of this test, I will either roll them back or work with Vlad to make them permanent (at least for LHO).

We cannot make a reasonable assessment of energy deposited in HAM6 when we had the pressure spikes (the spikes themselves are reported in alogs 78346, 78310 and 78323, Sheila's analysis is in alog 78432), or even during regular lock losses.

This is because all of the relevant sensors saturate badly, and the ASC-AS_C is the worst in this respect because of heavy whitening. This happens each and every time the lock is lost. This is our limitation in configuration. I made a temporary change to partly mitigate this in a hope that we might obtain useful knowledge for regular lock losses (but I'm not entirely hopeful), which will be explained later.

Anyway, look at the 1st attachment, which is the trend at around the pressure spike incident at 10W (other spikes were at 60W, so this is the mildest of all). You cannot see the pressure spike because it takes some time for the puffs of gass molecules to reach the pirani.

Important points to take:

• Marker shows when the Fast Shutter was actually fired, which is witnessed by GS13 (bottom left).
• ASC-AS_C cannot be relied on. It started saturating badly even before the FS fired (look at INMONs, 3rd from the top, left), and it kept saturating for a long time.
• OMC QPDs cannot be relied on. They started saturating at about the same time as AS_C though they seem to have stopped saturating earlier  (2nd from the bottom, left).
• AS_B and AS_C cannot be relied on. They're much better than AS_C and OMC QPDs, they started railing some milliseconds earlier than the FS, but they railed (right middle for SUM, one below that for INMONs).
  • But the beam was NOT clipped by the Fast Shutter after it closed as the power dropped to zero, we can see that after WFS stopped saturating.

This is understandable. Look at the second attachment for a very rough power budget and electronics description of all of these sensors. QPDs (AS_C  and OMC QPDs) have 1kOhm raw transimpedance, 0.4:40 whitening that is not switchable on top of two stages of 1:10 that are switchable. WFSs (AS_A and AS_B) have 0.5k transimpedance with a factor of 10 gain that is switchable, and they don't have whitening.

This happens with regular lock losses, and even  with 2W RF lock losses (third attachment), so it's hard to make a good assessment of the power deposited for anything. At the moment, we have to accept that we don't know.

We can use AS_B or AS_A data even though they're railed and make the lower bound of the power, thus energy. That's what I'll do later.

(Added later)

After TJ locked the IFO, we saw strange noise bump ffrom ~20 to ~80 or so Hz. Since nobody had any idea, and since my ASC SUM connection to the PEM rack is an analog connection from the ISC rack that also has the DCPD interface chassis, I ran to the LVEA and disconnected that.

Seems like that wasn't it (it didn't get any better right after the disconnection), but I'm leaving it disconnected for now. I'll connect it back when I can.

J. Kissel, E. von Reis, D. Sigg

Circa March 2023, Daniel had installed some never-used first attempts at further filtering the high frequency noise present in the 524 kHz OMC DCPD channels (never aLOGged because it was never used, but I call them out after I found the work in LHO:68098). 

Now, because I'd like to characterize the existing, in-use, digital AA filtering, running into some unknown noise (LHO:78516), and hoping to install 2 to 4 more parallel filter banks that would also be quite full of filters (LHO:78956), there is worry that there won't be enough computation time in the 524 kHz system.

Remember, that "the 524 kHz system" is actually a modified "standard" 65 kHz system, which is reading out 8 samples from the 524 kHz ADC each 65 kHz clock cycle and computing everything at 65 kHz. Thus, in principle, the max turn around time is
    1 / (2^16 Hz) = (1 / 65536) [sec] = 1.5258789e-5 [sec] = 15.3 [usec]

However, we *think* the practical limit is somewhat less that this. I don't think I understand what those limitations are enough to say definitively and/or to quote a limit quantitatively, but I think they're related to "the copy of the OMC DCPD channels which are demodulated at high frequency to create PI channels that are shipped to the end station -- in other words, the IPC sending demands a bit of computational time, and if there isn't enough turnaround time left in the OP to write to the IPC network, then the end-station SUS PI models throw an IPC timing error."

Anyways -- this morning, I looked at the 524 kHz system's computation time as is before doing anything (via the channel H1:FEC-179_CPU_METER), and it's sitting at 9 [usec] (out of the ideal 15 [usec]), occasionally popping up to 10 [usec].

But -- this led me to remember that -- regardless of whether the filter is turned ON -- the front-end computes the output of the filter -- sucking up computation time.
So, I've removed these unused prototype notch filters from the DCPD A0 and B0 filter banks.
In addition, I've also removed the old "V2A" filter from a previous version of the digital compensation for the OMC DCPD transimpedance amplifier response.

Removing the notch filters drops the computation time from "9 [sec] occasionally bopping up to 10 [usec]."
See attached time series of the CPU meter.

These filters are, of course, available in the filter_archive, under the latest previous archived file before today's work:
    /opt/rtcds/lho/h1/chans/filter_archive/h1iopomc0/
        H1IOPOMC0_1401558760.txt
but for ease of use, I copy them here.

FM3 :: Notches1
    ellip("BandStop",3,0.5,30,12800,13200)
    notch(10216,50,30)
    ellip("BandStop",3,0.5,30,10380,10465)
    ellip("BandStop",3,0.5,30,12900,13100)

FM5 :: Notches2
    ellip("BandStop",3,0.5,30,8100,8200)
    notch(9101,50,30)notch(9337,200,30)
    notch(9463,50,20)
    ellip("BandStop",3,0.5,30,9750,9950)

FM8 :: Notches3
    ellip("BandStop",5,0.5,40,14384,18384)
    ellip("BandStop",5,0.5,40,30768,34768)

FM6 :: V2A
    zpk([5.699+i*22.223;5.699-i*22.223;32.73],[2.549;2.117;6.555],0.00501187,"n")gain(0.971763)
I've also posted a plot of the magnitude of these notch filters -- mostly just to demonstrate how many second order sections that these filters had -- sucking up computation time.