J. Kissel, O. Patane

A follow-up from yesterday's work on installing the infrastructure of the upgrades to the ETM and TMS watchdog systems, in this aLOG I cover with what I've filled out the infrastructure in order to obtain the calibrated BLRMS that forms the trigger signal for the user watchdog.

Remember, any sensible BLRMS system should 
    (1) Take a signal, and filter with with a (frequency) band-limiting filter, then
    (2) Take the square, then the average, then square root, i.e. the RMS, then
    (3) Low-pass the RMS signal, since only the "DC" portion of the RMS has interesting frequency content.

As a bonus, if your signal is not calibrated, then you can add 
    (0) Take the input to the band limiting filter, and calibrate it
(and through the power of linear algebra, it doesn't really matter whether you band-limit first and *then* calibrate)

This screenshot shows the watchdog overview screen conveying this BLRMS system.

Here're the details of the BANDLIM and RMSLP filters for each of the above steps:
    (0) H1:SUS-ETMX_??_WD_OSEMAC_BANDLIM_?? 
             FM6 ("10:0.4") and FM10 ("to_um") are exact copies of the calibration filters that are, and have "always been" in the OSEMINF banks. These are highlighted in the first attachment in yellow. 
             FM6  :: ("10:0.4") :: zpk([10],[0.4],1,"n")    :: inverting the frequency response of the OSEM satellite amp frequency response
      FM10 :: ("to_um")  :: zpk([],[],0.0233333,"n") :: converting [ADC counts] into [um] assuming an ideal OSEM which has a response of 95 [uA / mm], differentially readout with 242 kOhm transimpednance and digitized with a 2^16 / 40 [ct / V] ADC.

    In addition, I also copied over the GAIN from the OSEMINF banks and copied these over as well such that each OSEM trigger signal remains "normalized" to an ideal 95 [uA / mm] OSEM. These are highlighted in dark green in the first attachment.


    (1) H1:SUS-ETMX_??_WD_OSEMAC_BANDLIM_?? 
             FM1  :: ("acBandLim") :: zpk([0;8192;-8192],[0.1;9.99999;9.99999],10.1002,"n") :: 0.1 to 10 Hz band-pass

    (2) This is a major part of the upgrade -- the front-end code that does the RMS was changed from the nonsense "cdsRms" block (see LHO:1265) to a "cdsTrueRMS" block (see LHO:19658)

    (3) H1:SUS-ETMX_??_WD_OSEMAC_RMSLP_??
             FM1  :: ("10secLP") :: butter("LowPass",4,0.1) ::  4th order butterworth filter with a corner frequency at 0.1 Hz, i.e. a 10 second Low Pass.
        This is highlighted in magneta in the second attachment. These are direct copies from other newer suspension models that had this infrastructure in place.
           

I've committed the filter files to the userapps repo,
    /opt/rtcds/userapps/release/sus/h1/filterfiles/
        H1SUSETMX.txt
        H1SUSETMY.txt
        H1SUSETMXPI.txt
        H1SUSETMYPI.txt
        H1SUSTMSX.txt
        H1SUSTMSY.txt
are all committed as of rev 27217.

All of these settings were captured in each model's safe.snap. I've not yet accepted them in the OBSERVE.snaps.

Sheila, Nutsinee

A follow up from alog76294 we want to properly compare OMC REFL when we inject squeezing and without squeezing. The noise goes up by 1.1e-5 mW (I believe this is what the channel is calibrated to) udinrg NO SQZ time compared to SQZ time.

Now including the trace from O4a and more no SQZ traces after a better OMC alignment.

After looking at more traces from SQZ and NO SQZ time it seems OMC REFL can drift up and down on its own. I don't think injecting squeezing is doing anything to OMC REFL.

The no sqz time used was 13/03/2024 15:29:00 UTC (no sqz time ends 15:51:00 UTC)

sqz time (1) used was 13/03/2024 16:00:00 UTC

sqz time (2) used was 13/03/2024 09:00:00 UTC

Post OMC alignment

no sqz time (1)  13/03/2024 21:25:59 UTC

no sqz time (2)  13/03/2024 21:36:27 UTC

O4a time 23/10/2023 15:00:00 UTC (~164Mpc)

Also attached a psd of OMC trans and a coherence between CLF and OMC RF3.

Comparing the range BLRMs from December (cold OM2 observing time used for noise budget) to now (Monday night - Tuesday 12th 10:00 UTC), BLRMs are better <60Hz and >1kHz.  BLRMS are worse in the 100-450Hz region and a little worse in the regions around this. Plot attached.

Summary pages range ASD plot shows we're now worse 35 to 500Hz: see comparison plot attached between for Dec vs Now.

J. Kissel

Checking the last day's worth of BLRMS values of the Transmission Monitor and Telescope Suspensions (TMTS) TMSX and TMSY M1 or top-mass OSEMs to compare against the arbitrarily chosen threshold of 25 [um_RMS] in the 0.1 - 10 Hz band -- typical values are ~0.02 [um_RMS] and maximum excursions after lock losses are 0.3 [um_RMS].

We could probably safely decrease the WD threshold to something like 1 [um_RMS].

See attached ~24 hour trend of the TMSX and TMSY BLRMS channels vs. the ISC_LOCK guardian state.

E. Capote, L. Dartez, J. Kissel, R. Short

As you know, Oli and I installed a revamped watchdog system on the ETMs yesterday (LHO:76305). 
In doing so, we arbitrarily set the watchdog threshold for all stages at 25 [um_RMS] (over the 0.1 to 10 Hz band).

Elenna and Louis reported yesterday evening that we had *one* WD trip of the PUM stage after the lock-loss of the fourth of four successful locks (see the last phrase of the last sentence in LHO:76315). Further, they were concerned because, apparently, "the watchdog reset itself."

To the first point -- that there even was a watchdog trip -- the first attachment shows
    - The ISC_LOCK guardian state (H1:GRD-ISC_LOCK_STATE_N), remeber 600 is NOMINAL_LOW_NOISE
    - The 4 stages of OSEM watchdog "is it tripped?" channels (H1:SUS-ETMX_??_WDMON_STATE, with ?? = M0, R0, L1, and L2 for Main Chain and Reaction Chain top stages, the UIM and PUM stages)
    - The 4 stages of OSEM watchdog "is the trigger actively suggesting it should be tripped?" channels (H1:SUS-ETMX_??_WDMON_CURRENTTRIG)
    - The PUM (L2) stage BLRMS trigger signals themselves, now for the first time in physical units! (H1:SUS-ETMX_L2_WD_OSEMAC_RMSLP_??_OUT16)

One can see that lock losses are typically kicking the PUM, up to 10-20 [um_RMS], but otherwise, the quiescent BLRMS is around 0.5 [um_RMS].

To the second point -- we're still investigating, but the second attachment shows that the PUM WD is reset (H1:SUS-ETMX_L2_WDMON_STATE goes from 2 to 1) at 2024-03-13 06:44:32 UTC.

In general, the guardian systems has been build on the axiom of "GUARDIAN SHALL NOT TOUCH WATCHDOGS," so we're quite suspicious of that this was not human-related. The human action that seems to be coincident with the reset on *this* lock acquisition attempt, as opposed to the three attempts immediately prior is that the ISC_LOCK guardian was taken to "INIT." It's not a smoking gun, but it's at least a bread crumb. We'll keep looking.

Finally, the third attachment shows an overall assessment of typical maximum values of the BLRMS for all stages. Here's a summary of the assessment in tabular form:

Stage        Quiescent Average Value        Maximum Transient Value
                    [um_RMS]                     [um_RMS]
M0                    0.35                         6.35
L1                    0.32                        18.34
L2                    0.44                        29.02

(where out of laziness and worry of too crowded a plot, I assume R0 has the same behavior.)

In the mean time, the watchdog is working exactly as expected, and let's bump up the PUM WD threshold to 30 [um_RMS].

Using the low frequency line injections from this morning, we can estimate if the OMC alignment at the time was good. Note that at the time of the test, not all OMC QPD offsets were on.

The first attached plot shows a time series of the OMC QPD signals, and BLRMS in a few bands, computed on OMC-DCPD_SUM. The most interesting is the 410-411 Hz band, which track the amplitude of a DARM calibration line, the one used to compute the optical gain KAPPA_C. We see a clear large modulation of the optical gain with the OMC alignment, and the zero position for all QPD is not the one giving the highest KAPPA_C.

The lines were injected at slighlty different frequencies to map as well as possible the entire 4d space of OMC QPD A and B PIT and YAW. The second plot shows the BLRMS at the DARM calibration line as a function of each QPD vavlue, so four slices in this 4d space. The third plot shows two-dimensional slices, but this is somehow harder to interpreter.

Looking at the second plot, it looks like the optical gain would be improved by moving the QPD values to

QPD_A_PIT 0.1
QPD_A_YAW 0.1
QPD_B_PIT 0.05
QPD_B_YAW -0.07

to maximize the optical gain. Iterative adjustments might be needed. Also, we could inject lines ona single DOF at a time to finetune the offsets

Jeff K, Dave B, Oli P

All of the optics (except OFIS, OPOS, and HXDS) were originally built with a DACKILL that will only trip if all of that suspension's Watchdogs trip. This has resulted in some problems (74545, 74622). Because of this, there has been a push to update the logic for each suspension to have an OR instead of an AND, so that the IOP DACKILL trips as soon as any one of the stages trips.
The Watchdogs for these suspensions also rely on an outdated BLRMS to figure out if the stages are saturating, and people have been wanting this to be updated since 2017(FRS9392). This update to the watchdogs would remove the need for a USER DACKILL, so now is a perfect opportunity to do both at the same time! Suspension models created after 2019 (OFIS, OPOS, HXDS, and the new HXTS) were originally made with the updated watchdogs and without the USER DACKILL, and we haven't had any issues. The changes that we made line up with what was done for these previously changed models.
Today we updated the models for the ETMs and TMSs(76305), and we will gradually be updating all of the models.
Changes made:
QUAD MASTER and TMTS Master sus models (QUAD_MASTER.mdl, TMTS_MASTER.mdl):
    - Top level: Removed USER DACKILL and WD block output flags(attachment1)
    - Inside each block with watchdogs: removed the watchdog flags that connect the WD RMS to the DACKILL(attachment2)
    - Inside each WD block within stage blocks: swapped the old WD BLRMS block with the better BLRMSLP block(attachment3)
Individal sus models (h1susetmx.mdl, h1susetmy.mdl, h1sustmsx.mdl, h1sustmsy.mdl):
    - Removed top level WD_2_ISI output flags and connection of WD_2_ISI out to ISI model(attachment4)
Individual sus pi models(h1susetmxpi.mdl, h1susetmypi.mdl):
    - Found out during installation of these updated models that a WD value that we had removed in the individual models went out to the pi models. The sus pi models were updated to include constants as inputs in lieu of the removed WD values(attachment5)
ISI QUAD models (h1isietmx.mdl, h1isietmy.mdl):
    - Since the USER DACKILL connects to dolphin, we also changed the ISI QUAD inputs that feed into SUSWD_2_PAYLOAD from ground to both be 0 for WDMON_STATE_DOLPHIN and WDMON_STATE_DOLPHIN_ERR respectively(WDMON_STATE_DOLPHIN_ERR needs to be 1 because it later gets inverted)(attachment6)
Changes to medms (ETMs,TMSs adl files):
    - Removed the IOP DACKILL button on suspension medm and TO ISI arrow(attachment7)
    - Inside the watchdog subscreens: added in the filter banks for the low pass and removed the diagrams showing the USER DACKILL logic(attachment8).
All changes have been committed to svn.

No SQZ time taken this morning 15:29UTC to 15:51UTC. There was small glitches at 15:35:37 and 15:41:53. Best stretch of time is 15:42:00 to 15:51:00. Took no sqz time with hot OM2 in 74834 / 74640, Cold OM2 Dec 20th 18:00- 18:18UTC 74935.

Followed Naoki's 70642 instructions. Pushed to aligoNB git. There was a small EQ passing through just before I started this. Had ~3dB of constant SQZ.

Last ran and analyzes aligoNB injections with hot and cold OM2 in December 74943 / 74788. We haven't measured coupling of frequency and intensity noise since August 2023: 72140.

I was able to request and transition to the new DARM state without issue. ETMX still saw a pretty large kick. I'll have to circle back re: how large compared to previous attempts. We will also want to take a look at any effects moving the integrator on L2 LOCK L had.

Note: We do get a few SUS_PI warnings shortly after transitioning to this state.

Quiet time to monitor for non-stationarity:
Start GPS: 1394344327
Stop GPS: 1394346169

Turned off cal lines at GPS 1394346770.39

Elenna started a Bruco after the cal lines were turned off.  here is a screenshot of DARM in the new configuration. 20-90Hz looks high and below 15Hz looks low. It's hard to tell how much of this is real until CAL-CS is calibrated & that calibration is propagated to the DTT template.

I started an L2 LOCK IN1/IN2 injection using noise recorder at 1394347034.426. We used a tuned broadband measurement that Craig and I put together. Apparently it was too strong because we lost lock from this injection. It also tripped EX. I requested DOWN and the EX trip alarm reset. 

Sheila, Camilla, Jennie W, Keita remote

Stefan and Daniel suspect that our excess noise around 100 Hz might be due to intensity noise, and we did have a large shift in alignment of the beam transmitted through IM4 (76291) 76241.  I moved pico 1, first to center the beam on the ISS QPD, which made the power on the ISS array PDs drop, and didn't improve the spectrum of the ISS Sum inner or outer channels (we did this with the loop open).  We reverted this, then looked at the QPD position in O4a when we had 60W input power, went to 60W input power and pico'd to bring the beam to the same location as O4a.  This also didn't improve the spectrum, so we brought the pico back to where we started.

Naoki, Vicky, Nutsinee, Matt

We recovered the FDS and got 4.5dB squeezing in IFO with 40 times more CLF power as shown in the attached figure. 

First we restored the ZM1/2/3 and FC1/2 OSEM positions to the values in O4a. Then we found the FC green trans in camera and PD. We aligned FC1 and FC2 and successfully locked the FC green. We went to SQZT8 and centered the green camera and green QPD.

Since we aligned the green QPD, the green QPD offset value is not valid anymore so we removed the beam spot control from FC ASC. We made a flag for the beam spot control in sqzparams and it is set to False now. We also set the ADF servo flag to False since the ADF demod phase might not be correct. After we figure out the optimal green QPD offset and ADF demod phase, we should revert them.

We reduced the FC IR gain from -3.5 to -0.1 and reduced the FC ASC gain from 0.1 to 0.005.

We have not done any PSAMS scan so we will do it next week.