WP11492 SUSPROC PI PLL model changes

Vladimir, Naoki, Erik, Dave

A new h1susprocpi model was installed. A DAQ restart was required

WP11485 h1hwsmsr disk replacement

Jonathan.

The failed 2TB HDD disk which is part of the /data raid was replaced with a 2TB SSD drive. At time of writing it is 70% done with the new disk rebuild.

Jonathan also cloned the boot disk. I verified that /data is being backed up to LDAS on a daily basis at 5am.

WP11478 Add HOFT and NOLINES H1 effective range to EPICS and DAQ.

Jonathan, Dave:

On Keith's suggestion, the HOFT and NOLINES versions of the H1 effective range were added to the dmt2epics IOC on h1fescript0. The MPC and GPS channels were also added to the H1EPICS_DMT.ini for inclusion into the DAQ. A DAQ restart was required.

WP11488 Deactivate opslogin

Jonathan

The old opslogin machine (not to be confused with opslogin0) was powered down.

WP11479 Add zotvac0 epics monitor channels to DAQ

Dave:

H1EPICS_CDSMON was modified to add zotvac0's channels. A DAQ restart was required.

DAQ Restart

Jonathan, Dave:

The DAQ was restarted for the above changes. This was a very messy restart.

0-leg was restarted. h1gds0 needed a second restart to sync its channel list.

EDC on h1susauxb123 was restarted

8 minutes later fw0 spontaneously restarted itself. At this point h1susauxb123 front end locked up, all models and EDC crashed.

Jonathan connected a local console to h1susauxb123, but there was no errors printed, the keyboard was unresponsive. h1susauxb123 was rebooted.

After the EDC came back online, the DAQ 1-leg was restarted.

h1gds1 needed a second restart to sync up its disks.

Here's a BruCo repot for GDS-CALIB_STRAIN_NOLINES from last night lock: https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_1382152121_STRAIN/ 

Some highlights:

• CHARD_Y is likely the leading limiting factor between 10 and 20 Hz, while DHARD_Y is the main contributor to residual noise below 8 Hz (to be confirmed with another BruCo scan on LSC-DARM_IN1
  • maybe it's time to retune / recheck the A2L gains
• SRCL is also contributing to a good fraction of the noise between 10 and 20 Hz.
  • maybe we can try an iterative retuning of the SRCL FF
• MICH is also not too far between 10 and 50 Hz:
  • time to do an iterative retuning of MICH FF
• PRCL shows (surprisingly?) some coherence between 24 and 50 Hz: PRCL could explain the peaks and structures between 24 and 40 Hz
  • maybe coupling throuhg MICH? Or is it time to put back in operation a PRCL FF?

Large frequency bands are completely devoid of coherence.

Tue Oct 24 10:05:21 2023 INFO: Fill completed in 5min 18secs

Summary - Jim fixed it with a little shake.

We cycled through the various ISI state controls. The 1Hz line would start appearing when stage 1 is isolated. In any states below this point, we can clearly see the difference in the 'sensor response' of the local H1 L4C wrt T240 or CPS. While Jim was taking an olg measurement, the ISI tripped, which seem to have changed the response of the H1 L4C back to the nominal response, see the first pdf attached, showing the L4C/T240 local TF, before (p1) vs after (p2) the ISI trip. The 2nd pdf attached shows the ISI spectra in the isolated state before (p1) vs after (p2) the shake, no other changes.
We've had sticky L4Cs in the past and solved it in a similar way (see alog 38939), but the symptoms were much more obvious than a slight change in resonant frequency as we are seeing here.

Timeline of tests :

16:04 - 16:12 UTC ETMX ISI/HEPI OFFLINE
16:15 - 16:30 UTC HEPI ON / ETMX ISI OFFLINE
16:31 - 16:37 UTC HEPI ON /ETMX DAMPED
16:38 - 16:44 UTC HEPI ON / ETMX ST1 ISO - We can see the giant ~1Hz line
16:45 - 16:48 UTC HEPI ON/ ETMX ST1 ISO - H1 L4C gain 0.5 - sym filter off - ~1Hz line goes down
16:50 - 16:52 UTC HEPION/ETMX ST1/ST2 ISO - H1 L4C gain 0.5 - sym filter off
16:55 - 17:07 UTC Back to nominal
17:25 - Jim changing stage 1 Rz blend to noL4C
17:38 - Jim changing stage 1 Y blend to noL4C
18:00 - Jim measuring Rz olgf

Ref alog 73625

I replaced two disks on the h1hwsmsr system.

 * cloned a failing boot disk
 * replaced a failed raid drive that holds the image repository

I used clonezilla to replicate the boot disk.  Then after booting into the newly cloned disk I replaced the failed raid disk.  At this point in time the data is being automatically replicated onto the new disk, it should take a few hours.

I prepared a new filter module with resonant gains at 2.8 Hz and 3.4 Hz. This can be tried tomorrow (on during some commissioning time) to reduce the DARM RMS and see if it helps the non-stationary noise.

The new FM is loaded into DARM2 FM1 "RG2.8-3.4". It is not engaged now.

For future reference, here are the FMs that are used during lock acquisition:

DARM1: FM1 FM2 FM3 FM4 FM7 FM9 FM10

DARM2: FM3 FM4 FM5 FM6 FM7 FM8 FM9 FM10

leaving DARM1 FM5,6,8 and DARM2 FM1,2 unused

I have made alterations to the signal path for the 80.3 kHz PI ("MODE28").

In the PLL block I have made all the requisite changes to exactly replicate my LLO implementation.
This should get applied to all PLLs, but in the interest of not making changes to signal paths that LHO currectly depends on to stay locked, I don't want to change other MODE's feedback settings.

I have altered the H1:SUS-ETMX_PI_UPCONV_UC3_SIG bank's bandpass filter, to steeply roll off.
This is motivated by the work in this LLO aLog. I have implemented the exact same filters here.
Again, I only make this change in the signal flow of MODE28, so as not to perturb other already functioning, and depended upon, systems.

Vlad, Naoki

Yesterday I observed PI MODE24 ring up a large number of occasions. This was caused because the PI Guardian was falsely perceiving that the amplitude of PI was rising, inferred that the phase was "bad" and changed the phase to an actually bad phase.

In the time it took to finally find the "good" phase again, the amplitude rang up, and the saving grace was actually the hard-coded limit on the PLL outputs (which I am re-implementing to be soft-coded), which meant that while the PI phase was slewing from bad to terrible and back to good, the actuation was not exponentially increasing.
I have increased the RMS threshold for which the guardian will infer that the mode is "going up" from 3->4.

Another issue is that the first time this triggers for any mode, it reads out the RMS in the last 5 seconds and considers whether that is larger than the last time it makes that check. But the these "last times" had an initialisation value of 0: so it would always change the phase (to a now "bad" one).
The fix for this we think is to initialise these thresholds to the triggering thresholds for each of the modes being damped. This will at least prevent the guaranteed change on the first trigger.

Dust monitor pumps are running smoothly.

Fil, Daniel

We installed a common mode board, a dual demodulator chassis and replaced the single delay line phase shifter with a dual. This delay line is used by the filter cavity green locking. We also installed the new feedthrough panels on SQZT0.

WP #11489, and #11492 (not marked as Tuesday Maintinence as LHO apparently doesn't track that).

I have altered the h1susprocpi model in the follownig ways:

• Removed hard-coded amplitude limit of 10 (and the addition of a pre-limit amplitude offset and a post-limit amplitude offset)
• Added a filter bank AMP_FINAL in its place. This will serve as the output signal filtering/offsetting/limiting.
• Restored the PLL_SEL_MTRX size to 3 and reinstated the option to bypass PLL completely, linked to the input to the PLL blocks. Now you can use raw signals in the damping loop once again if you are having difficulties with the PLL locking.

These changes affected libraries:

• PLL_MASTER/PLL (first 2 changes)
• PI_MASTER_V2/PLL_BY_MODE (last change)

Changes will come in when the model is restarted on the h1oaf0 and the DAQ is restarted, as the new filter block adds channels.

As per WP 11488 the opslogin system was turned off.  This was an older Debian 9 system that was kept around when we transitioned to Debian 10+ in case we needed access to older tools.  At this point the system is not being used and causes confusion when people login onto opslogin instead of opslogin0 (the current system to use).

The band limited RMS (BLRMS) (H1:OMC-BLRMS_32_BAND#_BP for #={1 to 7}) bands that monitor DARM (Actually OMC DCPD A) frequencies in the range 2.5 kHz through to 32 kHz were being used in the lockloss monitoring tools to assess if a lockloss was due to PI.

These were not working correctly, so I poked my head in this morning.

The BLRMS used Elliptical filters, which have a flat response down the DC, and we were seeing a large DC value in the outputs that was "saturating" any high frequency signal we might care about.

These BLRMS have been altered to use Butterworth filters instead of Elliptical. Butterworth filters have the highly valuable property of actually being AC coupled so that should do the trick.

The alog was rebooted as part of regular maintenance.  This is a test of the system.

Following instructions from Sheila, I stepped the CARM gains (H1:LSC-REFL_SERVO_IN{1, 2}GAIN) by 1 about every minute starting at 15:11:30 UTC. I started at a gain of 8 and we lost lock as soon as I got to 20. We lost lock at 15:24 UTC.

Using two periods of quiet time during the last couple of days (1381575618 + 3600s, 1381550418 + 3600s) I computed the usual coherences:

https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1381550418/
https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1381575618/

The most interesting observation is that, for the first time as far as I can remember, there is no coherence above threshold with any channels for wide bands in the low frequency range, notably between 20 and 30 Hz, and also for many bands above 50 Hz. I'll assume for now that most of the noise above ~50 Hz is explained by thermal noise and quantum noise, and focus on the low frequency range (<50 Hz).

Looking at the PSDs for the two hour-long times, the noise belowe 50 Hz seems to be quite repeatable, and follows closely a 1/f^4 slope. Looking at a spectrogram (especially when whitened with the median), one can see that there is still some non-stationary noise, although not very large. So it seems to me that the noise below ~50 Hz is made up o some stationary 1/f^4 unknown noise (not coherent with any of the 4000+ auxiliary channels we record) and some non-stationary noise. This is not hard evidence, but an interesting observation.

Concerning the non-stationary noise, I think there is evidence that it's correlated with the DARM low frequency RMS. I computed the GDS-CALIB RMS between 20 and 50 Hz (whitened to the median to weight equally the frequency bins even though the PSD has a steep slope), and the LSC_DARM_IN1 RMS between 2.5 and 3.5 Hz (I tried a few different bands and this is the best). There is a clear correlation between the two RMS, as shown in a scatter plot, where every dot is the RMS computed over 5 seconds of data, using a spectrogram.

I turned up CO2X 15:05 to 15:12 UTC to see if we could put noise into DARM, plot attached.

• Yellow - Nominal CO2 settings, CO2X and CO2Y 1.7W Annular @ 15:02:00 (after SUS_CHARGE stopped)
• Blue - CO2X Annular 6.6W in @ 15:05:45UTC
  • High frequency noise higher, not sure this is physical, only expect low frequency noise. 
• Red - CO2X No Mask 13.7W in @ 15:08:55
  • Lines appear at 20Hz, 28.5Hz, 32.5Hz, more low frequency noise. 
• Back to nominal @ 15:12:00 UTC
  • 20Hz, 28.5Hz, 32.5Hz lines stay and start reducing
  • Lower high frequency noise! DARM plot attached and HOM. We are trying to reduce high frequency noise using ring heaters in 73308.

We got DARM glitches when raised and lowered the annular mask, see t-cursors attached. This is surprising. We should check that this is from the CO2 laser being blocked while the mask  (on a flipper mount) passes through the beam, rather than any electronics issue. Can do this by turning off CO2 laser and then raising and lowering the mask. 

Still want to redo 72981 line driving once we understand which PD to use for readback.