After being at NLN for 1h30. 1382561229. No obvious reason for lockloss: environment, LSC and ASC signals look fine.

After IFO locked 11h29m. 1382549720 ~6Hz wobble in DARM before lockloss.

Sat Oct 28 10:10:35 2023 INFO: Fill completed in 10min 32secs

TITLE: 10/28 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 1mph Gusts, 0mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

IFO has been locked for 9 hours. Range seems to have dropped in last hour, plot attached, unsure why. Environment and squeezing is fine.

The TCSX laser unlocked this morning at 13:38 UTC 10/28. Before I had a chance to get into NoMachine to check it out, the laser relocked itself again at 13:45 UTC.

TITLE: 10/28 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC

STATE of H1: Observing at 151Mpc

INCOMING OPERATOR: Austin
SHIFT SUMMARY: Two large close earthquake plus an unknown lockloss at a highup state made relocking take quite a while. We've been locked for almost an hour as of 07:00UTC

03:10 lockloss from a 5.0 in Canada, I held us in down till the ground motion subsided (~10 minutes, then sei went back to calm at 03:21)

After the EQ, there was no light on green arms. I did a restore on the QUADS (ETM, ITM, TMS) to when we were locking for the previous lock (10/26 13:30, it barely changed the sliders) and then I tapped ETM{X,Y} in pitch a bit and then the beams appeared and increase flashes took care of the rest. The COMM & DIFF beatnotes were terrible at LOCKING_ALS, 24 & 18 and the ITM Error signals looked very off, so I decided to go down to run an initial alignment at 03:47 just to hear that theres another incoming earthquake from Canada, from the same spot as previously and this ones even bigger! I decided to hold off on starting the IA till after the EQ passed.

03:53 EQ mode activated, back to CALM at 04:13

04:13 started IA, finished at 04:33

05:14 lockoss at LASER_NOISE_SUPRESSION

06:25 in Observing

LOG:

DCPD saturation then lockloss during LASER_NOISE_SUPRESSION

Lockloss from a large close earthquake, a 5.0 in Northern BC, Canada (linear velocity of 12!) We went to EQ mode 40 seconds after the LL

STATE of H1: Observing at 159Mpc

• We've been locked for 32:30, in observing since 22:55UTC
• The winds calmed down and brought the seism down

I ran a calibration measurement (operator instructions). The report generated for this set of measurements is at /ligo/groups/cal/H1/reports/20231027T203619Z/ (PDF here).

The model parameter set populated with Hc, Fcc, and the ETMX actuation strengths can be found at https://ldas-jobs.ligo-wa.caltech.edu/~cal/archive/H1/reports/20231027T203619Z/. This report is using the new model parameter set template updated in LHO:73783.

The 20231027T203619Z report was used to export a new calibration to the front end in the control room. The updates to the CALCS foton filters can be seen here and the same can be found for the EPICS records that used to calculate TDCFs in SDF screenshots attached to this log entry (page1, page2, page3).


TDCFs:

The calibration update brought KappaC closer to 1 (from 0.976 to 0.994). The PUM and UIM Kappas were also improved but not by much (<1% improvement). However, the TST Kappa saw a hard reset by about 4% to 1. Certainly the actual ESD charge on the TST stage is continuing to build up. We should keep an eye on it.

Systematic Error lines:
The reported systematic error for all our systematic error lines improved as well. See sys_err_scope.png. Line1 (17.1 Hz) and Line5 (33.43 Hz) still report ~3% error. 

GDS:
Today's new report was committed to the LHO cluster and the GDS pipeline was restarted to make sure it knows about the new DARM boost filter.


I'll follow up with a comment over the weekend detailing the commands I used for today's cal work.

Jenne's new DARM2 low frequency boost filter (LHO:73745) occupies the FM2 slot in the LSC-DARM2 bank. I've updated the pyDARM model parameter set ("ini") to include the new boost filter. The file template, without any fitted parameters (Fcc, Hc, and actuation strengths), can be found at https://git.ligo.org/Calibration/ifo/H1/-/blob/main/pydarm_H1.ini (git hash d713a4e9).

Lines 48-51 now read (changes in bold):

digital_filter_bank = LSC_DARM1, LSC_DARM2
digital_filter_modules = 1,2,3,4,7,9,10: 2,3,4,5,6,7,8
digital_filter_gain = 400,1
digital_filter_file = Common/H1CalFilterArchive/h1omc/H1OMC_1382307261.txt


Relevant Links
===============
LHO:73745: Jenne proposes new boost filter with low frequency gain
LHO:73776: DARM boost activated

This aLog hopes to be something similar to this LLO aLog concerning why they do not currently wish to attempt to actively actuate on their 80.4 kHz PI.

I attempt the same procedure, calibrating to a fairly arbitrary amplitude, which in the case of LHO corresponds to 1 at the average mechanical mode amplitude at a well thermalised time.

I project a rate of change of amplitude at the maximum DAC rate, based on observing amplitude growth rates at the exact optimised actuation phase to drive up the mode as fast as possible. Its not very constant like it was in LLO, but the rate is something about 60 times higher at about 120 "amp"/s.

On the other hand; the PI gain is significantly stronger here (260 vs LLO's 55), almost certainly fully attributable to the higher Q factor of the mechanical mode (1.7million vs LLO's 0.67 million).
At the end of the day, the stronger actuation allows you to damp the mode back down even if it has grown in excess of 100x its normal ampliutude [red line is the actuation limit], compared to LLO's limit of about 2x visible amplitude (which is probably about 5x its normal amplitude).

The bad news:

The rate of change of amplitude of this mode still means that if something goes wrong, its game over within 5 seconds of fumbling around (much less actually as what I model here is only gain of 17, not the expected maximum gain of 260).

More-over there are issues on the sensing side:

1. Changes to high frequency noise, e.g. Jenne's changes this week, decrease the SNR in locking onto this mode, causing un-reliable PLL locking.
2. Unlike the isolated-by-many-Hz-on-both-sides mechanical mode of LLO, LHO's troublesome mode has very nearby neighbors. More-over, this mechanical mode, and it's friends move several Hz around the place based on thermal couplings that are not obvious to me at this stage. So much so that they almost overlap the mechanical mode in frequency, meaning that you are not going to have good clear actuation on this mode, unless it rings up significantly, and by the time the PLL locks, it will be un-dampable.
   • Here is a screenshot of 2 very well thermalised times (each time locked for several days). The vertical red line is the problematic mode. You can see drastic frequency changes of all modes, and the lower frequency neighbor is much too close to for the band-pass filters we use.
   • Outside temperatude was dramatically differenent on these 2 days. (~15 Celcius).
3. You cannot rely on setting up a static narrow band filter to cover this mechanical mode for 2 reasons:
   • It moves several Hz on a day to day basis.
   • You need as broad of a bandpass as possible to reduce the phase impact of a peak that is moving through its band.

Bottom line: while theoretically this PI can be handled, it would be impossible in practice. Avoid at all costs.

Summary:

It seems that the supply voltage for thermistors is oscillating, the frequency depends on whether or not the supply is loaded with thermistors (830mHz open to 1.66Hz fully connected to the in-chamber thermistors), the amplitude of the oscillation jumps seemingly randomly, and this is also happening for the unused Beckhoff module for the yet-to-be-installed second T-SAMS unit, all measured at the back of the Beckhoff chassis. (Can't we measure this from Beckhoff itself, without me going to the floor?)

Fil and Fernando quickly set up the same Beckhoff module in the lab and didn't observe this. Could we swap or maybe restart the modules in the chassis?

As of now, Beckhoff cable is disconnected from the back of the heater driver chassis. (I'm applying DetChar-Request tag just so people know, but we're just changing from one no-comb configuration to the other.)

Detaisls:

Since the past findings about OM2 and 1.66Hz comb (alogs 73367, 73233, 72967 72241 and 72061) didn't make sense, I went to the floor and remeasured the comb in the Beckhoff heater output (which goes to the heater driver input) as well as thermistor pins in the back of the heater driver chassis while Beckhoff connection was intact.

Turns out that all of these things share the same frequency but the voltage across thermistor pins was ~3 orders of magnitude larger than Beckhoff heater driver output pins (pin 9 and 19) (the latter were referenced from the driver board ground as it's common mode for both pins). I used a scope on battery and the thermistor voltage was like roughly 1Vpp 1.66Hz rectangular wave (1st pic). Yellow is the voltage across pin10 and 23 (across thermistor 1), blue is pin9 and 22 (thermistor 2) of the DB25 at the back of the driver chassis when the Beckhoff cable was still connected.  Voltage difference seemed to have come from the temperature difference of the thermistors (I disconnected the Beckhoff cable and measured the thermistor 1 and 2 resistance incl. cables to be 7.41k and 4.08k, respectively). When I disconnected the cable from the chassis and just measured the pin10-23 and pin9-22 voltage coming from the cable (picture 2), they were both about 1.2V pp. This is supposed to be the source voltage for thermisters. The frequency slowed down by about a factor of 2 (832mHz) when the thermistors were disconnected.

For your convenience, below is a table of which pins are what (see e.g. E1100530 and D2000212). Note that thermistors themselves only have two pins, therefore supply and readback pins are bundled together in chamber as shown. Supply is presumably a reference voltage supplied through a reference resistor.

Went to the CER, disconnected the cable from the back of the Beckhoff chassis and did the same measurement. Frequency didn't change but the amplitude was much smaller (~280mVpp instead of 1.2Vpp) for a while, but suddenly the amplitude of the thermistor 1 supply changed back to 1.2V (pic 3). Nuts. When the beckhoff cable was reconnected (and the connection to in-chamber thermistor was restored) the frequency went back to 1.66Hz (picture 4).

Picture 5 shows pin 10-23 (thermistor 1 supply) and pin12-25 (thermistor 1 readback, which is not connected to anything). Picture 6 is the same thing but for the unused Beckhoff unit for the second T-SAMS. It's strange that the same thing is happening in two independent units. Picture 7 is the thermistor 1 supply and pin 6-19 (voltage output for the heater driver). It really seems that this is a problem of the supply voltage.

I checked the 24V power strip for the Beckhoff chassis but it was good (pic 8 and 9).

Fil and Fernando set up EL3692, which is the Beckhoff unit used for Thermistors. They didn't observe this oscillation behavior.

I wanted to do some injections into thermistors to see how this couples to DARM but didn't have time.

Using two periods of quiet time during the last couple of days (1381575618 + 3600s, 1381550418 + 3600s) I computed the usual coherences:

https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1381550418/
https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1381575618/

The most interesting observation is that, for the first time as far as I can remember, there is no coherence above threshold with any channels for wide bands in the low frequency range, notably between 20 and 30 Hz, and also for many bands above 50 Hz. I'll assume for now that most of the noise above ~50 Hz is explained by thermal noise and quantum noise, and focus on the low frequency range (<50 Hz).

Looking at the PSDs for the two hour-long times, the noise belowe 50 Hz seems to be quite repeatable, and follows closely a 1/f^4 slope. Looking at a spectrogram (especially when whitened with the median), one can see that there is still some non-stationary noise, although not very large. So it seems to me that the noise below ~50 Hz is made up o some stationary 1/f^4 unknown noise (not coherent with any of the 4000+ auxiliary channels we record) and some non-stationary noise. This is not hard evidence, but an interesting observation.

Concerning the non-stationary noise, I think there is evidence that it's correlated with the DARM low frequency RMS. I computed the GDS-CALIB RMS between 20 and 50 Hz (whitened to the median to weight equally the frequency bins even though the PSD has a steep slope), and the LSC_DARM_IN1 RMS between 2.5 and 3.5 Hz (I tried a few different bands and this is the best). There is a clear correlation between the two RMS, as shown in a scatter plot, where every dot is the RMS computed over 5 seconds of data, using a spectrogram.