Lockloss from a large close earthquake, a 5.0 in Northern BC, Canada (linear velocity of 12!) We went to EQ mode 40 seconds after the LL

STATE of H1: Observing at 159Mpc

• We've been locked for 32:30, in observing since 22:55UTC
• The winds calmed down and brought the seism down

I ran a calibration measurement (operator instructions). The report generated for this set of measurements is at /ligo/groups/cal/H1/reports/20231027T203619Z/ (PDF here).

The model parameter set populated with Hc, Fcc, and the ETMX actuation strengths can be found at https://ldas-jobs.ligo-wa.caltech.edu/~cal/archive/H1/reports/20231027T203619Z/. This report is using the new model parameter set template updated in LHO:73783.

The 20231027T203619Z report was used to export a new calibration to the front end in the control room. The updates to the CALCS foton filters can be seen here and the same can be found for the EPICS records that used to calculate TDCFs in SDF screenshots attached to this log entry (page1, page2, page3).


TDCFs:

The calibration update brought KappaC closer to 1 (from 0.976 to 0.994). The PUM and UIM Kappas were also improved but not by much (<1% improvement). However, the TST Kappa saw a hard reset by about 4% to 1. Certainly the actual ESD charge on the TST stage is continuing to build up. We should keep an eye on it.

Systematic Error lines:
The reported systematic error for all our systematic error lines improved as well. See sys_err_scope.png. Line1 (17.1 Hz) and Line5 (33.43 Hz) still report ~3% error. 

GDS:
Today's new report was committed to the LHO cluster and the GDS pipeline was restarted to make sure it knows about the new DARM boost filter.


I'll follow up with a comment over the weekend detailing the commands I used for today's cal work.

Jenne's new DARM2 low frequency boost filter (LHO:73745) occupies the FM2 slot in the LSC-DARM2 bank. I've updated the pyDARM model parameter set ("ini") to include the new boost filter. The file template, without any fitted parameters (Fcc, Hc, and actuation strengths), can be found at https://git.ligo.org/Calibration/ifo/H1/-/blob/main/pydarm_H1.ini (git hash d713a4e9).

Lines 48-51 now read (changes in bold):

digital_filter_bank = LSC_DARM1, LSC_DARM2
digital_filter_modules = 1,2,3,4,7,9,10: 2,3,4,5,6,7,8
digital_filter_gain = 400,1
digital_filter_file = Common/H1CalFilterArchive/h1omc/H1OMC_1382307261.txt


Relevant Links
===============
LHO:73745: Jenne proposes new boost filter with low frequency gain
LHO:73776: DARM boost activated

This aLog hopes to be something similar to this LLO aLog concerning why they do not currently wish to attempt to actively actuate on their 80.4 kHz PI.

I attempt the same procedure, calibrating to a fairly arbitrary amplitude, which in the case of LHO corresponds to 1 at the average mechanical mode amplitude at a well thermalised time.

I project a rate of change of amplitude at the maximum DAC rate, based on observing amplitude growth rates at the exact optimised actuation phase to drive up the mode as fast as possible. Its not very constant like it was in LLO, but the rate is something about 60 times higher at about 120 "amp"/s.

On the other hand; the PI gain is significantly stronger here (260 vs LLO's 55), almost certainly fully attributable to the higher Q factor of the mechanical mode (1.7million vs LLO's 0.67 million).
At the end of the day, the stronger actuation allows you to damp the mode back down even if it has grown in excess of 100x its normal ampliutude [red line is the actuation limit], compared to LLO's limit of about 2x visible amplitude (which is probably about 5x its normal amplitude).

The bad news:

The rate of change of amplitude of this mode still means that if something goes wrong, its game over within 5 seconds of fumbling around (much less actually as what I model here is only gain of 17, not the expected maximum gain of 260).

More-over there are issues on the sensing side:

1. Changes to high frequency noise, e.g. Jenne's changes this week, decrease the SNR in locking onto this mode, causing un-reliable PLL locking.
2. Unlike the isolated-by-many-Hz-on-both-sides mechanical mode of LLO, LHO's troublesome mode has very nearby neighbors. More-over, this mechanical mode, and it's friends move several Hz around the place based on thermal couplings that are not obvious to me at this stage. So much so that they almost overlap the mechanical mode in frequency, meaning that you are not going to have good clear actuation on this mode, unless it rings up significantly, and by the time the PLL locks, it will be un-dampable.
   • Here is a screenshot of 2 very well thermalised times (each time locked for several days). The vertical red line is the problematic mode. You can see drastic frequency changes of all modes, and the lower frequency neighbor is much too close to for the band-pass filters we use.
   • Outside temperatude was dramatically differenent on these 2 days. (~15 Celcius).
3. You cannot rely on setting up a static narrow band filter to cover this mechanical mode for 2 reasons:
   • It moves several Hz on a day to day basis.
   • You need as broad of a bandpass as possible to reduce the phase impact of a peak that is moving through its band.

Bottom line: while theoretically this PI can be handled, it would be impossible in practice. Avoid at all costs.

TITLE: 10/27 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: IFO has been locked 28h30. Just finished up a commissioning period from 20:00 to 22:55UTC
LOG:

During commissioning:

• DARM2 Boost turned on 73776
• Calibration updated
• SQZ touched and IR beam centered on FC2 73777
• MICH FF remeasured but no changes made 73773

This morning we had to touch FC2 alignment when the FC unlocked 73770, but since then Naoki has adjusted the FC2 beam centering 73777 so we except this won't be an issue in the future.

TITLE: 10/27 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
OUTGOING OPERATOR: Camilla
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 20mph Gusts, 15mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

• All systems look good (seism and wind are a little high but not too bad), comissioning just wrapped up for the day
• We've been locked for 28:26

Took MICH FF measurements following instructions in lsc/h1/scripts/feedforward/README.md but running MICHFF_excitation_ETMYpum.xml. Data saved in lsc/h1/scripts/feedforward/ as .txt files. Last tuned MICH in 73420. 

I saved a new filter in FM6 as 27-10-23-b (red trace) but it made the MICH coupling worse between 20 and 80Hz so we left the original (pink trace). We could try to re-fit the data to load in Wednesday's commissioning period.

A recent suggestion is that we see if the BS coil drivers are a limit to our sensitivity now.  The way we'd check is to put the BS back to it's acquisition state, and see if we see noise in DARM.  Any excess noise can then be projected down with the ratio of the coil driver filters. 

I took an old version of a coil state switching script (from userapps/isc/guardian/bs_m2_switch_fast.py), and added to it turning off the output gain (upon recommendation of LLO who suspects that as contributing to their lockloss).  We don't have time to try it out today, but the script (attached here, as well as checked in to userapps) should be ready to try next week.  We can consider running it on Tuesday morning just to check that there are no issues with it, but then actually try it Wednesday afternoon.

To turn the BS to it's actuation state, line 12 setting the 'actuatorState' variable should be set to 2, then run the script.  Then, to go back to the lownoise setting, change that variable back to 3 and re-run the script.

Note that the BS suspension guardian can take us from the acquisition state to the low noise state gracefully, but it doesn't look like it's got built-in a graceful way to go back to the high noise state.

Since the FC2 centering is not good now and it could cause the FC2 L2A coupling issue in alog73770, I centered FC2 using IR dither. The first attached figure shows the FC IR error signal with FC1/2 dithered. FC2 is dithered at 7.1Hz in pitch and 16.1Hz in yaw. You can see the FC2 dither peak at 7.1Hz, which indicates that FC2 centering is not good in pitch. I changed the green QPD offset at FC transmission while running FC ASC to minimize the peak at 7.1Hz. I changed the offset of INJ ANG P from 0.08 to 0.18 as shown in the second and third attached figures. After the centering, the dither peak at 7.1Hz is removed and the FC2 centering is as good as one in May.

I activated Jenne's DARM2 boost from LHO:73745 (in FM2) at 10/27/2023 13:25:54. The FM2 change has been saved in SDF.

The file that includes this new DARM2 boost filter is at /opt/rtcds/lho/h1/chans/filter_archive/h1omc/H1OMC_1382307261.txt. I've copied it to /ligo/svncommon/CalSVN/aligocalibration/trunk/Common/H1CalFilterArchive/h1omc for the Cal group. The file has been committed to the SVN. The corresponding repo has been updated on the LHO cluster under the cal account too.

Jenne, Vlad, Louis


Vlad noticed that the injection on H1:CAL-INJ_CW_EXC stays on during NLN CAL MEAS. This is a note to say that we should have ISC_LOCK turn it off/on when moving into/out of NLN_CAL_MEAS.

Fri Oct 27 10:14:19 2023 INFO: Fill completed in 14min 14secs

Jordan confirmed a good fill curbside. Outside air +6C, TC mins -138C, -129C.

Vlad and I noticed a set of 20Hz peaks in DARM at 16:50UTC, see attached. Maybe similar to the peaks Robert and Tyler mitigated by changing AC settings in 73430

Vicky, Camilla

IFO was Out of Observing 15:40 to 15:56UTC as SQZ FC Unlocked. After 5 failed automatic SQZ_FC TRANSION_IR_LOCKING attempts, Vicky and I kept the FC GRD in DOWN, closed the Input 1 loop (SQZ Overview > FC Servo > H1:SQZ-FC_SERVO_IN1EN) to manually lock FC on green and aligned FC2 using the GR camera and maximizing H1:SQZ-FC_TRANS_C_LF_OUTPUT. Accepted FC2 alignment sdfs and went back to observing. Updated "Issues with SQZ FC Guardian" wiki

Think the issue could have been due to alignment drifts over the long 21 hour lock and as the M3 length was cleared, this couples in with pitch and yaw alignments, plot attached. Enough that the FC2 alignment needed to be slightly tweaked to relock. From plot, can see that M2 Pit (grey trace, bottom left) changed ~40urad on FC unlock?! This is a lot of drift over 20 hours, zoomed out plot here.

TITLE: 10/27 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 162Mpc
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 13mph Gusts, 10mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

IFO has been locked 20h30. There is plans for commissioning time 1-3pm this afternoon.

TITLE: 10/27 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 163Mpc
INCOMING OPERATOR: Austin
SHIFT SUMMARY:
Very Quiet night. Violins look great! Everything running smoothly.
 

H1 Current Status: Locked in NOMINAL_LOW _NOISE & OBSERVING for 12.5 hours. 


LOG:
No Log

Summary:

It seems that the supply voltage for thermistors is oscillating, the frequency depends on whether or not the supply is loaded with thermistors (830mHz open to 1.66Hz fully connected to the in-chamber thermistors), the amplitude of the oscillation jumps seemingly randomly, and this is also happening for the unused Beckhoff module for the yet-to-be-installed second T-SAMS unit, all measured at the back of the Beckhoff chassis. (Can't we measure this from Beckhoff itself, without me going to the floor?)

Fil and Fernando quickly set up the same Beckhoff module in the lab and didn't observe this. Could we swap or maybe restart the modules in the chassis?

As of now, Beckhoff cable is disconnected from the back of the heater driver chassis. (I'm applying DetChar-Request tag just so people know, but we're just changing from one no-comb configuration to the other.)

Detaisls:

Since the past findings about OM2 and 1.66Hz comb (alogs 73367, 73233, 72967 72241 and 72061) didn't make sense, I went to the floor and remeasured the comb in the Beckhoff heater output (which goes to the heater driver input) as well as thermistor pins in the back of the heater driver chassis while Beckhoff connection was intact.

Turns out that all of these things share the same frequency but the voltage across thermistor pins was ~3 orders of magnitude larger than Beckhoff heater driver output pins (pin 9 and 19) (the latter were referenced from the driver board ground as it's common mode for both pins). I used a scope on battery and the thermistor voltage was like roughly 1Vpp 1.66Hz rectangular wave (1st pic). Yellow is the voltage across pin10 and 23 (across thermistor 1), blue is pin9 and 22 (thermistor 2) of the DB25 at the back of the driver chassis when the Beckhoff cable was still connected.  Voltage difference seemed to have come from the temperature difference of the thermistors (I disconnected the Beckhoff cable and measured the thermistor 1 and 2 resistance incl. cables to be 7.41k and 4.08k, respectively). When I disconnected the cable from the chassis and just measured the pin10-23 and pin9-22 voltage coming from the cable (picture 2), they were both about 1.2V pp. This is supposed to be the source voltage for thermisters. The frequency slowed down by about a factor of 2 (832mHz) when the thermistors were disconnected.

For your convenience, below is a table of which pins are what (see e.g. E1100530 and D2000212). Note that thermistors themselves only have two pins, therefore supply and readback pins are bundled together in chamber as shown. Supply is presumably a reference voltage supplied through a reference resistor.

Went to the CER, disconnected the cable from the back of the Beckhoff chassis and did the same measurement. Frequency didn't change but the amplitude was much smaller (~280mVpp instead of 1.2Vpp) for a while, but suddenly the amplitude of the thermistor 1 supply changed back to 1.2V (pic 3). Nuts. When the beckhoff cable was reconnected (and the connection to in-chamber thermistor was restored) the frequency went back to 1.66Hz (picture 4).

Picture 5 shows pin 10-23 (thermistor 1 supply) and pin12-25 (thermistor 1 readback, which is not connected to anything). Picture 6 is the same thing but for the unused Beckhoff unit for the second T-SAMS. It's strange that the same thing is happening in two independent units. Picture 7 is the thermistor 1 supply and pin 6-19 (voltage output for the heater driver). It really seems that this is a problem of the supply voltage.

I checked the 24V power strip for the Beckhoff chassis but it was good (pic 8 and 9).

Fil and Fernando set up EL3692, which is the Beckhoff unit used for Thermistors. They didn't observe this oscillation behavior.

I wanted to do some injections into thermistors to see how this couples to DARM but didn't have time.

Using two periods of quiet time during the last couple of days (1381575618 + 3600s, 1381550418 + 3600s) I computed the usual coherences:

https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1381550418/
https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1381575618/

The most interesting observation is that, for the first time as far as I can remember, there is no coherence above threshold with any channels for wide bands in the low frequency range, notably between 20 and 30 Hz, and also for many bands above 50 Hz. I'll assume for now that most of the noise above ~50 Hz is explained by thermal noise and quantum noise, and focus on the low frequency range (<50 Hz).

Looking at the PSDs for the two hour-long times, the noise belowe 50 Hz seems to be quite repeatable, and follows closely a 1/f^4 slope. Looking at a spectrogram (especially when whitened with the median), one can see that there is still some non-stationary noise, although not very large. So it seems to me that the noise below ~50 Hz is made up o some stationary 1/f^4 unknown noise (not coherent with any of the 4000+ auxiliary channels we record) and some non-stationary noise. This is not hard evidence, but an interesting observation.

Concerning the non-stationary noise, I think there is evidence that it's correlated with the DARM low frequency RMS. I computed the GDS-CALIB RMS between 20 and 50 Hz (whitened to the median to weight equally the frequency bins even though the PSD has a steep slope), and the LSC_DARM_IN1 RMS between 2.5 and 3.5 Hz (I tried a few different bands and this is the best). There is a clear correlation between the two RMS, as shown in a scatter plot, where every dot is the RMS computed over 5 seconds of data, using a spectrogram.