LOCKLOSS @ 6:53, looking at the lockloss select, potentially movement in CSOFT P, potentially INP1 P/Y and DHARD Y - scope attached - Tagging ISC

H1 has been locked for just over 5 hours. There has been a couple of DCPD and EX saturations alongside what looks to be CHARD P ringups, but the IFO looks to have stabilized.

Picket Fence was updated to replace the seismometer used on the channel 1 stream, which had not been working for some time.

Following up on EJ's alog, I have extended the code to check if the unresponsive filter is under local control by the model. I ran it this afternoon while H1 was in observe, running in 'nice' mode so as to not hammer the frontends with CA requests (delays between requests).

If we disregard locally controlled filters, the number of unresponsive filters is reduced to 3 (see attached). The top one runs on h1lsc, the other two on h1lscaux.

Here is the full list, with locally controlled filters marked with *

Total numer of filtermodules = 13367
num unresponsive 14

h1lsc {'LSC-EXTRA_AI_2': 'FM2  '}

h1lscaux {'LSC-LOCKIN_1_DEMOD_9_I': 'FM1  ',
          'LSC-LOCKIN_1_DEMOD_9_Q': 'FM1  '}

h1sqz {'SQZ-RLF_VCXO_SERVO': 'FM2* '}

h1alsex {'LSC-X_ARM_DRIVE': 'FM3* FM5* '}

h1alsey {'LSC-Y_ARM_DRIVE': 'FM3* FM5* '}

h1sussqzin {'SUS-ZM1_M2_COILOUTF_LL': 'FM1* FM6* ',
            'SUS-ZM1_M2_COILOUTF_LR': 'FM1* FM6* ',
            'SUS-ZM1_M2_COILOUTF_UL': 'FM1* FM6* ',
            'SUS-ZM1_M2_COILOUTF_UR': 'FM1* FM6* ',
            'SUS-ZM3_M2_COILOUTF_LL': 'FM1* FM6* ',
            'SUS-ZM3_M2_COILOUTF_LR': 'FM1* FM6* ',
            'SUS-ZM3_M2_COILOUTF_UL': 'FM1* FM6* ',
            'SUS-ZM3_M2_COILOUTF_UR': 'FM1* FM6* '}
 

TITLE: 06/09 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 137Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 7mph Gusts, 5mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.10 μm/s
QUICK SUMMARY:

At 20:10 UTC the SQZ_MANAGER went from FREQ_DEP_SQZ to SQZ_READY_IFO for some reason

20:15 UTC lockloss.
 
Relocking Notes: I had to Touch up X-ARM ALS Flashes.

Doors to Air handlers 1 & 2  were open from 1:35-138 local so if there is som etrending of the Air handlers during that time the plots may look diiffent than expected during those time.

Took ISC_LOCK to Initial Alignment 20:46 UTC
Started the Next Locking attempt 21:07
21:55 UTC Nominal Low Noise was reached.

Saved some SDF Changes to H1:ASC-ADS_PIT3_DEMOD_I_TRAMP  for PIT 3-5 and Yaw 3-5 after speaking to Naoki about some changes.

Current IFO Status: Nominal Low Noise And Observing for 1 Hour.

TITLE: 06/09 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 137Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 2mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

- IFO locked and in observing for ~1 hr

- CDS/DMs ok

- Seismic motion a bit elevated from recent EQ, will monitor

J. Kissel, S. Koehlenbeck, B. Lantz

Sina and I are putting together material to help motivate an SPI installation some time in the future (see most recent material in G2300655 and G2300846), and one of the open questions is whether we can demonstrate (via model, or extrapolated measurement) that the P & Y input to, say, PR3 would be better after linking the platforms together in PITCH & YAW than the local table PITCH (RY) & YAW (RZ).

This aLOG shows the first attempt at demonstrating this with data, sticking with PR3, and thus the HAM2 - HAM3 ISI system. 

Executive Summary: This initial look shows great promise for an SPI's ability to improve HAM2 and HAM3 platform motion across the 0.01 to 10 Hz frequency band -- differential X (longitudinal), as well as differential Y and Z converted to pitch and yaw motion, is factors of 2 to 200 better than local X (longitudinal), RY (pitch), and RZ (yaw).


The key concepts here are
    - As it's *always* worth repeating, the input to the suspension point is different from table motion, and the optic motion is different that optic motion, *especially* when it comes to cross-coupling between degrees of freedom, like suspension point Longitudinal and optic Pitch, or platform X / RZ vs. optic Yaw. Even within the ISI, there's a good amount of cross coupling between RX/RY tilt DOFs and Y/X translation degrees of freedom, so using the SPI to improve ISI RY could improve ISI Y, which benefits PR3 suspoint L, with benefits PR3 optic P. So the motivation for an SPI, via looking at only one diagonal DOF at once will never make a clear argument for "we need an SPI!" So, we're doing an incomplete job if we look at only one DOF at once and say "improvements in this one DOF will clearing improve that one DOF." We'll only really know the true quantified benefit once we install one and see. But it's still interesting to look at the data in this way to get a feel for if it makes sense to even try.

    - The SPI may be used for different frequency regions for different purposes. PR3, for example, may not be limited by residual seismic motion at some frequencies for some degrees of freedom. We just need to be conscious of this when make sweeping statements like "PR3 is completely dominated by OSEM noise, you won't see any improvement unless we improve the local sensors on the top mass of PR3 -- so why should we spend energy on an SPI?" because that sweeping statement is true at 10 Hz, but not true below ~3 Hz, and an improvement below 3 Hz might be benefit to the IFO in subtle ways like "we can reduce the bandwidth of the ASC loops, which means the ASC sensor noise doesn't couple into DARM at 10-15 Hz." 

    - In modern parlance, when we say an "SPI" we really mean a 3 DOF collection of sensors rather than "just" a 1 DOF seismic platform interferometer: 
        :: SPI L :: a longitudinal sensor -- the interferometer part, which uses an asymmetric michelson IFO (short arm stays local to the launching platform, and the long arm spans the 16 m distance between the two platforms) to determine the relative distance between the platforms. 
        :: SPI P :: an "optical lever" like pitch sensor measuring differential pitch of the platform (expected to be dominated by the difference between local Zs rather than local pitch RYs) 
        :: SPI Y :: an "optical lever" like yaw sensor, measuring differential yaw of the platform (expected to me dominated by the difference between local Ys, rather than differential yaw RZs)

    - Fundamentally, why an SPI wins in P and Y is the lever arm of sensing. The local sensing of pitch (RX / RY) or yaw (RZ) is the difference or sum of inertial sensors across the width of the table, a lever arm of ~1 [m]. One the other hand, if we can tie pitch/RY or yaw/RZ one platform (say HAM2) to the other (HAM3), but using the lever arm *between* the platforms -- L = ~16 [m] -- and this is *better* than the local interial sensing, we improve both local platform's motion by something like sqrt(2) / L. The details of of the improvement depend on a lot of things, and we'll get there, but at least conceptually, this is the promised improvement.

    - Another point to emphasize is that *we don't yet know the optimal controls topology and/or frequency region that these 3 DOFs will be used.* Depending on the results, it may be interesting to compare the local inertial sensors to the SPI sensors, or the local CPS to the SPI sensors. What frequency region we blend in the SPI vs. the local platform's inertial and displacement sensor array is still unclear. So the plots below compare both against the SPI. You'll see that the differential signals are better in some frequency regions and worse in others. This is all still to be understood and schemed upon, so my commentary below should be treated as "first impressions."

    - Remember that depending on the frequency band, the CPS or GS13s are "in loop," forming the error signal for the feedback loops, which have UGFs of around 30 Hz. I indicate a "CPS Rolloff Frequency" with vertical lines on all the plots -- where the vertical line is set at the frequency above which the CPS contribution to the super sensor is less than 1.0. (Note, this is different from what has traditionally been referred to as "the blend frequency," which is instead a center point of equal contributions of GS13 and CPS. Because 
        - gain peaking is involved at that "blend frequency" cross-over point, 
        - that gain peaking can be large and/or span across a lot of frequency space, 
        - GS13 blend filters are not easy to plot because of the embedded inertial sensor calibration and
        - AFAIK, no one's plotted them in their easy-to-digest complementary form recently, 
    I chose the more straight-forward, easily findable, "when does the CPS contribution to the feedback loop start to roll off" frequency. These frequencies are close, of course, but not the same.)

    - I'm concocting "differential motion" between the HAM3 and HAM2 platforms by coherently subtracting one platform's DOF from the other platform's corresponding DOF. Since the data is recorded and synchronized by the CDS system, one simply needs to download the two platform's data over the same time period and subtract one time series from the other and it's a "differential motion coherent subtraction."  

    - Of course, sensor noise is always of concern when doing differential measurements. As best I can, I use the estimates of the GS13 and CPS sensor noise from our old friend SEI_sensor_noise.m (T0900450), and I 
        (a) Assume every colocated sensor of a given type has identical noise
        (b) Scale them to the appropriate cartesian degree of freedom, assuming all sensors are incoherent, by taking the quadrature sum of the given sensor type's input matrix from 
            /ligo/svncommon/SeiSVN/seismic/HAM-ISI/Common/Basis_Change_Matrices/aLIGO_HAM_ISI_2_3.mat
        (c) Look at the results, find where the *Z* DOF is limited by sensor noise the real data, and scale every type's sensor noise in (a) by a single scalar fudge factor to match the real data
        (d) For differential sensor noise between the 2 platforms, I then multiply that DOF's sensor noise by a factor of sqrt(2).



What motivating data I'll show and why
    (I) SPI Longitudinal (the "interferometer" part of the SPI) would measure the differential distance between the HAM2 and HAM3 in X. So, to motivate the benefit of SPI L we'll compare an individual platform's X motion against the differential X, (HAM3 X - HAM2 X).

    (II) SPI Pitch (one "optical lever" part of the SPI) would primarily measure the differential Z (vertical) motion between HAM2 and HAM3 in Z. So, to motivate the benefit of SPI P we'll compare an individual platform's RY motion against the differential X, (HAM3 Z - HAM2 Z) *divided by the lever arm between the platforms, L.*

    (III) SPI Yaw (the other "optical lever" part of the SPI) would primarily measure the differntial Y (transverse) motion between HAM2 and HAM3. So, to motivate the benefit of the SPI Y, we'll compare an individual platform's RZ motion against the differential Y (HAM3 Y - HAM2 Y) *divided by the lever arm between platforms, L.*



OK, Now to the data.
For starter's I grabbed *one* 3 hours of data starting at -- last night, 2023-06-09 at 09:30 UTC (02:30a PDT, GPS 1370338218). The wind is low (< 10 mph), the microseism is around 0.1 [um/s], no earthquakes, and the IFO is locked, observing, in nominal low noise busy detecting gravitational waves. Note, that the RZ loops include Jim's new-and-improve blend filters that improve the local RZ performance by factors of 5 to 10 between 2 - 20 HZ.

LOCALX_vs_DIFFX shows the performance metric comparisons for SPI L.
    - The differential GS13 signal is about the same as the HAM3 local sensor for most of the frequency band. There's *some* areas below 0.5 Hz where the differential X GS13 would be better, but out of the nominal frequency region where the GS13 is used in loop.
    - Below 7 Hz, the differential CPS is better than the local CPS by factor ranging from 2 to 20. 
    - Below the "CPS rolloff frequency" for this DOF, 0.34 Hz, and SPI measuring this same differential motion (without being limited by its own sensor noise) would improve things over each local CPS by a factors of 2 to 5. 
    
Promising!

LOCALRY_vs_DIFFZoverL shows the performance metric comparisons for SPI P.
    - Here, we see a HUGE difference in performance between the local sensors and platform-to-platform differential sensors.
    - Assuming that the SPI optical lever has sensor noise below this level of rotation, would should be able to get a huge amount of improvement, and we'd have our pick of which sensor to use where, since both local CPS RY and GS13 RY sensors are MUCH worse than differential Z.
    - The only place where the differential motion is comparable (note -- comparable, not worse!) is in the ~10 Hz region where differential HEPI pier motion between the two platforms is, well, different.

*Very* promising!

LOCALRZ_vs_DIFFYoverL
    - Similar to the SPI P plots there's lots of frequency regions where SPI Y, informed by differential HAM3 - HAM2 Y would be better than local RZ. However, this DOF is probably the most confusing, because we know that each local table Y is limited in the 0.07 to 5 Hz region by local RX tilt (from local RX sensor noise across the ~1 [m] lever arm). So differential Y is limited by RX tilt, where differential Z is not in anyway limited by tilt. So we don't expect to get *as much* benefit out of an SPI Y, but we'd definitely still get some if the chambers are left as is. However, we also know that the ISI local RX sensor noise can be improved by installing better CPS sensors, like we've done for the HAM7 HAM8 systems. So, the story is a little longer here, but still...

*Very* promising!

Next on the to-do list:
    (1) Gather HAM3-HAM2 data at different environmental times, to see whether any of the above conclusions from this one data set change.
    (2) Compare this differential GS13 and CPS data against Sina's projections for sensor noise
    (3) Use this (and other data we gather) data to try to concoct a control strategy 
    (4) Use that control strategy to estimate the improved platform and suspension performance in various scenarios including
        (i) If we also improve the HAM-ISIs to use fine vertical CPS sensors 
        (i) If we also improve PR3's top mass sensing noise 
    (5) Present this kind of progress to the systems group

Some H1:ASC-ADS Y&P DEMOD Channels had some SDF changes that I accepted After speaking to Naoki about some change he had made today. Please See Screen shot below.
 

Naoki, Vicky

In alog70212, Vicky stepped ZM5/6 P/Y by 100 count (-100 count for ZM6 P) to check the sensing matrix of SQZ ASC. We checked the response of AS A/B RF42 as shown in the attached ndscope. The measured sensing matrix is as follows.

It seems the AS B RF42 has some P/Y cross coupling (or ZM5 driving has P/Y cross coupling?), but apart from that, the input matrix of SQZ ASC should be proportional to the inverse matrix of the measured sensing matrix, which can be calculated as follows.

The current input matrix for SQZ ASC is shown in the second attached figure. The POS and ANG correspond to the ZM5 and ZM6, respectively. Compared with the calculated matrix above, the current input matrix seems not correct. The correct input matrix should be proportional to the calculated matrix above and I propose a new input matrix as follows, which is larger than the calculated matrix by a factor of -10. This factor of -10 is chosen to have the same order of input matrix as the current one and it can be adjusted depending on the desired UGF of SQZ ASC. We will try this new input matrix soon.

Right Before this Locklockss at 20:10 UTC the SQZ_MANAGER went from FREQ_DEP_SQZ --> SQZ_READY_IFO for some reason

20:15 UTC lockloss tool suggests that the cause was an earthquake but the analysis says that it's "Failed" for some reason.

 

Fri Jun 09 10:11:49 2023 INFO: Fill completed in 11min 48secs

Travis confirmed a good fill curbside.

J. Kissel

Since I just got through reviewing the last two months of LVEA HVAC system woes (LHO:70284), I happened to have the LVEA HVAC's Air Handler Unit metrics still up on my computer. 

As such, I noticed that on June 07 2023 14:46 PDT (Wednesday afternoon) that the LVEA Zone 1B (think 3IFO storage area) took a ~0.4 deg F dip, and then recovered by ~20:00 PDT (that same Wednesday night). Not a big deal, but it's suspicious because no other metric -- the zone heaters percent "on-ness", the airflow meters from the fans, the air handler damper percent "openness", the coiling coil temperatures, i.e. the human/computer controllable temperature actuators -- show any similar excursion. 

Further, at an indeterminate amount of time ~20-24 hours later, but definitely by June 08 2023 18:00 PDT (Thursday afternoon), Air Handler Unit 2 (AHU2) Fan 3 shows a slow and steady increase in air flow (see H0:FMC-CS_LVEA_AH_AIRFLOW_3) from 7800 cubic feet per minute (CFM) to 8500 CFM -- see purple trace in 3rd panel of the last week.

You can also see that the Zone 1B (orange) and even Zone 1A (blue) in the top panel of that same trend may also be starting to increase.

Again, probably not a big deal in terms of LVEA temperature, but given the last couple of months of HVAC issues, I want to call folks attention to this early since we're about to head into a weekend.

This has been kind of hard to pick out, but it definitely seems like the IFO is less robust now against earthquakes that it was before increasing power over 60W in the IMC. Basically since then, we haven't been able to stay in NLN for any earthquakes over ~.5um/s, as measured by the peakmon eq witness channel.

Attached plot shows peakmon vs lock state trends since Feb 10 this year. The blue trace is the peakmon ground velocity minute trend, each green point indicates where the IFO was still locked 2 minutes after a local maximum ground velocity (found using the matlab peakfinder routine with some prominence and time separation requirements). The two marked points indicate where IMC power was increased above 60W (X=79730 minutes after Feb 10th) and the start of the run (X=148255 minutes after Feb10). Before power up we had multiple locks survive multipe 500nm/s eqs, a few around 1micron/s and one at almost 3micron/s. After the power up, the IFO doesn't ride out any eqs over 500nm/s and we basically not survived any notable eqs since the start of the run.

This is further reinforced by TJs alog from earlier today, when SEI_CS transitioning knocked us out of OBSERVE. Unless SEI_CS got dropped off the exclude recently, I suspect we never noticed because we were losing lock before the seismic system switched.

There have been no changes to the SEI eq controls. I looked at the small eq Camilla noted this morning and didn't see anything suspicious in the seismic systems, but I'll keep digging.

I looked again at DARM at 60W vs. 75W, comparing sqz and no-sqz for both. Each DTT trace uses 8 seconds, 50% overlap, 100 averages.

Lighter colors. 60W times used from 68251, w/15dB generated sqz.

• cyan,  FDS    @ 60W -  1364158671
• yellow, No SQZ @ 60W -  1364159804

Darker colors. 76W times from 6/4, ~3 hours into lock (semi-thermalized, past fast initial thermalization).

• red,    FDS    @ 76W  - 6/4 18:25 UTC
• purple, No SQZ @ 76W  - 6/4 19:00 UTC

See the summary page with this time window-- comparing red vs. black in this screenshot, I don't see obvious + excess noise with the squeezer on below 50Hz, when we are past fast initial thermalization.

Re: "fast initial SQZ thermalization" -- For reference, see the V-shape in blue that DARM takes with the fast 0.5-1 hour SQZ thermalization from a cold IFO (even unlocked for just 1-2 hours). Notably, in the first 0.5-1 hour, esp if the IFO was cold, the fast-thermalizing-SQZ (blue) exceeds the no-sqz noise (red) both below 50 Hz, and above 1kHz. I think this is related to the rapidly thermalizing SRCL detuning, which rotates squeezing across the band for the first 0.5-1 hour. To fix this SRCL detuning problem for sqz misrotation, I think we'd need to thermalize the SRCL offset; it wouldn't be sufficient to thermalize the sqz angle. If we have to pick an angle, we should pick the sqz angle that optimizes the bucket noise ~100 Hz; this is the most stationary as SRCL thermalizes. I think for calibration, Jeff sees this too as wildness in the rapidly thermalizing sensing function at the start of locks. This effect seems more prominent from an cold IFO, as it seems more muted of a V-shape when IFO relocks quickly (not much down/cold time).

Note the excess noise 20-50 Hz in the 60W vs. 75W configuration. Brina is now looking into understanding the calibration status and LSC coherences for these sqz and no-sqz times.

There are a number of models with filters in modules that have been engaged but no filter has been defined for that stage. This causes some confusion when scanning for unresponsive filter stages, as these will clutter the results. 
Attached is an example of what this looks like on the medm screen. FM2 is enabled, but nothing is loaded in that stage so the 2nd box never turns green. 

My guess is that these stages used to have a filter defined for them, but were removed. The solution is to finish the removal of these stages, buy disabling the stage and saving the new state with SDF. 

Full listing of filters/stages that are enabled but don't have the enabled stage defined in the filter file. 

h1susetmy : [('SUS-ETMY_L2_DAMP_MODE19', 'FM1'), ('SUS-ETMY_L2_DAMP_MODE9', 'FM4')],

h1sussqzin : [('SUS-ZM1_M1_WD_OSEMAC_RMSLP_LL', 'FM6'), ('SUS-ZM2_M1_LOCK_L', 'FM6')]

h1hpietmx : [('HPI-ETMX_3DL4CINF_C_Z', 'FM1'), ('HPI-ETMX_3DL4CINF_C_Y', 'FM1'), ('HPI-ETMX_3DL4CINF_C_X', 'FM1'), ('HPI-ETMX_3DL4CINF_B_Z', 'FM1'), ('HPI-ETMX_3DL4CINF_B_Y', 'FM1'), ('HPI-ETMX_3DL4CINF_B_X', 'FM1'), ('HPI-ETMX_3DL4CINF_A_Z', 'FM1'), ('HPI-ETMX_3DL4CINF_A_Y', 'FM1'), ('HPI-ETMX_3DL4CINF_A_X', 'FM1')]

h1lsc : [('LSC-EXTRA_AI_2', 'FM2')]

h1lscaux : [('LSC-LOCKIN_1_DEMOD_9_I', 'FM1'), ('LSC-LOCKIN_1_DEMOD_9_Q', 'FM1')]

h1oaf : [('OAF-CAL_SUM_DARM_L1', 'FM3'), ('OAF-CAL_SUM_DARM_L1', 'FM6'), ('OAF-CAL_SUM_DARM_L3', 'FM3')]

h1calcs : [('CAL-CS_DARM_ANALOG_ETMY_L1', 'FM4')]

h1susproc : [('SUS-ETMY_L2_DAMP_MODE19_BL', 'FM1'), ('SUS-ITMY_L2_DAMP_MODE18_BL', 'FM1'), ('SUS-ITMY_L2_DAMP_MODE19_BL', 'FM1')]

Edited to remove any filter stages under local control.