I will also add that this ~.5micron/s eq-band velocities are a level that the primary microseism can hit for a week or two at a time during the winter. Conditions that were already challenging for us in prior runs.

Another possible culprit from around the time of April 7: removing 12 dB of low-frequency (< 1 Hz) gain from the Michelson loop to try to reduce the amount of sensor noise reinjection in the GW band (LHO:68432).

It would be a relatively simple test to (1) increase the EPICS gain of the loop by 3 dB to place the UGF back around 10 Hz, and then (2) re-engage the 4:1 Hz boost (LSC-MICH2 FM3). If the noise increase in DARM is acceptably low, perhaps you could run like this for a week to see if the duty cycle improves.

If this extra gain helps during EQs, it could also be made part of the EQ guardian response.

Brina, Sheila,

We ran some excitations while locked, need to review plots (will come back to alog to update info)

Here is are a few plots comparing the LSC coherences from MICH, PRCL, and SRCLbetween these times with/without the SQZ, it seems that the SQZ/noSQZ plots for the same days look similar in some regions, but there is more coherence in the 76 W dates.

(just updated the image to match the range from the DARM plots and updated the H1 live to a recent time today (06/09/23)).

Additional screenshot including DARM 76W at 20mA here, no-sqz = dark green ; fds = purple. Looks like these DARM times were taken during a contrast defect scan, seen by squeezer too.

Couldn't edit my previous comment to update the image so here are the updated coherence plots.

Nothing has changed on the SEI side of things that would affect our earthquake robustness. We should have been able to ride out this eq, but SEI_ENV didn't even transition until 5 secs after the IFO lost lock, but we have survived eqs this size without. I'm starting to look to see if we are falling off a pd somewhere or something.

On attached screenshot, left column are some ifo channels, right column are some SEI channels. I don't have much to say at this point, but none of the SEI channels look crazy, IFO is just starting to see the eq and we lose lock.

Adding another dimension to the problem...

I was thinking over night about this, and realized "well, maybe there's *one* part of the power increase that has impacted duty cycle; the 11 Hz ring ups from PRCL going unstable thru too much gain."

But, I've added the PRCL gain adjustments to the series of trends -- see new attachments in relative time and UTC time -- and I think the new-ish PRCL instabilities can also be explained by drifts and changes in alignment.
- We started using the THERMALIZATION guardian around Apr 24. This slow ramps the PRCL2 gain through the thermalization period

- We made a change, *reducing* the PRCL2 end-point setting on May 21 -- but this was reactive to the time when the ISI YAW drifts were at maximum. 

- We then restored that end-point setting on May 26

- Then, on May 31, after having starting to have trouble with 11 Hz ring ups of PRCL gain, we adjusted the "base" PRCL1 gain from 1.0 to 1.5, -- but this was reactive is after several days of LVEA temperature drifting around between May 25 and May 31, and an especially bad Tuesday on May 29th.

- On Jun 4th, the LVEA temperature controls settle on a "new normal" and we "find" we need to increase the "base" PRCL set point again to 1.7 on June 6

- Then on Jun 7th, we reset the "base" PRCL1 gain to 1.0, but instead increase the thermalization set point higher.

My vote is the following: we take the hit in time that this will mean:
- Restore (or change to re-create) the LVEA temperature to values we had consistently for months up until May 10th. Since the LVEA is en-mass cooler than before, we can use the individual zone heater settings to bring each zome back *up* to is "prior to May 10th" value.
- Once that's settled, we re-align PR3 YAW to the slider values we had up until May 10th of 151.6 "urad," and run an initial alignment.
- Once that's settled, we go out to ISCT1 and re-reset the alignment of the table (though it's not reproducible, hopefully, doing so will get us back to the ISCT1 alignment we've had for many moons prior to all this mess).
- Once that's settled, restore ETMX ring heater to its value of 1.3 W.
- Once that's settled, we go back to the May10th era PRCL gains and THERMALIZATION guardian set points of PRCL1 = 1.0 and PRCL2 = 23.0.

If all that works, then we re-calibrate PR3's sliders, optical levers, and OSEMs.

Since it's perhaps quite tough to see all these traces stacked vertically on top of each other (a regrettable "must" because the timing of all these changes is important to the story) I've capture this epic ndscope session as a .yaml template. 

/ligo/home/jeffrey.kissel/2023-06-07/
    2023-06-07_3motrend_IMCPWR_ISIYAW_LVEATEMP_GNDBLRMS_PR3YAW_PRCGAINs.yaml

I can't attach a file with the ".yaml" or ".yml" extension to the aLOG, but it's linux, so I've just changed the file extension to .txt (because, in the end, it *is* just a text file). So if you'd like to download it from here and look around, download and then change the extension back to ".yaml".

With the temperatures more stable then they had been we are waiting to get the cooling coil strainers cleaned before attempting other changes.  The major flucuations we were seeing we caused primarily by a cooling coil not getting to temperature creating an issue for mulitiple days.  Once the strainers are clean temperature control can be changed to try and recreate previous zone temperatures.  Though it was nice have all zones grouped together for once.

Some filters have front-end control over which filters stages are on. Typically, more than one filter stage is used for, say a automatic boost, but only a subset is actually loaded. So, a filter may be on, but empty. This is the expected behaviour. Changing which filter stages are participating would be a front-end model change.

A better solution may be to turn the 2nd box on for empty filters, when they are on.

After reviewing all the "SUS" filter banks mentioned above, these empty filter banks are either on because they were turned on by mistake (the ZM1 and ZM2 filters) and blindly accepted into SDF, or the bank had never been active use for control or monitoring (the violin MODE control and monitoring filters), so someone was probably playing around with a filter design and cleared it out but forgot to turn off the filter. 

In all SUS cases, the filter module should be turned off and accepted as such in SDF.

We'll make a point to clean this up and clear out the confusion next Tuesday, or during a next convenient lock loss.

h1sussqzin ZM1 and ZM2 filters in question have been turned off -- see LHO:70245.

h1hpietmx filters in question have been turned off -- see LHO:70251.

The h1oaf and h1calcs filters in question have been turned OFF -- see LHO:70255.

The h1susetmy and h1susproc filter modules in question have been addressed -- see LHO:70264.

To Daniel's point - Another choice is to populate the filter with a gain=1 stage. Then it turns on, but doesn't do anything. SEI does this with some of the calibrations. e.g.  FM1 is the the manufacture's calibration, and FM2 is to tweak the calibration based on measurements. If the sensor is very close to spec, FM2 can just be a gain=1. Then all the automation works more smoothly.

Adding plots of LVEA temperature over the last 10, 16 and 30 days. With cross-hairs showing dates temperatures changed. This plot is available with command: ndscope /opt/rtcds/userapps/release/cds/h1/scripts/fom_startup/nuc32/VEA_temperatures.yaml

Adding some quantitative time and information to this aLOG series.

First, I cover the HVAC system and the channels that I'm using for metrics of the LVEA HVAC system.
The LVEA has 2 giant cooling actuators, or "air handlers" that live in the Mechanical Room, +X -Y of the beam splitter.
These air handlers add together to cool the LVEA as a common actuator.
There are separate, local *heaters* in each zone of the LVEA that serve as "differential" actuators. 

The system is depicted pretty well in the MEDM overview screen found under sitemap > "FMCS OVERVIEW" > "Air Handler 1,2"

The channels that are useful metrics of the system are listed below with their function and units
External, outside the building temperature
    H1:PEM-CS_TEMP_ROOF_WEATHER_DEGF       Temperature on the LVEA building roof, in deg F (look for corresponding "DEGC" channels for deg C)

LVEA temperature sensors << our primary "canary in the coal mine" indicator that the suspensions will likely misaligned
    H0:FMC-CS_LVEA_ZONE1A_DEGF    Temperature, in deg F, around BSC2 (think H1 Beamsplitter and ITMs)    
    H0:FMC-CS_LVEA_ZONE1B_DEGF    ", ", around 3-IFO Area (think old H2 beamsplitter)
    H0:FMC-CS_LVEA_ZONE4_DEGF     ", ", Output arm (think SR3)
    H0:FMC-CS_LVEA_ZONE5_DEGF     ", ", Input arm (think PR3)

Local, corresponding heater units
    H0:FMC-CS_LVEA_HEATER_ZONE1A_PC    percentage of heating power being applied to the zone
    H0:FMC-CS_LVEA_HEATER_ZONE1B_PC
    H0:FMC-CS_LVEA_HEATER_ZONE4_PC
    H0:FMC-CS_LVEA_HEATER_ZONE5_PC

Common HVAC Air Handler 1 -- often referred to as just "AHU1"
    These channels are is human controllable via the HVAC control system
    H1:FMC-CS_LVEA_AH_DAMPER_1_PC            "Percentage of open" Intake air reduction "damper" valve (0% full closed, 100% full open)
    H0:FMC-CS_LVEA_AH_COOLTEMP_1_DEGF        Temperature of the "Coiling Coil"
    H0:FMC-CS_LVEA_AH_COOLTEMP_2_DEGF
    H0:FMC-CS_LVEA_AH_AIRFLOW_1              Output Air flow into the LVEA in cubic feet per minute (cfm, or CFM)
    H0:FMC-CS_LVEA_AH_AIRFLOW_2

Common HVAC Air Handler 2 -- often referred to as just "AHU2"
    These channels are is human controllable via the HVAC control system
    H0:FMC-CS_LVEA_AH_DAMPER_2_PC
    H0:FMC-CS_LVEA_AH_COOLTEMP_3_DEGF
    H0:FMC-CS_LVEA_AH_COOLTEMP_4_DEGF
    H0:FMC-CS_LVEA_AH_AIRFLOW_3
    H0:FMC-CS_LVEA_AH_AIRFLOW_4

The story starts earlier than the adjustment to the air-handler units to reduce the overall cooling airflow into the LVEA. Here's the timeline of things changing, given that IFO problems with temperature started late April 2023. Here's every event.

Apr 26 2023 10:56 PDT (Wednesday, mid-morning)
    Air handler 2's damper has a step-change in behavior, going from diurnal fluctuations between 50% and 80% open, to between 60% and fully 100% open.
    This only causes a minor "glitch" change in the LVEA temperatures, mostly in the diurnal fluctuations getting "reset," changing the diurnal pattern but no overall average temperature change.

May 02 2023 07:57 PDT (Tuesday, first-thing, minutes before the official start of maintenance day)
    Air handler 2's damper, and corresponding cooling temperature 3&4 glitches for 30 minutes. For the most part, the damper returns to the normal 50% and 80% open behavior
    This only causes a minor change in the LVEA temperatures, again in the diurnal fluctuations changing their pattern but no overall average temperature change.

May 10 2023 13:46 PDT (Wednesday afternoon)
aLOG: LHO:69516
Air handler unit 2, AHU2 fails and turns off.
Channels showing this:
    H1:FMC-CS_LVEA_AH_DAMPER 2_PC         Goes all the zero closed at 0%
    H0:FMC-CS_LVEA_AH_AIRFLOW_3           Goes to zero, as airflow ceases
    H0:FMC-CS_LVEA_AH_AIRFLOW_4           Goes to zero, "
    H0:FMC-CS_LVEA_AH_COOLTEMP_3_DEGF     Goes up from stable at 50-60 deg F to really high at ~65 deg F
    H0:FMC-CS_LVEA_AH_COOLTEMP_4_DEGF

    This causes a major, a mean value excursion in LVEA temperature, but after the air handler function is restored, all zones come "back to normal" by May 11 20232 12:38 PDT, ~24 hours later.

May 11 2023 12:38 PDT
With Air Handler 1 restored, the LVEA temperatures return to normal but the airflow is now much larger, with Fan 1 producing ~12500 cfm worth of flow, where it used to only put out 6000 cfm.
Channels that show this:
    H0:FMC-CS_LVEA_AH_AIRFLOW_1
    
    This doesn't change the LVEA temperature, but means that there's a lot more air flowing into the LVEA. Maybe this is what cause Robert to pay attention?
    
    
May 17 2023 09:42 PDT (Saturday morning)
Damper for Air Handler Unit 2 damper starts to open up, constantly at 100%, and has stayed like this since, only occasionally coming off the rails to 80% in the first few days.
Channels showing this
    H1:FMC-CS_LVEA_AH_DAMPER 2_PC

    This causes a marked change in Zone 4, 0.5 deg F decrease in temperature. That's not a lot, but work calling out. The diurnal pattern of Zone 4 changes as well. Further, the "coiling coils" for AHU2 also start to run warmer increasing from (diurnal fluctuations around) 51 deg F to (diurnal fluctuations around) 53 deg F.

Then we get to where Bubba Says - "It started with trying to adjust the airflows (per Robert S.)"

May 24 2023 08:48 PDT (Tuesday maintenance)
In reference to LHO:69894, where Bubba says:

"Fans 1 & 2 [of Air Handler Unit 1] supplying the LVEA were readjusted to [reduced] airflows for the Observation Run. 
 The **total** CFM for AHU 1 was reduced from ~ 26,000 to ~ 11,200 CFM"

With a bit more detail,
 - Fan 1 (H0:FMC-CS_LVA_AH_AIRFLOW_1) is reduced from ~12500 cfm to it's level that it was prior to the May 11 2023 fix of Air Handler 1, now back to ~6000 cfm.
 - Fan 2 (H0:FMC-CS_LVA_AH_AIRFLOW_2) is reduced dramatically from the value it's been in a long time ~11000 cfm down to 4500 cfm.
Thus Bubba's statement about the total from (12500+11000 =) 23500 cfm to (6000+4500 =) ~10500 cfm.

As a result of this we land on Bubba's statement "At that time, it was discovered that cooling coil 4 was much warmer than the other coils" (from the May 17th 2023 change in damper behavior) because you see both coil 3 and coil 4 from Air Handler 2 (H0:FMC-CS_LVEA_AH_COOLTEMP_3_DEGF and H0:FMC-CS_LVEA_AH_COOLTEMP_4_DEGF) at the higher, ~53 deg F level.
   
   This dramatic airflow change causes the LVEA temperatures to merge high for a bit until May 24 18:38 PDT -- Zone 4 (think SR3), the output arm gets warmer to meet the other zones.

May 24 2023 22:30 PDT yo May 25 2023 06:52 PDT (Late Night Tuesday to Early morning Wednesday)
Air Handler 1's cooling coils are slowly stepping up in temperature (by the control system? by a human?) from 46 deg F to as much as 54 deg F.
   
   Somehow, this slowly starts to bring all temperatures *down* in the LVEA in all zones together until the next morning at May 25 2023 06:45 PDT.

May 25 2023 06:52 PDT (Early morning Wednesday)
Air handler 1's cooling coils 1 and 2 get restored back to 46 deg F, and the heater in Zone 1A kicks on for 2.5 hours. (by a the control system? by a human?) 

   This brings all temperatures *up* in the LVEA to a relatively high value of 69 deg F until another change on May 26 2023 15:56 PDT.

May 26 2023 15:56 PDT (Friday Afternoon)
Air Handler 2 Fan 4's air flow drops to zero, and cooling coil 3 and 4 temperature drops from 54 deg F to 49 deg F.

These are Bubba's actions when he says "Towards the end of the day [Friday Afternoon, May 26th] coil 4 was still much warmer than normal and allowing warm air into the LVEA. I decided to turn Fan 4 completely off to see if I could get the temps to stabilize. This strategy helped."

   This brings the temperatures collective *down* in the LVEA from ~69 deg F, to 67 deg F, closer to the "normal" values for the LVEA, though Zone 5 (think PR3) and Zone 1B (think 3IFO) are lower than their "normal" values.

May 30 2023 09:50 PDT - 11:51 PDT (Tuesday, Maintenance Day)
All hell breaks loose with the HVAC as the Fire Alarm system overrides the HVAC controls -- only really mentioned in the operator log during maintenance that day -- see LHO:70003, and drives TJ to investigate drifts in temperature, LHO:70000 thinking "it's happening again!!"

Also, in the mean while Bubba's action he mentions,
" Yesterday, being maintenance day, I took the opportunity to investigate a little more into why coil 4 was not cooling. After checking functionality of all the valves, I decided to open the strainer and found that it was very full of debris. I cleaned the strainer, reinstalled and turned Fan 4 back on. The coil is now running at normal temperature and the LVEA is coming back to a stable temperature. "

    However, upon restoration of the HVAC settings, Air Handler 2's "Fan 4 is turned off" setting from May 26 2023 (Friday Afternoon) get lost, and FAN4 comes back on at full blast at 12000 cfm. Further, Air Handler 1's cooling coils come back on much hotter than they've ever come on before, changing from 45 deg to 54 to 65 deg.

    Once AHU2 Fan 4 comes back on, and air flow to the LVEA re-increases, the temperatures in the LVEA begin to drop again, tanking down as low as 66 deg F.

May 31 2023 06:23 PDT (Early Wednesday Morning)
    Zone 4 and Zone 5 heaters come in to play (by human? or by the control system?) turning on for the first time with Zone 4 at 40 percent, and Zone 5 left to oscillate diurnally between 20% and 40%.

    This brings all LVEA temperatures back up to a really tight cluster, for the first time in months around 67 deg F. This is great, we should hold here if we can.

June 04 2023 18:34 PDT (5 days later, Sunday Evening)
For reasons I can't determine (no other visible change in the metrics I've been using), Zone 1B (think 3IFO) jumps up in temperature 0.5 deg F from 67.2 deg F to 67.7 deg F.

    This is a pretty inconsequential change in temperature that shouldn't affect suspensions.

June 06 2023 09:53 PDT (2 days later, Tuesday Maintenance Day)
    The fire alarm maintenance team is back at it, and this causes the intake damper of Air Handler 2 (H0:FMC-CS_LVEA_AH_DAMPER_2_PC) to drop to zero.

    This causes the entire LVEA temperature to rise again from its stable 67 deg period starting on May 31 2023 06:23 PDT to zones going as high as 69 deg F.

June 06 2023 11:52 PDT (same day)
    The operations team realizes the fire alarm closed the AHU2 damper again, called in facilities and re-opened it, and the temperatures come back down, but overshoot because Zone 1B and Zone 4 configuration got lost.

June 06 2023 17:02 PDT (same day, but later in the evening)
    Some one, or some thing, turns the Zone 1B and Zone 4 heater configuration back on.

    Temperatures return to the "good" May 31 2023 06:23 PDT configuration change