Adding plots of LVEA temperature over the last 10, 16 and 30 days. With cross-hairs showing dates temperatures changed. This plot is available with command: ndscope /opt/rtcds/userapps/release/cds/h1/scripts/fom_startup/nuc32/VEA_temperatures.yaml

Adding some quantitative time and information to this aLOG series.

First, I cover the HVAC system and the channels that I'm using for metrics of the LVEA HVAC system.
The LVEA has 2 giant cooling actuators, or "air handlers" that live in the Mechanical Room, +X -Y of the beam splitter.
These air handlers add together to cool the LVEA as a common actuator.
There are separate, local *heaters* in each zone of the LVEA that serve as "differential" actuators. 

The system is depicted pretty well in the MEDM overview screen found under sitemap > "FMCS OVERVIEW" > "Air Handler 1,2"

The channels that are useful metrics of the system are listed below with their function and units
External, outside the building temperature
    H1:PEM-CS_TEMP_ROOF_WEATHER_DEGF       Temperature on the LVEA building roof, in deg F (look for corresponding "DEGC" channels for deg C)

LVEA temperature sensors << our primary "canary in the coal mine" indicator that the suspensions will likely misaligned
    H0:FMC-CS_LVEA_ZONE1A_DEGF    Temperature, in deg F, around BSC2 (think H1 Beamsplitter and ITMs)    
    H0:FMC-CS_LVEA_ZONE1B_DEGF    ", ", around 3-IFO Area (think old H2 beamsplitter)
    H0:FMC-CS_LVEA_ZONE4_DEGF     ", ", Output arm (think SR3)
    H0:FMC-CS_LVEA_ZONE5_DEGF     ", ", Input arm (think PR3)

Local, corresponding heater units
    H0:FMC-CS_LVEA_HEATER_ZONE1A_PC    percentage of heating power being applied to the zone
    H0:FMC-CS_LVEA_HEATER_ZONE1B_PC
    H0:FMC-CS_LVEA_HEATER_ZONE4_PC
    H0:FMC-CS_LVEA_HEATER_ZONE5_PC

Common HVAC Air Handler 1 -- often referred to as just "AHU1"
    These channels are is human controllable via the HVAC control system
    H1:FMC-CS_LVEA_AH_DAMPER_1_PC            "Percentage of open" Intake air reduction "damper" valve (0% full closed, 100% full open)
    H0:FMC-CS_LVEA_AH_COOLTEMP_1_DEGF        Temperature of the "Coiling Coil"
    H0:FMC-CS_LVEA_AH_COOLTEMP_2_DEGF
    H0:FMC-CS_LVEA_AH_AIRFLOW_1              Output Air flow into the LVEA in cubic feet per minute (cfm, or CFM)
    H0:FMC-CS_LVEA_AH_AIRFLOW_2

Common HVAC Air Handler 2 -- often referred to as just "AHU2"
    These channels are is human controllable via the HVAC control system
    H0:FMC-CS_LVEA_AH_DAMPER_2_PC
    H0:FMC-CS_LVEA_AH_COOLTEMP_3_DEGF
    H0:FMC-CS_LVEA_AH_COOLTEMP_4_DEGF
    H0:FMC-CS_LVEA_AH_AIRFLOW_3
    H0:FMC-CS_LVEA_AH_AIRFLOW_4

The story starts earlier than the adjustment to the air-handler units to reduce the overall cooling airflow into the LVEA. Here's the timeline of things changing, given that IFO problems with temperature started late April 2023. Here's every event.

Apr 26 2023 10:56 PDT (Wednesday, mid-morning)
    Air handler 2's damper has a step-change in behavior, going from diurnal fluctuations between 50% and 80% open, to between 60% and fully 100% open.
    This only causes a minor "glitch" change in the LVEA temperatures, mostly in the diurnal fluctuations getting "reset," changing the diurnal pattern but no overall average temperature change.

May 02 2023 07:57 PDT (Tuesday, first-thing, minutes before the official start of maintenance day)
    Air handler 2's damper, and corresponding cooling temperature 3&4 glitches for 30 minutes. For the most part, the damper returns to the normal 50% and 80% open behavior
    This only causes a minor change in the LVEA temperatures, again in the diurnal fluctuations changing their pattern but no overall average temperature change.

May 10 2023 13:46 PDT (Wednesday afternoon)
aLOG: LHO:69516
Air handler unit 2, AHU2 fails and turns off.
Channels showing this:
    H1:FMC-CS_LVEA_AH_DAMPER 2_PC         Goes all the zero closed at 0%
    H0:FMC-CS_LVEA_AH_AIRFLOW_3           Goes to zero, as airflow ceases
    H0:FMC-CS_LVEA_AH_AIRFLOW_4           Goes to zero, "
    H0:FMC-CS_LVEA_AH_COOLTEMP_3_DEGF     Goes up from stable at 50-60 deg F to really high at ~65 deg F
    H0:FMC-CS_LVEA_AH_COOLTEMP_4_DEGF

    This causes a major, a mean value excursion in LVEA temperature, but after the air handler function is restored, all zones come "back to normal" by May 11 20232 12:38 PDT, ~24 hours later.

May 11 2023 12:38 PDT
With Air Handler 1 restored, the LVEA temperatures return to normal but the airflow is now much larger, with Fan 1 producing ~12500 cfm worth of flow, where it used to only put out 6000 cfm.
Channels that show this:
    H0:FMC-CS_LVEA_AH_AIRFLOW_1
    
    This doesn't change the LVEA temperature, but means that there's a lot more air flowing into the LVEA. Maybe this is what cause Robert to pay attention?
    
    
May 17 2023 09:42 PDT (Saturday morning)
Damper for Air Handler Unit 2 damper starts to open up, constantly at 100%, and has stayed like this since, only occasionally coming off the rails to 80% in the first few days.
Channels showing this
    H1:FMC-CS_LVEA_AH_DAMPER 2_PC

    This causes a marked change in Zone 4, 0.5 deg F decrease in temperature. That's not a lot, but work calling out. The diurnal pattern of Zone 4 changes as well. Further, the "coiling coils" for AHU2 also start to run warmer increasing from (diurnal fluctuations around) 51 deg F to (diurnal fluctuations around) 53 deg F.

Then we get to where Bubba Says - "It started with trying to adjust the airflows (per Robert S.)"

May 24 2023 08:48 PDT (Tuesday maintenance)
In reference to LHO:69894, where Bubba says:

"Fans 1 & 2 [of Air Handler Unit 1] supplying the LVEA were readjusted to [reduced] airflows for the Observation Run. 
 The **total** CFM for AHU 1 was reduced from ~ 26,000 to ~ 11,200 CFM"

With a bit more detail,
 - Fan 1 (H0:FMC-CS_LVA_AH_AIRFLOW_1) is reduced from ~12500 cfm to it's level that it was prior to the May 11 2023 fix of Air Handler 1, now back to ~6000 cfm.
 - Fan 2 (H0:FMC-CS_LVA_AH_AIRFLOW_2) is reduced dramatically from the value it's been in a long time ~11000 cfm down to 4500 cfm.
Thus Bubba's statement about the total from (12500+11000 =) 23500 cfm to (6000+4500 =) ~10500 cfm.

As a result of this we land on Bubba's statement "At that time, it was discovered that cooling coil 4 was much warmer than the other coils" (from the May 17th 2023 change in damper behavior) because you see both coil 3 and coil 4 from Air Handler 2 (H0:FMC-CS_LVEA_AH_COOLTEMP_3_DEGF and H0:FMC-CS_LVEA_AH_COOLTEMP_4_DEGF) at the higher, ~53 deg F level.
   
   This dramatic airflow change causes the LVEA temperatures to merge high for a bit until May 24 18:38 PDT -- Zone 4 (think SR3), the output arm gets warmer to meet the other zones.

May 24 2023 22:30 PDT yo May 25 2023 06:52 PDT (Late Night Tuesday to Early morning Wednesday)
Air Handler 1's cooling coils are slowly stepping up in temperature (by the control system? by a human?) from 46 deg F to as much as 54 deg F.
   
   Somehow, this slowly starts to bring all temperatures *down* in the LVEA in all zones together until the next morning at May 25 2023 06:45 PDT.

May 25 2023 06:52 PDT (Early morning Wednesday)
Air handler 1's cooling coils 1 and 2 get restored back to 46 deg F, and the heater in Zone 1A kicks on for 2.5 hours. (by a the control system? by a human?) 

   This brings all temperatures *up* in the LVEA to a relatively high value of 69 deg F until another change on May 26 2023 15:56 PDT.

May 26 2023 15:56 PDT (Friday Afternoon)
Air Handler 2 Fan 4's air flow drops to zero, and cooling coil 3 and 4 temperature drops from 54 deg F to 49 deg F.

These are Bubba's actions when he says "Towards the end of the day [Friday Afternoon, May 26th] coil 4 was still much warmer than normal and allowing warm air into the LVEA. I decided to turn Fan 4 completely off to see if I could get the temps to stabilize. This strategy helped."

   This brings the temperatures collective *down* in the LVEA from ~69 deg F, to 67 deg F, closer to the "normal" values for the LVEA, though Zone 5 (think PR3) and Zone 1B (think 3IFO) are lower than their "normal" values.

May 30 2023 09:50 PDT - 11:51 PDT (Tuesday, Maintenance Day)
All hell breaks loose with the HVAC as the Fire Alarm system overrides the HVAC controls -- only really mentioned in the operator log during maintenance that day -- see LHO:70003, and drives TJ to investigate drifts in temperature, LHO:70000 thinking "it's happening again!!"

Also, in the mean while Bubba's action he mentions,
" Yesterday, being maintenance day, I took the opportunity to investigate a little more into why coil 4 was not cooling. After checking functionality of all the valves, I decided to open the strainer and found that it was very full of debris. I cleaned the strainer, reinstalled and turned Fan 4 back on. The coil is now running at normal temperature and the LVEA is coming back to a stable temperature. "

    However, upon restoration of the HVAC settings, Air Handler 2's "Fan 4 is turned off" setting from May 26 2023 (Friday Afternoon) get lost, and FAN4 comes back on at full blast at 12000 cfm. Further, Air Handler 1's cooling coils come back on much hotter than they've ever come on before, changing from 45 deg to 54 to 65 deg.

    Once AHU2 Fan 4 comes back on, and air flow to the LVEA re-increases, the temperatures in the LVEA begin to drop again, tanking down as low as 66 deg F.

May 31 2023 06:23 PDT (Early Wednesday Morning)
    Zone 4 and Zone 5 heaters come in to play (by human? or by the control system?) turning on for the first time with Zone 4 at 40 percent, and Zone 5 left to oscillate diurnally between 20% and 40%.

    This brings all LVEA temperatures back up to a really tight cluster, for the first time in months around 67 deg F. This is great, we should hold here if we can.

June 04 2023 18:34 PDT (5 days later, Sunday Evening)
For reasons I can't determine (no other visible change in the metrics I've been using), Zone 1B (think 3IFO) jumps up in temperature 0.5 deg F from 67.2 deg F to 67.7 deg F.

    This is a pretty inconsequential change in temperature that shouldn't affect suspensions.

June 06 2023 09:53 PDT (2 days later, Tuesday Maintenance Day)
    The fire alarm maintenance team is back at it, and this causes the intake damper of Air Handler 2 (H0:FMC-CS_LVEA_AH_DAMPER_2_PC) to drop to zero.

    This causes the entire LVEA temperature to rise again from its stable 67 deg period starting on May 31 2023 06:23 PDT to zones going as high as 69 deg F.

June 06 2023 11:52 PDT (same day)
    The operations team realizes the fire alarm closed the AHU2 damper again, called in facilities and re-opened it, and the temperatures come back down, but overshoot because Zone 1B and Zone 4 configuration got lost.

June 06 2023 17:02 PDT (same day, but later in the evening)
    Some one, or some thing, turns the Zone 1B and Zone 4 heater configuration back on.

    Temperatures return to the "good" May 31 2023 06:23 PDT configuration change