Overnight the end station BRS enclosures at both endstations heated up about .5 degrees C. This very nearly drove BRSX out of range, which would have probably tripped the ISI. This wasn't a big issue for BRSY, but BRSX went through 8k counts of the ~28k usable range, it probably also has an effect on the noise of the BRS. The damping for the BRS also goes unstable when the ccd image goes out of range.
Attached plot shows trends for the last week of each BRS driftmon, X is the left image, Y is the right. The BRS go out of range at about +/- 15000 counts, BRSX will need adjusted immediately, unless temperatures are changing back.
Was there some change to building heating or something? Nothing was changed on either BRS enclosure.
All 4 quads also see movement over the same time period. Attached trends are the last week of the quad topstage osems.
The FMCS controlled set point was not changed at either end station.
We've seen a similar issue before with the ETM temperature. It stems from the fact that the HVAC is stabilizing the average temperature of 4 temperature sensors: (A+B+C+D)/4.
The ETM chamber is down at one end of the building (near A & B). If, for example, the local temperature rises at the other end of the building (near temperature sensors C&D) then the HVAC reacts to keep the average temperature constant. A little bit of math shows that if C+D becomes hotter than A+B must become cooler. (Or it could be that the region near A is getting cooler and forcing C+D to get hotter - which is probably more likely given the current weather).
You can see that the ETM BRS DRIFTMON is much more strongly correlated with individual temperature sensors than with the average temperature. Stabilizing the VEA temperature to (A+B)/2 should minimize this issue.
Here's similar data from the LLO ETM - the RH RTD is much more strongly correlated with the 202B temperature sensor than it is with the average temperature.
I've done a quick rebalance of BRSX to get it closer to the middle of it's range, it's still damping down and temperatures will take a little while to stabilize. We may want to think about some weekly-ish checks of things that can get driven by temperature swings? I don't know what that would look like.
