Verbalalarms continues to complain about SR3 and OMC suspensions, off and on. So, a little investigations led me to notice that H1:SUS-SR3_M1_OSEMINF_RT_INMON signal is decaying. So, Austin and I were trying to figure out what was happening on the 14th of December by checking the alog. We saw that there was reboot of the front end computers. This is visible in the plot below as well. We are not sure what or why is SR3 RT signal decaying. It could be that the OSEM is problematic.
No idea why verbalalarms is complaining about OMC, we didn't find anything that could trigger the alarm.
Betsy suggested to trend the corner pressure with this signal. So, I found PT-120B in BSC2 that is closest to SR3 in HAM5. The attached images show the H0:VAC-LY_Y1_PT120B_PRESS_TORR pressure and H1:SUS-SR3_M1_OSEMINF_RT_INMON signal. Note the second image is zoomed in to Dec. 14, 2021, which is when the signal in SR3_M1_RT starts to decay.
After consulting with Rahul, I am including the plots of INMON and DAC OUTPUT for SR3 and OMC. These show data from Dec. 1-21, 2021. For SR3, it looks like LF and RT INMON and DAC_OUTPUT signals are not the same after Dec 14, 2021. Also, RT_INMON signal has dropped to and looks like it is leveling off to ~2000.
When looking in GDS TP for both SUS in question here, there are overflows in DAC 0 (CH 4) for SR3 and DAC 1 (CH 4) in OMC. This is when verbalalarms sends out an alert.
Austin pointed out yesterday that there is an FRS Ticket 21444 open for TIM errors in h1iopsush56 by Dave (tagging CDS). SR3 and OMC are under the computer sush56 (thank you Austin for generaing the Computer-Model-Suspension Map). I do not know anything about TIM errors to say that that could be the reason.
Anyways, more investigation is needed here by experts.
Georgia suggested to look at the SR3 heater and SR3 cage servo settings; in case those were turned on in the restart. So, from SITEMAP > TCS > ACTIVE WAVEFRONT CONTROL, the I and P MONs are zero and V MON is fluctuating in couple of mV. SR3_CAGE_SERVO is INIT state; its nominal state is CAGE_SERVO_OFF. So, not sure if that is the reason. To see if temperature inside the chamber is the reason, I trended the H1:PEM-CS_TEMPERATURE_BSC1_ITMY_MON. It looks like none of these could be the reason why SR3 LF and RT signals are noisy and RT INMON signal dropped to ~2000. Last thing we checked was changing the state of SR3 from ALIGNED to DAMPED to SAFE to back to ALIGNED. See the figure below. The noise in the DAC_OUTPUT goes to zero when SR3 is in SAFE state.
Opened corresponding FRS Ticket 21820, and tagging FRS.
H1:SUS-SR3_M1_OSEMINF_RT_INMON has now gone up to 9-10k as shown in the ndscope below, although it still looks very noisy. I wonder if this issue was due to ongoing pump down and relative change in the pressure (pressure plot is also shown below, currently showing 0.5*10^-6 torr). SR3 damping loops are working fine and SDF table is also clean.
