I caused a HEPI trip this evening, messing with arm cavity slow feedback to HEPI.
While trying to restet this, I once again became annoyed by how the isolate script sets the current setpoint to something other than the target setpoint for Ry and RZ. Forgetting this is part of why it took us several hours to recover from the hepi pump trip this afternoon, and it seems like by continuing to have the script do this we are shoting ourselves in the foot. I have heard that the script does this because of fear that HEPI can't handle these large offsets, but we have been using these large target positions for more than a month, so it seems HEPI can handle them. If that's true, is it possible to fix this script?
Having a closer look at the offsets, I noticed that the location was not coming back to the set point even after waiting a long time. I attached a plot of the target location over the last 60 days, it has really never been at the set point.
AS far as I understood, this after noon after our long struggle to get ETMX back after the hepi pump trip, Jim got it back one DOF at a time using 250mHz blends on stage 1, Tcrappy on stage 2, and using the 60 Hz notch in the stage 2 controllers (which is for level 3 but while we are still using level 2 controlers we need to engage FM5 by hand). (correct me if I'm wrong Jim)
Instructions for what worked for me now are:
HEPI:
untrip watchdog
Comands 2, Isolate level 1
Important! Check that DC bias current setpoints match the target setpoints.
ISI:
Set T240 watchdog thresholds to something large
Set stage 1 blends to 250, isolate level 3.
Wait a long time, get distracted by other work, come back in a half hour....
change stage 1 blends to Tcrappy
Isolate stage 2 level2 with Tcrappy
All set!
For the record, it's not that *HEPI* can't handle the large offsets, it's that the Stage 1 T240s on the BSC-ISI can't handle the large offset without saturating, and therefore tripping the ISI's watchdog. The seismic group has heard this complaint loud and clear and is considering if they even need to include the T240s in the watchdog system. We promise we're trying to make it better!
I forgot to say, the reason I was messing with the feedback from ISC to HEPI in the first place was that Brett and I noticed while he was making his measurement that the tidal relieve servo was unstable....
I will try to fix this in the morning.
Like Jeff said, we limit the transfer of the Target of HEPI in the Current Setpoint to 500 nrad to prevent saturating the ISI's T240s. One has then to copy the targets into the current setpoint manually if he wants to override this. We could consider removing this limit if we increased the T240 WD thresholds on the ISI at the beginning of the turn on process (starting with HEPI), and had the T240s removed from the blend when turning the ISI on.
Sheila mentioned another interesting behaviour that we noticed yesterday: even when the Target and Current Setpoint agree, the position on RY does not go all the way to the setpoint value along RY on ETMX. (See attached plot, and Cart Bias MEDM screen). It is not the case on ITMY and ITMX HEPI, but none of them request such high target values (see attached Cart Bias MEDM sceeens for HEPI ITMX and ITMY). One could argue that parts of HEPI may start contacting (each HEPI has a slightly different Range Of Motion which depends on how it was set up) and prevent the servo loops from pushing HEPI to the requested Current Setpoint. This would probably make the position loops go unstable though, and it is not the case. As a side-test, we increased the value of the current setpoint on RY to see if the location would follow and it does, which indicates that HEPI is not stuck contacting.
This oddity has been going on for 60 days, but has also remained consistent throughout, even though HEPI, and its pumps, got restarted more than once. HEPI alignment along RY, once controlled, has not changed for those 60 days.
As far as having trouble turning the ISI on, the shift between the floating, and requested controlled position are quite high. We tried making this smaller, which made the ISI turn on way easier, and request the needed alignment to SUS, which seems to have worked. Should we do it this way from now on, for the sake of making the ISI turn on process more repeatable on ETMX?
