While Betsy, Ibrahim and I were in BSC2 to work on the BBSS, we noticed that ITMY started shaking significantly. To give you a sense of the motion, we could see the UIM moving while sitting on the floor next to the BBSS. Betsy called the control room, and no measurements were happening. This happened a few times until Corey put ITMY in SAFE.
I came out and began to investigate the issue with Oli. I tried putting ITMY back in ALIGNED, which turns on top mass damping and alignment offsets for the main and reaction chain as well as the L2 to R0 wit damping. After a few minutes, we start seeing saturations and large motion. At first I was suspicious of the L2 to R0 damping, so I turned it off, but the problem persisted. I then turned off all R0 damping, still the same problem. We put ITMY back in SAFE and the oseminfs of the main and reaction chain still show oscillation, but it seems to be stable or slowly damping down. There are also small kicks that the osems see. We're wondering if the in-chamber work is causing ITMY to move excessively.
We put ITMY back in aligned again and the excess motion damped down, as expected. However, we're still seeing some random saturation on R0 F3, and we're not sure why. The alignment offsets are the same that they've always been. Trending back to the start of the vent, it looks like ITMY R0 F3 may have always been occasionally saturating.
Whatever the problem was, it has stopped for now. Oli is looking into why the watchdogs didn't trip.
Here's what the watchdogs looked like for the M0 and R0 osems during the first round of being rung up (when they were rung up for ~40 mins before we put them in SAFE the first time). The watchdog value for R0 SD, which was the osem that was moving the most by far, was steady around 43um. The trip threshold for R0 is 200um. The reason why the watchdog value was so low even though it was moving so much is because the flag for SD was completely obscuring the light, as suggested by the OSEMINF OUT values. SD was saturating almost constantly. The light being completely blocked and saturating on SD also messes up the T damping loop, which could've made everything worse. The rest of the R0 osems were also saturating, but not as often, and the M0 osems were barely saturating, if at all.
We have no idea why R0 was pushed so far into the SD coil driver.
We also don't know if the suspension would have tripped if the SD OSEM would've been able to see the movement, but it definitely should have. It would be good to find a way to have the suspension trip if it suddenly can't 'see' the flag (either fully out or in), but that can be hard to do since we don't want the suspension tripping every time an osem saturates for a split second.