Numerous entries of the last few days mention the BRS X being down.
First notice is LHO aLog 31792 when PEM went to EndX. A couple hours later Jim mentioned that it was very rung up and he disabled the damping. On Thursday Jim worked on the BRSX but was unsuccessful at getting it to damp reliably. On Saturday and Sunday, TJ logged that the BRS was ringing down--it was but just passively.
This morning after reenabling the Damping, it was going nowhere. The Damping neither helped nor rung it up further so I'm unsure what it was doing.
After IFO dropped lock, remote logged into the BRS computer and looking at the encoder file showed it contained gibberish. At the End Station, opened up the box and saw the Damper Masses sitting at the 0 position. I thought I had cleared out the gibberish in the Encoder file but upon restart, the damper just started 360s, this might be what was going on for the past few days.
With Krishna on the phone and some guidance that needs updating, I managed to clear the encoder file and reset the damper encoder angle. Once this was successfully done, the damper quickly did its job and got things under control.
For the operator--see T1600103 for some troubleshooting. After today's lesson, this needs some updating, and it contains nothing about the Encoder file problem. This is on my to-do list.
When I went to the end station on Wednesday, the damper table wasn't moving at all because it had been disabled. When I came in on Thursday, I got it to turn on once, and it seemed to servo normally for a while, but then it stopped and I couldn't recover it. It may have gone crazy on Wednesday afternoon while Anna-Maria and Robert were down there, but it wasn't moving at all after Wednesday night.
Jim's observation is not inconsistent with our conclusion that the problem was with the corrupted encoder file, I think. What I am more curious about is what caused this corruption in the first place. After the damper was upgraded, the last time we had a similar problem was when we had a series of restarts of the Beckhoff computer (see 29871). It would be interesting to look at the history of BRS status bits over the last ~2 months to see if we can get a clue.
Attached trends are all of the "bit" channels I could find associated with BRSX. First trend is the last 10 days, second is the last 90. Where the AMPBIT shows excursions away from 1 lines up with the two times when the encoder file has been corrupted. If this is diagnostic, maybe we could monitor this in the DIAG guardian.