This trip has just happened again.  But, this time we aren't able to fix it.  Sheila went to the CER to look, and all the lights on the coil driver chassis are green.  No overtemps are on, and fuse status is green.  We're calling Hugh right now, but if anyone sees this and knows how to fix it, please give the control room a call.

We tracked down the BS ISI trip to a single bit flip in the h1seib2 DIO_0 card, corresponding to the BS ISI ST1 V3 coil driver:

H1:ISI-BS_BIO_IN_CD_ST1_V3_STATUS

This bit being true was causing the ST1 watchdog to be permanentaly tripped.

We tried without success to clear the problem with the following:

• power cycling the relevant BIO chassis
• power cycling the relevant coil driver chassis
• power cycling the entire h1seib2 IO chassis, which involved:
  • shutting down all h1seib2 models
  • taking h1seib2 FE out of Dolphin
  • power FE down
  • power IO down
  • power AI down (to protect again DAC high output on restart)
  • power IO up
  • power FE up
  • power AI up

None of that cleared the bad bit.  Eventually Sheila checked the binary output from the coil driver and determined that everything looked good.  This gets the coil driver off the hook, narrowing the problem down to the BIO chassis or the BIO card itself in the IO chassis.

Given that we determined there was no actual problem with the coil driver, we decided to just bypass the problem by using the H1:ISI-BS_BIO_IN_BIO_IN_TEST EPICS channel to add in a value to the BIO input that would flip the single bad bit back to the desired value.  This solves the problem for the time being

Once the BIO chassis/card is fixed, the H1:ISI-BS_BIO_IN_BIO_IN_TEST channel will need to be set back to zero (0).  See the "BIO" screen from the ISI BS overview screen.

Unfortunately this bad bit is actually periodically flipping, and not staying flipped, so our test sum-in hack solution is not working.  We've had maybe two more locklosses caused by these bit flips.