While testing out new SEI/ISI/HPI guardian code on ITMY this morning (to be logged separately later), Fabrice and I encountered what may be a bug in the watchdog code associated with the T240s in stage 1.
We had applied a large pitch offset on ITMY HPI, and were attempting to restore the system to full isolation. The large pitch offset rang up the T240, causing it to saturate (expected). The new T240 monitor (H1:ISI-ITMY_ST1_T240_MONITOR_OUT) kicked in, and guardian waited for the T240 monitor to clear before it attempted to start isolating stage 1. As soon as the T240 monitor cleared, guardian started to enage the isolation loops and ramp up the gains:
20140514_17:27:05.582 ISI_ITMY_ST1 [WAIT_FOR_T240_SETTLE] 20140514_17:37:28.138 ISI_ITMY_ST1 transitioning state: WAIT_FOR_T240_SETTLE->ENGAGE_ISO_FILTERS_HIGH_RX_RY 20140514_17:37:28.139 ISI_ITMY_ST1 calculating path: ENGAGE_ISO_FILTERS_HIGH_RX_RY->HIGH_ISOLATED 20140514_17:37:28.139 ISI_ITMY_ST1 new target: RAMP_ISO_FILTERS_UP_HIGH_RX_RY 20140514_17:37:28.139 ISI_ITMY_ST1 executing state: ENGAGE_ISO_FILTERS_HIGH_RX_RY 20140514_17:37:28.139 ISI_ITMY_ST1 [ENGAGE_ISO_FILTERS_HIGH_RX_RY] 20140514_17:37:28.204 ISI_ITMY_ST1 [ENGAGE_ISO_FILTERS_HIGH_RX_RY.run] ezca: H1:ISI-ITMY_ST1_ISO_RX_SW1S => 21508 20140514_17:37:28.330 ISI_ITMY_ST1 [ENGAGE_ISO_FILTERS_HIGH_RX_RY.run] ezca: H1:ISI-ITMY_ST1_ISO_RX_SW2S => 1537 20140514_17:37:28.455 ISI_ITMY_ST1 [ENGAGE_ISO_FILTERS_HIGH_RX_RY.run] ezca: H1:ISI-ITMY_ST1_ISO_RX => ONLY ON: INPUT, OUTPUT, DECIMATE, FM4, FM5, FM6, FM7 20140514_17:37:28.456 ISI_ITMY_ST1 [ENGAGE_ISO_FILTERS_HIGH_RX_RY.run] ezca: H1:ISI-ITMY_ST1_ISO_RY_SW1S => 21508 20140514_17:37:28.582 ISI_ITMY_ST1 [ENGAGE_ISO_FILTERS_HIGH_RX_RY.run] ezca: H1:ISI-ITMY_ST1_ISO_RY_SW2S => 1537 20140514_17:37:28.708 ISI_ITMY_ST1 [ENGAGE_ISO_FILTERS_HIGH_RX_RY.run] ezca: H1:ISI-ITMY_ST1_ISO_RY => ONLY ON: INPUT, OUTPUT, DECIMATE, FM4, FM5, FM6, FM7 20140514_17:37:28.889 ISI_ITMY_ST1 transitioning state: ENGAGE_ISO_FILTERS_HIGH_RX_RY->RAMP_ISO_FILTERS_UP_HIGH_RX_RY 20140514_17:37:28.890 ISI_ITMY_ST1 calculating path: RAMP_ISO_FILTERS_UP_HIGH_RX_RY->HIGH_ISOLATED 20140514_17:37:28.890 ISI_ITMY_ST1 new target: ENGAGE_ISO_BOOST_HIGH_RX_RY 20140514_17:37:28.890 ISI_ITMY_ST1 executing state: RAMP_ISO_FILTERS_UP_HIGH_RX_RY 20140514_17:37:28.891 ISI_ITMY_ST1 [RAMP_ISO_FILTERS_UP_HIGH_RX_RY] 20140514_17:37:28.954 ISI_ITMY_ST1 [RAMP_ISO_FILTERS_UP_HIGH_RX_RY.run] ezca: H1:ISI-ITMY_ST1_ISO_RX_TRAMP => 10.000 20140514_17:37:28.954 ISI_ITMY_ST1 [RAMP_ISO_FILTERS_UP_HIGH_RX_RY.run] ezca: H1:ISI-ITMY_ST1_ISO_RX_GAIN => 0.010 20140514_17:37:28.955 ISI_ITMY_ST1 [RAMP_ISO_FILTERS_UP_HIGH_RX_RY.run] ezca: H1:ISI-ITMY_ST1_ISO_RY_TRAMP => 10.000 20140514_17:37:28.955 ISI_ITMY_ST1 [RAMP_ISO_FILTERS_UP_HIGH_RX_RY.run] ezca: H1:ISI-ITMY_ST1_ISO_RY_GAIN => 0.010 20140514_17:37:28.956 ISI_ITMY_ST1 [RAMP_ISO_FILTERS_UP_HIGH_RX_RY.run] timer['isolating'] = 10.0 20140514_17:37:29.077 ISI_ITMY_ST1 state returned jump target: WATCHDOG_TRIPPED_DEISOLATING 20140514_17:37:29.140 ISI_ITMY_ST1 transitioning state: RAMP_ISO_FILTERS_UP_HIGH_RX_RY->WATCHDOG_TRIPPED_DEISOLATING 20140514_17:37:29.140 ISI_ITMY_ST1 calculating path: WATCHDOG_TRIPPED_DEISOLATING->HIGH_ISOLATED 20140514_17:37:29.140 ISI_ITMY_ST1 executing state: WATCHDOG_TRIPPED_DEISOLATING 20140514_17:37:29.140 ISI_ITMY_ST1 [WATCHDOG_TRIPPED_DEISOLATING] 20140514_17:37:29.203 ISI_ITMY_ST1 [WATCHDOG_TRIPPED_DEISOLATING.run] USERMSG: WATCHDOG TRIP: DEISOLATING (2)
Immediately after the gains turned on, the T240 watchdog tripped. However, this appeared to not be associated with any actual activity in the T240 (see attached plot of T240 trip which shows nothing).
We suspect that this must be due to a bug in the watchdog code. A recent change activates the T240 watchdog based on the state of the stage 1 isolation loops: no T240 watchdog when the isolation loops are disengaged, and yes T240 watchdog when the loops are engaged with non-zero gain. It must be something to do with this logic. Fabrice and I are investigating.
The problem comes from the saturation counter that is not reseted:
- the T240 saturation flag turns on when we apply the HEPI offsets
- the saturation counters reach the treshold value (10 saturations)
- but the watchdog doesn't trips because the T240 are not in loop and therefore ignored by the top layer of the WD code. That's a feature we recently included so that we can turn on hepi without triping the ISI.
- as a consequence the saturation counters never gets reset
- so the WD trip when the T240 isolation is turned on, because the saturation counter flag is still on
Jamie and I have a plan to fix this. We need to modify the WD code and the simulink model. Dave says we can add those chances to work permit #4626.