Sheila, Hugh
IFO lost lock ~1426utc, Richard suspects it was his testing of the ESD. Almost 45 minutes later, the ISCINF_LONG stepped to its rail, LIMIT set to 250000nm. And I mean stepped, ~1 second or less. This of course put a bit much demand on the HEPI loop. It spiked its output and pretty much instantly tripped the watchdog on the Actuator followed by the ISI shortly thereafter. The HEPI and ISI reisolated normally.
The first attached plot shows the lock loss. This 30 minute trend has the DARM OUT with the ISCMON_X_INMON quickly going to zero and the HEPI Outputs moving in response. No problems here, HEPI does fine. You also see the arm or something grab again as you can see the Tidal drive to the HEPI start again but this only lasted for a couple minutes.
The second plot attached zooms out to 2 hours, with the IFO lock loss seen on the ISCMON on the left. ~45 minutes later the DARM steps, this time stepping the ISCMON and tripping HEPI and about 8 seconds later, the ISI (not shown.)
In the third attachment, Sheila dug around in LSC/ALS space and put together some trends showing the spike into HEPI originating when the ALS-X_LOCK_STATE steps from level 2 to 6 (PLL Lock to Red Lock.) This is unrealistic as the Lock state dropped from Red Lock just ~90 seconds earlier. Notice the TIDAL_CTRL_INMON steadily growing (DARM not Cleared.)
Sheila is looking deeper into the Guardian feeling it may have failed to do something it possibly should have to prevent the DARM OUT from going to the rail and the unreasonable rapid transitions of the ALS LOCK STATE.
There were two things in place that should have prevented this but didn't.
- First of all the guardian should have noticed that the lock was lost and gone to the down state, which clears the DARM drive to the ETM, so that the input to the tidal on ETMX sus should be zero, and therefore the input to HEPI tidal should have been zero. The guardian was in the state DRMI_ON_POP, which had a typo from some efforts to incorporate new functionality. I assume that the guardian didn't go to down because it would have been in error with this typo.
- Second, the logic in the end station state machine in beckhoff should turn off the tidal feedback when the arm is not locked. It did this, which is why HEPI didn't trip right away despite the guardian problem. However, the new Red Lcoked state can be entered when there is just a flash of red in the arms, without passing through the other states, which it should not be doing. This can be fixed, hopefully on tuesday.