Reports until 14:14, Thursday 01 December 2016
H1 SEI
hugh.radkins@LIGO.ORG - posted 14:14, Thursday 01 December 2016 - last comment - 15:43, Thursday 01 December 2016(32079)
WHAM3 ISI Corner2 CPS Glitches break IFO lock--Sat Rack Power Cycled and Satellite Cards Reseated

See the first attached, showing the ISC lock state (Ch16) dropping as the HAM3 ISI WD (Ch1) trips to State3 (Damping only.)

All the CPSs show a twitch and shift as the DC coupled ISO loops disengage but note the scale of the glitch on H2 and V2 CPSs.  What is curious to me is how the C2 sensors clearly exceed saturation level (20000) but the SAT_COUNT (Ch2) does not count them.  Ahhh, I think that any 'trip' of the watchdog, even if only to damping only, will clear the saturations.  I think we should change this in the code although I think someone coded it explicitly this way (BTL/HRAP.)  I think this hinders us though and maybe we can rework it.

I decided to Power Cycle the CPS and while off reseated the corner2 Gauge Boards in the satellite rack.  Did this several times.  ISI reisolated without difficulty.

Attached as well is another view a few hours ago of this glitching dropping the IFO out of some state better than zero.  Prior to the ISI trip is another glitch on the Corner2 CPS that was not enough to rile things up.

Finally, a 7 hour trend capturing all the recent occurrances of this.  Prior to this, 22 Nov saw a CPS saturation and trip but that was a Tuesday.

Images attached to this report
Comments related to this report
brian.lantz@LIGO.ORG - 15:43, Thursday 01 December 2016 (32083)
Hugh - 
Recall that we added the channel
H1:ISI-HAM3_WD_CPS_SAT_SINCE_RESTART
which can probably do what you want. It will have some big number in it, but should only count up, the delta will be what you want. 
I think we added this just for you, so merry christmas!
-Brian