Displaying report 1-1 of 1.
Reports until 08:16, Sunday 12 January 2020
H1 PSL (PSL)
corey.gray@LIGO.ORG - posted 08:16, Sunday 12 January 2020 - last comment - 05:12, Wednesday 15 January 2020(54445)
Another Lockloss And Another Case of FSS Oscillating/Not Locking

H1 had a 4hr lock and this was with sustained winds near 20mph, and then there was a random lockloss.  

Once again, FSS would not lock and the PSL_FSS said it was due to "FSS Oscillating".  (most recent alog of this I see is from June 2018)  

I did the same thing I did earlier in the shift for this:  toggled FSS autolocker, took ISC_LOCK to INIT/DOWN, took IMC_LOCK to INIT/LOCKING (& DOWN).  Then I took PSL_FSS to INIT/DOWN (then FSS locked).  This time it took 8min vs 20min after the last lock.

I marked this down time as CORRECTIVE MAINTENANCE since this isn't normal.  (although Ed has just walked in, he mentioned addressing this before and Jason mentioned to him that there was a special way to "toggle" the FSS autolocker....so don't turn it off and wait a while.  Just turn off and then on....but I'm pretty sure I tried that and other variants.)

Anyway, FSS during my shift wasn't happy.  Am curious to know if this is a trend.

Comments related to this report
jason.oberling@LIGO.ORG - 09:29, Monday 13 January 2020 (54467)OpsInfo, PSL
Link

Not only do you have to toggle the autolocker off/on in rapid succession (click OFF then immediately click ON), when you perform the toggle is important as well.  If it's having trouble relocking it will get to a point where it wants to grab, but for some reason can't (you'll see the 00 flashes on the PSL quad display, upper-left quadrant).  You want to toggle the autolocker during this state, when it should be grabbing a 00 mode but isn't.  If the autolocker is running through its temp search (seen by a triangular shape to the NPRO crystal temp graph on the FSS MEDM) then toggling the autolocker will do absolutely nothing.

There have been a few instances of the FSS RefCav having issues relocking in the last few days (a couple on Friday, and the two reported by Corey).  This seems to be a growing trend, but oddly enough we very rarely have problems getting the RefCav to lock when we perform work on it.  I suspect there may be issues with the archaic FSS RefCav autolocker code; it was written in one of the variants of C many years ago and hasn't been touched since.  I'll formalize this in a Wiki page, but here are some pointers if the autolocker continues to have problems (in no particular order):

  • Pause the PSL_FSS guardian node
    • This is the code that detects an oscillation in the FSS and drops then slowly raises the FSS Common Gain to attempt to clear the oscillation
    • If this code is yanking the common gain around when the FSS is trying to lock (which happens), it can greatly increase the time it takes to lock the RefCav
    • Be sure to re-enable this node once the FSS RefCav has locked
  • Look at the 1st loop ISS diffracted power, turn it OFF if it's not sitting at a steady value
    • Upon a lockloss and the sudden turning off of the 2nd loop ISS, the 1st loop ISS can have some residual "anger issues" that take some time to settle out
      • If the diffracted power is moving around, I've seen this increase the time it takes the RefCav to lock
    • Press the OFF button under Autolock in the ISS MEDM screen; the autolock indicator and the 1st loop indicator above it should both go red; if the 1st loop indicator doesn't go red, open Manual Mode (under the Autolock) and press the red OFF button in the small window that opens; when OFF, the diffracted power sits at ~2.8% and the graph above it is a flat line
    • Be sure to re-enable the ISS 1st loop once the RefCav has locked (press ON under Autolock in the ISS MEDM screen)
  • As metioned above, toggle the FSS autolocker off/on
    • Do this only when the FSS is attempting to grab a 00 mode (you'll see the flashes on the PSL quad display)
      • Done at other times it will not help (it also will not hurt, so don't worry if you don't time the toggle correctly)
    • The FSS will generally grab on the first toggle, but I have seen cases when it doesn't; if this happens, just toggle it again.
camilla.compton@LIGO.ORG - 10:41, Monday 13 January 2020 (54469)
Link

I also had the FFS not locking on Tuesday (alog 54329), the wind was very high at this time though I don't know if that could have any effect. 

How would you suggest pausing the PSL_FFS code? Will taking it to INIT do this as it is Managed by ISC_LOCK.

corey.gray@LIGO.ORG - 05:12, Wednesday 15 January 2020 (54515)OpsInfo, PSL
Link

Thanks for the notes about this, Jason---this helps alot!  It was the middle of the night, and I totally did not want to phone you for something like this.  I did think I tried different types of toggling of the Autolocker, but I definitely did not correlate toggling with looking at the RefCav flashing modes.  

Camilla:  (Correct me if I'm wrong Jason) One can PAUSE the PSL_FSS node (and any node) by going to the node's Main Screen & for the "OP" pull-down, click that and select PAUSE (other choices are STOP & EXEC).  I imagine we can do that even if PSL_FSS is MANAGED?

Displaying report 1-1 of 1.