the interferometer locked by itself after we left this morning. It stayed locked for 10 hours (until the 6.4 EQ in Tonga a couple hours ago).
Some troubles re-locking:
- arms locked green sort of OK although the ETMs are still swinging too much and there is lots of HOM green locks. The ALS unlocks drive up the ETMs; the damping settings ought to be made stronger for this part of the locking. ETMX in particular is kicked much more than ETMY.
- ALS DIFF breaks the lock too much. Also there is something not right about ALS WFS scripting. The gain ramping or sequencing is off and often it comes on too slow to keep the y-arm locked.
- sometimes there are Guardian thread exceptions related to channel access. I checked that these channels are responsive using 'caget' from terminal. Errors from log attached. Looking through the logs its looks like this is happening on many of the attempts to go DOWN. IF this is not caught by a person, it could mean that the Guardian is not putting us back into the correct state for initial locking. Since the SDF is not up to date, sometimes our acquisition problems could be due to this. Perhaps we need some error flag (besides the error logs with a zillion lines of messages) if the Guardian fails to make its connections?
Rana, Hang
Over the last 4 hours, the ALS DIFF turn on broke the lock dozens of times; we've modified the DARM loop to compensate.
We looked at the time series to diagnose the problem. There was a ~2 Hz oscillation in DARM at the turn on. At this point, according to my old log entry (and confirmed by sweeps today), the L1/L3 xover is ~1.2 Hz and the bottom of the DARM-DIFF phase bubble is 2 Hz. So it seems like when the alignment is bad (i.e. DIFF beatnote is ~3 dB low) we get a 2 Hz transient when the DARM gain is ramped on.
In the script there was a 20 dB filter ramping over 5 seconds and the filter module gain ramping over 2 seconds; both were happening simultaneously. We have now commented out the filter and just ramp the gain. WE increased the gain in this step by 3 dB so that it is more resistant to high wind, useism, EQ, etc. Also changed the 2 Hz integrator in DARM2 into a 0.2:1 boost to get more phase at 2 Hz; there is already an integrator in the SUS-M0 stage so don't need one in DARM.
Tested several times so far so good.
In the future would be better to:
- make the initial DIFF turn on with an unconditionally stable loop
- turn on the loop slowly (~10 seconds)
- Set the UGF to 5 Hz
- Check it by measurement or at least refuse to engage boosts if DIFF beat is too low
- use a filter to ramp the gain; set filter turn on to trigger when error signal is with +/- 50 counts of zero
The guardian often gives the error "unhappy with inconsistent use of tabs and spaces in interferometer", as seen in the logs whenever we reload ISC_LOCK.py.
This is because python sadly doesn't know what to do if you give it an extra space here or there in the code. Today I used emacs 'M-x untabify' to heal the whole file.
Errors are gone. Commited to SVN before and after.
** however, not sure if my SVN commits are getting recorded. The latest commit I did has #18435, but svn log only shows commits up to #18336 on Dec-13
Two points Guardian related here:
- The connection errors seen in the DOWN state of ISC_LOCK come from the use of fast_ezca.py. This will use threading to set many channels quickly (most often the violin damping filters, as seen in Rana's attached log). Though the guardian log shows a connection error, I haven't seen any evidence that Guardian is unable to make any cahnges due to this error. This issue is known in the control room, but I don't know if it has ever been posted in the log. It is on the list to fix.
-
"inconsistent use of tabs and spaces" - Thank you Rana for doing this. Unfortunately, as long as control room users continue to use improper setting on their text editor (most often gedit), this will continue. To all users that edit Guardian code, please make sure that your editor is set to use a tab with of 4 and make tabs into spaces. If anyone needs help with this, please let me know and I can gladly show you where this setting is.
Lastly, for the svn commits, to see the most recent logs from the command line, one must do a "svn update" on that directory. If you had already done this, then we have some issues to figure out...