Summary: Since we have made many changes to DRMI over the last few weeks and locking is much slower than it was, I started trying to restore us to the old configuration which worked pretty well. We have locked DRMI 6 times since, none of the acquisitions have taken longer than 8 minutes. (see second attachment)
Lesson for the future:
We should be much more careful about making changes to DRMI acquisition, and look at locking statistics to evaluate whatever we do.
TJ Massinger pointed out that there is a page that already displays times, ISC_LOCK summary. If you scroll to the bottom and click on the state LOCK_DRMI_1F [101] you can see a list of the locking attempts and their duration. Craig and I used this data to make the histograms in the second attachment. We were locking much more quickly in Sept 9-12 than in the first days of October, and after making the changes tonight the times are closer to what we had in mid September although the distribution will probably not be as good.
I think our approach going forward should be to wait a few days after making a change, and get several tens of locking attempts for statistics before we make a change. If we compare the locking statistics before and after the change, and they are better, we can go ahead try another change if we want to then.
Details:
I compared changes in the DRMI acquisition settings for LSC using conlog. The two times are used are Sept 12 2018 at 21:43 (1220823832), this was a week when we remember DRMI locking quickly and looking back at the data confirms that it was locking quickly. Now DRMI locking is consuming a lot of time, and we have made many changes and aren't sure which of the changes made are the problem.
| Sept 12 | Oct 4th | changes made tonight | |
| POP18 when locked | 145 | 179 (alignment onto diode) | |
| MICH trigg +FM trig on | 40 (27% of locked) | 40 (22% of locked level) | 48 (27%) |
| SRCL trigger on level | 40 (27%) | 30 (17%) | 48 (27%) |
| SRCL trig + FM trig off | 15 (10%) | 5 (3%) | 18 (10%) |
| MICH trigger threshold off | 15 (10%) | 5 (3%) | 18 (10%) |
| REFL9I to PRCL matrix | 3.5 | 2.2 | 3.5 |
| PRCL gain | 12 | 8 | 12 (ACQUIRE) |
| REFL9I to SRCL | -5.508 | -1.8(44318) | not changed |
| MICH FM trig wait | 0.1 | 0.2 | 0.1 added to guardian |
| PRCL FM trig wait | 0.2 | 0.3 | 0.2 added to guardian |
| REFLair RF45 phase R | 82 | 97 | not changed |
| Refl air RF9 phase R | -21 | -23 | not changed |
| SRCL FM1 | was triggered, +6dB, 0.2 second ramp | no longer triggered, +106 dB to cancel -100dB in FM2 | reluctant to change |
I was reluctant to make changes to SRCL FM1 because that is the filter that has twice given huge and inexplicable outputs when we modified it.
Things I haven't done which could be done (after collecting some stats about this state, if we want to try to get closer to Sept):
- Run conlog for these times for the suspensions, starting with BS, then PRM and SRM
- check the filter archive for differences in LSC, BS, then PRM and SRM
- check the guardian svn (or git) for differences Sept 12 to now.
