J. Kissel, D. Barker, R. Bork, S. Dwyer, J. Driggers, E. Merilh, J. Bartlett

While trying to recover today (I'm trying to get in separate aLOGs for all the problems, such that the summary and timeline is easier later), we've found a particularly nasty bug in the Settings Definition File (SDF) system. If you're model has more that 40 setting differences (i.e. the DIFFs run off the first page), and you wish to sort the list of channels on a user defined sub-string, the front end's EPICs server crashes -- making it *look* (as far as any MEDM screen goes) like the entire front-end has crashed. Since you can't access any EPICs settings, you have no chance to take down the running system -- that's right, the front-end code is still happily running away, you just have no way to look at it or control it -- in a nice graceful way. You just have to restart the front-end model process.

We had first this on H1 SUS ETMY, when at the time, we had no idea what happened, but Dave aLOGged the error message of the failure -- see LHO aLOG 19628. All of these front-ends had to be restarted, most of which happened around 3:30pm PT.

After reproducing the error (first on accident, and then finally on purpose) for SUS PR3, HAM6 ISI, ODC MASTER (and maybe a few others, I was talking calibration with Rick at the time, hopefully dave will aLOG the exact body count tomorrow), Rolf, Dave, and Jeff B. figured out the above described cause, searching on a sub string for an SDF system with more than 40 DIFFs.