J. Kissel, D. Sigg, D. Barker, J. Batch, B. Weaver, E. Merilh

We've upgraded all front-end models to RCG 2.9.1. Recovery of the IFO is on-going, given that we had our usual chaotic array of activities, but all signs point towards success. The upgrade was a little bit harder for some models / front-ends than others (details below), but we now have a completely green CDS overview screen (except for the saturation indicators that are always present on certain ISC models, and the currently-railing STS2 plugged into STS B causing all corner station seismic models to saturate -- more on that later). Details of the hiccups below.

Details
---------
Problem children:
- PSL ISS front-end model
This guy was the nastiest. The model some how did not have a safe.snap file or softlink in the 
/opt/rtcds/lho/h1/target/h1psliss/h1pslissepics/burt/
directory. We didn't notice this until much later, but at first this caused the model to simply not start because it couldn't find (what used to be) "the burt button" EPICs record that gave to OK status to get started. Daniel and I tried blindly restarting the model, recompiling-reinstalling-and-restarting the model, with no success. Finally, Daniel figured out that if he burt restored *while* the front end was coming up, he could get the model to start. Later, Betsy was making the effort to turn on the PSL ISS model's SDF system, created a table, but could not load it. After about 30 [sec] of trying, the front end model core dumped, seg faulted, and just died (not only this user model crashed, every other model on the front end survived). It was only after investigating this, that we found out about this missing target-area-safe.snap. The userapps repo had a safe.snap, so we softlinked the target-area to the userapps repo, and restarted the front-end model. 
All has been well since. 
We don't understand how a front end model can exist with out *something* in the target area called "safe.snap".

- The PCAL X and Y front-end model
Because it appears to be related (both in symptom and potentially in responsible parties) I mention this here. Last night, when I was capturing all settings in prep for today's model restarts, I captured new safe.snaps for the pcal front end models, because they were not yet under SDF control. In doing so immediately noticed that these models *also* didn't have safe.snap files in the target area. I didn't think much of it at the time, because I know the history of the pcal models, BUT now that I see a similar problem with the psliss model, I worry that the problem is systematic. Will investigate further.

- The end-station ISC, SEI, and SUS models
We had only planned to restart the end station's ISC and PEM front ends, because the SUS and SEI had been upgraded last week. However, when we restarted the ISC models, we found lots of continuous IPC errors between the ISC models and SEI and SUS. We think we traced this down to the clearing of the entire IFO's IPC file yesterday. Dave and Jim had thought they'd recompiled and reinstalled the SEI and SUS, which should have populated the IPC file *without* restarting the models, but this didn't appear to be successful. So, we ended up recompiling, reinstalling, and restarting the SEI and SUS models in addition to every other model at the end station (again). All errors are clear now, as mentioned above.

Also:
As mentioned by Daniel (LHO aLOG 17969), we had forgotten to update the some ODC library parts before getting started with the recompiling yesterday, so only some models received the upgrade to their ODC. There also seems to be a few bugs with the updated version, from what we can see. Will follow up with the ODC team. However, since we've run out of time, we'll include the remainder of these in the model recompiling already planned for next Tuesday. 
Models that need restarting to receive the ODC update:
- All corner station SEI and SUS models
- Corner station TCS model
- Corner station PEM
- Corner LSC, ASC, and OMC models
- Corner Station CAL model
(Note, all SUS AUX models don't have any ODC in them, so they do not need the update).