Reports until 07:04, Tuesday 05 May 2015
H1 CDS (DAQ, ISC, SEI)
jeffrey.kissel@LIGO.ORG - posted 07:04, Tuesday 05 May 2015 - last comment - 08:45, Tuesday 05 May 2015(18223)
Beginning CDS Maintenance 
J. Kissel

Plan for the morning:
- Recompile, reinstall, and restart BSC HPI models to absorb changes that were a residual from the HAM model updates
- Install ALS EX and EY models, which have been compiled against the tagged release of RCG 2.9.2 (it had only been compiled against the branch to date)
- Restart the frame builder / DAQ / h1cd0 to absorb new TCS Beckhoff channels and the updated channel list from the new BSC HEPI template.

Bringing all IFO guardians to DOWN and IMC guardian to OFFLINE, and bringing BSC Chambers to OFFLINE.
Comments related to this report
jeffrey.kissel@LIGO.ORG - 07:51, Tuesday 05 May 2015 (18224)
Link 

I've successfully installed and restarted ALSEX and EY, and successfully compiled, installed and restarted all BSC HEPIs. All chambers are back online, and I've confirmed that the IFO can reach the DRMI_LOCKED ISC_LOCK guardian state.

Sadly, the Framebuilder wasn't so happy with its restart. I tried twice, but the result is the same failure mode: the following front-ends show "02xbad" status:
h1sush56

h1susauxb123
h1susauxh2
h1susauxh34
h1susauxh56

h1asc0

h1susex

h1pemmx0
Note that all of these models are (at least superficially) unrelated to BSC HEPIs and ALS EX/EY.
I've reached the limits of my debugging abilities for the frame builder. I'll have to wait for the pros to deal with this issue.
Images attached to this comment
jeffrey.kissel@LIGO.ORG - 08:45, Tuesday 05 May 2015 (18229)
Link 

J. Kissel, J. Batch

The 02xbad error message seems to have been a result of a common occurrence -- a restart of the frame builder glitches the mx_stream process on some front ends. The first thing to try is to log on to the front end, and restart the mx_stream porcess, by running the command
sudo /etc/start_streamers.sh


Jim performed this command on all of the errant front-ends, and the bad DAQ status has cleared. 

Also -- he informed me that even installing the ALS models while the RCG is pointing to a different version will overwrite some necessary stuff and render the compilation against RCG 2.9.2 will cause badness. Jim changed the build machine's pointers briefly back to 2.9.2, recompiled and reinstalled the h1alsex and h1alsey models and I restarted them. All seems clear, and h1build pointers have been reverted back to RCG 2.9.1.