Carlos, Jim, Dave: WP5775
Yesterday (Monday 14th March) we moved most of the NFS exported files from h1boot onto a new server called h1fs0. This move was more invasive than last week's /ligo file system move as all front ends and DAQ computers were impacted.
The process was:
- put guardian in a down state
- put IFO into a state to ride through loss of front end computers
- stop all front end models and power down all front end computers
- perform final rsyncs from h1boot over to h1boot-backup, and then from h1boot-backup to h1fs0
- reconfigure h1boot to no longer export /opt/rtcds, /opt/mx, /opt/cdscfg and /opt/rtapps. Reboot h1boot
- reconfigure DAQ machines, power cycle them, get DAQ back for Vacuum trending.
- test FE reboot with h1susauxb123 (worked), then get all front end computers and models back up.
- start h1guardian0, hand IFO over to operator
- configure and restart all workstations and servers which mount /opt/rtcds (most of them)
The DAQ restart was messy. We saw the two framebuilders report different data CRC which we had not seen before. Also h1fw1 restarted with apparent slow file access for some time and then the problem apparently went away. We are monitoring this.
Today we are catching those machines which were overlooked yesterday.
For systems which only mounted /opt/rtcds from h1boot, I have a script to convert them to mount from h1fs0:
#!/bin/bash
# make changes to move the h1boot mounts to h1fs0
# in /etc/fstab, replace mount command with sed
cd /etc
cp fstab /tmp/fstab.$(date +%Y%m%d%H%M%S)
cp fstab /tmp/dave.tmp
cat /tmp/dave.tmp|sed 's/h1boot:/h1fs0:/rtopt/g' > fstab
# ensure h1fs0 has entry in /etc/hosts
cd /etc
cp hosts /tmp/hosts.$(date +%Y%m%d%H%M%S)
echo "10.101.0.13 h1fs0" >> /etc/hosts