WP8436 h1tw1 upgrade to new hardware and OS

Jonathan, Dave:

Summary: the original h1tw1 (V1 CPU, external SSD-RAID, Gentoo2.6.35) was replaced with the new style CPU+RAID unit running Debian8.

Details:

The replacement machine has been running as h1tw3 since February 2019. Today we renamed the new machine h1tw1 and turned off the old h1tw1 box. We reconfigured h1nds1 to get current raw minute data from the new unit.

Starting on Monday this week, the following was done:

1. h1tw1 daqd was stopped and not restarted, freezing its latest data block (24sep2019 - 21oct2019). At the same time h1tw3 was configured on-the-fly to write current data to a new disk location
2. h1nds1 was reconfigured to serve current data from h1tw3, sep-oct archive from h1tw1 and the rest of the archive from h1ldasgw1
3. found a 1-minute overlap between h1tw3 current and h1tw1 archive, which ndscope detected and alarmed on. Jonathan wrote code to shave off the last minute of data from each raw minute file, I ran that on the complete sep-oct archive on h1tw1.
4. the sep-oct archive data was copied from h1tw1 to h1ldasgw0 (to the compress ZFS file system used for permanent raw minute storage). This took 24 hours.
5. on h1tw3, the feb-oct data was deleted, it is a copy of what is on h1ldasgw1 (this took 2.5 hours)

Thursday work:

1. prep work: ran new ethernet cable from sw-msr-h1fe0 to h1tw3. We are keeping the new trend writer in its current location and not moving it to where h1tw1 was (will do this post O3). Verified switch is not mac-locked.
2. power down h1tw1(old)
3. rename h1tw3 as h1tw1 (name and IP address). The IP went from 10.20.0.18 (SERVER-LAN) to 10.101.0.45 (H1FE-LAN). Disconnected its SERVER-LAN ethernet cable, connected the new H1FE-LAN cable which uses the orignal h1tw1's sw-msr-h1fe0 port (port 14)
4. From now on h1tw1 refers to the new unit. h1tw1 froze (NFS problems) and needed a reboot to continue the upgrade.
5. renamed the target directories for the trend writers. h1tw1->h1tw1_oldv1, h1tw3->h1tw1
6. in the h1tw1 target daqdrc, changed TW3 to TW1
7. stopped puppet on h1tw1, reconfigure its 10GE port to reuse the h1tw1 IP address (192.168.1.200 -> 192.168.1.6)
8. reboot h1tw1 with new configuration and test reboot proceeds correctly
9. h1nds1 could not unmount the old h1tw3 NFS mount, so I rebooted it. This system had not done a disk check for 293 days, so one was forced.
10. h1nds1 daqd process did not start. Noticed its system clock was 15 minute in the future when checking log files, we fixed a misconfigured NTP configuration.
11. h1nds1 daqd process died a second time, log suggests a broadcast error?
12. h1nds1 daqd started on the third try. Noticed that its EPICS-monitor system was not running, because monit was not running. Started monit.

Tested that minute trend data is available from h1nds1 for past hour, 2 hours, day, month, 3 months (spanning all current and recent archive epochs), all looks good.