WP 3727, Alex, Dave, Jim
at 10:14 we shut down h1dc0 and replaced it with the new h1dc1. Jim activated a second 10GBE port on the H1 FE-DAQ switch and ran a second 10G FMM line to the DAQ rack. The new h1dc1 has three 10GBE pcie cards (compared with only two in h1dc0). The extra card allows the front ends to split their DAQ data over the two cards. Since each card has 16 MX end points, and LHO will have a total of 31 front ends, this means that we can ensure that no two front ends share an end point and interfer with each other's DAQ data on reboot.
The 10GBE PCIe card arrangement as seen from the rear of h1dc1 is
| top | IN1 | FMM-013-0001 |
| mid | OUT | FMM-007-0006 |
| lower | IN2 | FMM-013-0002 |
The pci bus mapping put the output card (connected to the Fujitsu switch) between the input cards.
The MX startup script for the front ends was modified such that the FE sent data to card IN1 or IN2 depending upon their location in the rtsystab file. We restarted all the front ends MX Streams (did not require any model restarts) and all DAQ data was back by 11:34
To test the new system we power cycled various non-dolphin and non critical front ends: h1pemmx, h1susauxb6 and h1susauxb123. During each reboot no DAQ data on any other system went bad.
Alex then upgraded the h1fw1/h1nds1 system and installed the following
- new frame-cpp to speed up frame file reads
- archive current raw-minute trend and start writing those files anew
- new Chan STAT status per channel in the frame files
- IRQ balancing over all CPUs
Tests between the unmodified h1fw0 and the new h1fw1 systems show significant speed up in trend data retrieval. Speed up in minute trend writing quickly dissipated.
The new h1dc1 machine initially only had 12GB of RAM compared with h1dc0's 24GB. Not wanting to touch h1dc0 since that is our fall back machine if h1dc1 has issues, we removed 3*8GB DIMMs from h0broadcast0 and reduced its data cache so it could work with 12GB (6*2GB) RAM. We were then able to make h1dc1 a 24GB machine. We have asked John Z to check that the broadcaster is still functioning ok with reduced memory.