recently h1fw1 has been quite unstable, restarting itself every two to four hours. During my investigation, I noticed that the raw minute trend files stopped getting updated about an hour before h1fw1 crashed, but the framed data (minute, second and full) kept being written right up to the crash. Also h1fw0 does not show this problem, or has it at a much lower rate. I extended the raw minute update rate from 5 mins to 15 mins on both h1fw0 and h1fw1 by setting in their daqdrc files:
set raw_minute_trend_saving_period=15;
The new configuration went in at 4pm Monday afternoon. As of time of writing, h1fw1 has been up for 17 hours.
The fact that the identical h1fw0 does not have the same behavior suggests a QFS issue with the SATABOY for this system.
Forgot to mention that the reason this instability has just appeared is that the H1 DAQ system was recently expanded from just the PSL to include all SUS and SEI systems for HAMS 1,2,3. The number of channels in the DAQ is now 42k, making it comparable with the H2 DAQ.
This begs another question, why isn't H2 DAQ, with a 5 minute raw min trend rate, not seeing this problem?