Jonathan, Jim, Dave:
Yesterday morning we increased the log level on h1fw0's daqdrc from 2 to 10. At this point it ran for over an hour writing all but commissioning frames. I then restarted it, now with a log level of 30 and writing both commissioning and science frames (but not trends). h1fw0 has not crashed since then through to today's DAQ restart (23 hours). Interestingly the log file has not increased its verbosity, and we are now seeing random retransmission requests similar to what LLO used to see.
In the mean time h1fw1 continues to restart. We increased its log level from 2 to 30, it still restarts roughly once an hour on average (today's restarts shown below). h1fw1 is writing all four types of frames.
We are building a third frame writer (h1fw2) which will be used to test new daqd code, for example mutex control of the writing threads to prevent more than one file to be written at a time.