This week we were put h1fw0 back into an unstable configuration by asking it to write out all the frames (full, science, strend, mtrend).

We were able to extract some useful information from the crashes.  The main point is that for some reason IO starts to become unstable and the raw frames end up taking too long to write.  This eats up all the internal buffers, and the system crashes.

I did some tweaking of nfs settings which brought the frame writing threads to have more stable write times.  Even with that a hiccup was observed which allowed the science and full frame to get a cycle out of sync, with the full frame writer being unable to catch up.

Today after looking at the code I have a new build of the framewriter running on h1fw2. This splits the framewriting thread into two parts, to allow parallel execution and allow the full frame writing threads the chance to catch up if they fall behind.  So far h1fw2 is stable.  We will test it on h1fw0 next week.