In looking at frame writer stability issues I installed a daqd build on h1fw2 that would report some timing information on the main producer thread.  This is the thread that reads the data in the data concentrator and checksums it and queues it up for the frame writing threads.

The four points I am measuring are:
full - a full cycle (should be under 1/16s)
recv - time to receive a 1/16s block of data
crc - time to checksum the data
xfer - time to move the data into the frame writing buffers

These numbers are do not include the time taken to output the timing samples to the log (so the full cycle time was really a bit longer).

Here is a particularly bad sample, time is in seconds:
 full min:0.030165 mean:0.0694159 max:0.371219
 recv min:0.00241995 mean:0.0398388 max:0.338558
 crc  min:0.0151799 mean:0.0171485 max:0.020812
 xfer min:0.0118489 mean:0.0124278 max:0.0145991

And another that was more typical

 full min:0.0603912 mean:0.0624495 max:0.0659258
 recv min:0.0284889 mean:0.0316754 max:0.0346711
 crc  min:0.0155029 mean:0.0183311 max:0.021708
 xfer min:0.012187 mean:0.012442 max:0.0149119

When a cycle on the producer thread goes over 1/16s we get retransmission requests on the network.  I had to stop running this build of daqd due to it triggering a large number of retransmission requests.

During a recent meeting at LLO to discuss daqd Keith had informed us that the producer thread is doing to much and it needs to be split into separate threads.  The timing test I ran today supports this.  I am working on code now to split the producer thread into two halves.

* reminder h1fw2 only writes to local disk and is being used to test fw changes under a production data load.