Jonathan, Dave
The DAQ was restarted at 19:06 Thursday 5/18 UTC (for vacuum controls IP6 work). If an NDS1 request for second trend data for certain channels is sent which spans the restart time, then the data preceding the restart is correct, but post data is bogus. In the case of H0:VAC-EY_Y3_410_PIRANI_INTLK, data at the tail end of the 19:00-19:10 second trend file is a static 2.2, and from 19:10 onwards is a noisy 2.2
We conclude that only the second trend frame from 19:00-19:10 is suspect. Both framewriters wrote the same frame, both nds servers show the same issues if spanning this frame. Investigation continues. I've saved the frame and its md5 file under h1nds1 control's homedirectory (dave subdirectory).
-rw-r--r-- 1 controls controls 434M May 22 11:27 H-H1_T-1179169200-600.gwf
-rw-r--r-- 1 controls controls 33 May 22 11:27 H-H1_T-1179169200-600.md5
Possible answer to Thursday's second trend issues:
Patrick noted that the interlock channel looked like it was showing a different channel's second trend data after the DAQ restart. I then remembered that the original NDS code had a performance feature built into it. When asking for second trends, it read the full channel list from the first GWF file, and then applied those channel offsets for all subsequent files it opened. In the initial system second trend files were one minute in length.
To cover the case where the channels configuration was changed during the requested time span, the code looked for any missing second trend files. At one minute per file it was not possible to restart the DAQ and not have a missing file. The file following the gap was read for the channel configuration and that was used from that point onwards.
Fast forward to today, with 10 minute trend files. It is now highly unlikely for a second trend file to be missing when the DAQ is restarted, and so the channel data pointers will be incorrect after a DAQ reconfiguration. We seem to be seeing this, with the interlock channel being replaced with a cold-cathode raw voltage channel (value of about 2.2V).
After reviewing the nds1 server code it looks like Dave's assesment is correct. The nds1 server implements a nice optimization where it reads in the frame table of contents at the start of a span of contiguous frame files. It only reads a new copy of the table of contents in when there is a gap in the frame files (thus saving a large amount of processing time). Now we can restart the daq fast enough that there need not be gaps in the trend frame files, so the nds1 server will not catch a restart of the daq (and thus the need to read a new copy of the table of contents in). I will work on correcting this on the DTS today.