Jonathan, Dave:
Yesterday we found that 3 of the beckhoff SDF faux-models were in an unresponsive state. It would appear that they run well for a while and break some time later.
Since this is a 'silent' failure (MEDMs show last good data) I am running a script (verify_beckhoff_sdf_is_working.py) once an hour which changes the displayed table from DIFFS to FULL-TABLE and back to DIFFS, verifying the number of displayed items change. If this switching becomes a nuisance I can reduce the test frequency.
The script is running in a tmux session on zotws6 for now. No failures on any node so far after over 24 hours of testing.