Reports until 17:02, Wednesday 24 October 2018
H1 CDS
david.barker@LIGO.ORG - posted 17:02, Wednesday 24 October 2018 (44798)
testing beckhoff SDF nodes every hour

Jonathan, Dave:

Yesterday we found that 3 of the beckhoff SDF faux-models were in an unresponsive state. It would appear that they run well for a while and break some time later.

Since this is a 'silent' failure (MEDMs show last good data) I am running a script (verify_beckhoff_sdf_is_working.py) once an hour which changes the displayed table from DIFFS to FULL-TABLE and back to DIFFS, verifying the number of displayed items change. If this switching becomes a nuisance I can reduce the test frequency. 

The script is running in a tmux session on zotws6 for now. No failures on any node so far after over 24 hours of testing.