I investigated the HWS glitching issue in more detail. I ran the following tests to try to reproduce the glitching seen previously:
-
Run each camera individually with the
take
command (number of ring buffers = 4): no glitching
-
Run each camera individually with the
take
command (number of ring buffers = 1): no glitching
-
Run both cameras at the same time with the
take
command (number of ring buffers = 1): no glitching
-
Run both cameras at the same time with the
take
command (number of ring buffers = 4): no glitching
-
Run both
Run_HWS_ITMX
and Run_HWS_ITMY
at the same time: no glitching
Since we couldn't reproduce the glitching after everything restarted, Kiwamu and I decided that we create some Guardian diagnostics to run continuously on the HWS data. The following channels will be monitored and will issue warning in the case of aberrant behaviour:
-
H1:TCS-ITMX_HWS_TOP100_PV: the mean value of the top 100 brightest pixels in counts - currently around 1040 counts for ITMX. This cannot exceed 4095 in a normal image. Corrupted images routinely show pixel values much larger than this. We can also monitor this to see if the value falls below a threshold (which it will do in the case of misalignments or SLED issues).
-
H1:TCS-ITMX_HWS_LIVE_ACQUISITION_GPSTIME: the GPS time of the most recently recorded HWS image. If this falls more than 15 seconds behind the current GPS time, guardian can issue a warning.