We are now in day 113 of O4 and we have not had any spontaneous IPC receive errors on any model throughout this time.
During Tuesday maintenance this week I forgot to issue a DIAG_RESET on h1oaf after the pem models were restarted, and therefore it is showing latched IPC errors from this time which I just noticed today.
To elevate the visibility of latched transient IPC errors, I have added a new block on the CDS overview which will turn yellow if the model has a latched IPC error. This block does not differentiate between IPC type (shared-memory, local-dolphin, x-arm, y-arm). The new block is labeled lower case "i". Clicking on this block opens the model's IPC channel table.
The upper case "I" block remains as before which turns red if there are any ongoing IPC errors (reported as a bit in the model's STATE_WORD)
To make space for this new block (located at the end by the CFC) I have reduced the width of the DAQ-STAT and DAQ-CFC triangles to the same width as the blocks (10 pixels).
I have added a legend to the CDS Overview, showing what all the model status bits mean.
Clicking on the Legend button opens DCC-T2300380 pdf using the zathura image viewer.