00:19 UTC H1 Dropped out out of OBSERVING and into COMISSIONING due to the CAMERA_SERVO Guardian detecting that the ASC-CAM_PIT2_INMON & ASC-CAM_YAW2_INMON was "stuck" for 5 seconds. The Guardian node took itself to TURN_CAMERA_SERVO_OFF. The Guardian node was in this state for less than a minute. It then took itself back to CAMERA_SERVO_ON. This was also seen in the Guardian log. Please See attached ndscope Screeneshot.
00:34 UTC We went back to observing.
Looking back at yesterdays alog, and some ndscopes this appears to be the same issue we saw today, and it was a mere coincidence that Elenna happen to hit the load button at the same time as the guardian node transitioned down to get the camera unstuck.
For such short camera freeze, we implemented WAIT_FOR_CAMERA state in the camera guardian (alog68756). When the camera freeze happens, the camera guardian turns off the camera servo and waits for camera for 30s without moving to ADS. If the cameras are OK after 30s, the guardian turns on the camera servo again. So the camera guardian itself is working as expected.
In the camera guardian, the is_chan_static in static_tester.py is checking if the cameras are stuck or not. If the is_chan_static can skip this short camera freeze, that would be one of the solutions.