One of these observing drops was for 90seconds as the CO2Y laser unlocked and relocked at a lightly lower power. It is a known issue that the laser is coming to the end of it's life and we plan to swap it out in the coming ~weeks 79560.

Looks like the camera servo had the beamsplitter camera get stuck, causing CAMERA_SERVO to turn the servos off and then back on, which worked. Since Dave had just had to restart that camera an hour earlier because of it getting stuck (80555), I wonder if it came back in a weird state?

ISC_LOCK:

2024-10-09_08:24:57.089769Z ISC_LOCK [NOMINAL_LOW_NOISE.run] USERMSG 0: CAMERA_SERVO: has notification
2024-10-09_08:25:10.719794Z ISC_LOCK [NOMINAL_LOW_NOISE.run] Unstalling CAMERA_SERVO
2024-10-09_09:25:01.088160Z ISC_LOCK [NOMINAL_LOW_NOISE.run] USERMSG 0: CAMERA_SERVO: has notification
2024-10-09_10:25:04.090580Z ISC_LOCK [NOMINAL_LOW_NOISE.run] USERMSG 0: CAMERA_SERVO: has notification
2024-10-09_11:25:07.094947Z ISC_LOCK [NOMINAL_LOW_NOISE.run] USERMSG 0: CAMERA_SERVO: has notification
2024-10-09_12:25:10.092275Z ISC_LOCK [NOMINAL_LOW_NOISE.run] USERMSG 0: CAMERA_SERVO: has notification
2024-10-09_13:07:53.531208Z ISC_LOCK [NOMINAL_LOW_NOISE.run] USERMSG 0: TCS_ITMY_CO2_PWR: has notification
2024-10-09_13:25:13.087490Z ISC_LOCK [NOMINAL_LOW_NOISE.run] USERMSG 0: CAMERA_SERVO: has notification
2024-10-09_14:25:16.088011Z ISC_LOCK [NOMINAL_LOW_NOISE.run] USERMSG 0: CAMERA_SERVO: has notification
 

CAMERA_SERVO example:

2024-10-09_14:25:16.020827Z CAMERA_SERVO [CAMERA_SERVO_ON.run] USERMSG 0: ASC-CAM_PIT1_INMON is stuck! Going back to ADS
2024-10-09_14:25:16.021486Z CAMERA_SERVO [CAMERA_SERVO_ON.run] USERMSG 1: ASC-CAM_YAW1_INMON is stuck! Going back to ADS
2024-10-09_14:25:16.078955Z CAMERA_SERVO JUMP target: TURN_CAMERA_SERVO_OFF
2024-10-09_14:25:16.079941Z CAMERA_SERVO [CAMERA_SERVO_ON.exit]
2024-10-09_14:25:16.080398Z CAMERA_SERVO STALLED
2024-10-09_14:25:16.144160Z CAMERA_SERVO JUMP: CAMERA_SERVO_ON->TURN_CAMERA_SERVO_OFF
2024-10-09_14:25:16.144402Z CAMERA_SERVO calculating path: TURN_CAMERA_SERVO_OFF->CAMERA_SERVO_ON
2024-10-09_14:25:16.144767Z CAMERA_SERVO new target: DITHER_ON
2024-10-09_14:25:16.146978Z CAMERA_SERVO executing state: TURN_CAMERA_SERVO_OFF (500)
2024-10-09_14:25:16.148768Z CAMERA_SERVO [TURN_CAMERA_SERVO_OFF.main] ezca: H1:ASC-CAM_PIT1_TRAMP => 0
2024-10-09_14:25:16.149295Z CAMERA_SERVO [TURN_CAMERA_SERVO_OFF.main] ezca: H1:ASC-CAM_PIT1_GAIN => 0
...
2024-10-09_14:25:29.432120Z CAMERA_SERVO [TURN_CAMERA_SERVO_OFF.main] ezca: H1:ASC-CAM_YAW3_SW1 => 4
2024-10-09_14:25:29.683381Z CAMERA_SERVO [TURN_CAMERA_SERVO_OFF.main] ezca: H1:ASC-CAM_YAW3 => OFF: INPUT
2024-10-09_14:25:29.757840Z CAMERA_SERVO REQUEST: CAMERA_SERVO_ON
2024-10-09_14:25:29.757840Z CAMERA_SERVO STALL cleared
2024-10-09_14:25:29.757840Z CAMERA_SERVO calculating path: TURN_CAMERA_SERVO_OFF->CAMERA_SERVO_ON
2024-10-09_14:25:29.824193Z CAMERA_SERVO EDGE: TURN_CAMERA_SERVO_OFF->DITHER_ON
2024-10-09_14:25:29.824193Z CAMERA_SERVO calculating path: DITHER_ON->CAMERA_SERVO_ON
2024-10-09_14:25:29.824193Z CAMERA_SERVO new target: TURN_ON_CAMERA_FIXED_OFFSETS
2024-10-09_14:25:29.824193Z CAMERA_SERVO executing state: DITHER_ON (300)
2024-10-09_14:25:59.577353Z CAMERA_SERVO [DITHER_ON.run] ezca: H1:ASC-ADS_YAW5_DOF_GAIN => 20
2024-10-09_14:25:59.581068Z CAMERA_SERVO [DITHER_ON.run] timer['wait'] = 0
2024-10-09_14:25:59.767573Z CAMERA_SERVO EDGE: DITHER_ON->TURN_ON_CAMERA_FIXED_OFFSETS
2024-10-09_14:25:59.768055Z CAMERA_SERVO calculating path: TURN_ON_CAMERA_FIXED_OFFSETS->CAMERA_SERVO_ON
2024-10-09_14:25:59.768154Z CAMERA_SERVO new target: CAMERA_SERVO_ON
2024-10-09_14:25:59.768948Z CAMERA_SERVO executing state: TURN_ON_CAMERA_FIXED_OFFSETS (405)
2024-10-09_14:25:59.769677Z CAMERA_SERVO [TURN_ON_CAMERA_FIXED_OFFSETS.enter]
2024-10-09_14:25:59.770639Z CAMERA_SERVO [TURN_ON_CAMERA_FIXED_OFFSETS.main] timer['wait'] done
...
2024-10-09_14:26:33.755960Z CAMERA_SERVO [TURN_ON_CAMERA_FIXED_OFFSETS.run] ADS clk off: ASC-ADS_YAW5_OSC_CLKGAIN to 0
2024-10-09_14:26:33.761759Z CAMERA_SERVO [TURN_ON_CAMERA_FIXED_OFFSETS.run] ezca: H1:ASC-ADS_YAW5_OSC_CLKGAIN => 0.0
2024-10-09_14:26:33.763017Z CAMERA_SERVO [TURN_ON_CAMERA_FIXED_OFFSETS.run] ezca: H1:ASC-ADS_YAW5_DEMOD_I_GAIN => 0
2024-10-09_14:26:37.898356Z CAMERA_SERVO [TURN_ON_CAMERA_FIXED_OFFSETS.run] timer['wait'] = 4

From the h1digivideo2 cam26 server logs (/tmp/H1-VID-CAM26_gst.log), at 25 minutes past the error we see a timeout on the camera.

2024-10-09 12:25:16,806 UTC - INFO - startCamera done
2024-10-09 13:25:05,745 UTC - ERROR - Duplicate messages in the previous second: 0 ... psnap.snapCamera failed: timeout
2024-10-09 13:25:05,745 UTC - ERROR - Attempting to stop camera
2024-10-09 13:25:05,748 UTC - ERROR - Attempting to start camera
2024-10-09 13:25:05,750 UTC - INFO - Opened camera
2024-10-09 13:25:24,404 UTC - INFO - Duplicate messages in the previous second: 0 ... Opened camera
2024-10-09 13:25:24,427 UTC - INFO - setupCamera done
2024-10-09 13:25:24,441 UTC - INFO - startCamera done
2024-10-09 14:25:08,817 UTC - ERROR - Duplicate messages in the previous second: 0 ... psnap.snapCamera failed: timeout
2024-10-09 14:25:08,817 UTC - ERROR - Attempting to stop camera
2024-10-09 14:25:08,827 UTC - ERROR - Attempting to start camera
2024-10-09 14:25:08,828 UTC - INFO - Opened camera
2024-10-09 14:25:31,865 UTC - INFO - Opened camera
2024-10-09 14:25:31,888 UTC - INFO - Duplicate messages in the previous second: 0 ... setupCamera done
2024-10-09 14:25:31,904 UTC - INFO - startCamera done
2024-10-09 15:07:36,397 UTC - ERROR - Duplicate messages in the previous second: 0 ... psnap.snapCamera failed: could not grab frame
2024-10-09 15:07:36,397 UTC - ERROR - Attempting to stop camera
2024-10-09 15:07:36,446 UTC - ERROR - Attempting to start camera
2024-10-09 15:07:36,447 UTC - INFO - Opened camera
2024-10-09 15:07:36,463 UTC - INFO - setupCamera done
2024-10-09 15:07:36,476 UTC - INFO - startCamera done
2024-10-09 15:07:36,477 UTC - ERROR - setupReturn = True
2024-10-09 15:07:36,477 UTC - ERROR - Failed to get frame - try number: 1
2024-10-09 15:25:11,892 UTC - ERROR - Duplicate messages in the previous second: 0 ... psnap.snapCamera failed: timeout
2024-10-09 15:25:11,893 UTC - ERROR - Attempting to stop camera
2024-10-09 15:25:11,898 UTC - ERROR - Attempting to start camera
2024-10-09 15:25:11,900 UTC - INFO - Opened camera
2024-10-09 15:25:39,541 UTC - INFO - Opened camera
2024-10-09 15:25:39,564 UTC - INFO - setupCamera done
2024-10-09 15:25:39,580 UTC - INFO - startCamera done

Tagging DetChar - These glitches that happen once an hour around 11Hz are because of this camera issue (the glitches are caused by the camera ASC turning off and ADS turning on). We are still having this issue and are planning on trying a correction at the next lockloss.