TJ, Jonathan, Erik, Dave:
yesterday the camera client flashing-blue-screen for cameras whose servers are running on h1digivideo1 steadily got worse as time progressed. This was seen before on 21 Nov 2023, at that time reseating the ethernet cable which connects h1digivideo1 to the cameras fixed it.
We tried, with no success, the following:
- reboot h1digivideo1
- reseat the ethernet cable
- replace the ethernet cable
- move the ethernet cable to a different port on the switch (sw-msr-h1aux 1/1/16 -> 1/1/34)
In all cases the switch port utilization went close to 50%, packets were dropped and camera-clients flashed blue-screen
We then stopped the server processes of three cameras which were not being used often in the control room and were not on any FOM. The port utilization dropped to 36% (still higher than h1digivideo0 (28%) and h1digivideo2 (20%)) but it resolved the flashing.
Current status:
h1digivideo1 eth1 ethernet is temporarily connected to sw-msr-h1aux port 34. Should put this back to port 16.
h1digivideo1 monit is not running the camera service for three cameras:
- cam17: SRM
- cam18: SQZ SHG TRANS
- cam20: SQZ IR TRANS
Note that while the problem persisted, making any of the changes described above risked losing a camera completely, at which point the camera needed to be power cycled via POE and its server restarted.
When the ethernet cable was reseated, we lost cam11 (MC1) and cam13 (PRM)
When the ethernet cable was replaced, we lost cam13 (PRM) and cam17 (SRM)