WP11542 EX -18VDC Power Supply Replacement
Fil, Erik:
Towards the end of maintenance, coincidentally 2 seconds before the EDC was restarted, the -18V EX SEIS power supply failed. This caused all of the ADC channels to read 0.0 counts.
Fil and Erik replaced the power supply with a spare unit.
WP11534 h1digivideo1 causing flashing-blue-screens
Jonathan, Patrick, Dave:
Since the upgrade of sw-msr-h1aux last week most, if not all, of the cameras on h1digivideo1 flash their gstreamer viewers periodically with a blue screen. Last Friday I restarted MC1's server process, which did not fix that camera's viewers. First thing today I rebooted h1digivideo1, which did not change anything. This was not surprising, all the server machines were rebooted last Tuesday following the switch upgrade.
Looking at the port statistics on sw-msr-h1aux for the h1digivideo[0,1,2] connections we found that h1digivideo1's VLAN106 port (1/1/16) had a 54% utilization and a dropped packet count of about 9000. Comparing this with the other two machines, which had utilizations around 20% and only dozens of dropped packets.
Jonathan unplugged the ethernet cable for the VLAN106 at both h1digivideo1 and the sw-msr-h1aux ports. The cable appeared to have been seated correctly.
The utilization dropped from 54% to 42% and we have not seen any blue flashes on h1digivideo1's cameras. We will continue to monitor.
WP11541 Add Lock Loss Alert IOC channels to DAQ
Dave:
I created a H1EPICS_ALERT.ini file which contains all of the non-string EPICS records in the lock loss alert IOC. A EDC+DAQ restart was required.
DAQ Restart
Dave, Jonathan:
The DAQ and EDC were restarted. The restart process was interrupted by the loss of h1seiex coincident to the EDC restart, which required some investigation to ensure it was a complete coincidence.
Other than that, and gds1 requiring a second restart, it was a unremarkable restart.
EDC channel count increased by 1082 channels from 56469 to 57551
Tue21Nov2023
LOC TIME HOSTNAME MODEL/REBOOT
11:28:25 h1daqdc0 [DAQ] <<< 0-leg restart
11:28:36 h1daqfw0 [DAQ]
11:28:36 h1daqtw0 [DAQ]
11:28:37 h1daqnds0 [DAQ]
11:28:45 h1daqgds0 [DAQ]
11:29:45 h1susauxb123 h1edc[DAQ] <<< EDC restart (h1seiex failed at this time)
11:44:19 h1daqdc1 [DAQ] <<< delayed 1-leg restart
11:44:30 h1daqfw1 [DAQ]
11:44:30 h1daqtw1 [DAQ]
11:44:32 h1daqnds1 [DAQ]
11:44:41 h1daqgds1 [DAQ]
11:45:36 h1daqgds1 [DAQ] <<< gds1 second restart
11:50:26 h1seiex h1iopseiex <<< first seiex attempt, restart the models
11:50:40 h1seiex h1hpietmx
11:50:54 h1seiex h1isietmx
12:53:15 h1seiex ***REBOOT*** <<< power cycle h1seiex following DC power supply replacement.
12:54:55 h1seiex h1iopseiex
12:55:08 h1seiex h1hpietmx
12:55:21 h1seiex h1isietmx