Ryan, Jason, Patrick, Filiberto As part of troubleshooting the PSL we hardware power cycled the PSL Beckhoff computer in the diode room this morning, along with all of the associated diode power supplies and a chassis in the LVEA. I had guessed that everything would autostart, but I was wrong, so I took the opportunity to set it up to do so. This required putting a shortcut to the EPICS IOC startup script in the C:\TwinCAT\3.1\Target\StartUp directory (see attached screenshots), and selecting an option in the TwinCAT Visual Studio project to autostart the TwinCAT runtime. We software restarted the computer again to test this, and after logging in, the Beckhoff runtime and PLC code started, along with the EPICS IOC, but the visualization did not. I found documentation that pointed to the location of the executable that starts the visualization, and added a shortcut to that to the startup directory as well. We didn't have time to restart the computer again to see if that would autostart correctly. For some reason there seemed to be issues with processes reconnecting to the EPICS IOC channels. I tested running caget on the Beckhoff computer itself and got a message about connecting to two different instances of the channel, and a couple of pop up windows related I think to allowing network access, which I said to allow. caget worked, although it gave a blank space for the value, so I tried it again with an invalid channel name, which it correctly gave an error for. On the Linux workstation we were using, the MEDM screens were not reconnecting, even after closing and reopening them, but again caget worked. We had to restart the entire medm process for it to reconnect. The EDCU and SDF also had issues reconnecting, and they had to be restarted too.
As Patrick mentioned, channel access clients which had been connected to the IOC on h1pslctrl0 would not reconnect after its restart.
The EDC stayed in its disconnect state for almost an hour, even though cagets on h1susauxb123 itself were connecting, albeit with "duplicate list entry" warnings:
(diskless)controls@h1susauxb123:~$ caget H1:SYS-ETHERCAT_PSL_INFO_TPY_TIME_HOUR
Warning: Duplicate EPICS CA Address list entry "10.101.0.255:5064" discarded
H1:SYS-ETHERCAT_PSL_INFO_TPY_TIME_HOUR 18
The restart of the DAQ EDC did not go smoothly, I had added a missing channel to H1EPICS_CDSRFM.ini (WP12195) in preparation for next Tuesday maintenance and so the EDC came back with a different channel list to that of the rest of the DAQ. I reverted this file change and a second EDC restart was successful.
11:38:35 h1susauxb123 h1edc[DAQ]
11:46:17 h1susauxb123 h1edc[DAQ]
The slow controls h1pslopcsdf system was also unable to reconnect to the 4 PSL WD channels it monitors. This was restarted at 12:08 14nov2024 PST.
Erik found that MEDM on some workstations would continue to show white-screen for h1pslctrl0 channels and a full restart of MEDM was needed to resolve this.