Opened FRS 29441. This crash appeared to cause locklosss at 8:05UTC.

I first verified for the IOPs with DACKILLs that their IO Chassis could be seen, which was the case.

I restarted all the models (to clear IOP DACKILLS) on h1oaf0, h1lsc0, h1susb123, h1sush2a, h1sush34, h1sush56. I put all the corresponding SEI IOP SWWD into bypass mode before restarting the SUS IOPs.

All the models restarted with no issues, no reboots (with their associated Dolphin fencing) were needed.

In the morning I'll see if I can find which front end caused the Dolphin crash.

The EDC is slow in reconnecting to susmc3 and susprm channels, but it looks like it will get there in time.

Sat21Oct2023
LOC TIME HOSTNAME     MODEL/REBOOT
01:48:20 h1oaf0       h1iopoaf0   
01:48:34 h1oaf0       h1pemcs     
01:48:48 h1oaf0       h1tcscs     
01:49:02 h1oaf0       h1susprocpi 
01:49:16 h1oaf0       h1seiproc   
01:49:30 h1oaf0       h1oaf       
01:49:44 h1oaf0       h1calcs     
01:49:58 h1oaf0       h1susproc   
01:50:12 h1oaf0       h1calinj    
01:50:26 h1oaf0       h1bos       
01:52:02 h1lsc0       h1ioplsc0   
01:52:16 h1lsc0       h1lsc       
01:52:30 h1lsc0       h1lscaux    
01:52:44 h1lsc0       h1sqz       
01:52:58 h1lsc0       h1ascsqzfc  
01:55:25 h1susb123    h1iopsusb123
01:55:39 h1susb123    h1susitmy   
01:55:53 h1susb123    h1susbs     
01:56:07 h1susb123    h1susitmx   
01:56:21 h1susb123    h1susitmpi  
01:58:57 h1sush2a     h1iopsush2a 
01:59:11 h1sush2a     h1susmc1    
01:59:25 h1sush2a     h1susmc3    
01:59:39 h1sush2a     h1susprm    
01:59:53 h1sush2a     h1suspr3    
02:01:57 h1sush34     h1iopsush34 
02:02:11 h1sush34     h1susmc2    
02:02:25 h1sush34     h1suspr2    
02:02:39 h1sush34     h1sussr2    
02:04:13 h1sush56     h1iopsush56 
02:04:27 h1sush56     h1sussrm    
02:04:41 h1sush56     h1sussr3    
02:04:55 h1sush56     h1susifoout 
02:05:09 h1sush56     h1sussqzout