the probability a freeze-up does not impact at lease one dolphined FE is very small, so I'm using the h1boot dolphin node manager's logs to data mine these events. The dolphin manager was restarted when h1boot was rebooted Tuesday 7th July, so data epochs at that time.

As I was seeing with my monitoring programs, the restarts preferentially happen in the 20-40 minute block within the hour. The first histogram is the number of events within the hour, divided into 10 minute blocks.

We are also seeing more events recently, the second histogram shows number of events per day. The spike on Tue 14th is most probably due to front end computer reboots during maintenance. Friday's increase is not so easily explained.

FE IOC freeze up time listing:

controls on h1boot

grep "not reachable by ethernet" /var/log/dis_networkmgr.log |awk '{print $2r, $4}'|awk 'BEGIN{FS=":"}{print $1":"$2}'|sort -u

total events 197

minutes within the hour, divided into 10 min blocks

00-09 11  :*****

10-19 11  :*****

20-29 67  :*********************************

30-39 79  :****************************************

40-49 17  :*********

50-59 12  :******

events per day in July (start tue 07)

wed 08 09 :*****

thu 09 09 :*****

fri 10 08 :****

sat 11 07 :****

sun 12 08 :****

mon 13 09 :*****

tue 14 22 :***********

wed 15 10 :*****

thu 16 20 :**********

fri 17 38 :*******************

sat 18 16 :********