We had some trips durring the earth quake, the attached plot of ETMX is from the first trip, a later trip wthe plot of ITMX is from a trip that happened after Chris untripped the watch dog and the guardian tried to bring it back (the ground motion was still high and we were still using Tcrappy). 

Since everything is tripped and those we have untripped are tripping again, I am not going to try to post all of these plots  Anyone who is interested is free to find the data in the frames. 

some things about guardian:

It would be nice if the manager displays an more usefull error message if the ISI trips, the message we got was: node ISIS_BS_ST2: NOTIFICATION

The ITMY manager says WATCHDOG TRIPDAQKILL, a more usefull message. 

We tried moving ITMX to T750, it tripped again tryng to isolate.  Chris was able to get ITMX to damped, he had to figure out first that it is necessary to go to init, which is not obvious. ETMX did suceed in isolating with the blends on Start.   Since we have had huge earth quakes the last two nights it is not surprising that guardian couldn't bring the ISIs back on Tcrappy.  We will need to wait for more normal trips to test that more.  

Also, Rich said that I should write about problems in the alog in red letters, so here are a few, but I'm making them yellow since they are more like annoyances than huge problems:

ISI overview screens bring the control room work stations to a halt.  73% of my cpu time is curently devoted to medm (Xorg), and all I have open are 2 ISI screens and 2 ISI watchdog screens.  I'm not sure if this is a problem with the workstations or the screens, but it is pretty annoying.

 It does seems like the control room workstations are crashing much less than they were, thank you for that CDS!

The plotting tool for WD trips doesn't alway work, here is a sample error message:

  File "/opt/rtcds/userapps/release//isi/common/scripts/wd_plots/main.py", line 170, in _BufferDict
    abs_threshold_mask=abs_threshold_mask, host=host, port=port)
  File "/opt/rtcds/userapps/trunk/isi/common/scripts/wd_plots/pydv/bufferdict.py", line 243, in __init__
    self.add_channels(channel_names, wd_state_mask=wd_state_mask, abs_threshold_mask=abs_threshold_mask)
  File "/opt/rtcds/userapps/trunk/isi/common/scripts/wd_plots/pydv/bufferdict.py", line 382, in add_channels
    abs_threshold_mask=abs_threshold_mask)
  File "/opt/rtcds/userapps/trunk/isi/common/scripts/wd_plots/pydv/bufferdict.py", line 344, in __fetch_data
    self.conn.clear_cache()  # clear any saved information from a previous connection
RuntimeError: Input/output error

It would be nice to have the brief guardian screen in the ISI screens, so you can tell what is going on.  I would greatly prefer if this is the brief version that still allows you to select the requested state (at least for the manger), because these screens allow us to use gaurdians despite the bugs that plauge us with other guardian medm screens.  

It would also be nice if the guardian medm screens themselves worked no matter which workstation you use,  how you opened medm, or if they worked at all at an end station.  There seems to be no way to open a guardian screen from an end station, not even using guardmedm.  My current approach durring the day is to call the operator to ask them to misaling/ realign optics, I'm sure they are going to get sick of this soon.