david.barker@LIGO.ORG - posted 13:50, Friday 27 December 2019 - last comment - 16:09, Friday 27 December 2019(54149)
seismon not updating
Corey, Jim, Dave:
we are currently investigating a non-updating seismon.
~23:28 Notice the SEISMON Dead RED box is gone and DIAG MAIN for this is also gone. And Seismon has more recent events listed on its medm screen.
But I do not see the box flashing on the SEISMON medm screen (it was flashing when we had the issues though).
It looks like it's back up and running now. Not sure at all what happened. Will try to summarize here.
I let Dave and Corey try to recover following the instructions on the wiki (which are still accurate as far as I know) : https://cdswiki.ligo-wa.caltech.edu/wiki/seismon
I then tried following the same steps, but also deleted the old data for the USGS productclient (which took many minutes to complete, the last time this may have been done is July or maybe February).
I then told Dave it would be worth trying to restart the VM. After he did that, I expected the seismon processes to restore on their own, but they didn't after several minutes. So I follow the wiki procedure again.
Seismon screen and DIAG_MAIN still reported dead, which can be due to the seismon calculated GPS time not updating. I poked around in the epics code and added a couple lines to us gpstime to get the right gpstime, then just write that. Still didn't seem to work. Then a couple restarts later (from me messing with just the seismon epics client) it just started working. The only material change to the code was something like adding:
gt=gpstime.now.gpstime() -> whatever the right call is to get the current gps time, I don't remember
counter=int(gt)
caput(seismon_gps_epics_time_channel, gt)
I tried to get the epics code to print the "counter" value in STDOUT, so I could see it was updating in the terminal, but couldn't get python to be happy with doing that. I think that would be a useful indicator in the code. The only thing I can think of is that sometimes after deleting the USGS data, it can take a little while for the USGS code to get to recent enough earthquakes for the seismon code to be happy.
Also, when I try to plot the eq reponse plot from home, python complains about not being able to find the epics package. I don't remember seeing that before.
We are able to bring up the EQ Response plots in the Control Room.