Displaying report 1-1 of 1.
Reports until 14:22, Tuesday 05 February 2019
H1 SEI (CDS, OpsInfo)
jim.warner@LIGO.ORG - posted 14:22, Tuesday 05 February 2019 - last comment - 09:17, Wednesday 06 February 2019(46795)
SEISMON got broken by the FS issues yesterday, running again

I noticed yesterday that Seismon was not recovered after the file system issues yesterday, but I caught it after Carlos had left for the day, so I didn't get to ask if they were still recovering. Working with Carlos this morning I was able to get it back up and running, but I wanted to capture the process here.

1. ssh onto the seismon vm as controls:

    ssh controls@seismon

2. Launch 2 screen sessions for the seismon scripts, this is not completely straight forward. The scripts live in /usr/local/seismon/seismon/bin, but 2 of them need a virtual environment, which I launched using an alias:

controls@seismon:~ 0$ cd /usr/local/seismon/seismon/bin
controls@seismon:bin 0$ screen

Activate the virtual environment for seismon:

controls@seismon:bin 0$ seismon_ve 

Then run the traveltimes script:
 (ve) controls@seismon:bin 0$ ./seismon_run_traveltimes.py

This will start outputting stuff to the terminal, so leave it running and CTRL+A+D to disconnect from the screen session, launch a new screen session and repeat for seismon_run_info, including the seismon virtualenv (seismon_ve alias).

3. The run a third screen session for the epics client seismon_epics_client_v5.py. this does NOT use the seismon virtualenv, and won't run with it set. When done there should be 3 screen sessions running.

controls@seismon:bin 0$ screen

controls@seismon:bin 0$ ./seismon_epics_client_v5.py

4. There is also an epics IOC on h1fescript0, ssh to that machine as controls and ps -elf | grep seis to see there is only one copy of seismon_epics_ioc_v3.py running.

controls@h1fescript0:~ 0$ ps -elf |grep seis
0 S controls  2952  2590  1  80   0 - 32756 poll_s  2018 pts/2    18:20:36 python ./seismon_epics_ioc_v3.py
0 S controls  4532  4301  0  80   0 -  2350 pipe_w 14:21 pts/0    00:00:00 grep --color=auto seis

That should be it.


 

Comments related to this report
jeffrey.kissel@LIGO.ORG - 14:34, Tuesday 05 February 2019 (46796)
FS = "file system"

These recovery instructions are for issues with Seismon are related to cdsfs0 going down on Monday afternoon. See some details from Carlos / Jonathan in 46767.
jim.warner@LIGO.ORG - 09:17, Wednesday 06 February 2019 (46823)CDS

I forgot to add that the USGS java client needs to be running on the seismon machine as well. This lives in /usr/local/seismon/ProductClient, there is a script called init.sh. This is launched with the command:

controls@seismon:ProductClient 0$ ./init.sh start

However, the client often gets confused or something by old data in the /usr/local/seismon/ProductClient/data folder, it's probably best to remove this folder:

controls@seismon:ProductClient 0$ rm -rf data/

When the client starts, it will populate this folder and subfolders. I think it works okay to remove this folder while the client is running, but if that doesn't work the USGS client can be stopped with:

controls@seismon:ProductClient 0$ ./init.sh stop

 

Displaying report 1-1 of 1.