16:05-16:31: H1 Out Of OBSERVING To Correct ISC_LOCK.py code (Range was 0Mpc during this time!)
Yesterday during my shift, I noticed that the ISC_LOCK log was continually running a line of script (a few times a second). I sent an email to commissioners & JeffK pointed me to the appropriate alog related to this change (Heather/Sheila alog33437). It sounds like this Reset button was being continuously looped & this is the channel:
ISC_LOCK [NOMINAL_LOW_NOISE.run] ezca: H1:OAF-RANGE_RLP_4_RSET => 2
So, yesterday Heather & TJ took a look at these lines of code to fix this (We just wanted this Reset to happen once after a lock). And then the operator was to wait for an appropriate time to hit LOAD on the ISC_LOCK. This morning I saw that L1 was down, so I took H1 out of OBSERVING & hit the LOAD, but I received an ERROR for the ISC_LOCK guardian node.
Instead of delving much into the code, I immediately made a call the the Guardian Help Desk (i.e. Jamie). I texted him photos of the error message & then he found the issue (it was a missing Closing Parenthesis). Once this was corrected, saved ISC_LOCK.py, hit LOAD on ISC_LOCK node, & the RED ERROR went away. I then went back to OBSERVING.
NOTE: During this time, H1 Range went to 0Mpc, BUT we were still at NLN the entire time. So our current lock is approaching 24hrs.
Here is the Error which came up on the ISC_LOCK log, when I initially pressed LOAD:
2017-02-03T16:27:08.58122 File "/opt/rtcds/userapps/release/isc/h1/guardian/ISC_LOCK.py", line 3994
2017-02-03T16:27:08.58123 for blrms in range(1,11):
2017-02-03T16:27:08.58123 ^
2017-02-03T16:27:08.58124 SyntaxError: invalid syntax
2017-02-03T16:27:08.58134 ISC_LOCK LOAD ERROR: see log for more info (LOAD to reset)
Here are the lines in question from ISC_LOCK.py code (the parenthesis which was missing is highlighted below)
3992 subp.call(['/opt/rtcds/userapps/trunk/isc/h1/guardian/./All_SDF_observe.sh'])<--------- !
3993 #clear history of blrms
3994 for blrms in range(1,11):
3995 ezca['OAF-RANGE_RLP_{}_RSET'.format(blrms)]=2
3996 #ezca['OAF-RANGE_RBP_{}_RSET'.format(blrms)]=2
Oh, forgot to mention another change. When Jamie was looking for issues with line 39994, we did make a change to:
3994 for blrms in range(1,11):
"range" used to be something else, but I didn't capture that before we changed it. (it was something like "gnumpy.range(1,11):" before). Will this change anything from what was initially intended?
The error condition was caused by a SyntaxError exception in the code. A simple load of the code will catch these exceptions, so you can easily avoided these load-time exceptions by parsing the code with e.g. guardutil first:
$ guardutil print ISC_LOCK
If there are any syntax errors in the code, that call to guardutil will catch and print it for you, before you hand it over to the operators.
Also, we've been trying to avoid calling out to shell scripts with subprocess calls. Is there some reason this 'All_SDF_observe.sh'
script can't be properly integrated?
I also note that the return code of the script is not being checked, so if the script fails nothing will catch it. That's not good. That's one of the main reasons we avoid the subprocess calls.
"range" is similar to "numpy.arange", except it returns a simple python list instead of a numpy.array. There's no reason to use numpy for this operations.
This was definitely my fault. I'm not sure how, but I somehow deleted that parenthesis when I cut and pasted the blrms code from run to main.
I'm also a bit confused when I saw that shell script call. There is a pass directly before it, almost as if someone didn't want the script to actually be ran, but a pass doesn't work for that purpose and the script has been ran every time.