Displaying report 1-1 of 1.
Reports until 10:26, Tuesday 23 February 2016
H1 SEI
hugh.radkins@LIGO.ORG - posted 10:26, Tuesday 23 February 2016 - last comment - 09:13, Thursday 25 February 2016(25679)
All LHO HEPI Guardians report USER ERROR CODE after guardian machine restart

DaveB restarted the guardian computer to correct a tconvert issue, WP 5740.

After this, every HEPI platform guardian reports this error:

2016-02-23T17:52:24.50084 HPI_HAM1 [ROBUST_ISOLATED.enter]
2016-02-23T17:52:24.57873 HPI_HAM1 W: Traceback (most recent call last):
2016-02-23T17:52:24.57874   File "/ligo/apps/linux-x86_64/guardian-1485/lib/python2.7/site-packages/guardian/worker.py", line 459, in run
2016-02-23T17:52:24.57875     retval = statefunc()
2016-02-23T17:52:24.57875   File "/ligo/apps/linux-x86_64/guardian-1485/lib/python2.7/site-packages/guardian/state.py", line 240, in __call__
2016-02-23T17:52:24.57876     main_return = self.func.__call__(state_obj, *args, **kwargs)
2016-02-23T17:52:24.57876   File "/ligo/apps/linux-x86_64/guardian-1485/lib/python2.7/site-packages/guardian/state.py", line 240, in __call__
2016-02-23T17:52:24.57877     main_return = self.func.__call__(state_obj, *args, **kwargs)
2016-02-23T17:52:24.57877   File "/opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/decorators.py", line 86, in wrapper
2016-02-23T17:52:24.57878     return func(*args, **kwargs)
2016-02-23T17:52:24.57878   File "/opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/decorators.py", line 86, in wrapper
2016-02-23T17:52:24.57879     return func(*args, **kwargs)
2016-02-23T17:52:24.57879   File "/ligo/apps/linux-x86_64/guardian-1485/lib/python2.7/site-packages/guardian/state.py", line 240, in __call__
2016-02-23T17:52:24.57880     main_return = self.func.__call__(state_obj, *args, **kwargs)
2016-02-23T17:52:24.57880   File "/ligo/apps/linux-x86_64/guardian-1485/lib/python2.7/site-packages/guardian/state.py", line 240, in __call__
2016-02-23T17:52:24.57881     main_return = self.func.__call__(state_obj, *args, **kwargs)
2016-02-23T17:52:24.57882   File "/opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/isolation/states.py", line 490, in run
2016-02-23T17:52:24.57882     if (iso_const.ISOLATION_CONSTANTS['SWITCH_GS13_GAIN']) & (self.has_requested_gs13_gain_switch == False) & (top_const.CHAMBER_TYPE != 'BSC_ST1'):
2016-02-23T17:52:24.57883 KeyError: 'SWITCH_GS13_GAIN'
2016-02-23T17:52:24.57883
2016-02-23T17:52:24.57888 HPI_HAM1 [ROBUST_ISOLATED.run] USERMSG: USER CODE ERROR (see log)
2016-02-23T17:52:24.62596 HPI_HAM1 ERROR in state ROBUST_ISOLATED: see log for more info (LOAD to reset)
 

I've reloaded and restarted both HAM1 and HAM6 and the error returns.  It sure looks like it is related to the updates to get the ISI guardian to switch the GS13 gains depending on the state.  I can not find documentation (logs) that the HPI nodes have been restarted since the gain switching updates which were intended for the ISI.

I've elected to not attempt any code corrections at this time; I suspect Hugo will correct this easily.  Meanwhile, I don't know if the guardian will actually function if it trips.  This can be done easily however with the command scripts isolating to level1.

Comments related to this report
hugh.radkins@LIGO.ORG - 11:32, Tuesday 23 February 2016 (25681)

II 1206 FR 4418

hugh.radkins@LIGO.ORG - 09:13, Thursday 25 February 2016 (25710)

Code oversite fixed.  Guardian restarted.  All HPIs are good.  Tested on HAM6, positive function.  ISI guardians not restarted--should do next Tuesday to confirm no issues there.

 

files updated:

hugh.radkins@operator3:~ 0$ userapps
hugh.radkins@operator3:release 0$ cd isi/common/guardian/
hugh.radkins@operator3:guardian 0$ svn up
U    isiguardianlib/isolation/const.py
U    isiguardianlib/isolation/states.py
U    isiguardianlib/damping/states.py
Updated to revision 12715.
 

Displaying report 1-1 of 1.