T. Sadecki, J. Kissel, T. Hardwick, J. Rollins

After heroic efforts by the PSL team (LHO aLOGs 30217, 30216), with a restored laser Travis was able to get us most of the way up to nominal low noise. We got stuck going through LOWNOISE_ASC due to what we believe is a (new?) bug in the guardian code regarding the check for the input laser power. More details below, but we believe it's now fixed. 

We had paused for a while after increase power to make sure PI's were in check, but then we noticed suddenly, and without request, the laser power began to slowly decrease from ~48 W to ~42 W. Thinking (wildly guessing) that this had something to do with one of the many layers of ISS loops going awry, we decided to march the rest of the way up the acquisition sequence. That's when we hit the LOWNOISE_ASC bug. Once we fixed that bug, we were able to continue all the way up at ~42 W input power. Once we got to nominal low noise, I put the ISC_LOCK node into manual, and used the LASER_PWR node to push the input power back up to 48.1 W. Note, this required requesting the power to be as high as 58 W. Once at what we think is the ER10/02 desired power (and we got a big glitch in the IFO when requesting higher), I resigned the machine over to Robert to begin PEM injections.

LOWNOISE_ASC state problems:
After entering the LOWNOISE_ASC state, the ISC_LOCK guardian went red. The log file error:
    2016-10-05_00:53:38.914210Z ISC_LOCK LOAD REQUEST
    2016-10-05_00:53:38.914670Z ISC_LOCK RELOAD requested.  reloading system data...
    2016-10-05_00:53:40.781560Z ISC_LOCK module path: /opt/rtcds/userapps/release/isc/h1/guardian/ISC_LOCK.py
    2016-10-05_00:53:40.781620Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/lscparams.py
    2016-10-05_00:53:40.781720Z ISC_LOCK user code: /opt/rtcds/userapps/release/sys/common/guardian/ifolib/down.py
    2016-10-05_00:53:40.786580Z ISC_LOCK user code: /opt/rtcds/userapps/release/sys/common/guardian/ifolib/__init__.py
    2016-10-05_00:53:40.786700Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/ISC_library.py
    2016-10-05_00:53:40.786800Z ISC_LOCK user code: /opt/rtcds/userapps/release/als/common/guardian/alsconst.py
    2016-10-05_00:53:40.786900Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/fast_ezca.py
    2016-10-05_00:53:40.787000Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/ISC_GEN_STATES.py
    2016-10-05_00:53:45.923730Z ISC_LOCK RELOAD complete
    2016-10-05_00:53:47.826060Z ISC_LOCK W: Traceback (most recent call last):
    2016-10-05_00:53:47.826090Z   File "/ligo/apps/linux-x86_64/guardian-1.0.2/lib/python2.7/site-packages/guardian/worker.py", line 461, in run
    2016-10-05_00:53:47.826100Z     retval = statefunc()
    2016-10-05_00:53:47.826100Z   File "/ligo/apps/linux-x86_64/guardian-1.0.2/lib/python2.7/site-packages/guardian/state.py", line 246, in __call__
    2016-10-05_00:53:47.826110Z     main_return = self.func.__call__(state_obj, *args, **kwargs)
    2016-10-05_00:53:47.826120Z   File "/ligo/apps/linux-x86_64/guardian-1.0.2/lib/python2.7/site-packages/guardian/state.py", line 246, in __call__
    2016-10-05_00:53:47.826120Z     main_return = self.func.__call__(state_obj, *args, **kwargs)
    2016-10-05_00:53:47.826130Z   File "/opt/rtcds/userapps/release/isc/h1/guardian/ISC_LOCK.py", line 3334, in run
    2016-10-05_00:53:47.826130Z     if self.low_power:
    2016-10-05_00:53:47.826140Z AttributeError: 'LOWNOISE_ASC' object has no attribute 'low_power'                                     <<< HERE'S THE PROBLEM
    2016-10-05_00:53:47.826140Z
    2016-10-05_00:53:47.862420Z ISC_LOCK ERROR in state LOWNOISE_ASC: see log for more info (LOAD to reset)
    2016-10-05_00:59:34.968070Z ISC_LOCK LOAD REQUEST
(Note for future reference, I was able to pull up the log file from the command line using guardlog --hours 2 ISC_LOCK > ISC_LOCK_log.txt, and then grep'ing for the error I knew was there. I'm not sure where guardian log files live such that you can just browse them.)

Looking into the ISC_LOCK code, we found this weirdness in the LOWNOISE_ASC state after CTRL+F searching for "low_power":
class LOWNOISE_ASC(GuardState):
    index = 507

    @ISC_library.get_watchdog_IMC_check_decorator(nodes)
    @nodes.checker()
    def main(self):
        if ezca['IMC-PWR_IN_OUT16'] > 45:             % This if statement is bonkers, and contradictory to the run state's conditional
            notify('power too Low!')
            self.low_power=True
        self.timer['LoopShapeRamp'] = 5
        self.counter = -1

    @ISC_library.get_watchdog_IMC_check_decorator(nodes)
    @nodes.checker()
    def run(self):
        if self.low_power:
            if ezca['IMC-PWR_IN_OUT16'] > 45:
                self.low_power=False
            else:
                notify('power too low!')
                self.low_power=True
        else:
            if self.counter ==-1:
                ezca.switch('ASC-CHARD_Y', 'FM6', 'ON') # Move compensation peaks for CHARD Yaw
        ...
        ...
        (etc)


Also, the real problem was that self.low_power wasn't initialized in the main portion of the code. So, Terra added the following initialization, and flipped the sign of the conditional statement in the main portion:
class LOWNOISE_ASC(GuardState):
    index = 507

    @ISC_library.get_watchdog_IMC_check_decorator(nodes)
    @nodes.checker()
    def main(self):
        self.low_power=False                # Initialize low_power attribute, 10/04/2016, see LHO aLOG 30222
        if ezca['IMC-PWR_IN_OUT16'] < 45:             # changed from > 10/04/2016, see LHO aLOG 30222
            notify('power too Low!')
            self.low_power=True
        self.timer['LoopShapeRamp'] = 5
        self.counter = -1

    @ISC_library.get_watchdog_IMC_check_decorator(nodes)
    @nodes.checker()
    def run(self):
        
    ...
    ...
    (etc)

With this bug fix, we were able to cruise through LOWNOISE_ASC. Really weird that this bug cropped up. Maybe some half-finished improvement to the state?