TJ, Jonathan, Dave:
Late last week Jonathan installed his system monitor EPICS-IOC on h1guardian1 and we striptool'ed it over the past 4 days. The trends have revealed a memory leak, with the available memory dropping from 80% to 40% and occasionally resetting back (see attached striptool plot).
The times the memory is recouped is coincident with seismic blend switching, which is usually associated with operator earthquake switching. TJ is taking the investigation further.
I forgot to mention the system load fluctuates at these times as well.
After Dave found that the SEI_CONF node was changed to EARTHQUAKE the time that the memory % freed up a bit, I tried each of the 3 types of nodes changed in that state: HPI sensor correction, ISI sensor correction, and ISI blends. The first didn't change anything, but changing the ISI_BS_ST1_BLND node made an immediate change.
These nodes will sit in its idle state after making the blend transition, and the only thing running is a decorator checking the configuration. This decorator is creating a new LIGOBlendManager class reference every cycle. I thought that the end of each new cycle would release this from memory, but perhaps I was wrong. I will test this in the coming days, when given the opportunity.
In the mean time, if memory becomes an issue, we can always re-request the same blend. It will blend to the same one, and no harm will be done to the IFO.
Potential offender here:
def check_blend_config(config):
"""
dof_dict - Dictionary with the dofs as the keys and
what should be the correct fm#s as their values
"""
config_name, dof_dict = config
for dof, fm in dof_dict.items():
if fm == 0:
continue
cur_fm = LIGOBlendManager([dof], ezca=ezca).get_cur_blend_name(dof)[0]
if cur_fm != fm:
return False
return True
I ran a test today of commenting out that decorator, reloading all of the blend nodes, and then watching the available memory rate and CPU load. This definitely seems to be the problem.
In the attached trend the large drop in the CPU load (blue) was me reloading the blend nodes without the decorator. The available memory (yellow) from before this point was steadily declining, but after was basically stable. I did not get the chance to change blend states, so it makes sense that we did not free up more memory like we had seen previously.
