evan.hall@LIGO.ORG - posted 19:40, Wednesday 06 May 2015 - last comment - 07:48, Thursday 07 May 2015(18285)
PSL tripped
Tripped around 2015-05-07 02:31:30 UTC. I reset it. Aside from the usual diode flow bit flipping issue, the diode chiller again appeared to have some unphysically fast jump in temperature for about 30 s.
Comments related to this report
evan.hall@LIGO.ORG - 20:37, Wednesday 06 May 2015 (18286)
PSL tripped again, around 2015-05-07 02:47:50 UTC. This time, the computer in the diode room showed a momentary dip in the diode chiller flow rate (see attached photo). I talked to Rick on the phone, and we agreed to restart it again.
Images attached to this comment
evan.hall@LIGO.ORG - 01:38, Thursday 07 May 2015 (18287)
These trips are really an impediment to progress, not only because of the time it takes to restart the PSL but also the time it takes to relock the IFO and get back to low noise. If it is an option to replace the bad sensor soon, that would be good, although I think I remember from when Michael Rodruck was struggling with the same issue it was not a simple fix.
These recent PSL trips, as well as the power watchdog trips, are an example of how we have made the system less robust by being overzealous in trying to protect it.
Both the recent trips and many trips a few years ago when Michael Rodruck was still here were due to bad sensors. These sensors broke and indicated that the flow was low, when the flow was not actually low and there was no real danger to the laser.
Do we have any redunant sensors, so that we could check that mutliple sensors agree that the flow is low before we turn off the laser?
We see H1:PSL-IL_DCHILFLOW flipping for a while and then it stops flipping after 30 seconds in these recent spurious laser trips. Does this bit go to 0 and stay there if the flow is actually low? If so we could wait to trip until this bit is low for at least half a second, or when a rolling average of the bit drops below half.
The power watchdog seems like another example of overzealous protection of the hardware. (The power watch dog has tripped less often recently, I assume that is because the PSL maintence is now being done) This is a watchdog that goes off when the PSL power becomes low. It seems to me like a drop in the power is not a dangerous situation. I think it would be best to downgrade the power watchdog to an alarm or warning, or anything else that does not bring a multi million dollar facility to a standstill.
richard.savage@LIGO.ORG - 07:48, Thursday 07 May 2015 (18290)
These laser trips are, of course, a big concern to the PSL "team." We are actively pursuing solutions.
For the past month or so we have been opeating without engaging the watchdog on the Front End laser.
We are trying to understand the functionality of the Beckhoff interlock which is appears is responsible for initiating the shutdowns. This week, we received some information from Maik Frede at LZH. He wrote the control software. It appears that the system is not functioning as designed and described in LIGO-T1000005.
Yesterday, Peter King and Jeff Bartlett replaced the flow sensors in the spare chillers with units that don't have moving parts. We have some indication that the flow sensors originally installed and currently operating (the ones that Michael Rodrick) was replacing, are the cause of the triggers. As soon as possible, we plan to either swap in the spare chillers (with the new sensors) or shut down for an hour or so to replace the sensors in the operating chillers.
Jeff, Peter, Jason, Ed Merilh and I are meeting at 9:00 this morning to discuss progress on several fronts in trying to understand and address these shutdowns.
PSL tripped again, around 2015-05-07 02:47:50 UTC. This time, the computer in the diode room showed a momentary dip in the diode chiller flow rate (see attached photo). I talked to Rick on the phone, and we agreed to restart it again.
PSL tripped a third time, around 04:23:00 UTC. I reset it again.
These trips are really an impediment to progress, not only because of the time it takes to restart the PSL but also the time it takes to relock the IFO and get back to low noise. If it is an option to replace the bad sensor soon, that would be good, although I think I remember from when Michael Rodruck was struggling with the same issue it was not a simple fix.
These recent PSL trips, as well as the power watchdog trips, are an example of how we have made the system less robust by being overzealous in trying to protect it.
Both the recent trips and many trips a few years ago when Michael Rodruck was still here were due to bad sensors. These sensors broke and indicated that the flow was low, when the flow was not actually low and there was no real danger to the laser.
Do we have any redunant sensors, so that we could check that mutliple sensors agree that the flow is low before we turn off the laser?
We see H1:PSL-IL_DCHILFLOW flipping for a while and then it stops flipping after 30 seconds in these recent spurious laser trips. Does this bit go to 0 and stay there if the flow is actually low? If so we could wait to trip until this bit is low for at least half a second, or when a rolling average of the bit drops below half.
The power watchdog seems like another example of overzealous protection of the hardware. (The power watch dog has tripped less often recently, I assume that is because the PSL maintence is now being done) This is a watchdog that goes off when the PSL power becomes low. It seems to me like a drop in the power is not a dangerous situation. I think it would be best to downgrade the power watchdog to an alarm or warning, or anything else that does not bring a multi million dollar facility to a standstill.
Sheila, Evan, et al.,
These laser trips are, of course, a big concern to the PSL "team." We are actively pursuing solutions.
For the past month or so we have been opeating without engaging the watchdog on the Front End laser.
We are trying to understand the functionality of the Beckhoff interlock which is appears is responsible for initiating the shutdowns. This week, we received some information from Maik Frede at LZH. He wrote the control software. It appears that the system is not functioning as designed and described in LIGO-T1000005.
Yesterday, Peter King and Jeff Bartlett replaced the flow sensors in the spare chillers with units that don't have moving parts. We have some indication that the flow sensors originally installed and currently operating (the ones that Michael Rodrick) was replacing, are the cause of the triggers. As soon as possible, we plan to either swap in the spare chillers (with the new sensors) or shut down for an hour or so to replace the sensors in the operating chillers.
Jeff, Peter, Jason, Ed Merilh and I are meeting at 9:00 this morning to discuss progress on several fronts in trying to understand and address these shutdowns.