The PSL tripped 4 times last night, 3 of them in quick succession. I've done some preliminary forensics, and all 4 were trips of the "Head 1-4 Flow" watchdog, with the Head 3 flow sensor being the culprit in every trip.
Trip #1: 5:07 UTC (21:07 PST 2017-3-5)
PSL tripped due to a trip of the "Head 1-4 Flow" interlock, caused by the flow reading from the Head 3 flow sensor dropping below the trip point. While restarting the laser after this trip, it took a couple attempts to get the crystal chiller to start and stay running. After the first attempt, the reported flow from the Head 3 flow sensor did not get above the trip point, so the chiller shut off. The chiller restarted without issue on the 2nd attempt and the PSL restart went smooth from there. Once the system was up and running, Jim had to reset the noise eater.
The first attachment shows the flow readings from the active laser head flow sensors along with the "Head 1-4 Flow" interlock signal. The second attachment shows a slightly zoomed in view of the reading from the Head 3 flow sensor at the time of the trip. As can be seen, the flow signal from Head 3 became ragged and dropped below the trip point, shutting down the system.
Trip #2: 5:47 UTC (21:47 PST 2017-3-5)
The PSL tripped again, for the same reason as trip #1. This was also caused by the flow reading from the Head 3 flow sensor dropping below the trip point. The third and fourth attachments show info for this trip, showing that again Head 3 was the cause. The restart went smooth, and Jim had gone to the LVEA to reset the noise eater when...
Trip #3: 6:02 UTC (22:02 PST 2017-3-5)
The third PSL trip occurred while Jim was in the LVEA resetting the noise eater. This trip was identical to the last 2. The fifth and sixth attachments show that once again the flow reading from the Head 3 flow sensor was the cause of the trip; I also included the power output from the HPO in the fifth attachment. As can be seen in the fifth attachment (PSL_Trip3_Head1-3_Flow_2017-3-6_06:02:06.png) there was another issue with restarting the crystal chiller; just over 30 seconds after restarting the chiller it shut down due to the flow reading from the Head 3 flow sensor. The second chiller restart attempt was successful. There were then zero issues in restarting the PSL; Jim had to reset the NPRO noise eater again once the rest of the system was up and running.
Trip #4: 7:59 UTC (23:59 PST 2017-3-5)
The fourth and final trip occurred just as TJ was taking over for the OWL shift. Once again this was a trip of the "Head 1-4 Flow" interlock caused by the flow reading from the Head 3 flow sensor dropping below the trip point; this is shown in the final 2 attachments. There were no issues restarting the PSL after this trip. Once everything was up and running, TJ had to reset the noise eater twice and was then on to locking the IFO.
Filed FRS 7559 for these trips.