Corey called shortly after 14:00 PDT and informed me that the TCSy laser was off. Looking at the MEDM screen, the RTD/IR SENS. ALARM was red and the laser was off. I went to the LVEA and reset the laser at the front panel; I also took a picture of the front panel before I reset it, I'll attach it as a comment (it's on my phone at the moment). This cleared the temp alarm and the laser restarted without issue.
Trending back with ndscope (see 1st attachment), the laser tripped off around 11:41am PDT. The laser temperature was holding steady at the time of the trip, so the laser did not overheat. While the laser was off, the chiller setpoint was stepping from 20 °C to 21 °C in 0.1 °C steps; it went through this a couple times, as seen in the attachment. Seeing this, I went out and checked on the chiller and found it was reporting a water temperature of 22.3 °C; the setpoint for the laser after restarting was 20.9 °C. The 2nd attachment shows the water temp reported by the chiller starting . At the time of the trip the chiller was reporting a water temp of 22.0 °C, and is currently reporting 22.1 °C (There appears to be a descrepancy between the chiller setpoint and the water temp reported by the chiller, which I have never noticed before. I.E. before the trip the setpoint was just over 20 °C, while the chiller was reporting a water temp of 22.0 °C. Is this normal? Or could this be part of the cause of the trip?). At this point it is unclear what caused the laser to trip. Investigation continues.
I should also note, while I was looking at the TCSy chiller, I ran into Jeff B. He pointed out something interesting, the small display screen on the TCSx chiller has gone blank. The chiller is still running and the TCSx laser appears to be working fine, so it appears the display has simply quit working (I've never seen them blank before (like one would expect when a display goes to sleep), everytime I go to check on the chillers they displays are active).
Edited at 15:35 PDT as I hit the POST button a little too early.
A couple pictures. The first is the front panel for the TCSy CO2 laser. Notice that the IR Fault light is not lit, but the Temp alarm is. I seem to remember that when the viewport IR sensor trips, the IR Fault light flashes but does not hold, but I don't think the Temp alarm accompanies this. So I think this rules out the viewport IR sensor.
The 2nd attachment is the blank display on the TCSx chiller, while the 3rd is the active display on the TCSy chiller.
Entered FRS 13309 for this trip.
The chiller screen has been blank since January when Patrick reenabled the serial communication (alog46289).
The discrepencies in the tempuratures is interesting. I think this may be a good lead because these TCSY lock losses have become more frequent when we put the spare chiller in.
I think TJ might be on to something here. I trended the power and temp of the TCSy CO2 laser, and the temp reported by the TCSy chiller, through the month of January 2019, bookending the chiller swap that occured on Jan 15 2019. After the swap the laser temp increased from ~24.5°C to ~25 °C. It hovered there for about 3 days or so, then jumped up to ~ mid-26 °C, with jumps up to over 27 °C. In addition to this, the laser temperature became much more erratic after the swap, and that behavior has continued to today. It should be noted that the TCSx CO2 laser sits at a laser temperature of ~24.3°C, around where the TCSy CO2 laser temp used to sit before the chiller swap. Next step is to look back into 2018 to see if the stable pre-chiller-swap TCSy laser temperature behavior is consistent or a fluke. Another interesting thing to check is if the TCSy CO2 PZT is moving more than the TCSx CO2 PZT. I seem to recall seeing an alog about this, I'll hunt it down and link it here if it's relevant.
More data mining.
1st attachment is the laser power and temperature, as well as the chiller setpoint and the temp reported by the chiller itself, for the 4th quarter of calendar year 2018 (Oct-Dec 2018). As can be seen, even though there are spikes, the temp is consistently in the low-24s °C, not bouncing around as seen after the chiller swap (in the plot I posted in the 3rd comment above). Also of note, the discrepency between the temp reported by the chiller and the chiller setpoint is present even on the original TCSy chiller. That said, this is further evidence that the spare TCS chiller doesn't seem able to hold the laser at a consistent operating temperature.
Also, I found the alog regarding PZT movement between the 2 TCS CO2 lasers; TJ noted this on July 10th in this alog. Taking this a step further, I plotted out the PZT signal for each CO2 laser for the month of January 2019 (as I did with the laser temp yesterday), as well as from Dec 2018 to Feb 2019 (inclusive; the cursors indicate the TCSy chiller swap). The CO2y laser PZT (PZTy from here on, easier to type) does move more than the CO2x laser PZT (PZTx), but the difference in PZTy movement pre- and post-chiller swap is less pronounced but noticable. This also holds when looking at the longer 3 month trend (interesting note: at times, PZTx moves more than PZTy, but it's rare). I then looked at the PZT movement from Jan 1 2019 to now (final attachment, chiller swap again indicated by the cursors). PZTy in general is more active that PZTx and has been for some time. There are also stretches after the chiller swap where PZTy is moving more than its new norm, but this does not appear to be getting worse as time goes on.
