Likely due to wind which is posting at 40+ mph right now. Looks like it could be a rough night with the Red Flag warning for thunderstorms and high wind throughout the night.
About 30 seconds after receiving the GRB alert, we dropped out of Observe due to TCS issues. I consulted with Keita who advised to treat this as a normal GRB alert even thought we were out of Observe. I thus began a stand down.
(Filiberto, Gerardo)
Pulled the controls signal cables for the new turbo/roughing pumps for Y-Beam manifold, X-Beam manifold and to a location near HAM5.
All six ends of the cables still need to be terminated.
At the same time, we removed from the cable trays some controls signal cables for the old roughing pumps that were removed several years ago.
Work done per WP#8290.
At 0:44 UTC, we were kicked out of Observing by TCSy CO2. At Keita's request, I trended the Laser Temp, PZT Readback, and Laser HD PD around the time of the dropout (see attached screenshot). Something definitely happened here as the Laser HD PD output went from 49.4 before the dropout to 49.8 afterwards.
Dick Gustafson, Marc Pirello, and I collected profiles of unwanted RF emission on ISC-R1, ISC-R2, ISC-R3, and ISC-C4 in preparation for a future RF noise injection study. We focused on cables carrying 9.1 MHz, 45.5 MHz, and 79.4 MHz. Measurements occurred between 16:00 UTC and 19:00 UTC (9:00 PDT and 12:00 PDT).
Alogging that when Peter, Karen and Vanessa were in the PSL enclosure, I tagged along and took IR pictures of the usual IO subjects, and a few new optics/sensors. Will add images showing beams.
Accepted a few tiny SDF diffs and one (see attached) for ECAT PLC1.
WP8280 h1tw0 raw minute data offload
Dave:
I completed the raw minute trend data offload from SSD on h1tw0 to spinning media on h1ldasgw0. h1nds0 was restarted to serve the past 5 months of min trends from their permanant location and the temporary directory on h1tw0 was erased (266,000 files took 2hr15min).
DAQ E80 Raid time zones
Dan, Dave:
We standardized the E18 controllers to use local time zone for its logs and reports.
After DRMI locked, I started getting saturation alarms related to OM1 and OM2 Pit. We lost lock at this point, but I tracked them down and cleared the histories in the locking filters.
TITLE: 07/23 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
Wind: 11mph Gusts, 8mph 5min avg
Primary useism: 0.06 μm/s
Secondary useism: 0.05 μm/s
QUICK SUMMARY: Relocking after the discovery of the TCSy CO2 laser trip and re-thermalizing.
TITLE: 07/23 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Travis
SHIFT SUMMARY:
During a fairly busy Maintenance, I did not hear that the TCSy laser went off at 11:42am. Noticed it at about 2:14pm, so ITMy was cold for almost 3hrs. These extra hours of downtime needed to heat up ITMy were totally my fault!
LOG:
Since we had a cold ITMy for almost 3hrs (I did not hear the verbal alarm warning about the TCSy laser during Maintenance.), we are waiting for ITMy to thermally equalize....we're getting there, but there continues to be some drift.
To help speed up recovery, Keita changed the Requested Power for the TCSy Laser from
While locking, PRMI flashes looked fairly ugly. When we did lock, POP18/90 had a slow thermal drift.
Eventually locked DRMI (but had a lockloss). Now that we may be equalizing thermally, it might be useful to run a new alignment (I ran one when the ITMy was cold!). But Travis is going to give it a go without running an alignment for a first attempt.
Corey called shortly after 14:00 PDT and informed me that the TCSy laser was off. Looking at the MEDM screen, the RTD/IR SENS. ALARM was red and the laser was off. I went to the LVEA and reset the laser at the front panel; I also took a picture of the front panel before I reset it, I'll attach it as a comment (it's on my phone at the moment). This cleared the temp alarm and the laser restarted without issue.
Trending back with ndscope (see 1st attachment), the laser tripped off around 11:41am PDT. The laser temperature was holding steady at the time of the trip, so the laser did not overheat. While the laser was off, the chiller setpoint was stepping from 20 °C to 21 °C in 0.1 °C steps; it went through this a couple times, as seen in the attachment. Seeing this, I went out and checked on the chiller and found it was reporting a water temperature of 22.3 °C; the setpoint for the laser after restarting was 20.9 °C. The 2nd attachment shows the water temp reported by the chiller starting . At the time of the trip the chiller was reporting a water temp of 22.0 °C, and is currently reporting 22.1 °C (There appears to be a descrepancy between the chiller setpoint and the water temp reported by the chiller, which I have never noticed before. I.E. before the trip the setpoint was just over 20 °C, while the chiller was reporting a water temp of 22.0 °C. Is this normal? Or could this be part of the cause of the trip?). At this point it is unclear what caused the laser to trip. Investigation continues.
I should also note, while I was looking at the TCSy chiller, I ran into Jeff B. He pointed out something interesting, the small display screen on the TCSx chiller has gone blank. The chiller is still running and the TCSx laser appears to be working fine, so it appears the display has simply quit working (I've never seen them blank before (like one would expect when a display goes to sleep), everytime I go to check on the chillers they displays are active).
Edited at 15:35 PDT as I hit the POST button a little too early.
A couple pictures. The first is the front panel for the TCSy CO2 laser. Notice that the IR Fault light is not lit, but the Temp alarm is. I seem to remember that when the viewport IR sensor trips, the IR Fault light flashes but does not hold, but I don't think the Temp alarm accompanies this. So I think this rules out the viewport IR sensor.
The 2nd attachment is the blank display on the TCSx chiller, while the 3rd is the active display on the TCSy chiller.
Entered FRS 13309 for this trip.
The chiller screen has been blank since January when Patrick reenabled the serial communication (alog46289).
The discrepencies in the tempuratures is interesting. I think this may be a good lead because these TCSY lock losses have become more frequent when we put the spare chiller in.
I think TJ might be on to something here. I trended the power and temp of the TCSy CO2 laser, and the temp reported by the TCSy chiller, through the month of January 2019, bookending the chiller swap that occured on Jan 15 2019. After the swap the laser temp increased from ~24.5°C to ~25 °C. It hovered there for about 3 days or so, then jumped up to ~ mid-26 °C, with jumps up to over 27 °C. In addition to this, the laser temperature became much more erratic after the swap, and that behavior has continued to today. It should be noted that the TCSx CO2 laser sits at a laser temperature of ~24.3°C, around where the TCSy CO2 laser temp used to sit before the chiller swap. Next step is to look back into 2018 to see if the stable pre-chiller-swap TCSy laser temperature behavior is consistent or a fluke. Another interesting thing to check is if the TCSy CO2 PZT is moving more than the TCSx CO2 PZT. I seem to recall seeing an alog about this, I'll hunt it down and link it here if it's relevant.
More data mining.
1st attachment is the laser power and temperature, as well as the chiller setpoint and the temp reported by the chiller itself, for the 4th quarter of calendar year 2018 (Oct-Dec 2018). As can be seen, even though there are spikes, the temp is consistently in the low-24s °C, not bouncing around as seen after the chiller swap (in the plot I posted in the 3rd comment above). Also of note, the discrepency between the temp reported by the chiller and the chiller setpoint is present even on the original TCSy chiller. That said, this is further evidence that the spare TCS chiller doesn't seem able to hold the laser at a consistent operating temperature.
Also, I found the alog regarding PZT movement between the 2 TCS CO2 lasers; TJ noted this on July 10th in this alog. Taking this a step further, I plotted out the PZT signal for each CO2 laser for the month of January 2019 (as I did with the laser temp yesterday), as well as from Dec 2018 to Feb 2019 (inclusive; the cursors indicate the TCSy chiller swap). The CO2y laser PZT (PZTy from here on, easier to type) does move more than the CO2x laser PZT (PZTx), but the difference in PZTy movement pre- and post-chiller swap is less pronounced but noticable. This also holds when looking at the longer 3 month trend (interesting note: at times, PZTx moves more than PZTy, but it's rare). I then looked at the PZT movement from Jan 1 2019 to now (final attachment, chiller swap again indicated by the cursors). PZTy in general is more active that PZTx and has been for some time. There are also stretches after the chiller swap where PZTy is moving more than its new norm, but this does not appear to be getting worse as time goes on.
(Corey & Richard)
All three were swept post-Mainteance.
Today I went out to End-Y to take a look at the ALS green path on ISCTEY to try to hunt down the source of the bad mode matching to TMSy (and therefore the Y-arm) that we've been seeing here. Based on the work done back in June (at this alog), the first thing I did was take a detailed look at the beam alignment up to ALS-M7. To my eye, the beam looks to be off-center horizontally on ALS-L4 by roughly 1 beam diameter in the -X direction (towards ISCTEY's door, away from the chamber). In addition, the beam looks to be slightly off (1 beam radius or less, also horizontal) on ALS-L5, in the -Y direction (ALS-L5 is oriented 90° w.r.t. ALS-L4). And in addition to that, I suspect that the beam may be clipping on the entrance aperture to the EOM. This is very difficult to tell as the laser safety goggles render the green beam invisible, but the beam on the card (which is a horribly inaccurate measure of beam size, but it's what I had to work with) is definitely larger than the EOM entrance aperture (next time I'm at EY I'll try to get a picture, hopefully my old phone can catch it). A second set of eyes taking a look at this will help confirm or deny whether or not my eyes are accurately seeing this apparent misalignment. The maintenance window ran out on me so I didn't get a chance to get any beam profile measurements today; I'll head back down with the beam profiler next Tuesday for Round 2 to see if there's a point in the path where the beam becomes elliptical, or if the ellipticity seen in the alog linked above is coming from the laser itself.
I also want to mention that the beam is very off-center horizontally on ALS-HWP5 (after the 2nd Faraday isolator in the green path). It's so far off-center that my first thought is that it was done intentionally, but I didn't build the table layout so I don't know this for sure (hence my reporting it here).
{Kyle, Chandra}
Kyle taught me how to operate the Hilti core drill today and I was able to core out two 1-1/8" diam. holes at the y-beam manifold turbo station in preparation for new stand installation. I also hammer drilled the anchor inserts into holes.
We left the 100' long water hose coiled near sink in west bay and plan to use it at x-beam manifold turbo station next.
Performed semi-annual and quarterly lubrication on axial vane fans per WP 8286. Also rotated chillers at all out buildings and corner station. All out buildings are on AHU and Chiller 2. Corner station is on Chiller 1. All chillers and AHUs are rotated on at least a quarterly basis when lubrication is performed.
The Laser Room walls and floor were wiped down/mopped this morning. Some white flecks were
observed near where the floor standing PEM speaker is. Don't recall ever seeing them before.
Fire extinguisher checks out okay. Last inspection was back in March 2019.
Vanessa, Karen, Peter