I'm guessing that the TCS-Y symbols are related to the fact that this laser tripped right when you guys were refilling the water for some reason (9:45am).

Yes this is normal. Means that the chiller is heating with the arrows pointing up (or cooling when pointing down) so that the water is at the setpoint of the chiller

At Richard's request, I inspected the inside of both TCS tables, and found nothing, no drips and no sign of water, the tables are dry.

I inspected TCSY first, then TCSX.

Also, apparently both Gerardo and I, separately, walked the length of the piping and found no leaks.

 

Attached is this morning's story of laser power.

It looks like there is a small increase in temperature on the laser which causes a power output increase from the laser.  Following that the laser trips off, and the Guardian code then starts trying to relock the laser.  

We'll need to  think about the right way to get Guardian to "see" that the laser has tripped off - we should get it to check the laser error channels.  Anyway that's why the temperature of the chiller has those steps in it.

I'm not sure how sensibly Guardian will react if you just restart the laser without restarting Guardian.  The reason I say this is because it typically takes ~30mins for the laser to come into thermal equilibrium during which time Guardian won't be able to lock it.  Also while Guardian is hunting for a locking point it will occasionally be stepping the chiller temperature (as shown in the chart) which may in turn increase the time taken for the laser to reach an equilibrium temperature.  This logic isn't well baked in to the current Guardian code which at the moment locks the laser when it is reasonably stable (ie after 30mins of running unlocked).

I'm thinking the anomolous note Vern mentions in the first post on this thread is the culprit for the laser trip:  The indicator float bobbled way low and then rose while they were filling.  So, I think an air bubble pushed out from under the filter and dropped the level briefly, thus triggering a low water level trip.

 

This adds to my thought that maybe the mystery water "loss" issue is actually not a water loss at all, but an air loss.  Maybe we didn't fill the chiller all the way up when we found the chiller dry on Wed, and we are still "topping it off'.  as I pointed out last week, the indicator goes/stays low after you push air out from under the filter.  Today, I played with this more and was again able to bleed more air out from under the puffy filter fabric which then left the indicator low.  I had to fill another 225mL just in playing with it.  Attached are pix of the X and Y filter while water is running thru.  Note how the Y filter fabric is being pushed up towards the surface of the water, while the X one has more of a trough down the middle?  (Use your imagination, it is much more obvious when standing over it and pushing it around with a gloved hand or hose tool). 

 

They added 1.3gal (5L) on Wed when they found it dry.  We've added ~a gallon (3800mL) since then, in small ~500mL batches.  I'm looking up the capacity of the chiller...

The chiller reservoir volume is listed as 7.2L (1.9 gal).

We've added ~5L plus ~3L since Wed - given some error bar on these plus a little reasonable amount of bubble bleeding, we might find that the indicator reading settles out at "full".  That's one theory anyways...

We have inspected the tables, the piping and the chillers 3 times since last Tues and cannot find any loss of water.

Completed Recovery in ~30min with Jason on the phone.  Back to locking at 12:42pm.

NOTE:  LRA, stands for Long Range Actuator.

The Xtal chiller has tripped 3 times in the last week:

• 9/25 (29964) - added 200 mL water - "normal due to known slow leak"

then 4 days later:

• 9/29 (30063) - added a "few 100 mL" water - water sprayed onto door

now 3 days later:

• 10/2 (30150) - "cap blew off" - added 375 mL

The last two events imply an overpressure condition. The problem seems more significant than replacing lost fluid from a slow leak.

Was the cap which "blew off" in today's event the "bleeding cap" (section 4.2.2), the "filter sleeve cap" (section 6.2) or the "filler pipe cap" (section 6.3 of T1100374-v1, "200 W Laser Crystal Chiller Manual")?

 

BTW, LRA = Long Range Actuator; see section 7.2 of  T0900641-v5, "Under Manual 200 W Laser".

From my quick look through the alog I think the PSL has tripped more than this (for LHO to concur). From what I can see it has gone off 11 times in the last week (thats as far as I looked back).

 

I'm not sure if all the same problems, but alogs reporting the PSL laser off are LHO alogs:

10/2/2016

LHO alog 30160

LHO alog 30154

10/1/2016

LHO alog 30146

LHO alog 30143

9/30/2016

LHO alog 30118

LHO alog 30086  (due to power glitch)

9/29/2016

LHO alog 30076

LHO alog 30063 (this alog reports two different  instances of the laser going off)

9/27/2016

LHO alog 30016

9/25/2016

LHO alog 29964

 

The filler pipe cap popping out is a known thing (happens all the time at LLO) when the chiller turns OFF. At LLO at least this has not shown to be due to any problem (just a consequence for whatever reason when the chiller is turned off).  Its why we try to not turn these chillers OFF if can help it as they "burp" water over the floor and pop these fill caps even when trying to restart

 

I have as one of the main agenda items of this Wednesdays PSL meeting to discuss this problem and see if we can work out whats going on. Some statistics on how many times happened in say the last month or two, what the PSL trip was attributed to, and how many times happened before and after the chiller swap (to see if accelerating or at the same rate), would help this. Im not sure if FRS fault reports have been made for each laser trip to make this search easy for us remote to the site to do and work out how much lost observatory time we have had due to this issue.

Yes, Dennis, as Matt says it's the Filler Pipe Cap that pops off.  We have had (3) trips since Friday evening (so Matt is probably right about there being more trips over the whole week).  OH, and I should correct ourselves here because we just had a 4th Weekend PSL trip (this was just after I had H1 at NLN for 15min.  This time the cap was blown off and there was a puddle on the floor.

Able to get back in 30min this time (vs 60min this morning).

OK, back to locking.  

Another Note:

Something I wanted to add about the chiller was that when filling it, I noticed quite a bit of turbulence in the fill pipe.  And you could see a air bubble vortex/tornado in there.  Something we probably don't want if air bubbles are postulated as a trigger for flow sensor trips.  

We see these bubbles as well in the chiller fill pipe at LLO with no chiller trips due to it (hopefully haven't jinxed myself). Perhaps post a movie of it so can see if looks the same as here

Jason & crew will be investigaing tomorrow.  We should ask them to record a video of it.

Movie of the water turbulence of LLO's crystal chiller fill tube posted at LLO alog 28397

I took a video, but I can't post it; my phone only takes video in .mp4 format, which is apparently not a valid file type for upload to the alog.  Huh.

I attached a still from the video to give you some idea of what we've been seeing here for the last few weeks.  It's appearing to fluctuate though; the video was taken on Monday, 10/3/2016, but this morning our fill port looks very similar to what's seen in Matt's video of the LLO crystal chiller fill port.

Input Mode Cleaner:

IMC =
    lambda: 1.0640e-06
         q: 0.2325 +13.3832i
         N: 4
      dist: [16.2406 16.2406 0.2325 0.2325 0]
      ifoc: [0.0731 0 0 0]
     label: {'MC2'  'MC3'  'Waist'  'MC1'}
       Pin: 50

z= 00.0000 m, Gouy phase: 00.0000 deg after passing MC1;   Spot size w=2.1293 mm
z= 16.2406 m, Gouy phase: 49.9133 deg after passing MC2;   Spot size w=3.3764 mm
z= 32.4812 m, Gouy phase: 99.8267 deg after passing MC3;   Spot size w=2.1293 mm
z= 32.7137 m, Gouy phase: 100.822 deg after passing Waist; Spot size w=2.129 mm
z= 32.9462 m, Gouy phase: 101.8172 deg after passing MC1; Spot size w=2.1293 mm
Round trip Gouy phase:  101.8172 deg

Gouy Phase from input coupler (MC1) to output coupler (MC3): 99.8267 deg

Input mirrors: From PSL periscope to MC3:

IM =
    lambda: 1.0640e-06
         q: -0.2325 +13.3832i
         N: 6
      dist: [0.2325 0.5507 1.0613 1.1703 1.2071 0.4130 0]
      ifoc: [0 0 0.1562 -0.3199 0 0.0409]
     label: {'Waist'  'IM1'  'IM2'  'IM3'  'IM4'  'PRMlens'}

z= 0.0000 m, Gouy phase: 0.0000 deg after passing MC3;         Spot size w=2.1293 mm
z= 0.2325 m, Gouy phase: 0.99527 deg after passing Waist;      Spot size w=2.129 mm
z= 0.78319 m, Gouy phase: 3.3516 deg after passing IM1;         Spot size w=2.1308 mm
z= 1.8445 m, Gouy phase: 7.8634 deg after passing IM2;           Spot size w=2.1444 mm
z= 3.0148 m, Gouy phase: 13.8099 deg after passing IM3;         Spot size w=1.7841 mm
z= 4.2219 m, Gouy phase: 19.9983 deg after passing IM4;         Spot size w=2.1257 mm
z= 4.6349 m, Gouy phase: 21.6763 deg after passing PRMlens; Spot size w=2.2471 mm
 

Input beam: From PSL periscope to MC3:

(The PSL periscope to input viewport distance is an educated guess.)

INP =
    lambda: 1.0640e-06
         q: -7.3931 +13.3832i
         N: 6
      dist: [0 2.7100 4.0180 0.2118 0.1400 0.2563 0.2895]
      ifoc: [0 0 0 0 0 0]
     label: {'PSLperiscope'  'INPUTViewPort'  'PeriscopeTop'  'PeriscopeBottom'  'Steer1'  'Steer2'}
 

z= 0 m, Gouy phase: 0 deg after passing PSLperiscope;                       Spot size w=2.4323 mm
z= 2.71 m, Gouy phase: 9.6308 deg after passing INPUTViewPort;         Spot size w=2.2556 mm
z= 6.728 m, Gouy phase: 26.0719 deg after passing PeriscopeTop;        Spot size w=2.1316 mm
z= 6.9398 m, Gouy phase: 26.9771 deg after passing PeriscopeBottom; Spot size w=2.1302 mm
z= 7.0798 m, Gouy phase: 27.5759 deg after passing Steer1;                Spot size w=2.1296 mm
z= 7.3361 m, Gouy phase: 28.6729 deg after passing Steer2;                Spot size w=2.129 mm
 

Power Recycling Cavity:

PRCrt =
    lambda: 1.0640e-06
         q: 7.1228 + 5.2551i
         N: 8
      dist: [16.6128 16.1551 24.8880 0 0 24.8880 16.1551 16.6128 0]
      ifoc: [-0.4391 0.0556 -2.3189e-04 -0.0010 -2.3189e-04 0.0556 -0.4391 -0.1818]
     label: {'PR2'  'PR3'  'ITMlens'  'ITMback'  'ITMlensr'  'PR3r'  'PR2r'  'PRM'}

z= 00.0000 m, Gouy phase: 00.0000 deg after passing PRM;      Spot size w=2.2471 mm
z= 16.6128 m, Gouy phase: 23.9355 deg after passing PR2;       Spot size w=6.1716 mm
z= 32.7679 m, Gouy phase: 24.8769 deg after passing PR3;       Spot size w=53.9602 mm
z= 57.6559 m, Gouy phase: 25.0458 deg after passing ITMlens;  Spot size w=52.9743 mm
z= 57.6559 m, Gouy phase: 25.0458 deg after passing ITMback; Spot size w=52.9743 mm
z= 57.6559 m, Gouy phase: 25.0458 deg after passing ITMlensr; Spot size w=52.9743 mm
z= 82.5438 m, Gouy phase: 25.2148 deg after passing PR3r;      Spot size w=53.9602 mm
z= 98.6989 m, Gouy phase: 26.1562 deg after passing PR2r;      Spot size w=6.1716 mm
z= 115.3117 m, Gouy phase: 50.0916 deg after passing PRM;    Spot size w=2.2471 mm
Round trip Gouy phase: 50.0916 deg (one-way Gouy phae: 25.0458 deg)

I see you've used 2.71 [m] as your distance between the PSL periscope an the HAM1 viewport, and mentioned this was a big uncertainty.

If I take the corner station Rack/Cable tray layout D1002704, which tells the distance between the HAM1 viewport and the +X edge of the PSL table is 86 [in], and add it to the distance from the edge of the table to the periscope mirrors from D0902114 -- 14 [in] -- I get an even 100 [in].

Thus, 2.54 [m] is likely a better number.

I tag Systems just in case they have an even more accurate/precise number, but I think past the ~inch level precision, we'd need to measure it.

T1000696-v2	PSL table to HAM1 door flange	88in
T1000696-v2	HAM door depth	18in
picture	top periscope mirror to edge of table	14in
total	top periscope to input viewport	84in, 2.134m

T1000696-v2 has verified distances.

D1002704 is measuring PSL table North edge to HAM1 door flange, and states it's 86 inches.

D1102219-v1, Micheal Rodruck's document, aptly named H1_table_v1.pdf, has what I believe are measured values, though not in a form that's easy to use, but what I've looked at is consistant with T1000696-v2.

Using this combination of documents, the outer PSL wall to the input viewport is calculated to be 70 inches, something that could easily be measured.

Checking the calibration of the WFS DC readouts I noticed a calibration error of a factor of 0.065. So, all angles measured by the WFSs should be scaled by this number. This still makes the jitter after the PMC dominant, but one might expect to see some of the HPO jitter peak show through in places where the downstream jitter has a valley. In any case, we should repeat the PSL jitter measurement with the IMC unlocked.

A report of the measured beam jitter at LLO is available in T1300368.

An earlier measurement at LHO is reported in alog 21212. Using an IMC divergence angle of 1.6x10^–4 rad, the periscope peak at 280 Hz is around 10^-4/√Hz. This is closer to the first posted spectrum with the "wrong" calibration. Here I post this spectrum again and add the dbb measurement of the jitter out of the HPO propagated through the PMC (1.6%), but scaled by a fudge factor of 2. The Sep 11, 2015, spectrum shows a more or less flat noise level below 80 Hz, whereas the recent spectrum shows 1/f noise. The HPO spectrum also shows as 1/f dependency and is within a factor of 2 of the first posted spectrum. If jitter into the IMC is the main coupling mechanism into DARM, the HPO jitter peaks above 400 Hz are well below the PSL table jitter after the PMC and the would not show up in the DARM spectrum.

In the last paragraph I meant "offset" when I wrote "gain". Just to clarify I added -0.05 offset to CSOFT pitch and yaw. Sorry about the confusion.