But don't worry, we are back to Observing at 10:36 UTC.

I haven't seen anything of note for the lockloss. I checked the usual templates, with some screenshots of them attached.

This seems like another example of the SR3 problem. (alog 32220 FRS 6852)

If you want to check for this kind of lockloss, zoom the time axis right around the lockloss time to see if the SR3 sensors change fractions of a second before the lockloss. 

J. Kissel, B. Weaver, T. Sadecki

Just for reference, I include a lockloss that was definitely *not* caused by the SR3 glitching for future comparison and distinction of whether this SR3 glitch has happened or not.

Also, remember, although the OPS wiki's instructions suggest that one must and can only use lockloss2, not everyone has the alias yet for this more advanced version. You can make the plots, and do everything you need with the more basic version:

    lockloss -c /ligo/home/ops/Templates/Locklosses/channels_to_look_at_SR3.txt select

It would also be great to get the support of @DetChar on this one. The fear is that these glitches begin infrequently, but get successively more frequent. Once they do, we should consider replacing electronics.

The fishy thing, though, is that LF and RT are on separate electronics chains, given the cable layout of HAM5 (see D1101917). Maybe these glitches are physical motion? Maybe with statistics of two, it's unclear whether it's that LF and RT just *appear* to be the culprit whether it may be a random set of OSEMs glitching.

See my note in alog 32220, namely that Sheila and I looked again and we see that the glitch is on the T1 and LF coils, which share a line of electronics.  The second lockloss TJ started with in this log (12/06) are somewhat unconclusively linked to SR3 - no "glitches" like the first one 12/05, but instead all 6 top mass SR3 OSEMs show motion before lockloss.

Sheila, Betsy

Attached is a ~5 day trend of the SR3 top stage OSEMs.  T1 and LF do have an overall step in the min/max of their signals which happened at the time of that lockloss which showed the SR3 glitch (12/05 16:02 UTC)...

Seems to have stablized itself now, back at 3.0Gpm

Eh, this actually is kinda the usual "thing" going on with this sensor of late...  It drops low for a while ~30-60 mins and then recovers eventually.  The work permit we have open is to fix next Tuesday.

I was wondering if this was coming from the CO2 controller chassis, D1200745. I had a look to see if there were accompanying glitches on the CO2Y laser temperature channel which might indicate a common cause. Whilst there are some glitches that show up simultaneously on both channels, there are many that are not simultaneous. It is far from obvious that there is a common cause for the flow rate variations.

% check to see if the CO2Y laser flow meter is broken or if there is a

% problem with electronics instead. Compare to CO2Y laser temperature

t0 = 1165166030 - 72*3600;

t1 = t0 + 2*24*3600;

chan = {'H1:TCS-ITMY_CO2_LASERTEMPERATURE.mean,m-trend', ...

    'H1:TCS-ITMY_CO2_FLOWRATE.mean,m-trend'};

[data, t, info] = get_data(t0,t1,chan,[]);

subplot(2,1,1)

plot((1:size(data,1))/60, data(:, 1)); grid on

axis([0 48 23.72 23.82])

xlabel('Time (hr)')

ylabel('CO2Y laser temperature (C)')

title('CO2Y laser channel glitches - correlated or not?')

subplot(2,1,2)

plot((1:size(data,1))/60, data(:, 2)); grid on

axis([0 48 2 3.5])

xlabel('Time (hr)')

ylabel('CO2Y flow rate (gpm)')

orient tall

print('-dpdf', 'CO2Y_flow_rate_errors.pdf')

Jason noticed that the chiller was complaining about low water level as he was bringing it back up.  This is apparently due to the fact that when the chiller trips off, it burps a bunch of water onto the floor.  I topped the Xtal chiller off with 300 mL H2O.

Filed FRS ticket 6853 for this trip.

Also, back to Locking now.

Also, Rana and photographer are on site.  I let them onto the Observation Deck to take pics while we are relocking.

Sorting through the myriad of signals leads me to think that the laser trip was due to the NPRO passing out.
Although it is possible that the flow rate in head 3 dipped below the 0.4 lpm limit.  Head3FlowNPRO.png
suggests that the NPRO tripped out before the flow rate in head 3 reached its limit.

When restarting the laser last night, the status screen on the PSL Beckhoff PC indicated a trip of the "Head 1-4 Flow" interlock; although looking at the graphs Peter posted above it appears that the laser lost power before any of the flow sensors dropped below the trip threshold.

Further forensics:  Attached are trends of the laser head temperatures around the time of last night's PSL trip.  To my eye nothing looks out of the ordinary.

Sheila and I looked again at this particular lockloss (2016-12-06 10:05:39 UTC) and agree that the glitch that likely caused the lockloss are actually on the T1 and LF top stage OSEMs.  These are indeed on the same set cabling, satellite amp, and driver run.  See attached for updated lockloss plot this time with the OSEMINF channels.  We'll keep watching locklosses to see if this happens more.