There has been an intermittent issue with the OMC_LOCK Guardian in which it runs a portion of code twice.  I've observed this most often in the OMC_LSC_ON state, where the LSC controls for the OMC are ramped up.  It's easy to notice, because if the gain steps are repeated for the OMC length servo the loop quickly becomes unstable and the cavity unlocks.  I've noticed this happen a handful of times in the past few weeks.  I've also seen lines of code in other main() function get executed twice, although I don't have screenshots to prove it.

Attached are screenshots of the OMC_LOCK log, from an event last week (March 14), and another tonight.  In both screencaptures, the OMC guardian enters the OMC_LSC_ON state, completes the instructions in main()...and then starts all over again.  In both cases the requested state was well downstream of OMC_LSC_ON, the guardian should not have looped there.  (And anyways, how does it repeat the main() function?)

I've committed the latest version of the OMC_LOCK guardian to the SVN, if experts want to check the code to make sure I'm not doing something heinous in the function calls or definitions.

In other locking notes from tonight...

• Kiwamu and I investigated the source of BS/SRM saturations, and a few locklosses, in the analog CARM handoff.  These saturation events had been variously blamed on the DRMI_ON_POP transition or the ASC loops turning back on, all of which happen in a short window of time.  Tonight we discovered it's due to the analog CARM handoff, specifically when input 2 of the REFL CM servo board is enabled with a large (14dB) input gain.  (This is the input of REFLAIR9I from the summing board to the CM board.)  When the input is enabled there is a big kick to the DRMI optics, and the MICH ASC loops (still engaged in this state) experience a large transient that rings for several seconds due to the inverse BS plant filter.  Screenshot of a typical lockloss is attached.  We've smoothed out the transition by putting a 10-second sleep in after the REFL_SERVO_IN2EN step.  No more locklosses from analog CARM.
• Violin mode damping is nearly finished.  There's one very tricky mode on ETMX that is a very problematic phase.
• We've continued to use Kiwamu's gains from Monday night to damp the ETMY and ITMX roll modes using AS_A_RF45Q, although I inserted a -100dB gain stage for ETMY to make the value more human readable.  Gain for ETMY_M0_DARM_DAMP_R is +40, gain for ITMX_M0_DARM_DAMP_R is +20.  These settings have not been propagated to the Guardian yet, I'd like to see if they continue to be robust after we touch up the ITM alignment to increase the recycling gain.
• We've been locking at 10W with the new DHARD_P filter.  Seems legit.

Dan, Keita, Kiwamu, Sheila

During a long, patient lock this evening I was able to measure the DHARD pitch loop down to 0.2Hz.  This follows Keita and Sheila's filter modifications to get some additional phase around the 3Hz UGF.  The attached plot is a record of the measurement (look at the RED trace), I have saved the xml file with the filename at the top of the plot.

The phase margin at the UGF is good (~40deg), and the loop does not cross unity gain at higher frequency.  There is almost a unity gain crossing at lower frequency, we have about 3dB of gain margin at 0.9Hz. 

Sheila, Nutsinee

This is the follow up from alog17278. I have attached five day worth of correlation plots between PEM wind channel, ALS control signal, and ground motion from two different sensors. Both ground and ALS correlate with the wind starting around 10 mph. The data point where PEM, ALS, and ISI are zeros and when ALS is constant has been removed in the correlation plots.

Todat we have two small changes to the initial alingment procedure:

there is now a clear history script that works for the arms, it can be used by clicking the button on the ALS OVERVIEW SCREEN or the end station screens.  

The X arm green transmission has been normalized, so now a transmission of 1 really does mean that the arm power is maximized.  

Also, as we have seen several times the OFFLOAD GREEN WFS doesn't really work for the Y arm. 

The status started reading low battery around 3:05 UTC 3/18.

Quiet day

09:42 Bubba running forklift outside by metal recycling container
09:45 Jody and Gary to mid Y, then transferring stuff to mid X
13:27 Hugh to BSC2
13:34 Hugh back

Scott L. Ed P. Chris S.

58 meters of tube cleaned today towards X-1-6 double doors.
Continuous monitoring of beam tube pressures by control room during cleaning operations.

So far not much has been done to do tilt decoupling on the the HAM ISI's, because we weren't sure it was necessary. I took some measurements yesterday to check, and I think that's still right. If you just look at my first attached plot, it seems reasonable to think that maybe we should. The GS13-Y/ISO-Y transfer function shows what looks like a  ~1/f^2 low frequency component then the normal f^3 component above 10mhz. But looking at Hugh's measurments on the BSC's the tilt component for the ETMX T240-X/ISO-X looks different (second attached plot). I also looked at Fabrice's calculations in alog 8284 and his data there looks like Hugh's BSC data, not my HAM data. Made me think I'm looking at something else. My third plot shows calbrated spectra, and I think I just looking at sensor noise at 10mhz. My calibration could be a bit  off, but the specrta going flat at 10mhz on the red trace is suspicious. I don't know what the bubble below is, maybe actual tilt? If it is, that doesn't look like an easy measurement to make, or worth doing.

Today we saw the second instance of what seems like a serious guardian bug, where the guardian executes the line of code above a return True statement, but doesn't return True and exit the state.  Screen shots of both incidents are attached.  After the first inicident (shown in the first attached screenshot) Jamie suggested that possibly the requested state could have been erroneously set to be the state LOCK_DRMI_1F.  

Today I am sure that the requested state was DRMI_ON_POP, so it should have changed states after returning true. 

Below are this weeks past 10 day trends.

The usual procedure for clearing this trip condition is to change the reset value on the MEDM screen from 1 to 0 and then back again. This had no effect, in this case. After consulting with Stuart Aston, it seemed the only other recourse was to power cycle the PUM coil driver for ITMX. This was done ~10:20PST. THis action was successful. This smells like a software issue?

PSL DBB/ISS scans for this week.  No significant change from last week.

Dan, Sheila, ITMX

While testing the vioin mode damping tonight, we had an abrupt lockloss, and for a short period my bandpass filter at 501.094Hz was sending noise x120dB into the ITMX L2 coils.  This tripped the coil current watchdog, just like a previous event on ETMY a few weeks ago.  (There are now limiters on all the violin mode damping filters...)

Unlike ETMY we can't reset this watchdog by toggling the RMS Reset epics channel.  Anyone know how to clear this error?

We suspect something in hardware is preventing a software reset.  In the CDS highbay, the SUS Independent Watchdog electronics box has red lights that won't reset when we use the 'Fault Reset' button.

Sheila, Kiwamu, Dan

We have had several occasions where we are trying to make a driven measurement, we can clearly see our drive orders of magnitude above the DARM noise, but we get no coherence according to DTT.  The first screenshot attached shows the coherence from my drive into the DHARD pitch loop to several channels that in principle all contain the same information:

H1:CAL-DELTAL_EXTERNAL_DQ, H1:LSC-DARM_IN1, and H1:LSC-DARM_OUT_DQ.  You can see all of these channels have different coherences, but I don't understand why.  Since I'm not driving the DARM filter bank, DARM IN and DARM OUT should be exactly the same information with a linear filter applied, and the CAL channel is the sum of these two channels with some calibration applied.  

Dan, SHeila, Jeff K remotely

The coil driver for ETMY L1 shut itself off this afternoon, this might have been while I was trying to measure the L2P coupling there.  We switched the rocker switch and things are fine.  The symptom was that we didn't seem to be drivig even though everything on the medm screen looked fine, the OSEM readback all had low values.  This is the same situation we had at End X a while ago alog 16511

This is one of the "errors" that we probably need better monitorinng of.  As a short term solution maybe we could add this to TJ's list, to check if all OSEM readbacks are low for a certain stage. 

Hugh and Krishna had found the BRS software had crashed pretty much on schedule, so Sheila and I went to the end-station to restart it. Pretty straightforward. Alogs 13817 and 15005 will get you everything you need to do it. The BRS is rung up right now because we went out to check the mass damper positions, but it should settle down in 30 minutes or so.