This morning, I started running ETMX and ETMY charge measurements in an effort to utilize today's PSL downtime window instead of tomorrow.  While looking around at SUS SDF settings during the measurements, I found all 4 test mass L2 EUL2OSEM matricies in a funny state.  The snapshot attached shows the ITMY settings as left from Saturday - note all 4 test masses show the same matrices settings (and therefore have the same SDF screen diffs).  Jenne points out that these setting diffs are from a relatively new implementation of the 3-coil switching alog 28959 and had not been put into the Guardian DOWN because they hadn't been fully exercised yet.   She has now added them to the appropriate place in Guardian to be restored back to nominal.

ITMX seemed to be extra sensitive to larger ground motions--it was often the only ISI to trip.  Of course somebody had to be the most sensitive so could be a wild chase.  FRS 5948.

LHO aLog 28567 -- Investigation first report finds wierd reversal of current in Stage2 H2 CD.  Some more data from EQ (only ITMX tripped) 28621.

LHO aLog 28649 -- Direct excitation of CD leads to finding bad voltage regulator on coil driver.  Entire unit replaced (no spare voltage regulator available.) No funny behavior seen on Current monitor.

LHO aLog 28948 -- 9 Aug: Original Coil Driver replaced (with replaced voltage regulator) but no excitation drive experiment done to confirm good behavior.

Today, drove Coil Driver w/ 3000 Ct 0.05 Hz sine, I_INMON responds to greater than 110 counts with no ill behavior observed, see attached.  Before repair, the I_INMON reversed and spiked to very large output (340cts) when the current reading hit 50-60 counts.  This suggests the coil driver is back to nominal behavior.  And, there have not been any ground motion events where ITMX was the only platform which tripped.

Deem this problem fixed and will move FRS 5948 to pending and recommend closure.

• PSL is DOWN & investigation work is ongoing to bring back the PSL.
• There was a 7.1 Earthquake off Africa last night (tripped all four Test Masses & TMSy)
• Various non-ifo work is ongoing:  Charge measurements, ITMx ISI Stage2 check, a couple contractors on-site....

PSL Team currently investigating the lastest tripping of the LASER. The trends don't really show any strangeness until AFTER the interlock trip. 29356

• SEI:  --
• SUS:  Charge measurements?  (Betsy)
• CDS:  --
• PSL:  Laser down due to cooling system; work in the PSL room
• VAC:  Vertex RGA baking (contine through the week); possible CP4 tests; VAC team possibly working with SUS wrt charge
• Facilities:  S&K Electric pipe work in Mechanical Room

General Announcement about testing of Remote Access Controls During DAY (8am-4pm, M-F) shift (Jonathan work permit 6121).

Went over the Work Permits.

Sat amp S1100173 was brought to the shop missing  -14VDC. A short between this supply and ground was found as well as a burnt trace. C340 was found to be shorting -14VDC to ground. Replaced 10µF 25VDC Tant cap and repaired burnt trace. See aLog 29304 . This action has also been recorded in E-traveler.

Let the saga commence...

Jenne called me this morning and told me that while checking the status of the IFO before she came in today she noticed that the PSL was off.  I logged in remotely to confirm and drove out to investigate.  I found the laser and the crystal chiller off, diode chiller still running, and the crystal chiller flow, FE flow, and NPRO OK interlocks all tripped.  Trends indicate this happened on 8/28/2016 at 01:19:38 UTC (8/27/2016 at 18:19:38 PDT). 

Attempt #1

I turned the crystal chiller on, and it turned off within 5 seconds, this time with a head 1-4 flow error (H1:PSL-OSC_FLOWERR) interlock trip.  The cause here was the HPO laser head 4 flow meter was not getting above the minimum threshold of 0.4 lpm.  I called Peter to consult, and we decided to lower the threshold to 0.2 lpm.  I then attempted to restart the crystal chiller and everything came up fine.  The flow for head 4 was sitting at ~0.38 lpm, and over the next 10 or so seconds it increased to 0.51 lpm.  After a minute or so it dropped to ~0.46 lpm and then slowly worked its way back to ~0.53 lpm (maybe something working its way through the head 4 laser head water circuit?).  I let the chillers run for a few minutes to ensure everything was working properly.  Trends indicate that this was NOT the original cause of the PSL trip.

Attempt #2

I then restarted the HPO, and everything came up OK so I let it warm up for a few mintues.  After about 7 minutes or so the laser turned off again, this time with the HPO DB heatsinks overtemp (H1:PSL-OSC_DBHTSNKOVRTEMP)  interlock tripped.  Looking at the chiller screen, the temp of the water in the diode chiller had slowly increased from its setpoint of 20 °C to ~28 °C, which caused the interlock to trip.  Checking on the diode chiller, the front panel was showing a 'F-3 error' error message.  I don't know what this message means, but coupled with the fact the water had gotten so warm, I'm assuming something with the chiller cooling system stopped working (fans maybe?  F-3 could be fan 3?  Will look into it more tomorrow).  I had to power cycle the diode chiller to clear the error.  Restarting the diode chiller, and everything appeared to work fine.  Trends indicate that this was also NOT the original cause of the PSL trip.

Attempt #3

I restarted the HPO and let things sit for a while to warm up.  The front end laser came up without issue, and the system injection locked as expected so I let things warm up for a bit before activating the PSL stabilization subsystems.

While waiting for everything to warm up, I trended every crystal chiller based interlock we monitor at the time of the trip to try to see which one killed the laser first (see first attachment).  As can be seen, the only interlocks that tripped were H1:PSL-IL_XCHILFLOW (crystal chiller flow interlock), H1:PSL-AMP_FLOWERR (Front End flow error interlock), and H1:PSL-AMP_NPROOK (is the NPRO running interlock).  They all tripped within the same second, so my best guess at this point is that something happened with the Front End flow that tripped the interlock.  What that could be, I'm not sure; trending the humidity over the same period shows no evidence of a water leak.

While the injection locked system was warming up the laser tripped out again, once again with the HPO Head 1-4 Flow Error interlock tripped.  Full data trends show no reason why the interlock would trip, none of the channels appear to drop below 0.2 lpm (see 2nd attachment).  The MEDM version of the Beckhoff interlock screen also showed that the HPO Diode temp guard interlock had tripped, although the Beckhoff screen did not indicate a trip of this interlock and a full data trend also showed no trip of this interlock.  The discrepancy is unclear at this time.

Attempt #4

I turned the chiller back on and let it run, all OK.  I brought up the HPO (no issues), the front end (also no issues), and turned on the injection locking (also also no issues).  While waiting for the system to warm up, I set up a StripTool of the 4 laser head flows and sat here and watched it.  Less than 10 minutes later the laser tripped out again, same HPO Head 1-4 Flow Error interlock tripped.  I happened to be looking at the StripTool when it happened and noticed the flow for laser head 4 drop, come back up, and then the laser tripped.  Zooming in on the spot (see 3rd attachment) shows this.  It doesn't show the flow drop below the threshold of 0.2 lpm, but I don't think StripTool is fast enough to catch it.  It appears we have an issue with the water circuit for the 4th laser head in the HPO.  Humidity trend doesn't indicate a leak, so something is likely going wrong with the flow sensor.

I've left the laser and the crystal chiller OFF.  We will go in to the PSL tomorrow and attempt to figure out what the problem is.  Please do not try to restart the laser.

As an aside

I took the opportunity while the laser was down to watch the current for HPO diode box 3 (DB3).  We have had an issue where the current shows 100.2A drawn, which is impossible as the power supplies max out at ~60A (originally reported here and FRS 5955).  I watched the reported current for HPO DB3 during the PSL restart process to see if it always reports 100A or if there is a spot during the startup procedure where it changes.  See the below table:

Unfortunately I couldn't get any farther in the startup procedure (see above), but there was no point today where the reported current draw of HPO DB3 was 100A.  Will monitor this the next time we restart the laser to see if there's a point where the reported current draw changes.

C. Cahillane

I had a look at the ER9 actuation measurements vs. the new DARM model, looking for a comparison to O1 covariance values.  
The results are shown in the hard-to-see plot below.

I cut off the frequency vectors below 6 Hz because the noise was very high and the DARM model did not fit well with the measurements there.

The results indicate greatly inflated variance and covariance, but this is probably due to the lack of linear fitting.  The UIM stage, for instance, is a factor of 2 off at 100 Hz again, just like in O1.  This time I haven't yet removed these systematics, I'm just looking purely at measurements/DARM model

For a better idea of what the covariance values mean, I have made a correlation matrix.  The correlation of two variables has the following definition:

               cov(x,y)
corr(x,y) = -------------,     -1 <= corr(x,y) <= 1
            std(x) std(y)


So if I take the covariance matrix and divide it by the square root of its diagonal, I will end up with a correlation matrix.  The covariance matrix is shown in the titles of the plot attached to this alog.  

The correlation matrix is shown here:

           Re(A_U)   Im(A_U)   Re(A_P)   Im(A_P)   Re(A_T)   Im(A_T)
         -                                                           -
Re(A_U) |   1.0000   -0.7111    0.5866    0.1474   -0.3010    0.1088  |
Im(A_U) |  -0.7111    1.0000   -0.2323    0.2400    0.1260    0.1124  |
Re(A_P) |   0.5866   -0.2323    1.0000    0.0672   -0.0510    0.3610  |
Im(A_P) |   0.1474    0.2400    0.0672    1.0000   -0.4935    0.3717  |
Re(A_T) |  -0.3010    0.1260   -0.0510   -0.4935    1.0000   -0.3554  |
Im(A_T) |   0.1088    0.1124    0.3610    0.3717   -0.3554    1.0000  |
         -                                                           -

The diagonal of the correlation matrix is unity, as expected.  The largest correlation is between the UIM real and imaginary stage, which is to be expected since the UIM stage has a large systematic error polluting it's statistics.  The other stages do not suffer from great systematic errors, and all have correlation statistics below 0.5.  

Once I've made the systematic error fits I'll post this matrix again in a comment for comparison.

New ISS outer loop prototype board was modifed and put in. Modification details will be in a separate alog.

PD5, 6, 7, and 8 anode and cathode cables were disconnected from the old transimpedance box under HAM2 and was connected to the PD5-8 cable for the new prototype board. One strange thing about the old setup was that the "LR C" and "LR A" cable that come from the chamber feedthrough were connected to the anode and cathode input of the old box, respectively. Since apparently they gave us a correct signal (no forward bias across the diode), I connected "LR C" chamber side cable to "LR A" rack side cable for the new setup.

I connected PD5-8 cable to PD1-4 input in the new prototype box. We can conveniently switch back and forth between the old and the new by just switching one cable that goes to the "outer loop input" connector on the 1st loop chassis.

I briefly connected the output of the new box to 1st loop, and enabled the servo, but it didn't work. It's not clear if the sign is correct, but there's no way to change the sign externally, I need to open the box. I'll make more measurements next week to see if the sign is indeed wrong.

I connected the old second loop back to the 1st loop.

While I was working on ISS outer loop at the PSL field rack, H1:PSL-MIS_FLOW_OK tripped.

Peter King walked me through the diagnostics over the phone and I was able to open the shutter.

2.5 hours into being locked at 50 W, we saw two PI ring up: ITMX 15522 Hz (Mode 2) and ETMX 15541 (Mode 17). We couldn't damp them and they broke the lock (22:22 UTC). 

We had about 10 minutes before lockloss to try different damp settings. For Mode 2, any phase/gain changes I made rang up the mode more; the existing settings were already providing the most shallow slope. For Mode 17, a slight phase change helped but instability still bounced back. Below is Mode RMS monitor after the struggle. (Green and Red modes only appear to be ringing up from bleed over).

I've also attached the trend of the two modes. 

Something looks like it really kicked on Mode 2; there's strong slope changes before we started trying to adjust damp settings. Both modes stayed rung up into the next lock aquisition which I've never seen before; we could see them on the RMS monitor before DC READOUT. I stopped at 30 W to tweak phase settings again and allow to damp. These settings have been saved to the guardian. After that we continued to 50 W and stayed locked for almost 2 hours without incident (lockloss unrelated to PI). Overall, after tweaking at 30 W for full peak minimalization, I've lowered gains (perhaps the high gains were 'kicking' things too much?). 

This is the same time scale into the lock as we saw the other night, though note that ETMX did not ring up and ITMX did not have any active damping that night. Also note the oscillation in Mode 17 ~1156310300 in the trend.

We're leaving the interferometer locked at 50 W with the tweaked PI settings so we'll see in the morning. 

Most of tonight's work was measuring the HARD Pit loops, to give them some low noise settings to match the pre-existing Yaw low noise settings.  I also tried moving around the spot position on the AS_C QPD, which is pretty much the only QPD whos offsets we haven't tried tweaking lately.

It's clear on the AS camera as well as on the ALS X green camera that something moves significantly between 2W and 50W in yaw.  I was hoping that perhaps adjusing the SRC alignment would help with this somehow, but it didn't.  What I do think it might help with though is the fact that previously, we've seen our sideband powers drop significantly when moving POP and Soft offsets to increase the carrier power recycling gain.  I haven't tested this yet though. The first two attachments show the improvement in AS90 and the AS90/POP90 ratio when the AS_C spot is moved.  The max value in yaw that I moved was to 0.94, but this was obviously too far.  Perhaps tomorrow I'll picomotor a little bit so I can go a little farther, if this indeed helps the sideband buildups.

I moved the filters in the CHARD loops around today so that I could fit in a new lowpass at 200Hz.  CHARD has been acting funny lately, and is likely why we've been tripping the quads' rms watchdogs the last day or so.  To at least help, I put in a high-freq cutoff so that we can at least limit the amount of sensor noise we send to the quads.  Probably we should add this to the DHARD loops too, since none of them have had cutoffs of any kind during the acquisition sequence for a long time now.  I modified the ISC_LOCK guardian, as well as the sensing matrix code to be aware of the new locations of the different filters.

I measured the HARD pitch loops several times while modifying the loop shapes.  Both Chard and Dhard now have 50W modifying filters that move the plant compensation frequency response to match the 50W resonant frequencies rather than the 2W freqs.  They also both have a 25Hz cutoff.  CHARD Pit I can lower the gain by a factor of 2, but DHARD pit I can't. Most of the measurements in the attached screenshots are with the BoostHBW still on.  These should have been turned off, and are turned off for the Yaw loops when we go to low noise mode, in order to win back enough phase to reasonably have our cutoffs.  I need to think more carefully about how much gain we need at what frequencies before I finalize these loop shapes, but it seems like (from the DHARD pit measurement with the boost removed) that I should have enough phase to move the cutoff lower, if I don't require all that gain at low frequency. 

The guardian is up to date with these changes - other than helping the still-cooling IFO with some alignment while trying to catch DRMI lock, you can just request Lownoise_ASC, and the IFO will go up to 50W and hang out.  (I've commented out all of the offsets in the Adjust_offset state, but other than that, things are all as they were, modulo the changes mentioned above.)

Terra is writing up our PI experiences and mysteries from today, but I expect that the IFO will stay locked for just under 3 hours, so I'm going to put it in Observe just in case the data groups want to test pipelines.  The noise is still bad though, especially the intensity noise coupling as the IFO thermalizes.  To do this, I changed the nominal IMC_LOCK state to ISS_TR_CLOSED rather than the usual ISS_ON (which implies 2nd loop on), since we can't close the 2nd loop right now - we need the new hardware so that we can have both 2nd and 3rd simultaneously.

Oooh, I almost forgot.  Sheila and I tried closing a dither loop around the PRM to keep the spot position on PR2 constant.  This seems like it was doing good things, however once she fixed the 3rd loop's SR560 we decided that it would be good to get back to a campaign of persuing low noise for a while, rather than chasing offsets around.  We should consider coming back to this though when we come back to offset moving for PRCgain improvement.

After the nearly 3hr long lock at ~50W, we were paused at Locking Arms Green due to an X-arm which started low in power and slowly recovered (probably due to cooling).

Backing up:  During the previous long lock at high power, Jenne noticed that ALS X was visibly misaligned in yaw (could easily see in camera).  

After the lockloss and attempting to get H1 back up, we could see the ALSX was slowly struggling to recover.  The theory was this is due to some sort of cooling effect after the 50W lock.  Attached is a screenshot  of trends for  oplevs, TMS pit/yaw, ALS transmission, and the PSL power.  The story here (starting from left to right) is:

1. 2-hrs of locking at low power, 
2. 2-hrs of the 50W lock
3. ALS locking for 7min after tonight's 50W lock

(For the plots, oplev pitches are the plots on the top & yaws are on the bottom)

• Notice the 7min plots' ALS-X_TR & ETMX yaw oplev (circled in blue).  Jenne hadn't seen this issue before.
• Compare the above effect with locking at 2W earlier in the day (circled in red).  Everything is flat.
• SIDE NOTE:  Also, for the BS pit oplev (circled in green), there is a cooling effect here----but this is a known effect.

It appears that something goes wrong when zooming in that creates random lines across the plots.

[Sheila, Jenne]

This is starting to feel a bit like we're using popsicle sticks and duct tape to make a splint and hold things together, but we're still having trouble with the vertex ASC at high powers, so we have tried implementing a dither/ezcaservo combo loop for MICH pitch.  As of now, Sheila has a script that will turn on a dither line for BS pitch and set up the demodulators in the ADS.  The script then sets up a cdsutils ezca servo to move the offset of ASC-MICH_P to zero the output of the demodulator. 

Earlier, we tried just a regular dither servo, moving the BS to minimize the demodulated version of AS90, but that wasn't working.  Note that earlier today we removed the AS90 element from the SRM dither loop, so the SRM dither now only looks at POP90. 

Since adding MICH offsets worked okay over the weekend to fix the sideband buildup after the POP offsets are adjusted, we thought we'd give the demodulator-adjusted offsets a try. This is well after fixing the TCS situation described in alog 29237.

On at least one occasion, this new system didn't keep up with the IFO heating when starting the offset from zero, so now it starts the pit offset from the value that Sheila et al. found the other day.

We've tried it now with both BS pitch and yaw under this new additive-offset-like configuration, but the sideband buildups still seem to be decaying.  Perhaps SRM dither should go back to the AS/POP ratio, and BS should be servoed to maximize POP18?

Side note:  We have got to find time to re-look at the OMC ASC.  Engaging the dither loops once again rails the OMC suspension.  This was happening last summer, and then the problem somewhat mysteriously went away, so I don't have any magical solution right now.  But this certainly can't be good for our noise.