over 120 days

3:30pm local

36 min. 26 sec. to overfill CP3 with 1/2 turn open on LLCVBV. Based on CP4 tests, the pump was probably right at 100% full. I'm going to leave LLCV at 19%.

Observatory Mode: taken from Commissioning to Corrective Maintenance (for PSL work) for most of the morning

Atlantic/Africa 7.1 earthquake last night around 10pm.  It tripped Test Masses & TMSy.

Once the PSL was brought back (around noon), I performed an Initial Alignment (lost the MC for a bit during the PRM portion of procedure).  After the Alignment was complete, H1 was handed over to the Commissioners & they've had a few 50W locks.  

Jeff K damped down some rung up violin modes (most likely due to the earthquake).

Following on from Jason's log from yesterday (29356).  The crystal chiller was engaged without turning the high power oscillator.  The flow rates for heads [1-4] were monitored.  Every now and then the flow rate for head 4 would drop.  Whilst monitoring the flow rates, a trend plot of the relative humidity was generated.  It indicated a slow rise in the humidity towards the alarm level of 60%, which from previous experience is the foretelling of a leak.

    The high power oscillator cover was removed and the insides of the laser was inspected.  No signs of a leak were observed.  No unusual bulges in the cooling hoses were observed either.  The output of the flow sensors for heads 2 and 4 were switched.  The signal noise followed the switch in flow sensor, leaving to the conclusion that the output electronics for head 4 was bad.  The LM2907-N frequency to voltage converter chip for that channel was replaced with one from an old board (the one closest to the edge of the table, labelled N4).  The chip labelled N3 was also replaced due to noise observed in the flow of head 3.  Since replacing the chips seemed to have improved the noise situation, the decision was taken not to remove flow sensor 4.

    The moisture sensitivity level (MSL) of the LM2907-N is 1 according to Texas Instruments, meaning that its lifetime is unlimited provided the relative humidity is less than 85% and the temperature is less than 303 Kelvin.  The chips that were replaced happen to be the two closest to the head 3 sensor leak that occurred recently.  It is not impossible that these chips got wet when the hose burst, and hence the erratic behaviour since.

    As an aside, the diode current for diode box 3 went back to reading ~50 A after the flow sensors were reconnected.  However since that time it has gone back to reading over 100 A.




Jason/Jeff/Peter

The new prototype board was modified to satisfy some of the things in the requirement (T1600064) as listed in 1., 3. and 5. below. I also listed things that were not modified but would have been good if recommendations in T1600064 were implemented.

I wouldn't claim that we won't modify it further, but this is just the current state.

In this entry you'll be able to see the electronics modifications in the attached, but the photos are too small to read the original component values, so read the original drawing D1600298 as necessary.

1. Readback of the PD array signal after the servo loop switch ("SUM PE MON")  needed to be DC coupled.

The "whitening" was originally AC-coupled (second attachment) and therefore it was not compatible with digital AC coupling described in the requirement.

We bypassed the big caps in the input of the opamps with some resistors so it acts as one single pole at 2.7k with DC gain of 50 (second attachment, mod 1).

To make the OLTF measurement easier, we made the same change to the whitening downstream of the summation for the excitation DAC output.

The servo board output readback was also AC-coupled, but we want to see if everything is working fine including DC injected into the inner loop board.

We made the output whitening a true whitening (second attachment, mod 2). 0.35Hz zero might be too low, though.

2. The board doesn't implement the analog compensation for the in the inner loop filtering upstream of the error point.

The inner loop error point "whitening" works as a very steep boost seen from the outer loop, roughly an equivalent of (z, p) = ([3; 3; 130], [0.07, 0.07]). T1600064 reccomends to implement the same thing in the outer loop somewhere to cancel this effect to make the loop design simpler.

Since this is not compensated in the prototype board, the outer loop OLTF would become huge for f<3Hz. This means that the digital AC coupling servo should work REALLY hard to effectively AC-couple the entire outer loop at, say, f<1Hz, without reducing the outer loop gain at 10Hz and up.

We do not implement this by analog cut and paste job (no opeamp available for this on the board), but we'll try to make it work by an aggressive digital AC-coupling filter.

3. Integrator as a boost doesn't go well with digital AC coupling so it was converted to a usual boost with finite DC gain.

The error point of the digital AC coupling servo is kept zero at DC (i.e. a pole at zero Hz). The outer loop prototype had an integrator as a boost.

Of course they don't go well as any tiny DC difference between the AC coupling error point and the integrator input will be integrated and eventually the servo runs away. But we don't really need a pole at zero Hz.

We changed it such that the first boost acts as 50Hz LPF with DC gain of 10, and when added to a flat unity gain it becomes 500:50 boost (third attachment) (but see the next entry).

4. Boosts were implemented as three parallel integrators added to a flat unity gain.

We had a choice of unity gain, zp=50:0, 100:0 or 150:0. See the first and the third attachment.

As described in 3. above, we already changed one integrator to []:50 LPF so the boost becomes 500:50.

We didn't fix the parallel summation, it might be OK to go without any boost or using just one.

But T1600064 shows our standard practice of connecting boosts in series.

5. Outer loop servo filters on the board were also changed.

Originally the servo filtering was something like (z,p)=(100, [20^2;40]) (fourth attachment). As described in 2. above, 1st loop effectively adds ([3^2; 130], [0.07^2]). IMC adds another pole at ~8kHz.

Everything added together it's like ([3^2; 100; 130], [0.07^2; 20^2; 40; 8k]), and this might be OK.

But we made a change so the board became ([380;9k],[20^2;600]) by changing one of zero-Ohm resistors in the second stage and one C-R pair in the third stage  (fourth attachment). Everything added together it would be   ([3^2;130;380;9k], [0.07^2; 20^2;600; 8k]).

This change might not have been necessary. We just felt that letting a big 4.7uF capacitor and a zero-Ohm resistor to determine AC gain at around 10kHz in the first as well as the second stage was maybe too much of an uncertainty in the AC gain.

6. Sallen-Key at 0.1Hz for Digital AC coupling path might be too aggressive

For the moment the Sallen-Key is disabled for the digital AC coupling path (so the only analog filtering is a 10Hz LPF) to make things easier as we try to make it work.

This morning, I started running ETMX and ETMY charge measurements in an effort to utilize today's PSL downtime window instead of tomorrow.  While looking around at SUS SDF settings during the measurements, I found all 4 test mass L2 EUL2OSEM matricies in a funny state.  The snapshot attached shows the ITMY settings as left from Saturday - note all 4 test masses show the same matrices settings (and therefore have the same SDF screen diffs).  Jenne points out that these setting diffs are from a relatively new implementation of the 3-coil switching alog 28959 and had not been put into the Guardian DOWN because they hadn't been fully exercised yet.   She has now added them to the appropriate place in Guardian to be restored back to nominal.

ITMX seemed to be extra sensitive to larger ground motions--it was often the only ISI to trip.  Of course somebody had to be the most sensitive so could be a wild chase.  FRS 5948.

LHO aLog 28567 -- Investigation first report finds wierd reversal of current in Stage2 H2 CD.  Some more data from EQ (only ITMX tripped) 28621.

LHO aLog 28649 -- Direct excitation of CD leads to finding bad voltage regulator on coil driver.  Entire unit replaced (no spare voltage regulator available.) No funny behavior seen on Current monitor.

LHO aLog 28948 -- 9 Aug: Original Coil Driver replaced (with replaced voltage regulator) but no excitation drive experiment done to confirm good behavior.

Today, drove Coil Driver w/ 3000 Ct 0.05 Hz sine, I_INMON responds to greater than 110 counts with no ill behavior observed, see attached.  Before repair, the I_INMON reversed and spiked to very large output (340cts) when the current reading hit 50-60 counts.  This suggests the coil driver is back to nominal behavior.  And, there have not been any ground motion events where ITMX was the only platform which tripped.

Deem this problem fixed and will move FRS 5948 to pending and recommend closure.

• PSL is DOWN & investigation work is ongoing to bring back the PSL.
• There was a 7.1 Earthquake off Africa last night (tripped all four Test Masses & TMSy)
• Various non-ifo work is ongoing:  Charge measurements, ITMx ISI Stage2 check, a couple contractors on-site....

PSL Team currently investigating the lastest tripping of the LASER. The trends don't really show any strangeness until AFTER the interlock trip. 29356

• SEI:  --
• SUS:  Charge measurements?  (Betsy)
• CDS:  --
• PSL:  Laser down due to cooling system; work in the PSL room
• VAC:  Vertex RGA baking (contine through the week); possible CP4 tests; VAC team possibly working with SUS wrt charge
• Facilities:  S&K Electric pipe work in Mechanical Room

General Announcement about testing of Remote Access Controls During DAY (8am-4pm, M-F) shift (Jonathan work permit 6121).

Went over the Work Permits.

Sat amp S1100173 was brought to the shop missing  -14VDC. A short between this supply and ground was found as well as a burnt trace. C340 was found to be shorting -14VDC to ground. Replaced 10µF 25VDC Tant cap and repaired burnt trace. See aLog 29304 . This action has also been recorded in E-traveler.

Let the saga commence...

Jenne called me this morning and told me that while checking the status of the IFO before she came in today she noticed that the PSL was off.  I logged in remotely to confirm and drove out to investigate.  I found the laser and the crystal chiller off, diode chiller still running, and the crystal chiller flow, FE flow, and NPRO OK interlocks all tripped.  Trends indicate this happened on 8/28/2016 at 01:19:38 UTC (8/27/2016 at 18:19:38 PDT). 

Attempt #1

I turned the crystal chiller on, and it turned off within 5 seconds, this time with a head 1-4 flow error (H1:PSL-OSC_FLOWERR) interlock trip.  The cause here was the HPO laser head 4 flow meter was not getting above the minimum threshold of 0.4 lpm.  I called Peter to consult, and we decided to lower the threshold to 0.2 lpm.  I then attempted to restart the crystal chiller and everything came up fine.  The flow for head 4 was sitting at ~0.38 lpm, and over the next 10 or so seconds it increased to 0.51 lpm.  After a minute or so it dropped to ~0.46 lpm and then slowly worked its way back to ~0.53 lpm (maybe something working its way through the head 4 laser head water circuit?).  I let the chillers run for a few minutes to ensure everything was working properly.  Trends indicate that this was NOT the original cause of the PSL trip.

Attempt #2

I then restarted the HPO, and everything came up OK so I let it warm up for a few mintues.  After about 7 minutes or so the laser turned off again, this time with the HPO DB heatsinks overtemp (H1:PSL-OSC_DBHTSNKOVRTEMP)  interlock tripped.  Looking at the chiller screen, the temp of the water in the diode chiller had slowly increased from its setpoint of 20 °C to ~28 °C, which caused the interlock to trip.  Checking on the diode chiller, the front panel was showing a 'F-3 error' error message.  I don't know what this message means, but coupled with the fact the water had gotten so warm, I'm assuming something with the chiller cooling system stopped working (fans maybe?  F-3 could be fan 3?  Will look into it more tomorrow).  I had to power cycle the diode chiller to clear the error.  Restarting the diode chiller, and everything appeared to work fine.  Trends indicate that this was also NOT the original cause of the PSL trip.

Attempt #3

I restarted the HPO and let things sit for a while to warm up.  The front end laser came up without issue, and the system injection locked as expected so I let things warm up for a bit before activating the PSL stabilization subsystems.

While waiting for everything to warm up, I trended every crystal chiller based interlock we monitor at the time of the trip to try to see which one killed the laser first (see first attachment).  As can be seen, the only interlocks that tripped were H1:PSL-IL_XCHILFLOW (crystal chiller flow interlock), H1:PSL-AMP_FLOWERR (Front End flow error interlock), and H1:PSL-AMP_NPROOK (is the NPRO running interlock).  They all tripped within the same second, so my best guess at this point is that something happened with the Front End flow that tripped the interlock.  What that could be, I'm not sure; trending the humidity over the same period shows no evidence of a water leak.

While the injection locked system was warming up the laser tripped out again, once again with the HPO Head 1-4 Flow Error interlock tripped.  Full data trends show no reason why the interlock would trip, none of the channels appear to drop below 0.2 lpm (see 2nd attachment).  The MEDM version of the Beckhoff interlock screen also showed that the HPO Diode temp guard interlock had tripped, although the Beckhoff screen did not indicate a trip of this interlock and a full data trend also showed no trip of this interlock.  The discrepancy is unclear at this time.

Attempt #4

I turned the chiller back on and let it run, all OK.  I brought up the HPO (no issues), the front end (also no issues), and turned on the injection locking (also also no issues).  While waiting for the system to warm up, I set up a StripTool of the 4 laser head flows and sat here and watched it.  Less than 10 minutes later the laser tripped out again, same HPO Head 1-4 Flow Error interlock tripped.  I happened to be looking at the StripTool when it happened and noticed the flow for laser head 4 drop, come back up, and then the laser tripped.  Zooming in on the spot (see 3rd attachment) shows this.  It doesn't show the flow drop below the threshold of 0.2 lpm, but I don't think StripTool is fast enough to catch it.  It appears we have an issue with the water circuit for the 4th laser head in the HPO.  Humidity trend doesn't indicate a leak, so something is likely going wrong with the flow sensor.

I've left the laser and the crystal chiller OFF.  We will go in to the PSL tomorrow and attempt to figure out what the problem is.  Please do not try to restart the laser.

As an aside

I took the opportunity while the laser was down to watch the current for HPO diode box 3 (DB3).  We have had an issue where the current shows 100.2A drawn, which is impossible as the power supplies max out at ~60A (originally reported here and FRS 5955).  I watched the reported current for HPO DB3 during the PSL restart process to see if it always reports 100A or if there is a spot during the startup procedure where it changes.  See the below table:

Unfortunately I couldn't get any farther in the startup procedure (see above), but there was no point today where the reported current draw of HPO DB3 was 100A.  Will monitor this the next time we restart the laser to see if there's a point where the reported current draw changes.

C. Cahillane

I had a look at the ER9 actuation measurements vs. the new DARM model, looking for a comparison to O1 covariance values.  
The results are shown in the hard-to-see plot below.

I cut off the frequency vectors below 6 Hz because the noise was very high and the DARM model did not fit well with the measurements there.

The results indicate greatly inflated variance and covariance, but this is probably due to the lack of linear fitting.  The UIM stage, for instance, is a factor of 2 off at 100 Hz again, just like in O1.  This time I haven't yet removed these systematics, I'm just looking purely at measurements/DARM model

For a better idea of what the covariance values mean, I have made a correlation matrix.  The correlation of two variables has the following definition:

               cov(x,y)
corr(x,y) = -------------,     -1 <= corr(x,y) <= 1
            std(x) std(y)


So if I take the covariance matrix and divide it by the square root of its diagonal, I will end up with a correlation matrix.  The covariance matrix is shown in the titles of the plot attached to this alog.  

The correlation matrix is shown here:

           Re(A_U)   Im(A_U)   Re(A_P)   Im(A_P)   Re(A_T)   Im(A_T)
         -                                                           -
Re(A_U) |   1.0000   -0.7111    0.5866    0.1474   -0.3010    0.1088  |
Im(A_U) |  -0.7111    1.0000   -0.2323    0.2400    0.1260    0.1124  |
Re(A_P) |   0.5866   -0.2323    1.0000    0.0672   -0.0510    0.3610  |
Im(A_P) |   0.1474    0.2400    0.0672    1.0000   -0.4935    0.3717  |
Re(A_T) |  -0.3010    0.1260   -0.0510   -0.4935    1.0000   -0.3554  |
Im(A_T) |   0.1088    0.1124    0.3610    0.3717   -0.3554    1.0000  |
         -                                                           -

The diagonal of the correlation matrix is unity, as expected.  The largest correlation is between the UIM real and imaginary stage, which is to be expected since the UIM stage has a large systematic error polluting it's statistics.  The other stages do not suffer from great systematic errors, and all have correlation statistics below 0.5.  

Once I've made the systematic error fits I'll post this matrix again in a comment for comparison.

New ISS outer loop prototype board was modifed and put in. Modification details will be in a separate alog.

PD5, 6, 7, and 8 anode and cathode cables were disconnected from the old transimpedance box under HAM2 and was connected to the PD5-8 cable for the new prototype board. One strange thing about the old setup was that the "LR C" and "LR A" cable that come from the chamber feedthrough were connected to the anode and cathode input of the old box, respectively. Since apparently they gave us a correct signal (no forward bias across the diode), I connected "LR C" chamber side cable to "LR A" rack side cable for the new setup.

I connected PD5-8 cable to PD1-4 input in the new prototype box. We can conveniently switch back and forth between the old and the new by just switching one cable that goes to the "outer loop input" connector on the 1st loop chassis.

I briefly connected the output of the new box to 1st loop, and enabled the servo, but it didn't work. It's not clear if the sign is correct, but there's no way to change the sign externally, I need to open the box. I'll make more measurements next week to see if the sign is indeed wrong.

I connected the old second loop back to the 1st loop.

It appears that something goes wrong when zooming in that creates random lines across the plots.

[Sheila, Jenne]

This is starting to feel a bit like we're using popsicle sticks and duct tape to make a splint and hold things together, but we're still having trouble with the vertex ASC at high powers, so we have tried implementing a dither/ezcaservo combo loop for MICH pitch.  As of now, Sheila has a script that will turn on a dither line for BS pitch and set up the demodulators in the ADS.  The script then sets up a cdsutils ezca servo to move the offset of ASC-MICH_P to zero the output of the demodulator. 

Earlier, we tried just a regular dither servo, moving the BS to minimize the demodulated version of AS90, but that wasn't working.  Note that earlier today we removed the AS90 element from the SRM dither loop, so the SRM dither now only looks at POP90. 

Since adding MICH offsets worked okay over the weekend to fix the sideband buildup after the POP offsets are adjusted, we thought we'd give the demodulator-adjusted offsets a try. This is well after fixing the TCS situation described in alog 29237.

On at least one occasion, this new system didn't keep up with the IFO heating when starting the offset from zero, so now it starts the pit offset from the value that Sheila et al. found the other day.

We've tried it now with both BS pitch and yaw under this new additive-offset-like configuration, but the sideband buildups still seem to be decaying.  Perhaps SRM dither should go back to the AS/POP ratio, and BS should be servoed to maximize POP18?

Side note:  We have got to find time to re-look at the OMC ASC.  Engaging the dither loops once again rails the OMC suspension.  This was happening last summer, and then the problem somewhat mysteriously went away, so I don't have any magical solution right now.  But this certainly can't be good for our noise.