[Sheila, Jenne, JeffK]

Frustratingly, we spent much of the day struggling to transition from ALS COMM to IR Trans.  We dug through old alogs, we measured various loops and crossovers, and started figuring out a new order of operations for doing the transition, and then finally discovered that it just worked the original way when we forgot to load the guardian after coding in there our new-fangled sequence.  Incredibly frustrating.  Even half an hour or so before we were not able to do the original sequence and keep the lock.  This is clearly something that we need to address eventually, but it's tricky if it seems to have just gone away.  We've always had occasional locklosses around this transition and just trying again would work, so maybe if we solve the greater problem that plagued us this afternoon, we'd also solve that. 

Anyhow, once we were able to get past the ALS->Trans handoff, we focused on trying to get the rest of the IFO up and working, since the problem seems to have mysteriously disappeared.

The first big lock we had, we let the guardian take us all the way to NomLowNoise.  At first the new AS36 SRM ASC input matrix elements from alog 36578 seemed okay, but something was clearly funny since the control signal kept increasing even though the error signals were centered around zero - the zero point of the error signals was moving.  That seemed to go away, and we had several minutes of nice lock.  Then, in the last 2 minutes or so, the SRC ASC control signals got very large, and we lost lock before we could open those loops.  So, re-doing the SRM ASC input matrix is back on the to-do list. The rest of the locks tonight we stayed on the dither loops for the SRM alignment.

The next 2 big locks were lost after the noise tunings state, with an oscillation that seems to originate in the INP1 ASC loops at a teeny bit over 1Hz.  One of those locks we had skipped the 9MHz modulation depth reduction, and one we didn't, and we can stay indefinitely long on LownoiseESD_ETMY, so I think it's something in the Noise_Tunings state that is causing trouble.  The 2nd of these two locks, when we did do the 9MHz reduction, we also saw oscillations in DHARD and MICH pitch.  So, we need to figure out why we can't hold the lock, but it does not seem to be related to the CSOFT loop (ex. when I had the gain up too high, and we lost lock with a 2.75 Hz oscillation) or the dP/dTheta, since this is around 1Hz which is too high for that.  Also, we don't see it in the CSOFT loop until much later than other loops. 

We ran A2L - we ran it 3 times because ETMX was so far off that it needed help by hand to get close, otherwise it would have taken forever.  I'll look at what this means for our spot positions tomorrow.

Jeff is writing a separate log post about all the CARM loop measurements taken tonight, since we still have lots of freq noise at high frequency.  We're suspecting that we need to retune TCS, and perhaps that will help some of these other problems (at least SRM ASC)?

Sadly, after losing our 3rd big lock for the night, we seem to be back at having trouble doing the ALS->Trans CARM handoff :( .  Sheila tried doing the nominal guardian state, but with an LSC-TR_CARM offset of -0.8, but we lost lock, so now we've gone back to the nominal guardian order, and we'll have to look into this further tomorrow.  Actually, we tried again doing the order listed below by hand, but without the 16% gain matching and later reallocation, and we lost lock.  So, either we need to include that gain, or this order doesn't actually work.

----------------------

Notes to self about the order for the ALS->Trans handoff that seemed to work (although it was probably just getting better on its own) :

* ALS-C_REFL_DC_BIAS lower than normal, 20 or 24 rather than nominal 28 (although the max gain we could use seemed to increase toward nominal as time went on......).  Do not let LSC-REFLBIAS FM4 come on.  FM6 still okay.

* LSC-TR_CARM offset to -1.0 rather than the nominal -0.5, to put us closer to IR resonance.

* Increase LSC-TR_CARM gain by 16% to make up the difference between gain of 24 and gain of 28, but putting the gain in front of the integrator in this series of 3 filter banks.

* Turn on LSC-REFLBIAS FM4.

* Jump to guardian's CARM_ON_TR state (skipping PREP_TR_CARM, which was just done by hand in a different order).

* Reallocate gain for rest of guardian states:  LSC-TR_CARM back to nominal of 1 and ALS-C_REFL_DC_BIAS gain to nominal of 28.  

* Allow guardian to go forward as normal.

J. Driggers, S. Dwyer, J. Kissel

We were finally able to get past the CARM reduction and recover some semblance of nominal low noise this evening (in that ISC_LOCK guardian state, but running on SRC1 / SRM on dither alignment, having not reduced the 9 MHz modulation depth, beam diverters not closed; call this "abnormal low noise"). We've been able to get there a few times (no clue yet on why the CARM reduction has started to work). 

Having done so, we measured the CARM loop in a few configurations:
 (1) 2.0W input power, DC readout (literally, the ISC_LOCK state DC_READOUT), with the IMC boost disengaged
 (2) 29.5W input power, abnormal low noise, (without NOISE_TUNINGS so) with no IMC boost, just arrived at high power
 (3) 29.5W input power, abnormal low noise, (without NOISE_TUNINGS so) with no IMC boost, after the IFO has cooked for ~10 mins
 (4) 29.5W input power, abnormal low noise, (without NOISE_TUNINGS so) with no IMC boost, after the IFO has cooked for ~20 mins
 (5) 29.5W input power, abnormal low noise, with IMC boost, after the IFO has cooked for ~20 mins

Attached are the results, and below are the observations:
 (1) UGF at 18.8 kHz 
 (2) 12.3 kHz
 (3) 7.97 kHz
 (4) 6.36 kHz
 (5) 6.46 kHz
There is no place where there is less phase margin than 45 deg.

Here's *a* supposition:
- Once we get up to high power, we're losing CARM plant optical gain. CARM plant optical gain is defined by the beat of the 9 MHz modulation and the carrier's "audio" frequency sidebands. While 9MHz is fine because it doesn't enter the IFO, we are losing REFL carrier (and corresponding audio sidebands) as the IFO heats up. This is because we're in an alignment / loss space in which we're closer to the critical coupling point, where all carrier power gets sucked up (and lost in) the IFO. That means there's less CARM signal side bands, which means lower CARM gain. We correspondingly / corroboratively have a lower recycling gain (29, where it used 31). 

See previous aLOG about critical coupling point: LHO aLOG 29474.

Measurement Notes:
 (1) 30 mVpk excitation for all above measurements
 (2) still captured on ASCII because GPIB isn't working.

J. Kissel

Since we've put the End-Station VEAs on the wall, we noticed that today at 14:12 UTC (Jun 05 2017 07:12:00 PDT) the noise in the XVEA temperature sensor we've posted (H1:CAL-PCALX_RECEIVERMODULETEMPERATURE) dropped by a factor of two. Then, at 23:45 (Jun 05 2017 16:45 PDT) the higher noise state returned. Today's OPS Report suggests that no one went to any end station. More detailed trending shows that this decrease in "temperature" noise did *not* show up in any other VEA temperature sensor. 

Given the rash of what appears to be faulty Beckhoff ADCs (see FRS Tickets 6024, 8250), I wonder if there may be a problem here...

In other news, the very peaky, higher amplitude diurnal temperature oscillations are back.

I'll talk with Bubba tomorrow.

The CP4 autofill did not run this morning from crontab because I had checked its pending changes into SVN earlier and it became a non-executable file. The file was originally non-executable when it was first committed to the repository. I subsequently made it executable so I could call it directly and not as an argument to python. Inadvertently I did this incorrectly by just changing the working-directory file's permissions (chmod 775 cp4_fill.py).

I should have made it executable with the svn command:

svn propset svn:executable on cp4_fill.py

this command makes the local file executable, and changes the file's metadata ready for a commit to the repository.

By the way, normally a commit like I did this morning will not change the file permissions, but in this case the file has the $Header$ keyword substitution enabled, so when committed both the file's contents and its permissions were "corrected".

TITLE: 06/05 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: Jim
SHIFT SUMMARY:  Commissioning all day hunting down ALS issues and inability to proceed past PREP_TR_CARM.
LOG: 

22:52 Jenne and Sheila to LVEA ISCT racks to look at electronics.

Similar to the work done on the BSC-ISI's, I've been working on tilt decoupling on the HAMs. So far I have most of the measurements needed for the the output HAMs, I just need a couple more measurements on the input HAMs to finiish. The first attached plot is the Z to X/Y coupling for HAM6. For this data, I took the Z tf at the isolation bank input with the ISI isolated with high blends (some measurements had sensor correction on, some off, it doesn't seem to matter much, off is probably best though). I then inverted the gs13 response (to get the units to nm/s), converted to nm and then divided by -g/w^2 to get the effective tilt. Doing this let me calculate the magnitude and sign decoupling element by getting the average of the Z to X/Y tf where the phase was either 0 or 180, usually over 10 to 80-100 mhz. I was also able to avoid getting the RX/RY tfs as I done for the BSCs by thinking a bit about the RX/RY to Y/X tfs. These transfer functions go as -g/w^2, so their signs are also possible to figure out, and after some thinking I realized I could model them by using (essentially) a +1 for the RX to Y tf and -1 for RY to X tf (something Jeff mentions in his eligo era log here). The second attached plot shows the improvements I was able to get for HAM6 Z to RX/RY decoupling.

It's interesting that while the Z tilt decoupling clearly needed to be done, the X/Y to RX/RY coupling does not seem so bad. I have taken several X or Y tfs, but all of the coefficients are pretty small (on order of .1%). In fact the coefficients are small enough that the X to RY or Y to RX coupling is hard to seen. The third attached plot is the HAM4 Y to RX/RY measurement. The blue trace is the Y to X tf and gives a coefficient of about .0015 for the Y to RY element. The red trace is the Y to Y tf and is dominated by the Y to Y closed loop tf. The first couple of frequency points around 5 mhz indicated a Y to RX coefficient of something like .00005. I don't fee like waiting around for a measurement long enough to resolve that any better. This seems to be typical of the "beam-line" coefficients for the HAMs.

So far the elements I've installed are:

HAM 4     Z        Y

    RX     .01    .0015

    RY    -.01

HAM 6     Z   

    RX     -.0181

    RY      .0178

I have a few more calculated coefficients, but I'm going to wait for a chance to do a before/after measurement to make sure I actually make things better.

JeffK, TravisS

We noticed a DIAG_MAIN message reporting that the EY PCal RX PD was more than 1% out of normal.  We had this warning put in place to indicate if clipping we have seen previously at EY returns.  This time, however, it turned out not to be clipping, which is a good thing, but rather that something had happened that railed the OFS loop.  Trending the TX, RX, and OFS PDs showed that they all railed at the same time (of course) on June 2.  Talking to Sheila and looking at Ed's OPS shift summary from that day, there was a lot of electronics work done at EY earlier in the day.  Not conclusive that the two things are related, but definitely possible.

Hopefully this doesn't happen very often, but if it does, the fix is very easy.  Open the PCal screen for the end station of concern (from the CAL tab on the Sitemap), click Shutter 'OFF' button circled in the attached screenshot, then click the Shutter 'ON' button.  This should fix the issue.  If it does not, please contact your friendly neighborhood PCal person.

I measured the output power as a function of laser crystal temperature for the ALS laser that was at EndX
up until recently (InnoLight Prometheus S/N 2011B).

    The laser crystal temperature was varied by ~7 degC, which corresponds to about 42 GHz of frequency
tuning.  At the time of the measurements, the laser settings were:
diode 1 temperature             21.00 degC
diode 2 temperature             20.30 degC
diode current                   1.837 A
doubling crystal temperature    32.9 degC

    The laser was set to the following in order to put it into the middle of the mode hop free region
laser crystal temperature       26.11 degC
doubling crystal temperature    33.09 degC

    The output power of the laser was 1.600 +/- 0.002 W at 1064 nm and 48.1 mW at 532 nm with these
settings.

CP4 Fill completed in 244 seconds. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 244 seconds. TC B did not register fill. LLCV set back to 41.0% open.

My bad!  I had neglected to remove this ladder from having used it to access IP3 and IP4 following the recovery of the ITMx incursion a few weeks ago.  It was also in contact with the "Thermos" cooler (laser enclosure?) located at the base of the "giraffe" pier on the South side of the OMC tube near HAM4.

J. Kissel
II 4705
WP 7010

After recent review of open FRS/Integration Issue tickets, I've found and remembered that we haven't yet connected the OMC SUS's USER Watchdog to the ISI HAM6 (and hasn't been since before the OMC SUS was initially installed before O1). 

A while back, when we re-remembered this we'd argued that, if/because the OMC SUS OSEMs are covered in the IOP / Independent Software Watchdog (SWWD), that we need not both connecting the USER watchdog. However, I think now (a) since LLO has their's connected, and (b) if we're going to rely soley on the SWWD, this should be done for all chambers, not just HAM6, and it should be raised to a more general review of watchdog philosophy / ECR / integration issue, that is more work than we want to do today.

So, in prep for making the change, I've connected the USER watchdog at the top-level model of 
   /opt/rtcds/userapps/release/sus/h1/models/h1susomc.mdl. 

We'll restart the model if/when convenient today or if not tomorrow during maintenance. This will not require a DAQ restart.

See attached before vs. after comparison of the simple change in the lower right-hand corner.

Laser Status:
SysStat is good
Front End Power is 34.06W (should be around 30 W)
HPO Output Power is 158.5W
Front End Watch is GREEN
HPO Watch is GREEN

PMC:
It has been locked 3 days, 19 hr 9 minutes (should be days/weeks)
Reflected power = 16.11Watts
Transmitted power = 59.46Watts
PowerSum = 75.57Watts.

FSS:
It has been locked for 0 days 0 hr and 45 min (should be days/weeks)
TPD[V] = 3.415V (min 0.9V)

ISS:
The diffracted power is around 3.2% (should be 3-5%)
Last saturation event was 3 days 19 hours and 9 minutes ago (should be days/weeks)

Possible Issues:
Head 1-4 flow error, see SYSSTAT.adl

Averaging Mass Centering channels for 10 [sec] ...
2017-06-05 10:43:01.360996

There are 5 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -1.017 [V]
ETMX T240 2 DOF Y/V = -0.987 [V]
ETMX T240 2 DOF Z/W = -0.743 [V]
ITMX T240 1 DOF X/U = -0.664 [V]
ITMX T240 3 DOF X/U = -0.685 [V]

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = -0.064 [V]
ETMX T240 1 DOF Y/V = -0.03 [V]
ETMX T240 1 DOF Z/W = -0.008 [V]
ETMX T240 3 DOF X/U = -0.05 [V]
ETMX T240 3 DOF Y/V = -0.161 [V]
ETMX T240 3 DOF Z/W = -0.052 [V]
ETMY T240 1 DOF X/U = -0.008 [V]
ETMY T240 1 DOF Y/V = 0.18 [V]
ETMY T240 1 DOF Z/W = -0.224 [V]
ETMY T240 2 DOF X/U = 0.147 [V]
ETMY T240 2 DOF Y/V = -0.231 [V]
ETMY T240 2 DOF Z/W = 0.032 [V]
ETMY T240 3 DOF X/U = -0.244 [V]
ETMY T240 3 DOF Y/V = -0.019 [V]
ETMY T240 3 DOF Z/W = 0.271 [V]
ITMX T240 1 DOF Y/V = -0.213 [V]
ITMX T240 1 DOF Z/W = -0.15 [V]
ITMX T240 2 DOF X/U = -0.162 [V]
ITMX T240 2 DOF Y/V = -0.135 [V]
ITMX T240 2 DOF Z/W = -0.182 [V]
ITMX T240 3 DOF Y/V = -0.131 [V]
ITMX T240 3 DOF Z/W = -0.075 [V]
ITMY T240 1 DOF X/U = -0.024 [V]
ITMY T240 1 DOF Y/V = -0.046 [V]
ITMY T240 1 DOF Z/W = -0.058 [V]
ITMY T240 2 DOF X/U = 0.125 [V]
ITMY T240 2 DOF Y/V = 0.138 [V]
ITMY T240 2 DOF Z/W = 0.008 [V]
ITMY T240 3 DOF X/U = -0.248 [V]
ITMY T240 3 DOF Y/V = 0.053 [V]
ITMY T240 3 DOF Z/W = -0.263 [V]
BS T240 1 DOF X/U = -0.166 [V]
BS T240 1 DOF Y/V = -0.03 [V]
BS T240 1 DOF Z/W = 0.219 [V]
BS T240 2 DOF X/U = 0.067 [V]
BS T240 2 DOF Y/V = 0.195 [V]
BS T240 2 DOF Z/W = 0.007 [V]
BS T240 3 DOF X/U = 0.028 [V]
BS T240 3 DOF Y/V = -0.068 [V]
BS T240 3 DOF Z/W = -0.118 [V]

Assessment complete.

2017-06-05 10:45:39.355333
All STSs prrof masses that within healthy range (< 2.0 [V]). Great!

Here's a list of how they're doing just in case you care:
STS A DOF X/U = -0.609 [V]
STS A DOF Y/V = 0.203 [V]
STS A DOF Z/W = -0.448 [V]
STS B DOF X/U = 0.492 [V]
STS B DOF Y/V = 0.314 [V]
STS B DOF Z/W = -0.307 [V]
STS C DOF X/U = -0.37 [V]
STS C DOF Y/V = -0.661 [V]
STS C DOF Z/W = 0.497 [V]
STS EX DOF X/U = 0.018 [V]
STS EX DOF Y/V = 0.624 [V]
STS EX DOF Z/W = -0.01 [V]
STS EY DOF X/U = 0.002 [V]
STS EY DOF Y/V = 0.213 [V]
STS EY DOF Z/W = 0.443 [V]

Assessment complete.

Ed, Jason

Everything looks normal. The ever degrading HPO diode box 1 is causing a strange anomalous increase in the DC output power due to the changing thermal lens in head 1 resulting in an operating point which is, ironically,  actually better for the laser.

Summary: 

After swapping a few electronics that we broke while investigating the ALS glitches at EY, the glitches are still there, but not nearly as bad as they were.  We have had these glitches go away on their own before when nothing was done, so that might be what is happening.  Now they are at End X, and about as bad as they were at End Y for most of the week.  ALS still will not lock for longer than 10 minutes.  It might be that this problem will go away on its own, but will probably come back again.  

• After seeing that the behavior of the VCO at end Y tune mon changed around 6:30 UTC June 2nd, and based on Keita's alog last night, I swapped it for a spare I found in the EE shop this morning.  The unit I removed is S1200566, the unit I replaced it with is S12000562.  The VCO is no longer railing and the tune mon is near zero when the tune input is terminated.  The tune monitor BNC however has about 900 mV output even with the tune input terminated.  After this the arm would lock and I was not getting any VCO errors
• I found that the Seismic configuration was left in Large earthquake mode since Friday, and changed it to windy.  I found this because the arm transmitted powers were fluctuating more than normal for both arms, and also the oplevs showed extra noise since Friday evening.  Now the arms are both locked and quiet.
• At 1:08-1:23 local time, I started 15 minutes of locking the Y arm in green without automation (the PLL automation is on, not the PDH automation).  At 1:32-1:47 I did another 15 minute test with the only difference being that the PDH CM board was controlled by the tester, not the automation. At 1:50 I locked the arm using the automation, in the normal way. During all of these tests, there are a few glitches, but not as many and not as large as we were seeing mid week and last weekend. 
• I attempted to lock ALS, which didn't hold for more than a few minutes.  I then noticed that we are having the type of glitches we had been struggling with in the Y arm in the X arm now. (1st attached screenshot shows the Y arm and X arm both locked in the normal configuration with automation).
• I went to end X, and put the cable for the PDH signal into IN1 of the PLL board.  The X arm locked for green with in 1 polarity negative, and -4 dB of gain.  (For Y arm, we used positive polarity and 6 dB of gain).  No glitches in this configuration. 
• I also swapped the cables to substitute the spare FET demodulator for the PFD.  I added a 20dB attentuator to the RF cable into the demod, with this in I got a ugf of 27 kHz for an in1 gain of 13 dB. The autolocker works fine to lock this, you just have to force it to ignore the error message from the PFD by hitting the "Force" button. The arm was locked this way for 15 minutes starting at 0:03:30 UTC June 5th for about 10 minutes, the glitches were just as bad as the normal configuration. 
• I used Keita's set up from End Y to use the CM board tester at End X.  The cavity was locked this way from 0:47 UTC to 0:54 UTC June 5th, glitches are just as bad as the normal configuration.  

Things that are not the cause of glitches at EX:

From PLL bypass test:

• PLL servo board
• PLL servo board controls
• Laser
• PDH sensing, including diode, demod, and LO
• PLL automation

Other things that are not the problem:

• PFD (glitches still seen when it was swapped for a mixer)
• PDH servo board control cables swapped (so PDH control cables and automation are not the problem)
• loop instabilities (X end PLL UGF is 24 KHz with 47 degrees of phase margin, PDH Ugf is 5.6kHz with 56 degrees of phase margin).
• The beatnote is a clean sine wave, there are 8MHz sidebands on it but they are 40dB smaller than the carrier. 

Things that we haven't checked:

• PDH servo board, but this is not likely to be the problem since it wasn't at End Y and swapping it has some risks.
• VCO, we could try swapping this for an IFR but I wasn't able to find one in the shop today. 
• BBPD
• A problem on the light coming out of the fiber. it still seems like glitches on the PSL are unlikely because the glitches are not at the same time in the two end stations. 

I recommend that the first thing in the morning operators take the ISC lock guardian to locking arms green, wait there for 10-20 minutes, and look at second trends of the arm transmissions.  You can compare them to the screenshot I've attached here, if the glitches are gone or much smaller it would be worth doing an initial alignment and trying to lock to DC readout, if the glitches are still there it is not worth trying.