Ops Shift Log: 10/27/2016, Owl Shift 07:00 – 15:00 (00:00 - 08:00) Time -  UTC (PT)
State of H1: IFO is relocking
Intent Bit: Commissioning
Wind: Is ranging from Calm to Light Breeze (0-7mph)
0.03 – 0.1Hz: Currently at 0.09um/s  
0.1 – 0.3Hz:  Currently at 0.8um/s
Outgoing Operator: TJ
Incoming Operator: Travis
 
Activity Log: Time - UTC (PT)
07:00 (00:00) Take over from TJ
07:20 (00:20) Relocked at NOMINAL_LOW_NOISE
08:18 (01:18) PI Mde-28 ring-up – Changed Phase from 20 to 150
09:12 (02:12) Mag 5.9 EQ in Indonesia – Established lock during the EQ
09:34 (02:34) Set Intent Bit to Observing
12:08 (05:08) Lockloss – 5.8 EQ in Alaska – Microseism up to 3.0um/s
14:52 (07:52) Ran initial alignment and relocked at NOMINAL_LOW_NOISE
15:00 (08:00) Turn over to Travis
	

Shift Details:
Support: Sheila, Jenne, Kiwamu 
Shift Summary: IFO is locked at NOMINAL_LOW_NOISE at 24.0W, range is 72MPc. After a commissioning Lockloss, relocked during a Mag 5.9 EQ in Indonesia with a R-Wave of 1.19um/s. Set intent bit to Observing.

Started running TCS noise study for Kiwamu (aLOG #30920)

Recovery after Alaska earthquake
Ran initial alignment and relocked at NOMINAL_LOW_NOISE, 22.6W, 74.2MPc	

Kiwamu saked the ops to run some TCS laser noise measurements.

SETUP:

• After IFO is locked at NLN for at least 30 minutes
• Open TCS_MASTER screen
• Open the CO2 X & Y screens
• Open the POWER CONTROLS screens for X & Y
• increase the power by 0.1W
• Wait 90 minutes, and increase the power by another 0.1W
• Repeat until at about 2W
• aLOG the times and power levels

Started the run: 

    TSC X - Initial Power = 0.2W                     TSC Y - Initial Power = 0.0W

At 05:08 Lost lock due to a Mag5.8 EQ in Alaska.

Sheila, Kiwmau, Jenne

Once the IFO relocked after our unexplained alignment change, we did a few quick tests.  We opened and closed shutters on the DBB, and looked for coherence between DARM and the rack power monitor that we set up yesterday. The DARM noise no longer changes with the state of the DBB shutters, this may be due to the alingment and TCS changes that got rid of our lump in DARM (even at 50W), or due to the PMC realingment that reduced the PMC HV noise in DARM. 

However, there is non neglible coherence between the rack power monitor and DARM, from about 320 to 380 Hz.  The spectrum of the rack voltage isn't especially noisy at these frequencies.

An attempt to re-measure the PMC HV noise broke the lock when I enabled the excitation.

For Jenne:

Injection for iterative tuning of SRCLFF: started at 8:08:33, went until 8:11:00 UTC on October 27th. 

[Jenne, Kiwamu]

Since we kept losing lock at Noise Tunings, on the third time around I went to step through the state line-by-line.  But, we lost lock in the same way at the first line:  DC coupling the ISS second loop. 

Kiwamu pointed out that since the light level on PDA changed yesterday (alog 30871), we might need to change the offset in the 2nd loop.  This caused us to realize that since we had forgotten to SDF-accept the offset we picked on Monday (alog 30817), there just wasn't any offset at all.  In the end, Kiwamu and I chose -0.25 for the offset H1:PSL-ISS_SECONDLOOP_AC_COUPLING_SERVO_OFFSET, since that is the value that kept the diffracted power constant when the 2nd loop was closed.  Note that on Monday we changed the way the AC coupling is turned off, so that this offset doesn't integrate up forever by turning it off at the output rather than the input.  This is in guardian.  

After this, we noticed that all of the power seemed to be on 2 of the inner loop PDs, rather than balanced between all 4.  This is certainly a result of my moving IM3 earlier today (alog 30910).  We found Robert's alog 29583, to remind ourselves which picomotor was which, and moved motor #1 on the HAM2+oplev controller, which is labelled PSL ISS QPD/PD (PM1).  This is the motor closest to the ISS array.  We used Large steps at a Sprint, since this was not in-loop.  The positions for the picomotor started at (0,0) for (X,Y), and ended at (-1900,0).

Although the QPD was at about 0.8 in yaw before yesterday's alignment shenanigans, we went for center.  As Robert pointed out in his alog, center on the QPD maximizes (and roughly equalizes) the power on all 4 diodes.  Kiwamu and I checked that this was better also by driving a line in IM3 and seeing that it did couple to IMC power when the beam was falling off the diodes, and then didn't couple to IMC input power when the beam was centered.  As one might expect since we didn't have to move in pitch, driving a pitch line didn't cause problems either time. 

At next acquisition we had no problems with the ISS, and are now sitting in NomLowNoise.

TITLE: 10/27 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: The start of the shift seemed bleak, things were basically still as they were when I left yesterday, but with Jenne and Sheila's help we managed to make it through IA by resetting the dark offsets, increasing the limiter on IM4, and lots of aligning work. After IA we spent more time aligning things further and eventually got past DC_READOUT for over an hour before moving on and losing it at NOISE_TUNINGS. The female commissioning crew and Kiwamu have been working the ISS since then, but are now finished and we will start to head up again.

 

We are havgin trouble using the lockloss tool, but it has been working since Jamie made a change on October 19th.  Is this related to problems with NDS2?  TJ and I used lockloss plot earlier in the day today without problems.

Here's what I get:

lockloss select
Segmentation fault (core dumped)
 

Manually finding a time to ask for:

lockloss -c channels_to_look_at_NOISE_TUNINGS.txt plot 1161580111

.......

INFO: plotting: Adding subplot with the following channels:
        H1:LSC-ASAIR_A_LF_OUT_DQ
2016-10-26 22:14:23,904 : NDSAxes : Initializing NDSAxes for the following channels:
        H1:LSC-SRCL_GAIN
INFO: plotting: Initializing NDSAxes for the following channels:
        H1:LSC-SRCL_GAIN
2016-10-26 22:14:23,938 : NDSBufferDict : Buffers requested at start time 1161580081 and end time 1161580110 for the following channels:
        H1:LSC-SRCL_GAIN
INFO: bufferdict: Buffers requested at start time 1161580081 and end time 1161580110 for the following channels:
        H1:LSC-SRCL_GAIN
terminate called after throwing an instance of 'NDS::connection::daq_error'
  what():  Low level daq error occured [13]: Requested data were not found.
None of the selected channels returned data.
Aborted (core dumped)
 

After a long struggle with initial alignment, we eventually went with the method that Sheila used yesterday (alog 30881).  Basically, get it locked, and then deal with fine-tuning the alignment.

Once at DC readout, I moved IM3 very slowly in yaw to get the IM4 trans QPD's yaw value back to Monday's alignment (pitch didn't need moving).  This brought the recycling gain from 24 to 31.  Offloaded the alignment while at 2W DC readout using the guardian shell:

• guardian -i ISC_LOCK
• opticList = ['ITMX','ITMY','ETMX','ETMY','BS','PRM','SRM','PR3','SR2','IM4']
• ramptime = 20
• ISC_library.smooth_offload_fast(opticList, ramptime)

Was able to go through most of the rest of the acquisition sequence, although we just lost lock at CloseBeamDiverters.  Not sure why. 

Now we're almost back relocked - not too much difficulty other than the usual needing to do PRMI before DRMI that we've been seeing a lot lately.  We still haven't tried an initial alignment with the new IM3 position.  Also, IM3 is very nearly at the edge of its actuation range for 2 of the coils.  We should probably offload this to IM1 and IM2 when we have some time to ensure that we don't clip through the Faraday.

J. Kissel, D. Barker

I've done several things to investigate the problems recently been seen with the binary switching of the H1 SUS ETMX L3 ESD LV/HV driver system (see FRS Ticket 6527, LHO aLOG 30879, LHO aLOG 30770).

1) I've toggled all of the relevant channels,
H1:SUS-ETMX_BIO_L3_[UL,LL,UR,LR]_VOLTAGE_SW
H1:SUS-ETMX_BIO_L3_[UL,LL,UR,LR]_HVDISCONNECT_SW
and confirmed that they are monitored in the 
/opt/rtcds/userapps/release/sus/h1/burtfiles/h1susetmx_down.snap
and they show up as DIFFs when toggled to the wrong state for high-voltage actuation.

2) I've confirmed that where the front end looks for safe.snap and down.snaps,
/opt/rtcds/lho/h1/target/h1susetmx/h1susetmxepics/burt/
lrwxrwxrwx 1 controls controls     64 Apr 27 12:23 down.snap -> /opt/rtcds/userapps/release/sus/h1/burtfiles/h1susetmx_down.snap
lrwxrwxrwx 1 controls controls     64 Jul 19 19:22 safe.snap -> /opt/rtcds/userapps/release/sus/h1/burtfiles/h1susetmx_down.snap
are pointing to the same down.snap reference file.

3) With the help of Dave, I've restarted the front end process for h1susetmx, and confirmed that upon reboot, the front end restores to 
lrwxrwxrwx 1 controls controls     64 Jul 19 19:22 safe.snap -> /opt/rtcds/userapps/release/sus/h1/burtfiles/h1susetmx_down.snap
and it restores correctly to the high-voltage configuration.

4) I've looked at the list time the down.snap file was changed, 
-rw-rw-r-- 1 jeffrey.kissel controls 103354 Oct 25 12:43 h1susetmx_down.snap
(that's Oct 25 12:43 PDT) and it was yesterday, when I was removing the ISIWIT channels from the reference files (as mentioned in LHO aLOG 30844)
so it's not as though these settings *were* wrong the last three reboots when there *was* a problem, and then yesterday Sheila (after I'd left for the day and found the problem) modified the down.snap to have the correct values.

5) I've checked the ISC_LOCK guardian for instances of "BIO_L3" looking for any incorrect guardian toggling of these switches, and found that only the DOWN and LOW_NOISE_ETMY states toggle those switches. The DOWN state correctly toggles the switches to high-voltage actuation, and the LOW_NOISE_ETMY state correctly toggles the switches to disconnect the high-voltage. In fact I grepped all guardian code that lives in /opt/rtcds/userapps/release/isc/h1/guardian and /opt/rtcds/userapps/release/sus/h1/guardian, and found that no other guardian touches these channels.

6) Since ISC_LOCK toggling uses guardian infrastructure that commands the switching with 1s and 0s instead of ONs and OFFs, I confirmed via the command line with good ol' "caget" and "caput" that what the guardian sends the correct commands.

In summary -- No Dice. I was not able to reproduce Sheila's problems, upon reboot the switches come up in the right state, and just in case the ISC_LOCK guardian forces the switches to be in the right state. The problem that Sheila and Ed saw should not have occured.

I have no choice but to throw up my hands and close the FRS ticket. Sorry! 

It has been noted that some ADC channels in the h1alsex model (e.g. H1:ALS-X_LASER_IR_LF) dropped slightly after the h1iscex machine was power cycled due to loss of network connection (cycle was at 11:33 PDT). This was not seen when the h1alsex model was restarted due to crashed epics process at 09:38 PDT. We think that because the computer was power cycled but the IO Chassis remained powered, that this is an unusual startup sequence and perhaps we should perform a full power cycle of both systems.

One hour second trend plot attached.

Jeff, Dave.

we restarted the h1susetmx model (no code or filter changes) as part of the binary IO investigation.

3:30 pm local

Took around 90 sec. to fill CP3 to the point where cold N2 vapor was coming out. I let it run another 3-1/2 min. to see if TCs would register LN2-ish temps (with LN2 trickling out). Never saw anything much below -50C. 

I filled by doubling LLCV to 36% open.

TITLE: 10/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: TJ
SHIFT SUMMARY:  First half of the shift was dedicated to Kiwamu/Jason chasing PSL alignment.  As soon as they determined that the pointing was OK, and gave me the go-ahead to start locking, we were hit by the 6.1 EQ in Italy.  After waiting for that to ring down, Jenne and I began diagnosing the issues with INPUT_ALIGN during Initial Alignment.
LOG:

16:07 Chandra, John to MY

16:50 Richard to EY

17:00 Jason, Kiwamu to PSL

17:37 Richard back

17:38 Betsy to LVEA

18:10 Jason, Kiwamu out

18:12 Betsy out

18:59 Jason, Kiwamu to IOT2L

19:02 Chandra back, going to EX

19:49 Betsy to LVEA TCSy table

19:55 Kiwamu to LVEA marking wall

22:08 Chandra to MY
 

As noted yesterday, LLCV and %full at CP4 are noisy (before and after Dewar fill). Things stabilized after I opened the exhaust bypass valve. Today I removed the check valve and closed bypass valve; noise again. Then I installed CP3's check valve; still noisy. So I reinstalled the original check valve and left bypass valve open. Suspect the PID parameters need adjustment. 

Stefan and I were wondering if someone could re-run Bruco on SRCL/PRCL/MICH on  the other night good data, similarly to what Gabriele did some time ago.

Also, I attach here the coherence between SRCL and IMC WFS at some point during O1; this seems much lower than what seen recently, which I think makes sense based on  Sheila's IMC WFS comparison  before and after the HPO was turned on.

I got suspicious about PMC length locking offset and changed H1:PSL-PMC_INOFFSET.

Increasing it by 1.7mV decreased the PMC length feedback by about a factor of 2, and 1st loop out of loop sensor by a factor of 4 or so, which doesn't make sense. In the attached, red and green are with nominal 3.1mV offset, blue and brown are with 4.8mV.

(After this measurement I noticed that Daniel increased the length gain from 16 to 28dB and forgot to bring it back. This measurement is with 28dB locking gain, but it still doesn't make sense.)

What's the nominal signal level for PMC demod? Is it tiny? When is the last time the PMC demod phase was optimized?

We have had trouble with alingment since maintence day today.  There were several things that happened that could potentially impact alingment: SUS models were restarted, HEPI work, and adding picomotors to the PSL before the PMC.  The PSL work is probably the most likely culprit.  I hope that we will be more selective about what we choose to do on maintence day from now on.

Alingment symptoms:

The PMC alingment is not as good as it had been.  There were also shifts in the alingments of MC1+3, especially in yaw.  Were these from model reboots or did someone intentiaonally change them to relock the mode cleaner?  The spot on IM4 trans move from 0.2 in yaw to -0.35 or so.  TJ and I trended all the IM osems and they hadn't moved much, but we restored the small changes.  Once Jeff B and I finally got the IFO locked (I engaged the soft loops one at a time by hand), our recycling gain was very low (below 24).  I tried to move MC1+3 in yaw to restore the position on IM4, both earlier with TJ and with the full IFO locked, the MC WFS generally drag it back to where it was.  With the full IFO locked I moved IM3 in yaw, which did help the recycling gain (I chose IM3 because it's after teh Faraday, not because I think it is what moved).  I moved it too fast and broke the interferometer lock, so we finished using it to restore the position on IM4 Trans with just the mode cleaner locked.  We partially reverted this because we saw no flashes in the arm, and are going to give up for the night now.

If we still have this problem in the morning, it is probably worth taking a look at the irises that Cheryl placed on the PSL table a while ago to see if the beam goes into the IMC with the same alignment.

REFL WFS don't work for PR2 at 2 Watts:

I changed the ISC_LOCK guardian back to using POPX WFS for acquisition.  The point of the POPX WFS is that they can be used even when inital alingment isn't great and the recycling gain is low, so we would like to keep them on for power up and switch to REFL WFS in LOWNOISE_ASC.  I've put some code in low noise ASC to switch back, but we haven't gotten to test this yet. I guess that the matrix to use REFL to control PR2 was probably tested at 25 Watts last night but set up in the guardian for use at 2 Watts, because Jeff B couldn't engage the REFL WFS yesterday morning after commisioners left.  Tonight it was clearly breaking the lock, which might have been partly because of our bad alingment, but when we changed it back to POPX it was fine. 

Initial Alignment changes:

Jeff B and I just made a few changes to initial alingment, after TJ, Evan, and I had difficulty with input align earlier in the night.  Although our alingment change today probably contributed to this, I think the changes are good and things I've been meaning to do for a while anyway.

• We re wrote the REFL WFS Centering states to be identical to the WFS centering states used in DRMI, without the AS loops.  This was beacsue the old refl wfs centering state would not always stop the operator from continuing to align if the DC servos were railed.
• We changed the INPUT align state from using REFL WFS, which have very little light on them but were the only WFS available when we first wrote this state, to using AS45I.  These are better signals and I've been meaning to make the change for a while now.  To do this we made AS WFS centering states which are a copy of the REFL WFS centering states. 

Stefan, Sheila, Terra, Ed and Nutsinee

We have found that alingment and TCS together can improve our noise hump from 100Hz-1kHz.  We have reverted both alignment and TCS changes to July, and we seem to be stable at 50Watts with a carrier recycling gain around 28. 

TImes: 

22:18:18 Oct 23rd (before alingment move, at 40Watts) and 22:28:12 (after)

ten minutes of data starting at 23:33:33 is with the better alingment at 40 Watts, 10 minutes starting at 0:19 UTC Oct 23 is at 50 Watts.  (we redid A2L at 40 Watts, but not 50 Watts, there is MICH and SRCL FF retuning still to be done.)

We saw that the TCS changes of the last few days made a small improvement in the broadband noise lump from 200Hz -1kHz, so we decided to retry several of the noise test we had done before without sucsess.  We only moved the POPA spot position in PItch, moving it in yaw made the carrier recycling gain drop but didn't help the noise.  The attached screenshot shows the spectra, and the coherence with IMC WFS, our best jitter sensors in lock.  We have lots of coherence with these signals, at frequencies above where the HPO changed the spectra.