Alexa, Evan, Koji, Arnaud, Sheila, Rana

The good news is that ALS is quite stable tonight, staying locked for up to 2 hours.  

• Earlier in the day we struggled for a while to recover alingment after maintence day.  We had difficulty locking DRMI plus the arms, the DRMI ASC didn't work well, with the arms.  We eventually turned off the ALS WFS, locked DRMI without the arms, used the ASC (MICH, SRC1, SRC2, and PRC2), and offloaded.  Since then we have not had much problem with the DRMI alingment.  It seems that actuating on the ITMs with the green WFS may be spoiling our DRMI alingment.  For now the green ASC is off, with the inputs turned off, so that even when the Gaurdian thinks the WFS are on, they are not. 
• We also have turned off the IMC ASC DOF4.  
• DRMI lock acquisition has been slow tonight, sometimes taking about a half hour.  We checked our open loop gains, which were all normal.  We would like to spend some time speeding this up tomorow so we can make progress faster.  
• We noticed that we were mostly adjusting the alingment of ETMX pitch in the positive direction after the arm transmission would drop, Arnaud noticed that the top mass moved by 10 um starting at 11 am this morning, the same time as a small temperature excursion.  
• We have had two trips of ETMX tonight, which were ISI, Hepi and suspension trips (the first was at 5:15:26 UTC Jan 21st, the second at 7:30:15 UTC) Arnaud investigated the second one, and found that there was a large (200 um) impulse sent to HEPI from the tidal feedback right before the trip. The six hour trend of the signal sent to HEPI shows that the tidal system has sent some relatively large impulses to HEPI several times tonight (second screenshot). 

We have been having trouble transitioning to 3F tonight.  We can transition MICH and PRCL fine, but we have seen a lot of noise at 300 Hz when we try to transition SRCL to 3F.  When we try to transition we see noise at 23.5Hz.  

ALS lock loss times: (all Jan 21st UTC) 0:03:02, 0:34:26

ALS lock loss +HEPI, ISI and sus trips: 7:30:15, 5:15:26 

Lockloss and bad misalingment of IMC due to DOF4: 6:35

Comparison of the OMC length noise for H1OMC and L1OMC.

Note that floor noise of the L1 measurement was limited by the intensity noise at the time.

Math puzzle: What's wrong with this UGF Servo?

its installed for use in the OMC and LSC at the moment, and could be used in the ASC if we find we want to hold the UGFs constant during TCS tuning.

Dan, Koji

This is a followup to Koji's post that described our OMC work on Sunday.  (Here I give details of the noise investigations; the measurement of the OMC actuator noise was described by Koji yesterday.)

The summary is that the excess ~1kHz noise in the OMC length error signal is not consistent with intensity or alignment fluctuations, and isn't being amplified by a resonance on the OMC breadboard.  This agrees with Zach's investigations from L1.  The peaks are consistent with cavity length noise that is driven by acoustic noise coupling into the chamber via the ISI, but our measurements aren't conclusive.  We will have to see if the noise appears in DARM -- at L1 this doesn't appear to happen.

1) Intensity noise suppression by ISS first loop.

First we investigated the stability and noise suppression of the ISS first loop.  We found we could reliably close the first loop once the input offset had been tuned, but oscillations would develop when the integrator was turned on.

The first plot is intended as a qualitative reference of the performance of the ISS first loop.  The intensity noise measured by the LSC REFL photodiode is improved above 300Hz with the first loop on, but the noise below 300Hz is unchanged, and there are peaks (at 700Hz, etc) that are not suppressed.  Is this noise coming from the IMC?

(We had to use the REFL PD to measure the intensity noise after the IMC because the beam going to the ISS second loop PDs is misaligned.)

2) 1kHz peaks in OMC noise - intensity noise, alignment jitter, or OMC length noise?

Next we locked the OMC on a half-fringe and studied the noise.  If the peaks at 1kHz are due to intensity noise (maybe due to a bad ISS), the coupling should increase as we move the OMC lock from the half-fringe to the full fringe.  If the noise is due to beam jitter, we can change the coupling by moving the alignment into the OMC.  If the peaks are due to real cavity length noise it should couple more strongly at the half-fringe lock.

The second plot tells the story -- there was no change in the peaks when we misaligned the cavity (compare purple and black), which seems to rule out beam jitter.  There was some change in broadband noise as we moved the locking point up the fringe (compare blue, to cyan, to purple); this indicates that we were limited by intensity noise.  In particular the peak at 817Hz in the lower plot is consistent with an intensity noise explanation.  But, the other peaks were mostly unchanged as we detuned the cavity (although the 877Hz peaks seemed to be slightly worse at the 50% fringe).  This appears to rule out intensity noise, but it's not really consistent with cavity length noise, either.  So, we're baffled.

(Regarding beam jitter: we had tuned the alignment into the OMC by using offsets in the QPD loops to minimize the linear coupling between excitations of the tip-tilts to the OMC TRANS spectrum.  Turning the offsets on reduced the height of alignment dither lines by 10x.  So, we should have seem *something*, if the peaks were due to beam jitter.)

For reference, the frequencies of the largest peaks are the following: 817, 877, 937, 991, 1055, 1132, 1165 Hz.  The location of the DTT file is at the top of the figure.

3) Comparison between PDH error signal and OMC TRANS.

Next we steered the OMC REFL beam onto ASAIR to use the PDH error signal as an out-of-loop sensor for the cavity length.  Here, the cavity was locked at 80% of a fringe, to move the PDH error signal into the linear region.  The third plot compares the PDH error signal (out-of-loop) to the OMC TRANS noise.  (We rotated the ASAIR_A phase to maximize the signal in the I_phase - see coherence in top panel.)  In the lower panel I've used a calibration factor to overlay the two curves.  The peaks around 1kHz appear in both signals, which also rules out intensity noise as a source.  The peak at 817Hz only shows up in OMC TRANS, not the PDH signal --> still consistent with an intensity noise source.

4) Transfer functions from OMC SUS and tip-tilt SUS to PDH error signal

The only hypothesis that remains is real length noise in the cavity.  This idea is supported by Robert's measurements of the ISI, which showed that noise from outside the chamber can be transmitted through the ISI due to resonances at ~1kHz.

We thought that the ISI resonances could be lining up with resonances in the OMC breadboard.  Koji had measured a breadboard resonance around 1kHz in bench tests, although the boundary conditions of the suspension points were different from the in-vacuum suspension.

So we tried to excite the modes using the SUS.  The fourth plot is a swept-sine transfer function of the OMCS vertical mode to the PDH error signal.  There's no coherence at the frequencies of the noise peaks; we believe this rules out the OMC breadboard resonances as the source of the peaks.  But at other frequencies our excitation was coherent with the length error signal, and the transfer function is pretty flat; the implication is that the OMC breadboard has a flat response, approximately, to excitations in the SUS, which means that noise from the ISI might couple into the cavity length.  Not good.

We also tried to measure the beam jitter coupling in the same band, using a pitch excitation on OM3.  This is the fifth plot, and the story is the same - coherence in some bands, but not at the peaks in the noise spectrum.  Also not good.   (It wasn't clear to us why beam jitter noise was coupling into the PDH error signal.  The cavity was locked using the length dither with a 30Hz UGF, so it's hard to imagine jitter at 900Hz being impressed upon the cavity length via the dither servo.)

5) Encore noise investigation: IMC DOF4 error point and sideband detuning

Right at the end of the day I was setting up a measurement of the IMC cavity pole.  With the OMC locked, we noticed that changing the frequency of the 9 and 45 MHz sidebands slightly off the IMC resonance would cause the OMC TRANS signal to drift down.  Eventually we realized that moving the sidebands by 20kHz changed the reflected power from the IMC, which moved the error point for the DOF4 loop and slowly misaligned the IMC.  See the sixth plot.  This isn't really surprising, but it was pretty confusing for a few minutes.  (The many dropouts in the plot are due to the ISS becoming unstable - these show up in the DC power signals but not in the ASAIR_A_RF45_I_ERR.)

h1calcs: Dave, JeffK, JoeB

WP5016. This install was postponed from today, the model is going to be reworked. WP is open through next Tuesday.

ETM SUS FE reboot tests; JeffK, Jim, Dave

Following some problems starting the ETM suspensions front end computers during last week's RCG update, we tested this further today by rebooting h1susex and h1susey from a powered down state. The procedure was: stop user models, stop iop model, take out of dolphin fabric, power down computer, via IPMI power computer back up.

We did not get a clean restart of all models and awg_tp_man processes. The restart problems were not reproducible on both end stations. Some manual intervention was needed.  Investigation is continuing.

Power cycle of DAQ Solaris QFS writers: Jim, Dave

the frame writer spontaneous restarts have become more frequent since last week's RCG2.9 upgrade. Today we power cycled both QFS writers in the MSR to see if this will make the DAQ more stable. The procedure was (for both h1ldasfw0 and h1ldasgw1 systems, staggered restarts so one framewriter is operational at all times):

on nds: stop monit, stop daqd process, stop nds process, un-mount ldasgw NFS mounted file system

on fw: stop monit, stop daqd process, un-mount ldasgw NFS mounted file system

on ldasgw: power down with "poweroff" command. Power up by pressing front panel button. Manually mount the QFS file system, manually share the file system

on fw: start monit. Verify daqd runs and writes frame files

on nds: start monit, verify daqd and nds process startup.

The only surprise was when h1ldasgw0 was powered back up, it started a continuous hardware diagnostics which I manually killed after 10 minutes of running. h1ldasgw1 did not do this.

We will know if this has helped in a few days of running.

DAQ reconfiguration, Dave:

I created the latest Beckhoff slow controls INI files and restarted the DAQ after the above solaris reboots had completed.

OS updates, Dave:

all servers accessible from outside of CDS were upgraded to the latest OS patches. cdslogin and cdsxegw0 required a reboot to complete. cdsssh was rebooted yesterday to complete.

Filter module FE load status, Dave, TJ:

The following models had partially loaded filter files and were fully loaded from the GDS-TP screen: h1sussrm, h1oaf, h1lsc, h1omc, h1iscey.

The following models have modified filter module files which have not been loaded, further investigation is needed (modifcation time is given): h1isiitmx (Tue 13jan2015 16:16), h1susmc2 (Sat 17jan2015 17:04)

FE code with local modifications pending SVN commit, Dave:

Attached file shows local mods to model mdl files.

inicheck fix: Dave, Jim

We found inicheck was reporting a warning if a channel's UNITS setting was different from that in the default block, problem arises from RCG2.9 upgrade. Jim has fixed this.

We have had SDF examples of a setting change which is smaller than the displayed resolution and so hidden by rounding. For example, I have changed a gain setting from its safe.snap value of 1.0 to 0.9999999999999998 and the SDF difference table shows both values being rounded to 1.000000. To get the actual value to 15 decimal places, use caget with the "-e 15" argument (without this argument it rounds to 1.0)

caget H1:PEM-MY_CHAN_17_GAIN
H1:PEM-MY_CHAN_17_GAIN         1
caget -e 15 H1:PEM-MY_CHAN_17_GAIN
H1:PEM-MY_CHAN_17_GAIN         9.999999999999998e-01
 

The snapshot of the table shows the two values are apparently the same when rounded to 6 decimal places.

6:45 Jeff - LVEA West Bay SUS Storage, LVEA East/West Bay Laser Safe
7:15 Cris – LVEA in
7:26 Mitch – LVEA in
8:12 Andres – LVEA West Bay w/Jeff
8:35 Mitch – LVEA out
8:45 Kyle, Gerardo – LVEA N. Crane
9:15 Filberto – To End X 
9:20 Kyle, Gerardo – LVEA out
9:30 Doug, Jason – To both ends
9:39 Christina – To both Mids for storage
9:57 Joe – End X
10:01 Corey – To End Y
10:14 David, Jeff – ETMX, TMSX, ETMY, TMSY brought down to Safe for restart
10:21 Truck in to take water samples
10:27 Cris – End X
10:29 Doug, Jason back from ends, to LVEA
10:45 Water delivery
11:00 Jeff, Andres – Out of LVEA, back to Full Laser Hazard
11:03 Doug, Jason out of LVEA
11:05 Kyle, Gerardo – Out of LVEA, they left a cart on next to BSC7
11:17 Sudarshan, Darkhan – To Ends to set up Pcal Lasers
11:39 Corey – Back from Ends
13:23 Kyle – Into LVEA to turn on power supply
13:52 Sudarshan, Darkhan - Done at Ends

J. Kissel, K. Izumi, A. Staley, S. Dwyer

ETMX, ETMY, and HAMs 2 and 3 were involved in maintenance day, and recovery was little-to-no different than any other Tuesday. Chasing alignment in the IMC and ARMs have been the most time consuming parts, though my impression is that it has been a little bit shorter / better today and the SUS are not nearly as far off as they have been. Indeed, in order to hard-reboot the SEI HAM23 front end and IO chassis, we needed to take down the IMC, all its associated suspensions (and the IM and PR suspensions), and the HAM 2-3 ISIs and HEPIs -- and this DID NOT shake the MC REFL path enough to cause any misalignment in pitch as we've seen in the past (see Integration Issue 854. Rather than going through my usual detail play-by-play, I've decided that its more productive to list what we should change as a result of today's experiences in order to keep getting better at the recovery.

Lessons (at least *I*) learned from today, in no particular order:
- We need to have guardian transition the BSC ISI Stage 2 Gains from HI to LO when a SEI manager requests to go from FULLY_ISOLATED to OFFLINE for ALL BSCs, because requesting to go from WATCHDOG TRIPPED (afer computer reboot) to OFFLINE to FULLY_ISOLATED after does *not* transition the gains to LO before getting started.
- We need to have the HAM ISI guardians *ensure* the GS13 gains are HI.
- We need to have an IMC "down" state that is actually DOWN, (or perhaps call it "OFFLINE") -- specifically the IMC WFS are turned off so no angular control is fed to the MC suspensions, and (less importantly) no longitudinal control is fed to MC2.
- The new(ish) IMC ASC DOF4 / MC WFS center servo really can pull the MC REFL pointing off into the weeds, so these should be the first thing turned off when ramping down the IMC.
- The transition from ISOLATED to OFFLINE on the HAM's watchdog trips the ISI halfway through the turn off process more often than not, and its likely because the GS13 gains remain in high-gain mode indefinitely.
- We need a guardian to manage the HSSS (today it was the IMs, but another day it was the RMs, etc.).
- We need to calculated / measure / install / save alignment offset calibrations for the HSSS. 
- We need a standardized, well-ordered check-list procedure for taking chambers / parts of the offline -- and then bringing them BACK online.
- We need to either resurrect / commission / trust the SUS DRIFT_MONITOR, or find another tool that we can quickly assess how all SUS alignments have changed from prior to the start of a maintenance day to after finishing.
- We need a tool that takes the output of the above (i.e. the assessment of the alignment), and *fixes* any SUS that are out of alignment.
- When the global "initial" alignment changes -- even by a few microradians here or there -- the optical gain of the IFO changes, which means we need to re-adjust ISC parameters.
- We need to include turning ON and OFF the ESD Bias into the SUS ETM guardians, such that it is ON in every state but SAFE. Further, we should pick a standard location to turn ON and OFF the ESD BIAS path.

We're not sure if we're *fully* recovered at this point, but we have gotten as far as a short DRMI locks with ARMs off-resonance, and we push onward.

Regarding the Guardian Logs: 
SEI_ETMX Manager Guardian was requested to go to OFFLINE from FULLY_ISOLATED between
2015-01-20 18:15:24.753 UTC
2015-01-20 18:16:43.581 UTC 

Rana, JeffK, Kiwamu,

We did some maintenance items on IMC ASC. The major activities are:

• We had a difficulty in restoring the IMC alignment back as we ran the offloading script when the HEPIs were down and therefore we screwed up the alignment.
  • Interestingly, restoring the suspensions using the witness sensors as references did not work at all. Simply burt-restoring the suspensions and ASCIMC frontends brought it back to a good lock-able alignment.
  • After the recovery, we ran the offloading script again and so all the MC suspensions now should be in good alignment.
• We set a limiter in each loop with a hope that this is going to reduce a risk of tripping the suspensions (alog 16138, 16128) . Here are the counts we set for the limiters:
  • DOF_1_P(Y)_LIMIT = 125
  • DOF_2_P(Y)_LIMIT = 125
  • DOF_3_P(Y)_LIMIT = 250
  • DOF_4_P(Y)_LIMIT = 3000
  • These values are derived such that they are roughly the same as their 5 sigma deviations in 100 days trend except for the DOF4 loop which had hit the new limiter once in this evening. So we set the DPF4 to a high value.
• We disabled the 2nd whitening stage in MC2_TRANS QPD because the signals were saturating most of the time when IMC was not locking.
  • Therefore the MC2_TRANS QPD now has one whitening filter.
• We cleaned up the control filters and installed a new low-pass in every DOF. Rana will report some more details about this.

Also, tomorrow or sometime this week, we might insert a lens in the REFL gigE path as the REFL beam now is a bit too big to be able to cover the spatial higher order modes.

J. Kissel, J. Warner, D. Barker, E. Merilh, TJ Shaffer

The one final quiver in our arrow against the sharp HAM3 0.6 [Hz] resonant feature was a computer (as suggested e.g. here), so we did so this morning. Sadly, it did not cure the problem. For now, we continue on, running in the configuration defined by Seb last week, see LHO aLOG 16100, in which the RY blend filters at a higher blend configuration, which is the only thing we can find that seems to help (see LHO aLOG 16001). This has now officially become a low-priority until it begins to quantifiably impact IFO performance.

A Recap of all of the things we've tried that DIDN'T make the feature go away:
- Restarting the front-end process LHO aLOG 15759

- Full power-down of Front-end and I/O Chassis (This aLOG)

- Coil Driver Swap LHO aLOG 16071, LHO aLOG 16061, LHO aLOG 15981

- Changing Suspension Damping LHO aLOG 16013, LHO aLOG 16001

- Changing Isolation Loops LHO aLOG 16001

- Location of Sensor Correction; ISI vs HPI LHO aLOG 15941. LHO aLOG 15933

- Changing from STS2 B to A; In analog LHO aLOG 15811, In digital LHO aLOG 15927, LHO aLOG 15894

- Checking between ADC and sensor correction bank LHO aLOG 15783

- Turning OFF CPS satellite amplifiers, checking jumper settings (oscillators, jumpers, etc.) LHO aLOG 15751

- Adding a large mechanical offset to relieve supposed mechanical rubbing LHO aLOG 15751

- Waiting LHO aLOG 15565

Sudarshan, Darkhan

Pcal Lasers at ENDX and ENDY are switched on but blocked at the transmitter module so that the light does not get into the test mass. These lasers will remain on until we report the change of status.

Andres R., & Jeff B.

   We craned the 3IFO Beam-Splitter storage container and the four (4) Quad Lower Storage containers (for Q6, Q7, Q8, & Q9) over the H1 Input arm and placed them into their final long-term storage positions. These containers are ready to be connected to the N2 system.  

Bubba G., Andres R., Mitch R., Jeff B.

   This morning Mitch and Jeff finished loading the Elliptical Baffles into HAM ISI Storage Container #2. Jeff added a 4" witness wafer. After securing the top on the container it was craned over the H2 input arm and placed into its final lone-term storage position. 

   We next craned the 3IFO BSC #4 container into its final long-term storage position. 

   This container was the last of the suspensions long-term storage containers to be filled and stowed. The cleanroom and dust monitor (#10) supporting this effort have been powered down.    

Pulled ETMX photodiode amplifier chassis from EX. Keita reported voltage outputs seemed to be reversed (negative output). Found the wiring for the outputs of the amplifier box wired wrong. Corrected wiring and placed unit back in rack. Patrick had to restart the Beckhoff computer for EX.

D1301017 SN S1400075

Kyle, Gerardo 

Craned pump cart out of Beer Garden with West crane -> Pump cart left running (near BSC7, west side of XBM) 

Alexa, Evan, Dan, Sheila

We have been having intermittent problems for the last two or three days.  This evening we traced the problems we've been having with ALS COMM (alog 16129 ) to an oscillation of the PSL noise eater.  The tell tale symptom was amplitude noise at around 900 kHz on the PSL light.  We don't know of a good indicator of this problem from the control room.

We do not know if this was the cause of our mode cleaner lock losses over the last few days (alog 16128 ),  or to tripping of MC1+MC3 suspensions and HAM2 ISI.  

After I toggled the noise eater switch the ISS first loop was unable to lock.  For now we have turned it off. 

We had the outputs held on the IMC WFS DOF4 from late last night until 8 pm today. We didn't see any trips of MC1+MC3 today.  Now we have turned DOF4 back on.