The h1susex models stopped running at 13:45 UTC (06:45 PDT) this morning (Sunday 27 Oct). This is a repeat of the last week's problem, in which h1susex appears to have lost an 18bit DAC (1-7 in that case, which was replaced with a spare).

I've stored all the system and Dolphin logs.

Looking at the PCI bus scan, there is a strange double entry for the 1-7 slot. Perhaps a board fault on this slot since this there is a new 18bit DAC card located there now?

Remember that even after replacing the suspect 18bit DAC, we got an 18bitDAC error on the first time the IOP model was started. I tested the DAC which was removed from h1susex on the DTS and it was not seen on the bus, suggesting it really was damaged.

Now that we're back in low noise, I tried to inject multiple sine waves into DARM using the PCAL to get a sensing function measurement.  

Using the result from alog 52526, I injected five sine waves spaced by factors of 3, starting with 5.62 Hz, to reduce the crest factor to avoid saturating the pcal.
The pcal saturates when the laser is amplitude modulating its full 2 W, basically when the orange line in the ndscopes posted below hits 0 at some crest.  
By injecting at factors of 3 in frequency spacing, and lowering the amplitudes of some of the injected lines to mirror the shape of DARM, I was able to get five going at once, with good coherence (PDFs attached). 


The following should work in any CDS computer ipython terminal 

import sys
sys.path.append('/ligo/home/craig.cahillane/Git/IFO/general/scripts/')
from dataUtils import *
freqs = np.array([   5.62,   16.86,   50.58,  151.74,  455.22])
amps = np.array([30000.,   5000.,   2500.,    573.,  16511.])
phases = np.zeros_like(freqs)
offsets = np.zeros_like(freqs)
exc_channel = 'H1:CAL-PCALY_SWEPT_SINE_EXC'
my_inj = SineMultiple(exc_channel, amps, freqs, phases, offsets)
my_inj.start() # only run if you actually want to inject the sine waves!
my_inj.stop()  # should ramp in 3 seconds

The cameras on IOT2L were not focused, and in the case of the REFL camera, the newly installed sutter prevents it from seeing the REFL sensor, so I moved that camera to look at the HWP/CW/2" steering mirror at the back of the table.

In looking at the WFS cameras, both are looking at WFS B, and there was clearly 2 beams when I focused, but our alignment was not set, so I revisitted this yesterday, and the two beams are still there.  See attached pictures.

The source of the second beam is known, the design of the beam dump and pick off just before the beam splitter to the REFL PD are sensitive to alignment, and when the alignment changes enough, the beams that should be picked off or dumped make it through and polute the rest of the beam path.  Attached is an annotated layout of IOT2L.

Unknown how long WFS has had 2 beams, but since we ended O3a, it's possible that PSL and MC1 changes could move the beam on IOT2L far enough to miss the beam dump and pick off.  Attached are changes in the IMC since O3a, and MC1 in pitch is ~8urad, which is about 10mm at the REFL PD beam splitter, which is enough position change to move the unwanted beams off ot the beam dump and/or pickoff mirror.  In the attachment, the beam changes are on the optics, MC1 and MC3.

Varun, Sheila, Georgia, Craig, Cao, Robert

Alignment:

• We ran initial alignment using the green references and AS_C references we did yesterday, which did bring us into lock with a recycling gain of around 42.
• SRY: We have been consistently having trouble with SRY, when things aren't well aligned it won't lock.  We took a couple of minutes today to measure the OLG of the SRY loop and saw that the gain is verry low compared to the reference I had saved.  It looks like we could use 20dB more gain in this loop.  I added a factor of three more gain for SRY locking, and this did help it lock for an alignment which it wasn't locking at earlier (second attachment red is measurement with SRY gain set to 30000, already increased by a factor of 3.  Blue is an old reference from July 2018).
• We with the new centering on AS_C, we were able to engage SRC asc in both DRMI and full lock without problems, so both of those are now back in the gaurdian.
• We were able to engage ADS without problems once the SRC ASC was working, when the ADS came on and brought the spots to the center of the optics our PRG monitor read 47.
• We have split the ENGAGE_SOFT_LOOPS guardian state, which had 8 steps, into two states.  ENGAGE_SOFT_LOOPS is now engaging the soft loops and ADS to bring the spots to the center of the optic, while MOVE_SPOTS moves the spots to the positions we use for power up, adjusts ASC gains, and engage's Hang's ISI to M0 feedforward.
• Using the newly split ADS states, we reset the green references with the spots centered on the optics (1st attached screenshot). 
• Because of a new peak in DARM that Robert found had some coherence with HAM2, we looked at alignment references for HAM2, and saw that IM3 yaw was moved almost a thousand sliders counts on Monday afternoon.  There's nothing about this in the alog, but it could have been related to SRMI work that afternoon maybe.  There was a large shift in the beam position on IM4 trans and the ISS 2nd loop QPD at that time compared to the positions in O3a (third attachment), so we decided to revert the moves of IM3+IM4. 
• After redoing the green references and moving the IMs, we redid initial alignment and locked with a recycling gain of 47 before any ASC came on.

Noise in DARM:

Once we had the ASC engaged, we had no difficulty wiith moving to the nominal state (other than the violin mode).  However, we say very bad noise in DARM. 

• The fan was on in SQZT6, now turned off.
• Robert found that one of the new peaks in DARM has some coherence with HAM2 table motion. 
• We looked at some alignments in HAM2 because we were worried about clipping, and undid the large move of IM3 that happened last week. 
• After relocking Georgia moved IM3 more, while driving a line on IM3 yaw.  She was able to reduce the amplitude of the line in DARM and broad noise in DARM also was reduced, she will make a comment.

Other things today:

• ETMY mode6 (510.7264Hz) was very rung up when we first locked, and the guardian does not damp it.  We engaged damping with +60degrees and -0.1 gain, which seems to be working well and not yet ringing up ETMY MODE1, which is at 510.715Hz.
• We wanted to be able to close the POP beam diverter, so Craig moved the IFO trigger back to in vac POP.  (It had been changed to in air pop in 52150) 
• Craig trended the thresholds which used to be used for in vac POP, and we thought that the old threshold for POP in vac was a little close to the value of the PD, the trigger was set to 800 but with today's pretty good alignment the value sometimes drops to 860.  To avoid the possibility of this causing a lockloss in the future we lowered this to 700.  Craig also found several places in the guardian where this was hardcoded, so he moved them to lscparams and removed a redundant setting of the thresholds in PREP_DC_READOUT.
• We lost lock once from a glitch moving ADS, which was a cause of about 40 locklosses in O3a.  Since there are no integrators in the ADS servos, we decided to try limiting their outputs once we arrive in nominal low noise to prevent the large excursions from being sent to the ASC.

There are frequently times when the build ups and ASC signals in DRMI become verry noisy about 1 minute after it locks.  The guardian does several things at the same time here, engages the DRMI ASC and waits for it to converge before offloading, requests BS stage 2 isolation, and changes DRMI from 1F to 3F locking.  Today we went through these steps slowly to try to understand which of these steps causes the big kick, and it seems that the BS ST2 isolation loops are the problem.  The attached screenshot shows that the problem happened durring the BS ST2 transition (there was not anything else going on at the time) although nothing seems to show up in the GS13 signals. 

We are often able to ride this out, but we sometimes loose lock due to this. Looking at the lockloss tool summary plot that Niko posted in the second attachment to 52514, this is probably responsible for the cluster of locklosses between the states 105 (TURN_ON_BS_ST2) and 111 (offload DRMI ASC) which is about 100 locklosses in O3a.

Edit:  In another locking attempt we waited again for the BS isolation loops, saw the big glitch but survived, then lostlock during the ASC offload, so there might be multiple problems in these states.  For now I've set ISC_LOCK to wait for the BS to finish isolating ST2 before it moves on to the other DRMI things, so that it will be easier to tell why we are loosing lock.

Also, watching the isolation loops, the glitch seems to happen at the time when FM1 is switched off and FM8 is switched on for the horizontal loops. (second attached screenshot.)

Lee, Kentaro, Sheila, Georgia

I've been tracking an regular glich that appears in the EOMRMSMON of the squeezer TTFSS. This glitch also corresponds to real phase noise seen in the beatnote between the seed and LO light on the homodyne. When the squeezer LO-loop is ON the glitch disappears.

SQZ_TTFSS_EOMRMS_Spikes.png shows the EOMRMSMMON channel while the LO loop is OFF. The glitch occurs near a voltage crossing of the PSL VCO. When the LO loop is enabled, the EOMRMS stays low, but you can see in SQZ_VCO_tunephase.png. That the SQZ VCO is tracking the PSL vco due to the LO loop. This was supposed to be prevented by the reconfiguration of the ALS fiber pickoff in 52381. Georgia helped us track the signals driving the ALS distribution AOM and it appears to be sourced by the PSL VCO OUT2 (split to also enter RF Patch Panel #11 on ISC-R1) rather than being sourced from a static REF 79.4MHz from the distribution source.

Currently, I think the squeezer is effectively still locked to the refcavity frequency, so we cannot easily measure seeding or backscatter, and the CARM-FF channel is needed until we reconfigure the ALS-distribution-AOM source.

I still do not understand the glitching though. It is appearing in the EOM monitor and doesn't correspond to any channels than the VCO frequency (not the PZT offset, slow offset or SQZ frequency counters). This suggests that it is a drifting whistle-like beatnote showing in the FSS PFD moving in and out of the control band and being driven into real phase noise by the servo (at a high enough frequency to be visible in the RMS monitor). While it is fortunate that it lessens when the LO loop is active, it is strange that it goes away only when the SQZ VCO moves with the PSL VCO. This suggests some underlying cause that could be injecting some phase noise through the TTFSS servo at other times, even if it is not so obvious as these glitches. We'll keep tracking it.

Sheila Varun

After acquiring full lock in nominal noise we observe excess noise in frequencies band from 10 Hz to 600 Hz. Reading out the accelerometer at the different chambers during the full lock from this morning (red) and comparing it against Sep 21 2019 19:03:38 UTC (dashed blue), we observe almost a factor of 10 more noise as compared to late O3a. Please refer to the plots attached.

The HAM6 shutter doesn't have high voltage.  (The medm screen says that the high votlage is OK, but the LED on the driver says the capactior is only charged to 13V.) With Varun as a phone buddy I went to the mezzanine, the HAM6 high voltage supply is off.  Attached is a trend of PT-110 over night, there is a spike to 4e-5 Torr around midnight .  It is reading 1.1e-6 Torr now.

Sheila Varun Cao TJ Dan B Jim Georgia Patrick Jenne

This morning we were able to get to resonance by changing some gains and offsets in the CARM offset reduction.

• TJ and I moved the parameters that I changed into a dictonary in lscparams called carm offset, to make it easier to undo this and to redo it if we need to again in the future. 
• We were sometimes loosing lock before making the transition to RF darm, so I moved the TR_CARM offset for the RF_DARM transition (from -5 to -3).  I tried to revert this once the recyclin gain was improverd, but we kept loosing lock, so it is at -5 for now. 
• We were engaging a DARM boost (FM4) part way through the CARM offset reduction, previously at -20 TR_CARM, I moved it to -12 because all the build ups became much more stable once that was engaged.  Now that the recycling gain is better, this boost still seems to help stabilize build ups lower on the frigne than -20, so I have left this at -12.
• Moved the TR_CARM gain increase and boost that normally happens at -40 to -32, based on measurements of the TR_CARM OLG. (This is so far still at -32)
• We were then able to move to TR_CARM of -38, manually skip the gaurdian state that calculates the TR_CARM offset to use for the TR_REFLAIR9 transition, and do the transition to REFL9. 
• We were able to go to resonance, but with a recyclcing gain of ~23 and very noisy build ups. We lost lock the first time that we locked this way attempiing the ANALOG_CARM transition.  
• Watching the ASC error signals, we were able to manually move the arms (CHARD P mostly) to increase the recycling gain. 
• We have had several iterations of this, closing more ASC loops each time combined with manual alignments.  
• The guardian is working until ENGAGE_ASC_FOR_FULL_IFO, if we manuly skip CARM_5_PM.  SRC1+SRC2 loops are commented out of ENGAGE_ASC for now.
• After our struggles with the ground moving (below) we were able to acquire lock with a recycling gain of 41, before turning on any ASC here Georgia servo'd the green QPD offsets to zero the green WFS signals, and we averaged the camera offsets to reset the inital alignment references. (1st attachment shows differences in references).
• We lost lock after this while trying to manually move SR2 +SRM to keep up with the ADS loops. 
• We decided to pico on AS_C to bring the beam back onto the diode so that we can try to close some loops. We got an EQ +wind after this and decided to leave closing these loops for another day.

Seismic/ISis:

• We have tripped the BS ISI 3 times, Jim thinks that the first two were the result of an incorrect setting in the GS13s, the third trip was due to walking near the sensor correction seismometer, and tripped both ITMs as well. 
• We had been runing 90mHz blends at EX, but we've had difficulty locking for the last 2 hours since the wind has been gusting to 40mph.  We switched the ISI config to WINDY no BRSX, although didn't make much difference to the ETMX pit oplev, which is about 0.4urad peak to peak
• After a few hours of struggle with wind, Patrick and Jim fixed the BRSX code with some remote advice from Michael, and added it back to the sensor correction, which helped quiet down ETMX optical lever. 
• Jim also turned on the differential CPS feedforward between HAM2/3 in the X direction, and between ITMX and BS in the X direction, and ITMY and BS in the Y direction.  We can turn this on or off using the master switching in the lower right hand corner of the output limiters screen accessible from the yellow ISI DIFF button on the ISI_CONFIG screen.  Jenne and Chiara are working on a guardian for this.  Things seemed much better after this is done, but this also coincided with the wind dying down a bit.
• The wind has now picked up, forecast to go until 11pm and there is now also an EQ, so I am done for the day.

Oher locking:

• Once, we had DRMI lock just a few seconds before the DRMI timer went off, but not long enough before for the DRMI guardian to have "arrived" in the requested state.  ISC_LOCK requested PRMI_ASC and unlocked DRMI.  We edited this so that it will not check the timer unless DRMI is not locked.
• We also saw that after the PRMI ASC runs, the OMs get railed sometimes (operators have been pointing this out).  We added a few lines in the ISC_DRMI gaurdian that will turn off the AS centering loops before attempting the transition to DRMI. 

Jenne, Chiara.

Today was very windy, so the locking team was not able to lock the interferometer, but it became much easier once the CPS DIFF was on.  In this case the HAM2 and HAM3 were locked together and also BS, ITMX and ITMY were locked together.

The plots in attachment show measurements in on (pink) and off (green) conditions of CPS DIFF; for HAM chambers plots, off traces are in green and red, on traces are in pink and blue. We got some data (MICH, PRCL,  SRCL, DARM) when the system was on and the interferometer was at full resonance, but  we can't get a reference of the system in off mode since we can't lock without it.

These results are similar to previous ones in earlier posts, we are showing that this can be helpful to reach the lock.

When looking at ISCT1 last week, I used a small flashlight to read some labels, and found particulate that might be a concern on the 2" lens, POP-L1.  Image attached.

Jenne Driggers, Adrian Helmling-Cornell

Today we updated the photodiode used for the frontend lockloss trigger from POP_A_DC to POPAIR_A_DC, as it responds slightly faster to the interferometer losing lock. The lockloss trigger is armed when Guardian enters the OFFLOAD_DRMI_ASC state. As Guardian goes through the CARM offset reduction sequence, if POPAIR_A normalized by the laser power falls below 2.5 the lockloss trigger will now trigger, stopping actuation. When Guardian reaches and passes the DRMI_TO_POP state, the trigger triggers if POPAIR_A normalized by the laser power falls below 108.

The appropriate SDF diffs were accepted and this change was implemented around 20:40 UTC today.