Evan and I spent most of the day trying to investigate the sudden locklosses we've had over the last 3 days.  

1) We can stay locked for ~20 minutes with ALS and DRMI if we don't turn on the REFL WFS loops.  If we turn these loops on we loose lock within a minute or so.  Even with these loops off we are still not stable though, and saw last night that we can't make it through the lock acquisition sequence. 

2)In almost every lockloss, you can see a glitch in SR3 M2 UR and LL noisemons just before the lockloss, which lines up well in time with glitches in POP18.  Since the UR noisemon has a lot of 60 Hz noise, the glitches can only be seen there in the OUT16 channel, but the UR glitches are much larger.  (We do not actuate on this stage at all).  However, there are two reasons to be skeptical that this is the real problem:

• There is nothing in the witness sensor or OpLev to indicate that the optic is really moving.  These just might not be sensitive enough, but last time we had sus electronics problems with SR3 there was a clear disturbance on the OpLev and witness sensor (21095 and 21081)
• These large glitches have been happening since May10th or so, and we have only started having locking problems in the last 3 days. The RF problem of the last few days seems to corespond better to our locking difficulties, although there aren't signs of problems in the RF readbacks at the exact lockloss times.

It could be that the RF problem that started in the last few days somehow makes us more senstive to loosing lock because of tiny SR3 glitches, or that the noisemons are just showing some spurious signal which is related to the lockloss/ RF problems. Some lockloss plots are attached. 

It seems like the thing to do would be trying to fix the RF problem, but we don't have many ideas for what to do. 

Not clear why. It may correspond to a PSL incursion the same morning.

Also, what used to be hooked up to outputs 7 and 8 on the 9 MHz distribution amplifier in the CER? Sheila and I found some terminated attenuators on these ports, but I recall it used to be cabled up somehow.

Evan H arrived on site this morning to find the TCS chillers tripped, reset them and found the same behavoir as described in alogs 27381 and 27374

On the OAFIOP GDS TP screen, both the DAC and ADC bits (as well as DK) were red, the ADC error cleared with a diag reset but the DAC error would not reset.  We called Dave and he asked us to check that the DAC outputs were all zero by looking at the DAC MON screens (accessed by clicking the blue buttons labeled D0). This means that it is the same DAC problem Nutsinee described.  

To fix the problem, you need to stop and start all the models on the front end, including the IOP model.  This can be done by:

1)maybe check SDF before you kill models (I frogot this step)

2) log in to h1oaf0 as controls 

3) run a script /etc/kill_models.sh, wait for all models to be shut down in the correct order, with the IOP model last

4) run a script /etc/start_models.sh

5) Dave said that for the PEM model only we restore the settings by loading the OBSERVE.snap and hitting LOAD TABLE +EDB.  Since I frogot to check SDF before killing the models I am using the automatic burts to restore the rest of the models. Confusingly, the automatic burts always appear to indicate that there are no diffs, because all channels are no mon in them, so to actually check you need to select full table, select all mon, then set the table back to setting diffs. Time machine doesn't work for the SDF screens, which would be handy in a situation like this.  

Go to the mezzanine and follow instructions here https://lhocds.ligo-wa.caltech.edu/wiki/TCS to restart the chillers, and the laser controllers near the TCS tables.

Kiwamu, Tega, Nutsinee

Quick Conclusion: We are done for the day. The clipping is pretty much fixed but the heating profile remains ununiform. We will resume the work on Tuesday. CO2Y power has been set to zero while CO2X remains at its nominal power. The beam dump has been put back in front of the FLIR camera.

Details: Today we went back to the table and tried to find the clipping point somewhere around M4 and M4A mirrors because the low transmissivity we saw the day before (alog27369). Seeing that the beam profile looks good reflecting off M4 we measured the power again using a power meter with a bigger aperture. We measured more power this time and the transmissivity from point 2 to point 3 wasn't as bad (78% instead of 35% -- see first attachment). We moved on to fix the horizontal clipping on M5 starting by adjusting M4A mirror. After centered the beam on M5 we moved M5, M6, and moved the annulus mask position sensor away from the beam path by about a centimeter. We adjusted BS1 to align the beam through both irises and fine tuned the alignment using the steering mirror between BS1 and the first iris. We adjusted M3 slightly to center the beam spot onto the FLIR camera screen. Looking at the camera image we noticed the beam profile has temperature gradient by 2 deg C from the lowest to highest when the power was 0.15 W at the screen. We are not sure if this is critical but we can improve this on Tuesday. The beam dump has been put back in place before we closed out.

ALso by fixing this clipping we improved the maximum CO2Y power to the ITM from ~2.5 W to ~3.7 W.

FLIR image yesterday

FLIR image today (I don't know why the image rotates. Use your imagination.)

The CP3 refill reminders (emails and cell phone text messages) have been rescheduled from every other day (actually every odd-day of the year) to Mon, Wed, Fri. This means that reminders will no longer be sent during weekends. The change was requested by the vacuum group.

13:55 UTC Chris S. opened high bay outside door for approx. 30 min to remove barrels
Mode cleaner lost lock, NPRO noise eater went out of range
15:22 UTC Jim W. to LVEA to toggle noise eater switch
15:28 UTC Jim W. done
15:55 UTC Betsy to LVEA to look for equipment
16:11 UTC Travis to HAM6 with tape measure
16:22 UTC Jim B. to staging building
16:28 UTC Travis and Betsy done
16:42 UTC Nutsinee and Kiwamu to TCSY CO2 table
17:02 UTC Jeff B. and Jason using forklift by mechanical room
17:32 UTC Jim B. back
17:55 UTC Keita and Haocun to HAM6 to take measurements
18:02 UTC Jeff B. and Jason done
18:09 UTC Kyle to LVEA to look for property tag
18:11 UTC Boom lift delivery through gate
18:55 UTC Kyle back. Kyle going to mid Y.
19:05 UTC Nutsinee and Kiwamu back
19:25 UTC Kyle back from mid Y
20:05 UTC Jeff B. using forklift near mechanical building
20:21 UTC Christina to open OSB receiving door
20:31 UTC Chandra and Gerardo to mid Y to fill CP3 and look at equipment in building
20:51 UTC Kiwamu and Nutsinee back to TCSY CO2 table
21:01 UTC Keita to CER to check ISC racks
21:13 UTC Jeff B. done
21:13 UTC Chandra and Gerardo done

Sheila and Evan H. have been troubleshooting locklosses. Nutsinee and Kiwamu have been working on TCSY CO2.

Lowered LLCV value from 20% to 19% because exhaust pressure was reading 1.2 psi. 

Chandra and Gerardo made arrangements for CP3 fill on Monday, Memorial Day.

2pm local

1/2 turn open LLCV bypass - took 22 sec. to overfill CP3. 

Jim, Dave:

Thursday afternoon and this morning we brought the Staging Building SUS test stand back to life. The front end machine (bscteststand2) and its IO Chassis started with no problems. Most of our work was in getting the workstation running to permit DTT and DATAVIEWER to run. The original SUS workstation, which used to be in the office area, has been repurposed. We located the old SEI workstation in the DTS and pressed that into SUS service. In order to run the workstation outside of the clean area but not in the office, we have temporarily setup in the communications closet. Next week when the area behind the large roll-up door becomes available we will move the workstation there to avoid the closet's cooling issues.

Originally a NAT router (bisbee) in the FEC rack was connected to the GC switch via a wall outlet. We have decommissioned the NAT router, and used the ethernet run to the closet as the way of hooking the workstation up to the iMac and the FEC. We will not be replacing the NAT router, this test stand will no longer be accessible from GC.

Next week Betsy will be able to use the iMac in the clean room to X-foward DTT sessions running on the workstation.

Attached are the trends for the BRSY Drift.  We are close to -10000 counts and doing about 5000 counts per week.  Next weekend power outage may suggest we just wait until after power recovery.  Will check with UW whether we should recenter before or after as well as power recovery procedure.

TCS CO2 Y arm alignment in the morning (Nutsinee)

Vacuum group will replace remaining pneumatic LLCV actuators with electrically driven actuators on Tuesday (Chandra)

Boom lift arriving on site this morning from Sun Valley rentals (Bubba)

To fix a problem with the readouts for the diode chiller flow and conductivity, the PSL Beckhoff computer was
rebooted this morning.

    I also took advantage of the opportunity to reset the clock on the computer from Central European time
to US Pacific.

    At first glance it looks like rebooting the computer fixed the signal(s).

    The laser was brought back to life, by the time I got back to the Control Room all the servos were locked
and engaged.



Jeff, Peter

Ross, Tega, Evan, Terra

Tonight we successfully damped a known parametric instability at 15540.6 Hz with the newly implemented ESD damping scheme. 

In April last year, this mode was detected in the X-arm during a 15W lock. Ultimately it was avoided by turning on the ETMX ring heater (0.5 W requested power top and bottom), shifting the optical mode peak down in frequency and away from ~15540 Hz mechanical modes. To test the new active damping scheme, we turned off the ETMX ring heater, allowed 15540.6 Hz to start to ring up during a 24W lock, and damped it by driving the UR and LL quadrants of the ETMX ESD.  

Below we tracked the amplitude of the ~15540.6 Hz mode. Leftmost action is the important part: first we briefly manually rang it up (gain -1000) before switching the gain sign (gain +500) to rapidly damp. Attached images show power spectrum before damping and immediately after.  We had planned to ring up and down again to get a better idea of the gain settings, but with the newly low magnitude peak, the line tracker got confused with another peak ~1 Hz away and then we lost lock shortly after, for unrelated reasons. 

Briefly, the damping set up: We grab the mechanical mode signal from the OMC transmission DCPDs (H1:OMC-PI_DCPD_64KHZ_A) and send it to the relevant end station, downconverting before the trip and upconverting after using synced oscillators set approximately to the known mechanical mode. There, the mechanical mode peak is tracked with iWave. Output is run through a damping filter for gain control and finally sent to actuate on the UR and LL quadrant of the ETM LNLV ESD. Overall, we get early detection of PI from the OMC and actuation on the test mass with the exact equal but opposite mechanical mode frequency that is ringing up, enabling damping to happen earlier in the lock aquisition process before PI has as much time to ring up. This is necessary as we increase power yet remain working with relatively low actuation force from the ESDs. 

PI at 15520: While working with 15540.6 Hz, we witnessed a mode at 15520 begin to ring up as well. During a second 24W lock, we allowed both to ring up ~15 min; they grew rapidly at similar rates, ultimately producing a strong 20Hz comb and breaking the lock. Will investigate (and attempt to damp) more this weekend.

We didn't get another good lock to test on tonight and we're still working out issues so I've left the damping system in manual mode and have turned the ETMX ring heater back on. 

Following up on yesterday's restart of CW HW injections with a new actuation scheme,
here are comparisons over 24-hour intervals of the excitation channel H1:CAL-PINJX_HARDWARE
with what it was previously when a time-domain inverse actuation filter was used.
One benefit for transient search groups is that if sporadic CW injection dropouts are seen again in O2, 
they should not induce nasty glitches in DARM (see figures 11-13 below).

The bottom line for CW searches is that things look close to what is expected, but the amplitude of the 
highest-frequency pulsar injections (above 1 kHz) are significantly lower than before. 

The small residual discrepancy does not seem to be explained by the difference
between the old and new inverse actuation filter curves that Evan G. posted yesterday. Perhaps both the old and new inverse actuation filters simply amplify
the 1000-2000 Hz band too much (by 20-30%)? 

The figures below show 24-hour second-trend plots of the excitation channel envelope and
4-minute spectrum snapshots taken at 6-hour intervals, along with samples of sudden shutting
off of the injections.

Figure 1 - 24-hour trend (min/mean/max) of the channel for old actuation, showing the envelope of injections,
which is affected by the rotating antenna pattern of the interferometer w.r.t. 15 different
points on the sky with various intrinsic source polarization and strengths.

Figure 2 - 24-hour trend for new actuation - one can see a small drop in amplitude, driven by the highest
frequency pulsars for which the inconsistency between old inverse actuation filter and new actuation function is largest

Figure 3 - 4-minute spectrum at 22:34 UTC on May 24 (old actuation)

Figure 4 - 4-minute spectrum at 22:30 UTC on May 25 (new actuation) - approximately one sidereal day later

Figure 5 - 4-minute spectrum at 04:34 UTC on May 25 (old actuation)

Figure 6 - 4-minute spectrum at 04:30 UTC on May 26 (new actuation) - approximately one sidereal day later

Figure 7 - 4-minute spectrum at 10:34 UTC on May 25 (old actuation)

Figure 8 - 4-minute spectrum at 10:30 UTC on May 26 (new actuation) - approximately one sidereal day later

Figure 9 - 4-minute spectrum at 10:34 UTC on May 25 (old actuation)

Figure 10 - 4-minute spectrum at 10:30 UTC on May 26 (new actuation) - approximately one sidereal day later

Figure 11 - Glitch induced by sudden shutoff of CW injections with old inverse actuation filter

Figure 12 - Vertical zoom of glitch

Figure 13 - No glitch induced by shutoff of CW injections with new direct application of inverse actuation function


Note that the new trend (Figure 2) is a little smoother than the old one, as expected, 
without the amplification of tiny glitches seen with the old inverse filter. Another
manifestation is the much cleaner noise floors seen in the new spectra, away from
the injected lines.