I have added the following three guardian DIAG nodes:

• DIAG_CRIT: "critical" diagnostics required for OBSERVATION (currently empty)
• DIAG_SDF: check that all SDF_DIFF_CNT for all active front end models are identically zero (0)
• DIAG_EXC: check that all front end models (except calcs) are reporting no active excitations (hardware injections go in through calcs).

DAQ restart required to acquire new channel names.

These nodes will all be folded under the IFO top node, but have not yet.  Waiting for go ahead from ops...

I'm also in the process of updating the GUARD_OVERVIEW screen to add these new nodes.  I will post when that screen has been updated.

At least twice it was just stage 2.  Is something going wrong with the ISI?  or is this a side effect of the SR3 problems, which could cause the MICH ASC to get a kick?

Over the weekend while it was windy Evan noticed that we were close to saturating PRM M3, which is because the offloading I did 19850 didn't do as good a job as the M2 offloading, even though it solved the M2 range issue.  

Tonight I had a second look.  I moved the roll off up in the sus comp filter, moved the complex zeros down a bit to gain some phase, and removed the 3 Hz pole (this design was originally intended for a crossover above the sus resonances and didn't include the low frequency pole we are using.) The filter comparison is the second attached screenshot.  The first one shows the current crossover measurement, the third shows the drives.  The prominent peak at 0.6 Hz could be from an ISI.  The last screenshot shows the current configuration of the filter. 

None of this is in guardian, so the next time things are locked it should revert to the old offloading. 

Tonight was our chance to work with PRMI.  

We had a look at the phasing of AS36, following the same procedure as we used with MICH bright the other day 20961.  (locked PRMI, aligned by hand, used OM1 +2 to align all the light onto one quadrant, maximize the Q signal. )

For AS A the results were satisfying,

• the phases needed to maximize I are similar for all 4 quadrants,
• the signal levels were all similar,
• once I re-engaged the centering the signals were still similar on all 4 quadrants. 
• Exicitng BS pitch (12 Hz 1000 cnts in Drivealign) and measureing a tranfer function gave a signal that has the correct sign for pitch in all 4 quadrants, exciting yaw gave the right sign for yaw in all 4 quadrants.  

For AS B the situation was also reasonable, but everything was a little less good (the phases are similar for all quadrants, but not quite the same, the signal levels varied more, and when exciting the BS the phases were almost right for PIT and YAW.   Overall this seems reasonable to use.  The resulting phases are in the attached screenshot, the new phases are the epics values, the old phases are the setpoints.  AS B quadrant 3 did not change, it is 63 degrees.  

I will revert these momentarily, but the next steps would be to set the matrix for AS A yaw back to something reasonable (the current set up doesn't use segments 1 and 2), set the phases to the values in the screenshots, lock DRMI, excite SRM and move the phase of all 4 quadrants on each WFS in common to minimize the SRM signal in the Q phase.  Since DRMI locking is frustrating tonight with the SR3 glitches this will have to wait.  

[Sheila, Anamaria, Jenne, Cheryl, Dan, Evan.  Richard via phone]

SR3 glitches are still causing us grief.  This is a continuation of the story started last night (alog 21046), and worked on earlier in the day (alog 21062).  After some heroic efforts, we have determined that we cannot lock in this state. 

To prove to ourselves that indeed it was a problem with the analog actuation chain we investigated turning different pieces of the analog electronics off.  The first attached Dataviewer screenshot shows the NOISEMON channels of the SR3 M1 stage throughout this investigation.  When the local damping is on, we cannot see the glitches very clearly in the noisemon channels - but we do see them in the voltmon channels (The second attached screenshot shows that the T2 voltmon channel does in fact see the glitches, so it's not a broken monitor).  When the local damping is off, we only see the glitches in the noisemon channels. 

• A:  04:57 UTC we turned off the local damping (M1 OSEMs and "cage servo"), but left the DC alignment.  After this we see significantly more noise in the M1 T2 NOISEMON than the other M1 channels. 
• B:  05:41 UTC we take SR3 to safe mode (i.e., we remove also the DC alignment bias on the M1 actuators).  Glitches seem to be gone?
• C: 05:45 UTC we power cycle the AI chassis for SR3 (which is shared with the OMC, so the OMC was first brought to safe) and the top coil driver chassis that drives T1, T2, T3 and __. After this we restore SR3 to aligned, but no local damping.  Noise isn't as good as other channels, but looks better than earlier.
• D: 06:07 UTC we turn off the AI chassis for SR3 and OMC.  Glitches and noise seem to be gone.
• Around 06:20 UTC we swap out the AI chassis (removed S1104370) with a spare borrowed from the H2 storage racks (at Richard's recommendation).  We put in S1108081. 
• 06:24 we are back from swapping, and the new AI chassis is turned on.  Glitches seem to still be gone.
• E:  06:26 UTC we restore SR3 alignment and local damping.

Since we do not see the glitches when the AI chassis is powered off, we infer that the noise is not generated in the coil driver board.  (Note that we also borrowed a triple top coil driver chassis from the H2 storage racks, S1100192, but did not swap it in since we don't think the noise is coming from there).  We have not, however, determined whether the noise is coming from the AI board (probably not, since it's still there after a swap?) or the DAC. 

We tried a few times to lock the IFO after the AI board swap, but we were continually losing lock before the CARM reduction is complete.  Lockloss investigations showed that the problem for most of these was SR3 motion. 

At this point, we have determined that we need more expert brains to have a look at the analog electronics.  The owl operator shift has been cancelled, since there will be no more full IFO locking happening until this problem is resolved. 

At LLO there are problems with engaging the PRC2 ASC loop when the recycling gain is low, but we don't seem to have this problem at LHO.  One theorey about the difference could be that we just arrive in lock with a higher recycling gain.  Although I believed this myself, the data I downloaded from the first 13 days of ER8 seem to indicate this is not the difference.  I looked at times when we first arrived on resonance (112 examples), when we transition to DC readout (after the ASC including soft loops has been on for about 1 minute, 85 examples), and after we power up to the maximum available power (60 examples).

• We have a 5% discrepancy between the recycling gains measured in transmission of the 2 arms, so that gives us an idea of the accuracy of this calibration into recycling gain.
• We seem to have a bimodal distrution of recycling gains when we first lock, centered around 34 (35 according to the Y arm) and 36 (38 according to the Y arm). 
• The ASC brings us to a more consistent high recycling gain.
• According to the X arm, we keep the high recycling gain when we power up, while the Y arm indicates that the recycling gain drops a bit as we power up.  This is interesting, it might be nice make the same plot for POP_DC. 
• I was curious if the locks that start with a lower recying galn survive to low noise, so I looked at the same plot made for only locks which reached the nominal low noise state (44 examples). It looks the same, meaning that we make it to low noise even when we arrive on resonance with a low recycling gain.

SR3 is still glitching - IFO did relock a few times, enough to verify that the porblem is not fixed.

OWL shift called off - Jim was OWL OPS, but staying home.

Sheila, Jenne, Anamaria - staying to make alogs.

shift started with PRM satellite box swaps (8/31 23:00 - 9/1 00:10UTC, one hour)

relocking issues: 9/1 00:10UTC to 5:14UTC

- alignment that I corrected without a full realignment (9/1 ended 1:00UTC)

- delays by BS ISI being tripped - happened twice (ended 2:14UTC)

- delayed by SR3 glitching, preventing IFO from going to CARM_5PM (2:14UTC to 5:14UTC)

Having called Vern, Richard, Dave: 5:14UTC

Dave restarted h1nds1, so it's back

- option later tonight is to power cycle the computer and then log in and restart the process

Richard

- options are to power cycle AI chassis and or coil drivers for SR3

- and then change the coil driver if power cycle doesn't work

Vern

- ok to call off PWL shift if no SR3 fix and therefore no locking 

- have emailed Jim who's on OWLs, to let him know

h1nds1 daqd process stopped running at 21:47 this evening

• [Mon Aug 31 21:47:22 2015] Ask for retransmission of 19 packets; port 7097
• [Mon Aug 31 21:47:22 2015] Ask for retransmission of 19 packets; port 7097
• [Mon Aug 31 21:47:22 2015] Ask for retransmission of 19 packets; port 7097
• [Mon Aug 31 21:47:22 2015] Ask for retransmission of 19 packets; port 7097
• [Mon Aug 31 21:47:23 2015] Ask for retransmission of 19 packets; port 7097
• [Mon Aug 31 21:47:23 2015] Ask for retransmission of 19 packets; port 7097
• [Mon Aug 31 21:47:23 2015] Ask for retransmission of 19 packets; port 7097
• [Mon Aug 31 21:47:23 2015] Ask for retransmission of 19 packets; port 7097
• [Mon Aug 31 21:47:24 2015] Ask for retransmission of 19 packets; port 7097
• Have to skip 19 packets (packet buffer limit exceeded)

computer did not lock up. I logged in and started the process but it quickly stopped again.

• [Mon Aug 31 22:15:14 2015] Ask for retransmission of 15 packets; port 7098
• [Mon Aug 31 22:15:14 2015] Ask for retransmission of 15 packets; port 7098
• [Mon Aug 31 22:15:14 2015] Ask for retransmission of 15 packets; port 7098
• [Mon Aug 31 22:15:14 2015] Ask for retransmission of 15 packets; port 7098
• [Mon Aug 31 22:15:15 2015] Ask for retransmission of 15 packets; port 7098
• [Mon Aug 31 22:15:15 2015] Ask for retransmission of 15 packets; port 7098
• [Mon Aug 31 22:15:15 2015] Ask for retransmission of 15 packets; port 7098
• [Mon Aug 31 22:15:15 2015] Ask for retransmission of 15 packets; port 7098
• [Mon Aug 31 22:15:16 2015] Ask for retransmission of 15 packets; port 7098
• Have to skip 15 packets (retry limit exceeded)

I restarted a second time and now it is running. Investigation is continuing.

Monit is not restarting the process, as root I started with /etc/init.d/daqd_nds1 start

• SR3 glitching enough to cause multiple lock losses
• IFO can't make it past CARM_5PM
• h1nds1 is down, making it hard to diagnose issues with SR3
• have called Vern, Richard, Dave

My detailed DQ shift report for August 27th - 30th can be found here:

https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20150827

Highlights:

• Overall very low glitch rate (almost as low as it can get)
• Range in the 65-70 Mpc range
• Overall very constant performance
• Only one CBC trigger with newSNR>8.

Main oustanding issues:

• PEM-EY_MAG_EBAY_SUSRACK_X_DQ seems to be associated with a good number of glitches, often with relatively high SNR
• Something may be wrong with PRM damping loops, as pointed out by Kiwamu here.
• on Aug 28, Kleinewelle reported (among others) 4 loud glitches all very well aligned at about 100 Hz. Origin is unknown, but may be common.
• on Aug 29th, a lock loss likely happened because of high winds, but the immediate cause has not (yet) been identified
• on Aug 30th, calibration lines appeared to be unstable throughout the day (including science segments)

In addition, many glitches picked up by Omicron or similar tools could use some further investigation, that I did not have the time to perform. I plan on following up on a few of them in the coming days.

The "noisemon pringle" actuator noise measurement from LLO alog 19853 has been calibrated to meters, and extended to the L1 and L2 stages of all test masses.

Total noise from these stages is estimated to be ~1e-20 m/rtHz around 30 Hz.  This includes driver noise, DAC noise, and glitches (to the extent they were present during the measurement).

A hierarchical noise budget PDF is attached. Here's how things look for ITMX L2:

From this plot you can see the following:

• Length control signal projections from the DAC drive and from the noisemon readback match well. (This is a calibration consistency check.)
• The pringle noise seems to closely approach DARM below 20 Hz, but this is due to contamination from ASC pitch signals (which has been present in DARM also).  Noisemon pitch and yaw are projected (using the length TF) to show the subtraction. The bottom of the roll bandstop suggests the true noise floor is consistent with the slope of the curve above 30 Hz.
• The DAC noise model from G1401399 is plotted for comparison. The model matches at high frequency but apparently underpredicts the noise below 200 Hz (though some of this band is subtraction-limited).

The calibration of these signals is approximate, based on the coil driver and suspension models -- not the precise measurements people have been taking during the past week. There may be some discrepancies especially in the L1 (UIM) stage.  The ETMY L1 plot definitely shouldn't be trusted, because its noisemons appear to be broken (see attached plot).

Scripts have been checked in to the NoiseBudget SVN.

(All time in UTC)

15:00 Jim has been having trouble locking DRMI all night. So I work on PRMI.

15:32 Managed to get pass DRMI_LOCKED, lockloss at REDUCE_CARM_OFFSET. Saturation alarm complained about PRM.

16:17 Hugh to end stations

16:29 Robert to CER and LVEA

16:40 SR3 is totally unresponsive to Gurdian. See alog 21057.

          DAC recalibratio and DAC restart.

16:53 Hugh back

17:05 Robert back

17:45 Begin initial alignment after several locklosses at random places.

          burtrestored the alignment to last night lock

18:20 Karen/crew to change room and optics lab for cleaning.

19:36 Since there's an operator training going on, Evan insisted I switch the Operating Mode to TRAIN.

21:39 Jeff/Evan to restart PR3, PRM satellite amplifier.

22:00 Jeff back

22:26 Gerado to LVEA hunting for parts.

22:28 Jeff K. brings IMC_LOCK Guardian to OFFLINE

22:39 Gerado back

This afternoon, Richard and Fil tried various hardware diagnostics of the cabling and boxes out on the floor for the shared PRM/PR3 chain noise.  While swapping sat boxes and reterminating cables a few times, the PRM RT TOP BOSEM flopped between flatlining, or being super noisy multiple times after they made connections.  There was additional confusion in the fact that the PRM misaligned state moves the PRM a huge amount in YAW, thus sending the RT BOSEM signal to near zero - we thought this meant it "died" a few times during our troubleshooting.  Long story short, the troubleshooting of the electronics this afternoon didn't seem to improve the noise much.

To be continued...

J. Kissel, Nutsinee

Dan found glitches in SR3 T2  last night and it seems like we have been having trouble locking ever since so we investgated. SR3 T2 problem started right before Aug 31 09:23 lockloss and never go away. DAC recalibration this morning didn't solve the issue and the power glitch didn't cause it. The noisy/glitchy feature looks just like what happened before the drive swap back in Aug 18.