Sheila, Nutsinee

After we got passed all the problems Sheila mentioned earlier, we are still trying to acquire lock. Ifo lost lock at SWITCH_TO_QPDS twice in a row.

Nutsinee started an inital alingment as part of recovering from the last lockloss.  The INPUT align WFS loops failed for an unknown reason, and caused the X arm to drop lock with out offloading the WFS.  At this point we noticed that the history in the suspension filters was not cleared in the down state of the ALIGN_IFO guardian, which caused the X arm to stay misalinged in this case.  I've now added clear history for the filters.  

All the environment monitors didn't show anything that could have caused the lockloss. Sheila's investigating.

Summary:

We had single-IFO time so I tested the new inverse actuation filter for PCALX. WP5530

Sudarshan and I believe we tracked down the factor of 2 and sign error from the initial PCALX test, see aLog 22160. We wanted to do this test to confirm that.

CBC injections:

The waveform file is: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/Inspiral/H1/coherenttest1from15hz_1126257408.out

The XML parameter file is: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/Inspiral/h1l1coherenttest1from15hz_1126257408.xml.gz

I did three CBC injections. The start times of the injections were: 1128303091.000000000, 1128303224.000000000, and 1128303391.000000000.

The command line to do the injections is:
ezcawrite H1:CAL-INJ_TINJ_TYPE 1
awgstream H1:CAL-PCALX_SWEPT_SINE_EXC 16384 coherenttest1from15hz_1126257408.out 1.0 -d -d >> 20151006_log_pcal.out
awgstream H1:CAL-PCALX_SWEPT_SINE_EXC 16384 coherenttest1from15hz_1126257408.out 1.0 -d -d >> 20151006_log_pcal.out
awgstream H1:CAL-PCALX_SWEPT_SINE_EXC 16384 coherenttest1from15hz_1126257408.out 1.0 -d -d >> 20151006_log_pcal.out

I have attached the log. I had to change the file extension to be posted to the aLog.

DetChar injection:

I injected Jordan's waveform file: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/detchar/detchar_03Oct2015_PCAL.txt

The start time of the injection is: 1128303531.000000000

The command line to do the injections is:
awgstream H1:CAL-PCALX_SWEPT_SINE_EXC 16384 detchar_03Oct2015_PCAL.txt 1.0 -d -d >> 20151006_log_pcal_detchar.out

I have attached the log. I had to change the file extension to be posted to the aLog.

Starting PCALX hardware injection tests. WP 5530. Intent mode is off. More details later.

TITLE: 10/7 [EVE Shift]: 23:00-07:00UTC (16:00-00:00 PDT), all times posted in UTC"

STATE Of H1: Observing at 72 Mpc. Been locked since yesterday.

OUTGOING OPERATOR: Ed

QUICK SUMMARY: Few people in the control room. Almost no wind. Nominal seismic activity in the earthquake band. Local seismic activities dropping to nominal.

TITLE:   Oct 7 DAY Shift 15:00-23:00UTC (08:00-04:00 PDT), all times posted in UTC

STATE Of H1: Observing

LOCK DURATION: Entire Shift

SUPPORT: None required

INCOMING OPERATOR: Nutsinee

Activity log:

16:42 Changed the time period on Video1 FOM to 24 hours

17:46 Jim noticed that the .6Hz peak in the PRCL Live spectra (NUC4) has gone away.

17:49 Jody and John traveling to the X/Y mid stations.

18:30  Jody and John are back from the Mid stations

19:24 Peter into optics lab to find a neutral density filter. (Landry’s approval)

19:38 Peter out of the optics lab.

20:45  Joe D checked the emergency lighting in the Control Room

Shift Summary: Quiet day: IFO locked at 75Mpc (avg) for the entire shift. No excessive wind speeds. No obtrusive EQ activity. Microseism was steady all day.

SUS E_T_M_Y saturating (Oct 7 16:9:29 UTC)
SUS E_T_M_Y saturating (Oct 7 17:33:6 UTC)
SUS E_T_M_Y saturating (Oct 7 19:8:7 UTC)
SUS E_T_M_Y saturating (Oct 7 19:54:24 UTC)
SUS E_T_M_Y saturating (Oct 7 20:57:5 UTC)

The frame wiper script which runs on h1fw0 was changed to alow the disk to become 95% full to match the configuration of h1fw1.  The previous setting allowed only 90% disk usage.  

The setting of 90% was most likely left over from a test configuration when the Ubuntu frame writer was first created.

Here are the LHO DAQ lookback times for framed data served by the internal NDS protocol-1 servers (data taken at 13:25 local time).

h1nds0 (disks in LDAS server room). Disk usage 91% (33TB used). 

h1nds1(default-nds) (disks in MSR). Disk usage 96% (35TB used).

Note underuse of disks on h1nds0, being changed to use 95% of available disk space.

Under WP 5527, yesterday I switched the sensors on two of the CS HEPI Pump Skids.

On a Pump Station Skid, see first attachment for a medm depiction, there are pressure sensor ports. 1--right after pump, includes first capacitor accumulator; 2 & 3--after laminar flow resistors & includes accumulator; 4--after final 3u filter. In previous logs, see first plot of 21366, I show that at the corner station, pressure 1 right after the pump is super quiet, again only at the corner.  I checked with Barker and we looked at things and there is no reason the trends would be different for these channels.

Here the attached plot shows just the four impacted pressure sensors.  The first two plots I've zoomed into the quiet portions before and after the switch showing that the "quiet" is tied to the sensor not the channel; the shift in the reading is due to the AOFF (zero) of the individual channel.  On the next two graphs (PS4_PRESS1 & 3) I've zoomed into the noisier after and before the sensor switch.  Bottom line, the noise level is at the sensor, not the channel.  The next two graphs (channels 7 & 8) are the End stations' Pressure1.  Clearly EndX is out to lunch with super ugly noise from something we can't figure out.  This shows up on the EX control out, CH10, and the servo'd PV (not shown.)  The Ch8 graph shows the EY PRESS1 is much noisier than the corner press1s.  The last 3 graphs are the control outs for the corner, L0, and EX and EY.  Interestingly, despite the much quieter pressure on the CS vs EndY PRESS1s, the CONTROL_VOUTs of the CS and EY are very similar as is their servo'd PVs (the EndY shows the diurnal much more than the others.)  Even though the pressure out of the pump is quiet or noisy (CS or EndY) the RC Manifold of the Pump Station makes the signal noisier for the CS.  The noise level on the EndY Manifold pressures are about the same or only slightly noisier down the manifold (I looked.)

Still wondering about what is going on...

1) The Accumulator charge on the Press1 volumes are similar although the fluid pressures are different at the three locations.  Why?  The zero AOFFs are coarse and may be variable over time.  The resistances to get to PV setpoint are likely different even at Ends and especially at corner.  Why? Different crap in the resistor stacks, different length pipe runs to BSC, more Actuators at corner, different actuators everywhere, etc.

Solution?  Be more precise at setting accumulator pressures based on operating pressures?  Given the spec'd nominal accumulator charge of 60 to 93% (I shoot for 90%) of operating fluid pressure and the very unlikely(hood) that I would be so good(lucky) at the CS versus the Ends, I'd say this is not the issue.

2) The Corner station has 24 HEPI Actuators per Pump Station while the Ends have just eight.  This leads the CS pumps to run about twice as fast as the Ends, see VOUT channels.

I could manually slow down the CS pumps to End speeds and see what that does.  HEPI performance likely compromised if not completely hosed but might be informative.  Could we introduce additional resistances at the Ends to require the Pump to speed up?

3) Could the laminar flow Resistors be contributing to the down stream noise increase?  Does it make sense that the pressure out of the pump is the quietest?  This certainly would not explain why the Ends are noisier but might explain the down stream noise.

4) Could the End Station Pumps be cavitating?  Yes.  I've had a pump cavitate in July, 19699 & 19794, after maintenance in the CS and had to restart it.  I did so over 5 minutes and the cavitation stopped.  I'd have to mine to find the last time the Ends were restarted and figure out how they were ramped back on.  I try to do so slowly manually or with the servo but it's not typically 5 minutes.  I don't suspect this as the pump made an audible rumbling when the CS was cavitating and I don't hear this at the Ends.  But, maybe there are varying levels of cavitation and there could be benefit to restarting the End Pumps very slowly.

It really does not matter anyway.  Given what the servo pressure and control drive look like comparing the CS and EndY, those results are the same.  More important and maybe still not that important is to figure out why the EndX pressures are so noisy.

I'll restore the pressure sensor cabling when able and close out this WP.

The LHO SEI team has known about a .6-ish hz peak on the HAM3 ISI for a long time (see my alog 15565, December of last year for the start, Hugh has a summary of LHO alogs in the SEI log for more). I was working with Ed on a DTT template for the operators when I noticed it was now gone. Very strange. Looking a little closer, it seems to have been decreasing over the last couple of days to a week, and disappeared completely this morning about 7-8:00 UTC. Attached spectra are from ~0:00 UTC (red) and ~16:00 UTC (blue, when I found it was missing). Looking at random times over the last week, it looks like it may have been trending down.

Could someone in Detchar look at this peaks longish term BLRMS, say over the last month, or even over the last year since we found it? Pretty much every sensor in that chamber saw this, but the GS-13s are the best witness.

Today we noticed that the EX beam diverter has been open since June 18th.  We don't normally switch this so it is not guardian controlled but we "usually" leave it closed.  It is beckhoff controlled so it is not monitored in SDF.  Today TJ added all the beam diverters we don't normally change in guardian to SYS_DIAG to prevent this from happening again (it was already checking the others).

We have now locked with the beam diverter closed, and see that the QPD spectra look the same between EX and EY.   (20418 and 21767 have some history.)  The peak in DARM at 78 Hz is also not here.  We will see if it stays gone, and check soon to see if the blinear coupling of ETMX ISI motion to DARM is gone with the beam diverter closed.  

Motivated by our locking difficulties at the end of last week, I downloaded guardian state data for the last two weeks to look into our locklosses durring the CARM offset reduction.

To start with the good news, the script that I use to do this automatically makes a few figures that tell us about our duty cycle, and we are doing well overall. As usual, I am only looking at the guardian state, and not considering any intent bits.  My reaasoning is that the hardest part is getting and keeping the IFO locked, and that is currently the important limit to our duty cycle. 

• We have been locked in the nominal low noise state for 83% of the time (compared to 68% for the last week of ER8).  A good part of the improvement is probabably because the ground has been quieter in the last 2 weeks, although I didn't try to quantify that. (guardian state pie chart is the first attachment).
• About half of our DRMI locks result in nominal low noise locks (33/64).  A lockloss pie chart is in the second attachment. We have fewer locklosses in the late stages of the guardian, which could be because we are waiting for the first sets of ASC loops to engage before we engage the soft loops (alogs 21583 21587)
• The median low noise lock stretch was 5.5 hours long, while the longest was 59.7 hrs long.  A histogram of the length of low noise locks is in the third attachment.

The main weakness in our lock acquistion sequence now seems to be the early steps of CARM offset reduction, as we saw yesterday. 

•  We certaintly have a harder time with these early steps of CARM offset reduction when the ground motion is high, which explains most of our down time from the 2nd to the 3rd.
  • The 4th attachement show a histogram of locking attempts bined by ground motion, which is the maximum of the sum of the Y direction blrms for 30mHz-100mHz and 100-300mHz bands durring the locking attempt. I chose the Y direction at Y end because that tends to have more wind induced tilt, so this should include the impact of wind, earthquakes and useism.  You can see that we struggle to lock when the ground motion in these bands is above 250nm/sec, although occasionally we do suceed.  In the version of the plot attached here, I called a locking attempt sucsesfull if it reached the low noise state, but the plot doesn't change much if you instead make it for locks which make it past the stage where the DHARD ASC loops come on. 
  • The 5th attachment shows the ground motion over the last week, with a plot of the range to illustrate where the long stretch of downtime was.  Clearly, this was mostly time when the ground motion was high enough to cause us locking difficulties.
• One thing that might be hurting us in these early stages of CARM offset reduction is OMC flashing, see alog 22237. We will pull this off resonance using the guardian soon.  These OMC flashes are mostly not bright enough for us to see them on the AS camera, and it may just be that we see more OMC flashes when there is more light at the AS port because of bad alignment or high ground motion.