The frame wiper script which runs on h1fw0 was changed to alow the disk to become 95% full to match the configuration of h1fw1.  The previous setting allowed only 90% disk usage.  

The setting of 90% was most likely left over from a test configuration when the Ubuntu frame writer was first created.

Here are the LHO DAQ lookback times for framed data served by the internal NDS protocol-1 servers (data taken at 13:25 local time).

h1nds0 (disks in LDAS server room). Disk usage 91% (33TB used). 

h1nds1(default-nds) (disks in MSR). Disk usage 96% (35TB used).

Note underuse of disks on h1nds0, being changed to use 95% of available disk space.

Under WP 5527, yesterday I switched the sensors on two of the CS HEPI Pump Skids.

On a Pump Station Skid, see first attachment for a medm depiction, there are pressure sensor ports. 1--right after pump, includes first capacitor accumulator; 2 & 3--after laminar flow resistors & includes accumulator; 4--after final 3u filter. In previous logs, see first plot of 21366, I show that at the corner station, pressure 1 right after the pump is super quiet, again only at the corner.  I checked with Barker and we looked at things and there is no reason the trends would be different for these channels.

Here the attached plot shows just the four impacted pressure sensors.  The first two plots I've zoomed into the quiet portions before and after the switch showing that the "quiet" is tied to the sensor not the channel; the shift in the reading is due to the AOFF (zero) of the individual channel.  On the next two graphs (PS4_PRESS1 & 3) I've zoomed into the noisier after and before the sensor switch.  Bottom line, the noise level is at the sensor, not the channel.  The next two graphs (channels 7 & 8) are the End stations' Pressure1.  Clearly EndX is out to lunch with super ugly noise from something we can't figure out.  This shows up on the EX control out, CH10, and the servo'd PV (not shown.)  The Ch8 graph shows the EY PRESS1 is much noisier than the corner press1s.  The last 3 graphs are the control outs for the corner, L0, and EX and EY.  Interestingly, despite the much quieter pressure on the CS vs EndY PRESS1s, the CONTROL_VOUTs of the CS and EY are very similar as is their servo'd PVs (the EndY shows the diurnal much more than the others.)  Even though the pressure out of the pump is quiet or noisy (CS or EndY) the RC Manifold of the Pump Station makes the signal noisier for the CS.  The noise level on the EndY Manifold pressures are about the same or only slightly noisier down the manifold (I looked.)

Still wondering about what is going on...

1) The Accumulator charge on the Press1 volumes are similar although the fluid pressures are different at the three locations.  Why?  The zero AOFFs are coarse and may be variable over time.  The resistances to get to PV setpoint are likely different even at Ends and especially at corner.  Why? Different crap in the resistor stacks, different length pipe runs to BSC, more Actuators at corner, different actuators everywhere, etc.

Solution?  Be more precise at setting accumulator pressures based on operating pressures?  Given the spec'd nominal accumulator charge of 60 to 93% (I shoot for 90%) of operating fluid pressure and the very unlikely(hood) that I would be so good(lucky) at the CS versus the Ends, I'd say this is not the issue.

2) The Corner station has 24 HEPI Actuators per Pump Station while the Ends have just eight.  This leads the CS pumps to run about twice as fast as the Ends, see VOUT channels.

I could manually slow down the CS pumps to End speeds and see what that does.  HEPI performance likely compromised if not completely hosed but might be informative.  Could we introduce additional resistances at the Ends to require the Pump to speed up?

3) Could the laminar flow Resistors be contributing to the down stream noise increase?  Does it make sense that the pressure out of the pump is the quietest?  This certainly would not explain why the Ends are noisier but might explain the down stream noise.

4) Could the End Station Pumps be cavitating?  Yes.  I've had a pump cavitate in July, 19699 & 19794, after maintenance in the CS and had to restart it.  I did so over 5 minutes and the cavitation stopped.  I'd have to mine to find the last time the Ends were restarted and figure out how they were ramped back on.  I try to do so slowly manually or with the servo but it's not typically 5 minutes.  I don't suspect this as the pump made an audible rumbling when the CS was cavitating and I don't hear this at the Ends.  But, maybe there are varying levels of cavitation and there could be benefit to restarting the End Pumps very slowly.

It really does not matter anyway.  Given what the servo pressure and control drive look like comparing the CS and EndY, those results are the same.  More important and maybe still not that important is to figure out why the EndX pressures are so noisy.

I'll restore the pressure sensor cabling when able and close out this WP.

The LHO SEI team has known about a .6-ish hz peak on the HAM3 ISI for a long time (see my alog 15565, December of last year for the start, Hugh has a summary of LHO alogs in the SEI log for more). I was working with Ed on a DTT template for the operators when I noticed it was now gone. Very strange. Looking a little closer, it seems to have been decreasing over the last couple of days to a week, and disappeared completely this morning about 7-8:00 UTC. Attached spectra are from ~0:00 UTC (red) and ~16:00 UTC (blue, when I found it was missing). Looking at random times over the last week, it looks like it may have been trending down.

Could someone in Detchar look at this peaks longish term BLRMS, say over the last month, or even over the last year since we found it? Pretty much every sensor in that chamber saw this, but the GS-13s are the best witness.

This activity is allowable as per T1500425. We are currently in Observing mode.

The timescale had changed to 6hrs. Jim pointed this out to me and requested to increase it to 24hours.

TITLE: Oct 7 DAY Shift 15:00-23:00UTC (08:00-04:00 PDT), all times posted in UTC

STATE Of H1: Observing

OUTGOING OPERATOR: Patrick

QUICK SUMMARY:IFO in Observation mode @ 72.5Mpc. All lights in LVEA, PSL and M/E stations are off. Wind is ≤5mph. EQ sie plot is nominal. Microseism plot is steady at around .25microns/s.

TITLE: 10/07 [OWL Shift]: 07:00-15:00 UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Locked. Observing @ ~75 MPc.
SHIFT SUMMARY: Locked and observing entire shift. Some SUS ETMY glitches.
INCOMING OPERATOR: Ed
ACTIVITY LOG:

11:37 - 11:51 UTC Stepped out of control room
11:57 - 12:01 UTC Stepped out of control room

SUS E_T_M_Y saturating (Oct 7 7:57:48 UTC)
SUS E_T_M_Y saturating (Oct 7 8:59:0 UTC)
SUS E_T_M_Y saturating (Oct 7 10:9:27 UTC)

SUS E_T_M_Y saturating (Oct 7 13:42:9 UTC)
SUS E_T_M_Y saturating (Oct 7 14:25:9 UTC)
SUS E_T_M_Y saturating (Oct 7 14:27:56 UTC)

07:39 - 07:46 UTC Stepped out of control room
07:59 UTC Acknowledged program errors reported by WaPac program on access card reader computer. Closed and restarted the WaPac program.
09:14 - 09:23 UTC Stepped out of control room

SUS E_T_M_Y saturating (Oct 7 7:57:48 UTC)
SUS E_T_M_Y saturating (Oct 7 8:59:0 UTC)
SUS E_T_M_Y saturating (Oct 7 10:9:27 UTC)

Still observing @ ~79 MPc.

TITLE: "10/06 [EVE Shift]: 23:00-07:00UTC (16:00-00:00 PDT), all times posted in UTC"

STATE Of H1: Observing at ~80 Mpc for the past 7 hours.

SUPPORT: Jenne, Sheila, Evan, Mike

SHIFT SUMMARY: Was acquiring lock during the first hour of this shift. Once the ifo was locked the rest of the shift was quiet. Wind stays below 5 mph. Couple of earthquakes but the ifo stays locked. Just me and Marissa here for most of the night. Mike called to check on the GraceDB alert make sure it's not missing anything (since the code update).

INCOMING OPERATOR: Patrick

ACTIVITY LOG:

0:12 Ifo back to Observing.

2:45 Robert left the control room

~6:30 Patrick came in

I checked GraceDB again to make sure the alarm didn't miss anything that might have popped up during my shift.

TITLE: 10/07 [OWL Shift]: 07:00-15:00 UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Observing @ ~ 79 MPc.
OUTGOING OPERATOR: Nutsinee
QUICK SUMMARY: From the cameras the lights are off in the LVEA, PSL enclosure, end X, end Y and mid X. I can not tell if they are off at mid Y. Seismic in 0.03 - 0.1 Hz band is around .015 um/s. Seismic in 0.1 - 0.3 Hz band is around .1 um/s. Winds are less than 10 mph.

We have finished generating and fine-tuning the filters we will use for the recalibration of ER8/O1 data.  The filters are generated using the LHO DARMmodel:

H1DARMOLGTFmodel_O1.m

with paramsters file H1DARMparams_1125963332 from SVN r1616 in the calibration SVN (aligocalibration/trunk/Runs/O1/H1/Scripts/DARMOLGTFs)

The filters file to be used for recalibration is located in the calibraiton SVN under aligocalibration/trunk/Runs/O1/GDSFilters:

H1DCS_1128173232.npz

The TST actuation filter was generated using

par.A.getFreqResp_TST(f)

The PUM/UIM actuation filter was generating using

par.A.getFreqResp_USUM(f)

The sensing filter was generated using

par.C.getFreqResp_total(f)

In addition, we have filters to dewhiten (IFO):CAL-DARM_ERR_WHITEN_OUT_DQ and (IFO):CAL-DARM_CTRL_WHITEN_OUT_DQ.  The whitening schemes for these filters can be found in DCC T1500463.  The dewhitening filters are made by switching the poles and zeroes and adjusting the gain appropriately.

The filters file also contains reference values of the calibration model needed for the computation of the time-varying calibration factors (kappa's).  These reference values are obtained from EPICS records in online mode, but since the EPICS records are not accurate for the entire time we are recalibrating, the reference values will instead be read from the filters file for recalibration.  

Attached are plots that compare the frequency response of the FIR filters for TST actuaiton, PUM/UIM actuation, and inverse sensing to their respective frequency domain models, obtained as described above.  

I have also attached a plot that compares the output of the online GDS-CALB_STRAIN channel to the output of DARM_ERR and DARM_CTRL calibrated using these FIR filters.  The two agree to better than 1% in magnitude and 1 degree in phase. 

TITLE: 10/6 [EVE Shift]: 23:00-07:00UTC (16:00-00:00 PDT), all times posted in UTC"

STATE Of H1: Observing at 79 Mpc. The range has been slowly increasing since the lock was acquired an hour ago.

OUTGOING OPERATOR: Ed

QUICK SUMMARY: We finally locked and Observed for the first time in 8 hours. Clean DMT Omega in the past hour. Wind below 5 mph. Nominal seismic activity in the earthquake band. Terramon reported a 3.9M earthquake off the coast of Oregon that should have arrived 17:33 UTC and dropped us out of lock. We seem to have ridden through it just fine.

Today we noticed that the EX beam diverter has been open since June 18th.  We don't normally switch this so it is not guardian controlled but we "usually" leave it closed.  It is beckhoff controlled so it is not monitored in SDF.  Today TJ added all the beam diverters we don't normally change in guardian to SYS_DIAG to prevent this from happening again (it was already checking the others).

We have now locked with the beam diverter closed, and see that the QPD spectra look the same between EX and EY.   (20418 and 21767 have some history.)  The peak in DARM at 78 Hz is also not here.  We will see if it stays gone, and check soon to see if the blinear coupling of ETMX ISI motion to DARM is gone with the beam diverter closed.  

This morning I did a few things:

First I made some injections to the EMTX ISI (WP5528).  This data can be used to make a projection to the normal level of noise. 

I made 3 guardian changes (WP 5533):

I added a new state to the OMC_LOCK guardian which tries to keep the OMC off resonance, and the ISC_LOCK guardian requests this for the early part of lock acquisition.

Added a function which checks that the ETM pit oplevs have an rms less than 0.5 urad before we attempt to start locking again.   This will prevent tidal from acculating junk while the optics are still swinging. 

Changed the decorators in the ISC_LOCK guardian so that we do not check the DRMI lock once CARM is locked on the transmitted power (this will reduce th confusion in lockloss logs).

I also copied a bunch of medm screens and filters in the ASC ADS system from LLO, as well as scripts with help from Carl, Adam and Marie.  This is in preparation to try using Marie's modifications to Hang's A2L script.  

Motivated by our locking difficulties at the end of last week, I downloaded guardian state data for the last two weeks to look into our locklosses durring the CARM offset reduction.

To start with the good news, the script that I use to do this automatically makes a few figures that tell us about our duty cycle, and we are doing well overall. As usual, I am only looking at the guardian state, and not considering any intent bits.  My reaasoning is that the hardest part is getting and keeping the IFO locked, and that is currently the important limit to our duty cycle. 

• We have been locked in the nominal low noise state for 83% of the time (compared to 68% for the last week of ER8).  A good part of the improvement is probabably because the ground has been quieter in the last 2 weeks, although I didn't try to quantify that. (guardian state pie chart is the first attachment).
• About half of our DRMI locks result in nominal low noise locks (33/64).  A lockloss pie chart is in the second attachment. We have fewer locklosses in the late stages of the guardian, which could be because we are waiting for the first sets of ASC loops to engage before we engage the soft loops (alogs 21583 21587)
• The median low noise lock stretch was 5.5 hours long, while the longest was 59.7 hrs long.  A histogram of the length of low noise locks is in the third attachment.

The main weakness in our lock acquistion sequence now seems to be the early steps of CARM offset reduction, as we saw yesterday. 

•  We certaintly have a harder time with these early steps of CARM offset reduction when the ground motion is high, which explains most of our down time from the 2nd to the 3rd.
  • The 4th attachement show a histogram of locking attempts bined by ground motion, which is the maximum of the sum of the Y direction blrms for 30mHz-100mHz and 100-300mHz bands durring the locking attempt. I chose the Y direction at Y end because that tends to have more wind induced tilt, so this should include the impact of wind, earthquakes and useism.  You can see that we struggle to lock when the ground motion in these bands is above 250nm/sec, although occasionally we do suceed.  In the version of the plot attached here, I called a locking attempt sucsesfull if it reached the low noise state, but the plot doesn't change much if you instead make it for locks which make it past the stage where the DHARD ASC loops come on. 
  • The 5th attachment shows the ground motion over the last week, with a plot of the range to illustrate where the long stretch of downtime was.  Clearly, this was mostly time when the ground motion was high enough to cause us locking difficulties.
• One thing that might be hurting us in these early stages of CARM offset reduction is OMC flashing, see alog 22237. We will pull this off resonance using the guardian soon.  These OMC flashes are mostly not bright enough for us to see them on the AS camera, and it may just be that we see more OMC flashes when there is more light at the AS port because of bad alignment or high ground motion.