On Tuesday all the HEPI Pumps were spun down to check system Accumulators' pressures.  Upon bringing them back into service, I noticed CS PS#8 sounded funny.  I had also greased this and another motor and suspect maybe this greasing has packed the motor too tightly.  See the first attachment of the Pump Controller Medm to follow along if you wish.

See the second attachment showing the last four days spanning the down time on Tuesday.  Notice how after the pumps come back on, PS1_PRESS1 (epics channel name) is lower by 10psi and is also much noisier.  PRESS1 is the sensor just after the pump before resistors and filters.

I was a little baffled at first seeing that the other Pump Stations PRESS1 was exactly the same but it now makes sense.  The system servos on the differential pressure across the actuators at BSC2 and that is a direct function of the total output of the four pump stations.  Working back upstream to the Pump, the resistances giving the pressure drops across the 3u filter and the two laminar flow resistors have not changed hence the Pressure out of the other three pumps have not changed.  However, now that Pump #8 is not putting out the same as before, the other pumps must spin faster to achieve our total desired output.  Hence the 20% increase in motor speed shown on the VOUT channel.

I propose I take this PS offline on Tuesday; it may stay offline for a week or I may get it back in one day.  I can do this with out having to deisolate any HEPI platforms although there may be a minor pressure glitch when I valve out this pump station as the motor stops.  Still this pressure glitch may be seen on the platforms, see the effect on the BS if you care here when I greased motors May 2014.  The glitches from this can be seen as 'deep' as the SUS ISI Witness channels although I don't know if these are actually strong.  Bottom line, this certainly should not be done during any measurements.

This morning I cleared some of the red out of the DAQ overview. The errors that were cleared with DIAG Reset were: TIM & ADC: SUSIOPEY / EX, SUSETMY / X; IPC: IOPSEIEY / EX, ISIETMY, ALSEY / EX, ISCEY / EX.

There was a ~3 hours lock at 7:00 UTC on July 13th.

Surprisingly, I did not find any aLOG entry related to this lock.

My full report is linked at the end of the page.

In brief:

• Although there seem to be issues with calibration, the range was "high" and relatively stable.
• The glitch rate is very low, although doubts have been raised about the interpretation of this data (the new version of Omicron may be reporting lower glitch rates than it used to)
• Lock loss was probably due to an earthquake

Outstanding issues:

• Whatever its value, the glitch rate increased by a factor ~10 in the last 30 minutes of the lock for unknown reasons
• In the last 50 minutes, a broad peak appeared (and was "stable") around 20 Hz. origin is unknown. BRUCO seems to suggest it could be related to the beamsplitter suspension.
• The laudest glitches seem all to have similar shape and frequency content. I was unable (partly for lack of data) to correlate them with any other channel.

Full report:

https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20150713

NOTE: for whomever was present at the DetChar call where i presented the report, I added a number of updates (you can find them by searching the page for "update") based on the feedback received during the call.

The following are a list of lock loss messages in the Guardian log. We've had a bunch of locklosses during the transition from locked to low noise this evening. As you can see there are a few different culprits, but one of the big ones is LOWNOISE_ESD_ETMY. It would be handy if someone can check out these lock losses and home in on what precisely went bad during this transition (e.g. ramping, switching, etc.). Then we can get back to SRCL FF tuning.

2015-07-17 06:47:05.694550  ISC_LOCK  LOWNOISE_ESD_ETMY -> LOCKLOSS

2015-07-17 06:48:22.162260  ISC_LOCK  LOCKING_ARMS_GREEN -> LOCKLOSS

2015-07-17 07:14:34.431170  ISC_LOCK  NOMINAL_LOW_NOISE -> LOCKLOSS

2015-07-17 07:26:40.249110  ISC_LOCK  CARM_10PM -> LOCKLOSS

2015-07-17 07:34:59.720030  ISC_LOCK  PREP_TR_CARM -> LOCKLOSS

2015-07-17 07:42:08.269350  ISC_LOCK  LOCKING_ALS -> LOCKLOSS

2015-07-17 08:02:41.773620  ISC_LOCK  LOWNOISE_ESD_ETMY -> LOCKLOSS

2015-07-17 08:21:58.665420  ISC_LOCK  LOWNOISE_ESD_ETMY -> LOCKLOSS

2015-07-17 08:31:29.035330  ISC_LOCK  REDUCE_CARM_OFFSET_MORE -> LOCKLOSS

2015-07-17 08:48:32.514870  ISC_LOCK  LOWNOISE_ESD_ETMY -> LOCKLOSS

Sheila, Jenne, Stefan

TOday we changed the calculation of TR CARM in the LSC model.  The model had a saturator before the square root, when we first turn on TR CARM we can be at low enough arm powers that we are below the lower limit of the saturator, (1e-4) when the powers increase above that the loop can get a large transient.  To smooth this out, we added a second path that has a saturator with an upper limit of 1e-4, subtracts a fixed offset  of 1e-4 and applies a fixed gain of 50 and is summed with the original path.  

This means that for values of the arm transmission above 1e4, we have the same signal as before, and for values below that we have a signal that is just the sum of the arm powers.  This should not cause the kinds of transients we were seeing last night.  

We have seen fringe wrapping due to our alingment drives to the OMC, which could be because the pitch drive really moves the OMC input coupler in the longitudnal direction.  To test this we can see if we can get rid of the fringe wrapping caused by a pitch drive by sending some of the pitch drive to the suspension longitudnal drive.  

I made an attempt to diagonalize the OMC drive for pitch.  First I drove the LOCK_P filter bank in the OMC SUS with 3000 counts at 0.2 Hz, producing the dark purple reference in the attached plot.  Then I adjusted the P2L gain to reduce the velocity.  The light purple trace shows the frequency of the shelf being reduced from 45 Hz to just over 20 Hz, so about a factor of 2 reduction in velocity.  

The OMC sus model doesn't send the signals from the OMC ASC through the driveAling matrix.  It would be nice to re-route them through driveAlign to make this easier.  For now I've added matrix elements to the OMC ASC DOF2TT matrix to send the pitch signals to the sus longitudnal.  This could be fine tuned more, but we lost lock as I was trying to fine tune the coefficient.  Hopefully leaving these elements in the matrix will reduce the glitches due to scattered light from the OMC.  

Stefan, Matt, Rana, Evan

We rephased REFL9 in full lock by driving a line at 5443 Hz into the error point of the common-mode board.

Then we switched control of CARM from REFLAIR9I to REFL9I in the usual fashion (by repeatedly measuring the OLTF of CARM while slowly swapping the relative strengths of the two sensors). Settings are shown in the attachment, along with an OLTF.

The noise in DARM is not better than with REFLAIR control, but at least this time it is not worse.

We were locked at 24 Watts for just over 2 hours before we rang up a PI that shows up in the Y arm QPDs at 15734 Hz.  I increased the ring heater power (for both arms )from 0.5 to 0.6 Watts.  template with the QPD IOP channels is attached.  I tried to reduce the power, but we lost lock when I did that perhaps because the ISS second loop was on.  The lockloss was at about 3:00 UTC on July 17

I have modified monit on h1pemmy to log system over-loads to its local /var/log file system. This will give us an indication of how system loading follows CA freeze up events.

Travis, Patrick

08:40 earthquake breaks lock, Leonid starting charging measurements
09:29 fire department through gate
09:41 fire department gone
09:48 Katie to LVEA to take pictures of magnetometers
09:55 Robert and SURF students to LVEA to work on PEM
10:16 Jeff B. and Karen to clean diode room
10:22 Vinny to mid stations to test PEM sensors
10:36 Jeff B. and Karen done
10:40 Leonid done charging measurements, Ops starting locking sequence
10:42 Robert and SURF students done
10:53 Kyle and Gerardo driving forklift in high bay
11:00 Jason taking training tours into diode room
11:31 Kyle and Gerardo done
12:07 Dave restarted h1oaf
13:47 fire department through gate
14:02 Nutsinee has had violin mode damping off for the last hour, turning it back on
15:00 out for Ops meeting

The lift pumps that service the LSB are not functioning. See FRS 3342.

 There will be a contractor on site tomorrow morning to remove the pumps and diagnose the problem. We may have to resort to porta-potties temporarily at the LSB depending on how long the repairs take. 

[WP 5359] LHO rtbuild was upgraded to RCG tag-2.9.5, this is now the default build release.

h1oaf was built against 2.9.5 and restarted. This will be used to test the fix of the True RMS part.

Leo, Jeff
Today morning we had some time after the earthquake to perform OPLEV charge measurements.
Results for ETMX are consistent with negative trend, now the charge is about 10 Volts on all the quadrants.
Results for ETMY do not not show a significant trend and below the 10 [V] Effective Bias Voltage.

Note. I'll check it again, but it seems that we had the wrong sign of applied bias voltage in charge measurements for ETMY. Applied bias was multiplied by -1 in filters.
Applied ETMY plots use the corrected sign for June 12 to July 14.

Summary: the effect of wind in the sub 0.1 Hz tilt band is very local (little coherence between seismometers 20m apart) and more than a factor of two greater in the HAM 2 and 5 seismometer locations then in the beer garden. We may be less sensitive to wind if the sensor correction seismometer(s) are located only in the beer garden. Also, because tilt is so local, real tilt meters, like Krishna’s at EX, should be as close as possible to the chambers.

Wind tilts our buildings, which produces spurious control signals from servo seismometers and can make it difficult to lock or maintain lock.  A previous log showed that there was almost no wind tilt at a location 40 m from the EY building, making it clear that wind tilt is a local effect (Link).  As a result, Hugh and I have been wondering if the HAM 5 seismometer location is better because it is down-wind for most storms or if the beer garden is better because it is furthest from walls. With Hugh’s help, I looked at chance coincidences between wind storms and seismometer huddles over the last few months as well as data with seismometers in the 3 locations. I think the answer is that the beer garden shows substantially less tilt than either the HAM 5 or 2 locations.

Figure 1 shows how local tilt is. The blue seismometer traces are for “huddled” seismometers (about 2m apart) in the beer garden and show high coherence below 0.1 Hz. But the red seismometer trace shows much lower coherence in this tilt band between the beer garden seismometer and the HAM5 seismometer, only about 20 m away. During high wind, I also found low coherence in the tilt band between the beer garden and the HAM2 seismometer locations.  The local nature of the tilt has implications for true tilt meters used to correct the tilt signal from seismometers. The tilt meter at EX is about 4 m from the chamber and, in Figure 1 we saw very little coherence at 20m. While it may not be enough of a return to move this one, it may be best to try and place the next one even closer, and, to the degree possible, engineer the BRS so that it can be as close as possible or even under the chamber.

Figure 2a and b show that tilt is very different at different locations in the LVEA and, of the 3 locations, the beer garden is the best. In both horizontal axes, the tilt in the beer garden is at least a factor of two better than the best of the HAM2 and HAM5 locations. It is about a factor of ten better than the worst of the HAM2 or 5 locations. I checked the 3 windstorms during the period when all 3 seismometers were working and, for each time that I examined, the beer garden seismometer was better. Figure 3 shows the two seismometers that were available during the windy period that caused locking problems last night: the tilt noise was half as much in the unused beer garden seismometer than in the HAM5 seismometer that was used for sensor correction. So, a sensor correction seismometer in the beer garden may be better than in the HAM2 or 5 locations in the frequency band dominated by tilt instead of real acceleration (roughly below 0.5 Hz). This morning Jim switched sensor correction to the beer garden seismometer.

Finally, when we have two STSs available, I think we should do a more detailed study of tilt-band coherence length, and attenuation with distance from the walls.

Robert, Hugh

For anyone who is interested in classifying locklosses, or in earthquakes, here is an example of a full low noise lock broken by an earthquake.