J. Kissel, B. Lantz, H. Radkins

Hugh and I, with the remote advice from Brian, have been investigating the coupling between HEPI differential pressure noise and HEPI platform motion (see preliminary studies in LHO aLOGs 16231 and 16309). In summary, the coupling is smattered between every chambers DOFs, limiting the performance between 0.02 and 2 [Hz]. There's no pattern as to how this coupling occurs except that the coherence for a given DOF happens in a similar frequency band in all chambers. All chambers show differential pressure moderate coherence to HP, but we're still working on estimating the transfer functions between this "pringle" mode and any translations / rotations to assess this impact. Further, we still have set up any successful long stretches of IFO / WFS data for the DRMI, so this pressure noise' impact can't yet be assessed. However, for now, I continue under the assumption that *any* low frequency improvement, especially to RZ / YAW, will improve the relative angular fluctuations of the IFO and stabilize optical gains, etc.

Hugh has already revealed the punchline: this morning's pump-servo OFF data shows that the problem is entirely because is of noise on differential pressure sensor. I'll attach a comment with complete set of plots for proof in a bit.

Further discussion on the noise as it stands with the pump servo (i.e. ON), based on the attached plots, which include

• ASDs of all Cartesian (plus pringle) DOFs' L4Cs -- out-of-loop witness sensors -- compared against the differential pressure and the expected sensor noise**,
• Coherence between all Cartesian (plus pringle) DOFs and the differential pressure,
• ASDs and RMS of Colocated Actuator Drive signals, compared against the differential pressure and the 2 [hr] mean, DC value of each signal,

is below. I post all of these discussion points for future reference and design considerations to show how nastily and diversely imposing sensor noise on the pump servo imposes the platform motion. But mostly, because I sorted through a TON of data and plots to make these conclusions, before we knew the source of the problem.

** For the HEPI L4C ASDs, I plot the *raw* sensor noise because (a) I'm trying to cram as much information on as few plots as possible, and (b) the scale factors for both the IPS and L4Cs are the same for each DOF, and are all between 0.3 and 1.1, so the estimation estimation is off by at most a factor of three. Mostly it's there just to guide the eye, as a primitive noise budget.

(1) Krishna (before coherence between all DOFs and chambers was fully flushed out), suspected that because we now run Z sensor correction is on HEPI, the extra drive force meant excess pressure noise coupling. However, compared to 23e3 [ct_{rms}] range, the Vertical RMS of 5 [ct_{rms}] is peanuts. Further discussion of DC / mean requested drive is below.

(2) Brian suspected that the passive filtering system, the accumulating bladders might not be operating at their nominal air pressure. However, pump 8's (out of 12 for the corner) bladder pressure was checked last week, and was within spec. There're still lots lots to check, but Hugh suspects they're also OK.

(3) I suspect the PID controller for the servo is likely not well-tuned for new differential pressure sensing, suggesting we need to remeasure plant (which could mean significant down time for the IFO). There's been some discussion offline in the SEI group as to how we should systematically characterize the pump servo plant, given there's no "fast" excitation point in the pump servo make a tranditional frequency-response transfer function difficult (see SEI aLOGs 684 and 686). For now, we suspect we can do a sufficient characterization with simple steps using the existing EPICs infrastructure. Characterization to-date (though very early on in aLIGO) was also made by step response (see E1100508), but the assumptions about the plant don't seem sound in retrospect, and were done when the pressure was still on an absolute "single ended" readback.

- Brian's "Allowable Pump Servo Noise" defined in T1100198, but contains a LOT of assumptions. One of which is that the open loop gain transfer function of the feed back loops is "100 at low frequency." As such, Brian suggests, if the HEPI feedback loops are gain limited, try adding some boost at these frequencies to suppress the noise. However, I argue, with Hugh's recent redesign of the controller (see LHO aLOG 15308, the relevant attachment re-attached here), RZ Loops for example have *plenty* of gain, upwards of 3000 - 4000 at 0.1 [Hz], typically asymptoting to a few million a DC. This is far more than Brian assumed, so I thing we're alright in the gain department. Regrettably, the differential pump pressure is an EPICs channel, so I can't get an ASD of the performance above ~8 [Hz], but I'll post more on that in the future comment.

- Differential Pressure Noise supposedly coupling increases with actuator force request, but I've not found this to be true empirically. Check out the table below. Colors of the text help indicate which requested DC offsets are small, and which are large (dark green is smallest, then light green, gold, orange, with bright red is largest, in bins of 50 [micro-whatevers] and anything higher than 200 [micro-whatevers] is marked as bright red). Cells shaded in gray show coherence between the differential pressure and the HEPI L4Cs.

There is no discernable pattern:

Here're what patterns I was able to glean from the above table and attached plots:

• Every chamber has a large coherence with HP
• All chambers coherence drops off gradually below ~0.1 [Hz], probably because the measurement is hitting the L4C noise floor. Similarly, the coherence drops off above ~2 [Hz] because the table motion becomes dominated by IPS noise reinjected via the feedback loops
• Pressure is typically coherent between 0.2 and 2 [Hz] in Z, where RZ is typically coherent between 0.02 - 0.2 [Hz] (though note that the sensor noise limitations of this statement from above)
• BSCs have stronger coupling to pressure noise than the HAMs. One might argue this is because the actuators are pushing more (~5 [ct_{rms}] on HAMs, 50 [ct_{rms}] on BSCs calculated down to 1 [mHz]). BUT HAM6 has a huge vertical offset, but now where near as heavy a coherence.
• Input HAMs have no RZ coupling, but output HAMs and BSCs do
• Only HAM2, HAM5, and ITMX have some X & Y coherence with pressure. HAM2 in X, HAM5 in Y (of course), and ITMX in both. All between 0.6-0.9 [Hz].
• It looks like a pretty bingo board, doesn't it? Thanks for reading this far!

- My best guess at this point is that it depends on the intricacy of how the piping is laid out between each chamber and the pumps. But it's only a guess, it's the only thing I know that significantly different between all chambers.

-------------------

DTT templates for this data live here:

/ligo/svncommon/SeiSVN/seismic/HEPI/H1/Common/

2015-01-23_H1HPI_ActuatorDrive.xml

2015-01-23_H1HPI_PumpControllerNoise.xml

The stopband filter we put in the ETMX OL servo yesterday seems to have done the job of preventing the ringups:

In the attached trend of the OLPIT 3-10 Hz BLRMS, you can see the ringup start around 6AM local time this Sunday, the 25th.

Around noon on Monday (1800 UTC), Evan and Alexa start damping the mode using the bandpass filter in the OL PIT servo. They stop around 2100 UTC when the mode is small. It then rings up for the next 7 hours until we turn the stopband filter on.

Then it rings down with a 1/e time of ~4 hours (which implies a bounce mode Q of ~440,000 if nothing else is driving it).

We should make sure to install these on all of the test mass OL servos.

06:30 Cris and Karen into LVEA- esit doors will be open

08:00 restarted digital camera images on FOM5

08:13 Rick, Darkham, Thomas and Nutsinee to EY for P-Cal work. Working close to STS.

08:14 :21, :22 Sensor correction is off @ EY to accomodate P-Cal work.

08:22 Sigg rebooting end station ISC

08:23 Gary moving ISS arrays from optics lab to H@ enclosure. Will be using exit doors from LVEA to accomodate. Sigg approves.

08:30 Cyrus installing new edge router for CDS. GC will be down for a few minutes.

08:34 Christina w/exterminator visiting all points to check mouse traps.

08:34 Hugh into CER

08:48 Cris and Karen to Mids

08:52 Jef to LVEA to get a LASER vibrometer.

09:00 Plumber on site for toilet maintanance.

09:02 Kyle called to infor about 2 Praxxair deliveries today.

09:08 Betsy out to W Bay

09:10 J Kissell ramping down SUS in HAM6 for ISI FE Model restart.

09:15 Restart HAM6 ISI FE Model. All sensor correction is turned off except for EX. No DAQ restaert required.

09:18 HAM6 restored and back online.

09:29 Karen leaving MY.

09:40 Praxxair called to announce arrival

09:44 Gary driving to EY to talk to Rick

10:08 Doug out to LVEA to move copper pipes under ITMY.

10:11 Rick et al back from the end station. Can't run excitations.

10:13 Jeff an TJ back from LVEA.HEPI pump ! needs electrial attention. Hugh will give more detail.

10:30 Jeff B back into LVEA by HAM2 vibrometer.

10:49 Betsy and Janine out of th LVEA.

11:15 Sudarshan to EX

12:00 Jef back out to LVEA HAM2

12:12 Jeff out of LVEA

12:15 Fil and co out of LVEA

12:30 Jim out to LVEA to get some parts

13:45 Kyle and Joe out tp EY mechanical room

14:20 Kyle ad Joe back from EY

15:20 Sudarshan out to EY

15:50 Sudarshan back from EY

JeffK is going to produce a much more extensive result and possibly conclusions but the simple view is here.  I selected a few chambers to look at BS, ITMX and HAM5.  HAM5 is closest to the Pump Stations, HAM1 is certainly the farthest but it isn't unlocked.  HAM2 might be a better candidate but too late, I chose ITMX, the BS is where the pressure sensors are which derive the differential pressure channels.

I looked at all DOFs (not VP) and we see/saw lots of coherence with HP RZ and Z on most chambers.  HP & RZ being the largest.  I show HP, RZ and the oplev YAW and Pitch of the optic on the platform in the attached plot.  The top group is the BS, the middle traces are HAM5 and SR3, and, the bottom group is the ITMX.

Pretty much nothing seen on the Oplevs but the coherence on the RZ and HP loops when the HEPI Servo is on is pretty clear compared to when the drive to the Pump Stations is fixed ignoring the pressure signals.

More Details:  Switched the Servo to Manual state at 1510utc this morning and made one output tweak at 1514.  The Sensor corrections to the BSCs were turned off at 1549--this may come into the spectra toward the ends of these 10 average 1mHz BW measurements.  The current traces are with the servo off and are started at 1515utc.

The servo was turned back on at 1716utc.  The reference traces (dashed) in the plots where started at this time.

My current conclusion from this is the noisy pressure channels is injecting this noise into the drive signal and it is getting to the platform.  We will tune the controller to help deal with this but the noisy signals must be addressed.

J. Kissel, H. Radkins, R. McCarthy

We tried a few things in order to debug STS B (GND_ITMY), which had been revealed to have a factor-of-2 too low gain after an event at Tuesday, 8:30a Dec 23 2014 (see LHO aLOG 16237). Nothing was successful at fixing the problem. However, we have ruled out the STS Distribution Chassis, the STS Interface Chassis, and the ADC that reads in the STS by swapping a few cables around at the rack: 
- The factor of two followed the field cabling, i.e. when STSB / ITMY was plugged into STSA / HAM2's readout, the factor-of-two moved to STSA / HAM2 readout. 
- Richard also made sure that cables were secure both going in and out of the satellite amplifier. 
- Finally, he injected (differential) +10 [V_{DC}] into the appropriate STS Interface Chassis channel, and found the readout displaying +16384 [ct], as expected (see T1100538). 
This implies that the fault is either in the STS itself, the orange cable from the STS to its satellite amplifier, the satellite amplifier itself, or in the long cable run from the satellite box to the racks. 

Richard suspects that the custom D50 connection at the STS Interface Chassis, but will continue to investigate tomorrow.

Attached is evidence of the mornings activities, but more importantly that the gain factor is *exactly* what is was before after restoring everything, so we'll continue to use the matched gains determined in LHO aLOG 16208.

This morning we ran a new field power cable from the CER DC power rack to the ISC racks by HAM6. This is to prep for the installation/testing of the Fast Shutter later this week. We also installed a HV Kepco power supply in the CER. In order to terminate the new field cable at the floor rack, we temporarily powered down the OMC HV Piezo power supply from around ~11am-12pm. The power for the OMC Piezo is now restored.

We turned off the sensor correction to do some trouble shooting on the STS2_B. Ed got these turned back on ~11pst.  All HAMs have ground XYZ to ISI-ST1, BSCs have ground Z to HEPI and ground XY to ISI-ST1.

 ISS erratic. I unlocked, adjusted REFSIGNAL and re-locked. Holding steady for now. https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=13781

The original log entry was made before 12:00P.

With permission from Jeff K I have remotely updated the EPICS settings related to the ODC for the PSL. This change had no impact on any non-ODC EPICS settings, or other settings. I have committed updated safe.snap files for h1psliss, h1pslfss, and h1pslpmc, as logged here.

This should bring the H1 PSL ODC in line with the equivalent at LLO.

When we center the REFL DC with the servos, we often find that the RM1 has tripped. This was true yesterday evening, and today Jan 27th, 2015 19:43 UTC. No investigation has been done by the commissioning team.

This noise source (pump) will likely be running for the next 6 - 8 weeks

Doug went to the floor and used a rope to pull the instrument air line etc. more from the oplev pyron and the 10-min oscillation is gone.

alog from yesterday: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=16276

This is yet another temporary fix that is good enough for a while, just like the original "fix" which stayed until today. This sounds like a good target for the detector group to keep track of.

Last week I found all of the BSC ISIs with their St2 GS-13s in low gain. Looking in conlog, I found that the gains had been switched at 07:05 PST on the 22nd, then I found the error and corrected it at 08:59 PST on the 23rd, meaning there was an entire night where the all of the ISI's were running in a different from nominal state. This would be a good window to look at the performance

Additionally, last week, Arnaud tried running the TM ISI's with all of their RZ loops turned on. The largest window I found for this was from about  10:20 to 20:00 PST.

All BSC ISI's have otherwise been in their nominal configurations since then (ie RZ loops off, rdr blends, sensor correction on, all instruments in high gain (except the BS)).

J. Kissel, A. Pele (WP #5025)

We are to make the IFO's fast shutter functional within the next week, in order to prepare and protect the OMC DCPDs from the energy leaked from a FULL IFO lock lock. In its current design / implementation, this shutter is fast and loud as it closes and reopens, which sends large impulses to the optical table on which it sits (in this case H1 ISI HAM6). In order to prepare for this "toaster" being activated, I've increased the number of consecutive saturations (computer cycles above the user-predefined threshold) that are acceptable for the H1 ISI HAM6 GS13s and Actuators before its watchdog trips to 4096 [samples] / 4096 [Hz]= 1 [sec] and 8192 [samples] / 4096 [Hz] = 2 [sec], respectively (where it had previously been 10 [samples] / 4096 [Hz] = 0.0024 [sec] for these channels). See attached screenshot of the relevant WD block innards.

This has been done and in place at LLO for a few weeks with no obviously adverse affects, (see 16430 and 15998), we were merely replicating their change. As such, they've seen a drastic reduction in HAM6 ISI trips after lock-losses, and we hope to see the same benefits. 

Note, the change involves *unhooking* (disabling and breaking) the 
/opt/rtcds/userapps/release/isi/h1/models/h1isiham6.mdl 
model link to the generic HAM ISI model library part
/opt/rtcds/userapps/release/isi/common/models/isihammaster.mdl
and replacing the WD_SAFE_COUNT tag input to each GS13 and ACT blocks (inside the HAM6 WD block) with hard-coded constants of 4096 and 8192. 
I've committed the unhooked model to the repository.

For the restart, I
- Brought the OMC suspension to SAFE
- Turned off all output to OMs 1-3
- Asked the SEI_HAM6 manager to take the chamber to DAMPED
- Paused the SEI_HAM6 manager (to make sure ISI_HPI is left alone and happily running)
- Changed the ISI_HAM6 subordinate to exec
- Requested ISI_HAM6 subordinate to READY
- Turned OFF ISIHAM6's master switch 
- Recompiled, Reinstalled, Restarted h1isiham6 front end process on h1seih16 computer
- Requested ISI_HAM6 subordinate to DAMPED, confirmed IOP output
- Changed the ISI_HAM6 subordinate back to managed
- Changed the SEI_HAM6 manager back to exec (who came back, happily fooled that it never left the DAMPED state)
- Requested SEI_HAM6 manager to return to ISOLATED.
- Restored OMC suspension to ALIGNED
- Restored alignment and damping to OMs 1-3 (and a random Y offset of 2000 [ct] on the TEST filterbank).
Note, also, since I merely changed constants in the model, the change did NOT require a DAQ restart.
I also attach trends of the P and Y of the HAM6 suspensions (as measured by their local OSEM sensors), confirming that they've been restored to the same alignment.

Switched to manual control at 1510utc.  Made one setpoint tweak at 1514.  It looks like it will probably run without further adjustment (maintaining the ~70psi differential pressure) for a hour or more.

Alexa, Elli, Sheila, Rana, Evan

Today we worked some more to make the sqrt(TRX+TRY) handoff more robust.

Now we can pretty reliably complete the transition by hand. We are working on implementing the transition in the guardian.

We have tried to reduce the CARM offset while locked on sqrt(TRX+TRY), but we cannot seem to get beyond 2 or 3 times the single-arm power without blowing the lock. The next step is to go through the CARM reduction sequence more carefully and characterize the OLTF of the CARM loop in this state.

Gain redistribution

As discussed in LHO#16252 et seq., we suspect that the common-mode board has gain-dependent offsets. If this is true, it explains why the interferometer can be knocked out of lock while ramping down or turning off ALS COMM.

To boost the CARM loop's immunity to these offsets, we redistributed gains as follows:

• The gain of the ALS COMM input on the common-mode board was increased from −4 dB to +16 dB (LSC-REFL_SERVO_IN2GAIN). Correspondingly, the gain on the AO into the IMC board was reduced from +16 dB to −4 dB (IMC-REFL_SERVO_IN2GAIN).
• The input matrix gain for the slow path out of the IMC board (going into the digital CARM control) was decreased from 1 to 0.1 (LSC-PD_DOF_MTRX_5_26).
• The digital gain for sqrt(TRX+TRY) was increased by a factor of 10 (LSC-REFLBIAS_GAIN was increased from 0.8 to 8).

Even with these changes, success is not guaranteed. The gain steps can be heard very clearly when listening to LSC-CARM_IN1, and turning off ALS COMM on the summing board will blow the lock about half the time. We can get better results by using the gain slider on the common-mode board, but we can still hear the gain steps.

Modified handoff procedure

• In LSC-REFLBIAS, we engage FM3 (1 Hz pole, 40 Hz zero) and FM9 (1.6 Hz pole, 40 Hz zero). The gain is 8 ct/ct.
• In LSC-TR_CARM, we set the offset to −0.5 ct.
• We engage the sqrt(TRX+TRY) path on the common-mode board.
• We engage LSC-REFL_DC_BIAS with a gain of 30 ct/ct.
• Once the buildup has stabilized, we turn on FM6 (0 Hz pole, 1 Hz zero). Then we turn on FM4, which undoes FM3. Then we ramp the gain of LSC-REFL_DC_BIAS to 50 ct/ct.
• We ramp down ALS COMM using the common-mode board input gain slider. Then we turn off this input on the common-mode board.

A lockloss plot is attached for an unexplained lock loss during the CARM offset reduction after handing off to sqrt(TRX+TRY). Other lock loss times, all 2015-01-27 UTC: 03:56:53, 04:05:25, 05:36:16, 06:40:59, 07:56:02, 08:37:20. The last four are during CARM offset reduction after the handoff is complete.

For a sqrt(TRX+TRY) offset of −1.5 ct, the gain we need for REFL_DC_BIAS is 50 ct/ct. Our OLTF looks good here. However, when we reach an offset of −2, we seem to lose lock, even though there is no obvious nastiness in the CARM spectrum.

Mystery

Today, locking DRMI without arms was pretty painless. In contrast, DRMI+arms lock acquisition was very, very slow for most of the morning and afternoon. After about 5pm local, it became painless as well. This may have been correlated with our changing the trigger settings to be twice as high with arms (compared to no arms), since the POP18 buildup is twice as high. But we haven't investigated this systematically.