Spliting the End Station ISC Models Into Two

Daniel, Jim, Dave:

WP5034. The problem: end station isc models are running long, so their RFM IPC to the corner station do not make it in time. The solution: split the ISC model up into ALS and ISC, with the latter running faster and able to send RFM IPC to the corner in the allocated time (60uS).

Because the h1iscex,ey computers did not have any spare cores (iop, pem, isc, odc, cal) for now we added CAL to PEM to free up a core. Hopefully a better long term solution is to combine isc and odc.

Daniel had done a lot of early legwork getting the models into shape. He hand edited the H1.ipc file to change the source model name for channels moving from ISC to the new ALS models. Daniel also handled the filter module files.

We identified all models which receive IPC channels (including Dolphin) from the new ISC and ALS models. These were also recompiled and restarted.

First thing this morning we moved the new H1.ipc file into place and did a round of make and make-install on the related models. The testpoint.par file needed editing during the "make install" process due to the model-dcuid change for the isc models.

The rtsystab file was modified for the new model layout in the end station isc front end.

Then all the related models were stopped:

h1lsc, h1asc, h1susetmx, h1susetmy, h1hpietmx, h1hpietmy, h1pemex, h1iscex, h1odcx, h1calex, h1pemey, h1iscey, h1odcy, h1caley

The h1iopiscex and h1iopiscey were restarted. We saw an awgtpman autostart problem with these models.

The end station models were restarted (h1pemex, h1iscex, h1odcx, h1alsex, h1pemey, h1iscey, h1odcy, h1alsey). We had to play around with the safe.snap files for the ISC and ALS models to get them to startup.

The receiving models were restarted (h1lsc, h1asc, h1susetmx, h1susetmy, h1hpietmx, h1hpietmy)

The IPC from the end station ISC models to the LSC are now without any errors. The ALS model still has some IPC to the LSC and this retains its error rate (this model was not substantially sped up).

The DAQ master file was reconfigured for the new model layout, and H1EDCU_DAQ.ini was resynced.

I checked the DAQ data for the PEM and found it was corrupted. This was tracked to the change of the ADC part name in the model when PEM and CAL were combined, and we forgot to remap the bus selector parts attached to the ADC. This was fixed, there is a two hour gap in PEM data.

CDS OVERVIEW MEDM

Dave:

The H1CDS_STATE_WORD_CUSTOM.adl overview MEDM screen was updated:

• model shuffle in end station ISC front ends
• addition of CFC (configuration file change) bit. A BLUE bit like the overflow.
• addition of the CONLOG section

Pump #8 back on line--FClara had to rewire the db9 end of the Control VOUT.

Pump #7 taken off line to check Accumulators--all good, none with low pressures, #7 back on line.

Added explicit ground to + & - power outputs of power supply.  All kinds of wires jiggled, poked, & twisted.  No change in the sensor noise.

7:10 Peter King and Richard McCarthy to PSL

8:06 Jim running measurements on ITMx ISI, Jeff K running measurements on HEPI pump servo

8:13 Cris and Karn to LVEA

8:21 Jim done with ITMx

8:31 Betsy to LVEA

8:41 Gerardo to LVEA

8:47 Jodi moving dessicant cabinet from high bay to VPW

8:48 Jim, Hugh, and JeffK to LVEA HEPI spring constant measurements

8:54 Corey to LVEA inventorying enclosures

9:14 Krishna was at EX from 7:30-9:00 working on BRS

9:16 Dave B starting ISC, ASC, and LSC work

9:19 Jodi done

9:40 Corey out of LVEA

9:45 Corey heading to EX and EY for enclosure inventory

9:45 Karen to MY

9:49 Jim, Hugh, and JeffK out of LVEA

9:49 reconnecting HEPI caused a short of pumps, Hugh swapping plugs while down

10:03 Gerardo done

10:03 Betsy done

10:19 Cris to MX

10:36 Betsy to LVEA

10:37 Richard and Peter out of PSL

10:39 Cyrus taking down Framewriter 1

10:40 Elli to EX, EY

10:41 Fil to EX

10:55 Cyrus done

11:03 Corey back from ends, heading to LVEA

11:06 Dave B restarting DAQ

11:13 Karen out of MY

12:05 Betsy and Corey done

13:11 John and Bubba to LVEA

13:33 John and Bubba done

J. Kissel, H. Radkins, J. Warner

To help better characterize and model the HEPI system (and to get some gratification out of using M*g = k*x), Hugh, Jim, and I measured the vertical spring rate / spring stiffness of several H1 HEPI systems in the corner station. 

Here're the results:
Chamber       Spring Rate (+/- Uncertainty) [N/m]
ITMX (BSC1)       7.52e6 (4.22e4)
ITMY (BSC3)       7.75e6 (8.09e4)
HAM4              7.55e6 (1.19e5)
HAM5              8.21e6 (3.23e5)

Method:
(1) Turn OFF HEPI and ISI feedback control for chamber (brought OFFLINE by guardian)
(2) Measure raw V1-V4 IPS signals (waiting for signals to settle enough that the one's digit is all that's changing, and then just caget the H1:HPI-${CHAMBER}_IPSINF_${DOF}_INMONs)
(3) Place 10 [kg] load on all four corners of the suspended payload (on the blue cross-beams directly vertical of the vertical actuators on the BSCs, and on the "do not step" cover for the vertical L4C, also directly vertical of the vertical actuators, on the HAMs)
(4) Measure raw V1-V4 IPS signals
(5) Remove masses
(6) Measure raw V1-V4 IPS signals
(7) Re-place masses
(8) Measure raw V1-V4 IPS signals
... repeat of all four chambers.
(9) "Offline" convert raw data to [m] of displacement, x,
    (a) convert raw IPS signals to [m], with 3.87e-08 [m/ct]
    (b) take average of V1 - V4 to get Z position
    (c) create three Z displacement measurement trials, (4) - (2), (6) - (4), and (8) - (6)
(10) Calculate spring rate for each trial, k = F / x, where F = m g, with m = 40 [kg] and g = 9.81 [m/s^2].
(11) Take the mean and std / sqrt(3) of three trials, to report spring rate mean and uncertainty on the mean in the above table.

Raw notes from data taking (includes load on each spring, which was also measured)
/ligo/svncommon/SeiSVN/seismic/HEPI/H1/Common/2015-02-03_HPI_SpringRates.txt


Script to process data:
/ligo/svncommon/SeiSVN/seismic/HEPI/H1/Common/HEPI_Spring_Rate_20150203.m

WP #5037

Dataviewer has been updated to version 2.9.1.2 for Ubuntu workstations in the control room.  This is to fix bugzilla 775, to prevent incorrect channel names from being entered by the user.  An incorrect string comparison of entered names to the channel list read from the NDS allowed certain bad channel names to be entered manually, and since data wasn't available for the incorrect channel, dataviewer would substitute different channel data, leading to confusion.

I want to get the HWS working.  This morning I installed HWS power cables on ISCTEY and ISCTEX, and plugged in the HWS fiber-optic cable at EY.  ETMY power cable was re-wired to match ETMx, as required.

It looks like everything is in place for ETMx HWS.  At EY, there are no cables running from the ISTEY feedthrough panel to the TCS rack.  I couldn't connect to the HWS computers from the control room, need to find out what state they are in.

(Peter K, Richard M, , Filiberto C, Daniel S)

We noticed that the fast channels corresponding to H1:IMC-PWR_IN and H1:IMC-PWR_EOM were never hooked up. This required installing the PD interface box in the PSL enclosure, running the DAQ cable into the PSL and installing a tee in the photodetector readback. (These channels have previously been hooked up to the EtherCAT system, so they that they are available to the rotation stage.) The EOM channel is currently railed and needs an adjustment of the PD gain.

EPICS updates:

• Added buttons for these channels to the IMC overview screen.
• Fixed the screen for the PSL rotation stage in the sitemap.
• Added rotation stage button to the IMC overview screen.

K. Venkateswara

In response to the problems reported in 16391, I centered the beam-balance in BRS within the nominal range of the angular readout. To do so, I physically moved a small mass on the balance using the bellows mounted on the front face of the vacuum can. After a few tries I was able to center it to within a ~1/3 of the full range. The balance was then damped to low amplitudes and I repositioned the damper.

BRS is now functioning correctly. The first attached plot shows the tilt-subtraction at work. The second shows the coherence between various sensors. The low coherence between the tilt-subtracted super-sensor and BRS RY shows the subtraction is working reasonably well.

Had to boot EX which seems to be a biweekly occurrence. The problem aren't the PLCs or the hardware interface but the communications. The tcioc stops updating with a "No DS mailbox available" error message. This is always preceded by numerous "CAS: UDP  recv error". So far, the only way to fix it is to reboot. I did notice that h1ecatx1 runs the Windows firewall whereas the c1 and y1 run a commercial product.

C. Reed, D. Moraru

In an effort to ameliorate the daqd crashes on framewriters 0 and 1, we have enabled jumbo frames on the private network links between the framewriters and the QFS gateways under WP 5035.  This was the last remaining major difference between our configuration and the LLO DAQ (that we know of).  If this doesn't help, we will need to keep looking for a cause for our framewriting woes.

Betsy, Travis

Because of new filters loaded by Rana last week, I wasn't sure there had been an update to the safe.snap file for the ITM and ETMs.  Since the IFO was down for Tuesday chaos, we therefore took new ones and committed them to svn.  While we were at it, we committed the safe.snaps for the 3 MC SUSes due to my addition of the SDF column to the files yesterday.

WP #5031

Dataviewer has been updated to version 2.9.1.1 for Ubuntu workstations in the control room.  This is to fix bugzilla 786, allowing dataviewer to display raw data of multiple signals in a single plot in playback mode.

• We edited the guardian so that it reliably gets us to a CARM offset of approximately 10 pm, (TR_CARM = 30)
• We are now using an OPLev servo at DC on PR3 to help with the alingment drifts we see everytime we get some decent power build up in the IFO. 
• We noticed that the SRCL and MICH gains needed adjsutment.  At a TR CARM offset of 30 we reduce the SRCL gain by about 50%, from -45 to -26, and increase the  MICH gain by a factor of 3 to 6
• We attempted to use the MICH WFS, they worked well without the arms, we used the highbandwidth configuration.  We tried the CARM offset reduction with these two loops on, the alingment indeed stayed better, but we also lost lock suddenly.  Elli saw in at least one of these lock losses that the MICH WFS were oscillating, so we are now leaving them off. It would probably be worthwhile to try the low bandwidth version. 
• We went to ISCT6, replaced the 2 0.5 ND filters with an absorptive ND 1.0 filter and checked the centering on ASAIR A. This turned out to be a different level of attenuation than we had with the 2 ND 0.5s so we changed the gain in ASAIR_A_RF45_I and Q to 5.4

We then made several attempts to transition off of TR_CARM.  As on friday, we are able to get to low CARM offsets, and sit there stably, but haven't been able to turn off TR_CARM yet.  We found that we could sit at about 5 pm (TR CARM = -39) stably with just TR_CARM on. 

Transition attempts:

At -36 in TR_CARM, engage analog REFLAIR 9 with 40 dB total gain on summing junction and CM board. 

At -36. engage REFLAIR9 with and offset of -1.8, tried turning off FM 9 (the complex pole compensation) in TR_CARM.  Lost lock at 7:23:45

We've had trouble engaging the DHARD WFS with the opposite sign at the small CARM offsets tonight, which on Friday seemed like it was key to being stable at low offsets.  

J. Kissel, H. Radkins,

In further pursuit of HEPI Pump Servo Noise -- found to be coherent with the IFO at low-frequencies (see G1500099, and LHO aLOGs 16386, 16342, and 16239) -- Hugh and I briefly terminated the pressure sensor inputs to the HEPI pump servo (see times in LHO aLOG 16419), in attempt to measure the ADC noise of the PC/104, Athena II, 16-bit ADC (see Diamond System's product website for data sheet and user manual). 

Sadly, there appears to be no difference between the channels when the sensors are plugged in and when the sensors are terminated.

What's confusing about this, is that we know the pressure preamp (D1101839, which adds a gain of 151 [V_diff/V_diff] to each supply and return sensor) and the input instrumentation amplifier inside the servo (pg 2 of D0901559, adding a gain of 2 [V_se / V_diff] for a total of 300 [V_se / V_diff]) are functional and calibrated correctly in the EPICs database. If that weren't true, the DC signal on each sensor's EPICs readback would not match the dial gauges on the system.

Further, as mentioned on pg 13 of G1500099, if the pressure sensor's calibration is 0.3 [mV/PSI] (0 to 100 [mV] over a 0 to 300 [PSI] range), typical supply pressure is 70-80 [PSI], then the pre-amplified, instrumentation-amplified, supply pressure signal reaching the Pump Servo ADC should be roughly 6.3-7.2 [V] single-ended, using plenty of the range of the 0-10 [V] ADC range. On the other hand, the return pressure, typically ~5 [PSI], would arrive at the ADC as only 0.45 [V]. Small, but still well above the 10 / 2^16 = 1.5e-4 [V/ct] resolution.

So, the DC signals add up and agree with dial gauges, the signals should fit well within the ADC's range well above it's bit resolution, and responds appropriately to little steps in the control pressure. So why don't we get a change in ASD between terminated and connected?

We'll continue to investigate tomorrow morning. 

I attach three things:
(1) 2015-02-02_H1HPI_PumpServo_ADCNoiseCharacterization.pdf 
The pudding. Amplitude spectral densities of the Supply Pressure (H1:HPI-PUMP_L0_BSC2SUP_PRESS), Return Pressure (H1:HPI-PUMP_L0_BSC2RET_PRESS), and the calculated Differential Pressure (H1:HPI-PUMP_L0_DIFF_PRESSURE) for three different times. RED is with the pump servo closed (the current, nominal running state), BLUE is the with the pump servo open, but the sensors still connected, and MAGENTA is with the pump servo open and the sensor inputs terminated. As you can see -- no change between BLUE and MAGENTA.

(2) 2015-02-02_HPI_PumpServo_Pics_A.pdf 
A collection of pictures of the HEPI Pump system, as built at LHO.

(3) 2015-02-02_HPI_PumpServo_Pics_B.pdf 
More pictures of the HEPI Pump system, including pictures of the terminating plug used and how / where it was connected (starting on pg 9).

Following the pin out shown in D0901559, where (on pg 1, upper left corner) connectors J1A and J1B are shown to bring in the field Supply and Return pressure signals, I crafted two, dummy, DB9, pressure sensor terminators by connecting the + and - legs (pins 4 and 6) of the differential input with a 100 [ohm] resistive load. All other pins (i.e. the +10 [V] supply voltage to the pressure sensor, and the +/- 15 [V] supply voltage to the pressure sensor preamp) were left open. Again, see pgs 9 and 10 of 2015-02-02_HPI_PumpServo_Pics_B.pdf.

On Saturday we closed the ISS second loop and dither-locked the OMC, with a single bounce from ITMX.  In this state we were able to measure the RIN at the OMC DCPDs with the 2nd loop open and closed.

With the ISS second loop closed, the relative intensity noise at the OMC DCPDs at 100Hz is about 5x10^-8 / rt[Hz].  Above 100Hz, the OMC DCPD RIN is limited by intensity noise; below 100Hz the noise is from something else.  See the first plot.  The OMC DCPD sum was ~14mA at the time of the measurement, and the shot noise RIN should be 5x10^-9/rt[Hz], well below the measured noise floor.

Compared to Gabriele's measurement from November, the noise with the ISS 2nd loop open is ~10x worse, compare the second plot, upper right, to his third plot, lower right & left.  Is the excess noise due to jitter into the IMC?  We did not take the time to fiddle with the IMC WFS offsets.  Gabriele measured 10^-8 RIN/rt[Hz] at 100Hz with the 2nd loop closed, but we measure 3x10^-8 /rt[Hz], so above 100Hz we may be limited by the gain of the ISS 2nd loop.  If this is the case, then improving the RIN out of the IMC could reduce the intensity noise at the AS port.

In the second plot, we compare some signals with the ISS loops open and closed.  The second loop suppresses the noise at the OMC DCPDs by about a factor of 100 (lower right plot).  Note that in the lower left plot, of the RIN observed by the IM4 trans PD, the ISS first loop only improves the noise at high frequency.

There is a peak in the intensity noise at 700Hz that is not suppressed by the second loop.  This also seems to be ~10x worse than what Gabriele measured in November.  It's not clear where the peak is coming from, it's seen by the IMC WFS (see third plot), but not by any PEM or IMC length channels that I can find.  This peak is broad enough that it may cause trouble with the OMC ASC dither lines.  (L1's dither frequencies are around 550-650Hz.)

The first attachment is a time series of the HAM6 Watchdog, the Fastshutter State, and a GS-13 trace on HAM6.  The switch to low Gain is clear on this last trace.  This is a 5 day trend and the ~20 minutes needed to get the below .005Hz BW data is comfortably unmolested by the ISI or the fastshutter--the start times of the Spectra calculations begin at the vertical black lines.  As for ISI trips and shutter fires--there were none.

The High Gain Spectra period begins at 0800utc on 29 Jan, the Low Gain Spectra begins at 0800utc 1 Feb.

The second attached graph has Horizontal GS13s on HAM6 in High and Low Gain on the upper plot.  The Verticals are in the lower plot. High Gain Spectra is Dashed, Low Gain traces are Solid.  Also shown is our ADC noise in counts.

One caveat is that the ground motion is different at the two times.  The third attachment shows the Cartesian DOFs for the HAM6 GS-13s in the upper two plots and the ground noise in the bottom plot.  We've had a ground motion decrease in the microseism frequencies and below during the low gain period.  You can see the Low Gain Spectra are at the ADC level below 50mHz.  And the 10x signal reduction expected is more like 5x below these frequencies.  If this conclusion passes mustard, we'll have to revisit the GS13 triggering delays/levels.

DTT template is /ligo/svncommon/SeiSVN/seismic/Common/MatlabTools/HAM6_GS13_GainNoiseTest.xml

J. Kissel, J. Warner, D. Barker, E. Merilh, TJ Shaffer

The one final quiver in our arrow against the sharp HAM3 0.6 [Hz] resonant feature was a computer (as suggested e.g. here), so we did so this morning. Sadly, it did not cure the problem. For now, we continue on, running in the configuration defined by Seb last week, see LHO aLOG 16100, in which the RY blend filters at a higher blend configuration, which is the only thing we can find that seems to help (see LHO aLOG 16001). This has now officially become a low-priority until it begins to quantifiably impact IFO performance.

A Recap of all of the things we've tried that DIDN'T make the feature go away:
- Restarting the front-end process LHO aLOG 15759

- Full power-down of Front-end and I/O Chassis (This aLOG)

- Coil Driver Swap LHO aLOG 16071, LHO aLOG 16061, LHO aLOG 15981

- Changing Suspension Damping LHO aLOG 16013, LHO aLOG 16001

- Changing Isolation Loops LHO aLOG 16001

- Location of Sensor Correction; ISI vs HPI LHO aLOG 15941. LHO aLOG 15933

- Changing from STS2 B to A; In analog LHO aLOG 15811, In digital LHO aLOG 15927, LHO aLOG 15894

- Checking between ADC and sensor correction bank LHO aLOG 15783

- Turning OFF CPS satellite amplifiers, checking jumper settings (oscillators, jumpers, etc.) LHO aLOG 15751

- Adding a large mechanical offset to relieve supposed mechanical rubbing LHO aLOG 15751

- Waiting LHO aLOG 15565