WP6287 Add PEM ADC card to h1oaf0

Jim, Dave, commissioning

this turned into a large task. Please see alog 31316 for details. Currently the ADC card is in the chassis and we are not seeing any errors. We will monitor for any recurrence of ADC/DAC errors on this system, or nf_conntrack table errors.

WP6295 Add PEM-EX chans to DAQ for ESD-PS monitoring

Ansel, Robert, Dave

h1pemex was modified to add three test ADC channels to the DAQ at 2048Hz.

WP6303 Run later version of Frame-cpp on DAQ FW2

Jonathan,

as a test of using frame-cpp version 2.5.2 in the DAQ, the new version is being tested on h1fw2. This is the latest version supported by Ubuntu-12. It does have the leap-second modification needed for 31Dec2016.

WP6307 Add BRS Beckhoff computers to CDS KVM

Carlos

The new Beckhoff branded BRS computers at EX and EY were added to the KVM so we don't need to support an additional monitor, keyboard and mouse. These computers do not have an on/off switch, so we are unable to stop them booting up when power is restored after an outage.

WP6306 Update Hartmann Wavefront Sensor channels into DAQ

Kiwamu, Nutsinee, Dave

I have updated the H1EDCU_HWS.ini file now all four hws systems are running the new python code. Starting with the EPICS database file, I made a list of non-string channels (i.e. binary and analog channels).

Interestingly ITMX was found to be missing 14 channels which appear in the other systems. These channels have ITMM in their names. After discussion with Kiwamu and Nutsinee we decided to remove these from the ini file for now.

DAQ Restart

Jim

DAQ was restarted to: add ADC to h1iopoaf0, add chans to h1pemex, add new H1EDCU_HWS.ini.

Mounted scroll pump to Turbo stand at X-end VEA but didn't run pumps as specified on WP due to lack of NW16-25 adapter.  I'll run the pumps and test the interlock next maintenance day.

Dave reported that since h1oaf continued to have issues this morning after some hardware card work, the "fix" he needed to do may or may not cause a dolphin network glitch similar to the events that occurred last Tuesday.  Sooo, before he started this "fix", Hugh, Kissel, Sheila and I reconciled all SDF diffs in the down and safe.snap files.  We captured many changes made by the commissioning team last night.  Hopefully 1) the dolphin glitch won't actually happen and or 2) the SDF cleanup will help with recovering the nice lock from last night.  In any case, it forced us to capture the latest and greatest commissioning settings, and a bunch of other mystery settings which have accumulated, so it will be less painful next time.

With the higher sideband power we were limited to ~2 kHz ugf in the PMC with the lowest gain settings. Today, we changed resistor R19 to 4.75K from 1.21K which reduced the electronics gain by about 2.

While having the servo board in the shop we check R16/R17 which are 9.09K/1.00K rather than the 19.6K/1.00K indicated in the schematics. This explains the gain mystery described in alog 31120. The calibration of the slider has been changed is now accurately going from 0dB to 40dB.

We measured the transfer function with 0 dB, 12 dB and 22 dB of gain. The servo goes unstable at 25 dB, so 22 dB should be the maximum used. The ugf was around 1.1 kHz, 3.2 kHz and 9.4 kHz, respectively.

The reason we are not able to go below 1 kHz ugf is todays PSL change which improved the mode matching into the PMC and increased the optical gain (maybe by a factor of 2).

Our new Inficon Bayard Alpert vacuum gauges have a drift issue. The plot shows three gauges in the corner; PT170, PT180 vs PT120 (cold cathode). The cold cathode is trending flat or down while the two inficons are trending up.

The two inficons located in the two end station are flat, however.

Perhaps the upward drift on the two LVEA gauges is due to some "wet" hardware the gauge was mounted with??Gerardo plans to talk with technical folks at Inficon. The LVEA gauges are mounted on BSC 7, BSC8 and BSC2(PT120 cold cathode)

WP6287 Add PEM ADC to h1oaf0, reconfigure h1iopoaf0 to read new ADC

Jim installed a 7th ADC into the h1oaf0 IO Chassis this morning. On power up, the IOP model did not start well and reported ADC and DAC errors.

Since we were to restart the h1iopoaf0 model to clear the DAC errors, we installed the new code which reads the new ADC.

After running for about an hour, the h1oaf0 machine stopped making new network connections with the console error

nf_conntrack: table full, dropping packet

repeating frequently. On h1lsc0 we issued the command to remotely take h1oaf0 out of the corner station dolphin fabric, and on h10af0's console we issued the command for it to prepare-shutdown from the fabric. With no connection between h1oaf0 and h1boot (dolphin master) we had little confidence we could reboot h1oaf0 without glitching most of the corner staiton.

Researching the error, we found that it is possible to expand the netfilter connection tracking table size on-the fly with the command

echo 256000 > /proc/sys/net/netfilter/nf_conntrack_max

(the max is at the default of 65536).

This cleared the error and new MEDM's and Guardian could establish CA links, and we could SSH onto the machine. At this point we again issued the dolphin prepare shutdown command with more confidence that it was successful, but there is still a chance of corner station crash.

We will make the larger table size the default by creating the file on the boot server (h1boot)

/diskless/root/etc/sysctl.conf

with one line

net.netfilter.nf_conntrack_max = 256000

will test this on the reboot of h1oaf0 (waiting for a good time if a CS crash is precipitated)

Note that around the time of the nf_conntrack errors the IOP reported ADC and DAC errors. It is still a possibility that the new ADC was the  cause of  these errors and may be removed it another error is seen.

The online h(t) pipeline has been restarted at LHO, following a software upgrade to gstlal-calibration (see https://versions.ligo.org/cgit/gstlal/tag/?h=gstlal-calibration-1.0.7-v1 and Redmine issue https://bugs.ligo.org/redmine/issues/4813#change-20027). We are applying a coherence uncertainty threshold of 0.004 for the online calculation of kappas, which is consistent with behavior observed during recent lock stretches (see the figure attached).

This update adds six new channels containing kappa values not gated by the calib_state_vector or by coherence uncertainties:

Note, the filters have not been updated for this pipeline restart. We are still using the filters documented in this aLOG: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=30210.

	[IFO]:GDS-CALIB_KAPPA_A_REAL
        [IFO]:GDS-CALIB_KAPPA_A_IMAGINARY

	[IFO]:GDS-CALIB_KAPPA_A_REAL
        [IFO]:GDS-CALIB_KAPPA_A_IMAGINARY

The work in alog 31308 (which involved an h1oaf restart) took the TCS chillers down, which subsequently took the lasers down.  Richard/Gerardo restarted the chillers and the lasers (~9:30am PT).  After a bad restart of h1oaf which caused some confusion, the 2nd or 3rd restart was finally successful and the FE servos took over to adjust the laser temps/powers to nominal.   Here's the notes we gathered from today which we will write in an ops wiki:

If h1oaf gets booted and the chillers go down, you only need to restart the chillers at the front panel and the lasers at the racks on the floor.  The TCS settings come back up properly with the safe.snap.  If the front end is not up yet, the SDF may show some stale setting diffs.  Ignore anything in SDF until the FE is back up  - most of these diffs will take care of themself as many are servos that do automatic adjustments.

Doh - 10:52am PT - h1oaf "just died" - CDS working on it...

As of 10:40PT, 18:40UTC

Work going on

• psl alignment
• pcal calibration
• charge measurements
• TCS electronics upgrade
• SEI EX deisolated/isolated
• Frame Writer 2
• LVEA 24V cabling
• OpLev recentering
• Two LN2 deliveries
• DMT update

Still to come:

• DAQ restart needed

I have rebuilt daqd fw on h1fw2 using a new ldas-tools build (2.5.2).  This is the version in use at LLO and contains updated leap second tables.

The plan is to compare the output from h1fw2 to h1fw0 and h1fw1 for a week and then migrate h1fw0 and h1fw1 to use the newer framecpp releases next week.

In the mean time the daq status page will show differences in the frames between h1fw2 and the other frame writers. This is due to framecpp encoding a different version string in the frame file.  Dave will update the daq overview screen to help minimize the differences.

This is under workpermit 6303.

Trended LVEA pressure gauge readings over 60 days. We have seen a trend upward in the two new hot cathode ion gauges PT 170 & PT 180 on BSC 7 & 8. Last Tuesday we degassed both to see if that helps stabilize the readings (cold cathode gauges, PT 120B, 114B, 134B, readings in corner are either flat or trend down!). 

PT 140 is the one and only gauge measuring diagonal volume, and is also a new hot cathode ion gauge. It was reading pretty flat, until Oct. 10th when the temperature in LVEA was increased at 2B area (west corner) by around 1/2 degF.

PT110 is a new hot cathode gauge on HAM 6.

WP 6287

Installed additional ADC in h1oaf0 I/O chassis, bus slot 2-8.  

8:05 PST powered down h1oaf0 computer, I/O chassis.  Installed new ADC and interface card. Powered down TCS AI chassis, powered up I/O chassis. Remotely started h1oaf0 computer, verified card was detected, powered TCS AI chassis back on.  Out of CER at 8:25.

I have switched chillers at the corner station from Chiller 1 to Chiller 2. This is simply to try and distribute the run time somewhat evenly between the three chillers.
Temperatures should remain the same.  

Nutsinee, Stefan

As suggested by alog 31304, we set up a 26:74 mixed input matrix element for PRCL in the next lock:

POP_A_RF9_I to PRCL = 0.0091
REFL_A_RF45_I to PRCL = -849 (yes, a minus sign)

This gave us the expected additionbal slight improvement in PRCL sensing, see the attached plot.

Indeed, this auxiliary noise reduction seems to have given us a range increase of about 5% to 75Mpc.
More importantly though, suddenly the range seemed to be rock-sold, without any up- and down drifts - see the last hour in the attached range plot.

Still to do: PRCL FF & possible modulation depth increase.
After this first step the return will not be as big in terms of range, but the improvement in range stability by itself makes it worth the effort.

We put all this into Guardian (NOISE_TUNINGS), but we didn't test the Guardian yet (the interferometer was too steady).

J. Kissel

I've taken the first round of all swept sine measurements typically done before an observing run:
(1) Sensing function, C
    (a) DARMOLGTF and Loop Suppression (IN1/IN2 and IN2/EXC); G and 1/(1+G)
    (b) PCAL2DARM (and PCAL2DELTAL); (C/1+G) and [m/m]
(2) Actuation Function, A
    (a) L1 to DARM & associated PCAL2DARM; (C A_uim)/(1+G) and (C/1+G)
    (b) L2 to DARM & associated PCAL2DARM; (C A_pum)/(1+G) and (C/1+G)
    (c) L3 to DARM & associated PCAL2DARM; (C A_tst)/(1+G) and (C/1+G)
where I've taken a separate PCAL2DARM or C/1+G transfer function with every corresponding excitation to destroy any covariance that had plagued us during O1. Also, I've worked to
    (i) reduce the template's measurement time by maximizing drive, 
    (ii) reducing the number of points over all, and
    (iii) keeping integration time in check,
such that each template only takes about 15 minutes. I've also modified the frequency vectors of the actuation functions to go out to 1 kHz, as LLO did for their O1 data sets. In doing so, I made sure to use frequency points that avoided any known high-frequency resonances from wire violin modes (characterized in LHO aLOG 24917). 

There's still some improvements to make, which I'll shoot for in the next round (so don't use/take these templates as cannon yet), but thanks to the high-duty cycle of the IFO, I had the opportunity to make really well-balanced templates tonight.

The data lives here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER10/H1/Measurements/
(1a) DARMOLGTFs/2016-11-07_H1_DARM_OLGTF_4to1200Hz_fasttemplate.xml
(1b) PCAL/2016-11-07_H1_PCAL2DARMTF_4to1200Hz_fasttemplate.xml

(2)
FullIFOActuatorTFs/2016-11-07/2016-11-07_H1SUSETMY_L1_iEXC2DARM.xml
FullIFOActuatorTFs/2016-11-07/2016-11-07_H1SUSETMY_L1_PCAL2DARM.xml
FullIFOActuatorTFs/2016-11-07/2016-11-07_H1SUSETMY_L2_iEXC2DARM.xml
FullIFOActuatorTFs/2016-11-07/2016-11-07_H1SUSETMY_L2_PCAL2DARM.xml
FullIFOActuatorTFs/2016-11-07/2016-11-07_H1SUSETMY_L3_iEXC2DARM.xml
FullIFOActuatorTFs/2016-11-07/2016-11-07_H1SUSETMY_L3_PCAL2DARM.xml

While it'll be days before we have a model and a complete, traceable uncertainty budget, preliminary results show the *mean* of *one measurement* of PCAL's estimated displacement agrees with the front-end calibrated sensitivity to roughly 5% and 5 [deg]. This confirms that the > 50 Hz discrepancies between PCAL and DELTA L EXTERNAL seen while we were at 50 [W] (e.g. LHO aLOGs 30391 and 30431) were likely a function of the poorly controlled signal recycling cavity, and therefore time-dependent RSE cavity pole that was quite low.
Recall that the uncertainty of the PCAL estimation is around 1%. 

We're right on schedule, and things are looking up. We'll see how the modeling goes...

The PMC length noise no longer shows up in DARM. Even for the 1 kHz and 4 kHz peaks there is a factor of a few safety margin. Increasing the PMC gain slider from 0 dB to 16 dB (actually from 0 dB to 12 dB), we see no significant change in DARM. The 4 kHz peak starts touching the noise floor with a 0.3 coherence.