fix h1susb2h34 DAC MEDM and verify correct drive of DACs

WP13479 Move last ADC in h1susb2h34 in front of the LIGO-DACs

Erik, Jonathan, EJ, TJ, Dave:

The 3rd ADC, which was after the LIGO-DACs on the PCI bus, was moved to be before the LIGO-DACs. EJ found that if a General Standards card followed the LIGO DACs, and there are other General Standard DACs in the chassis, then the DAC mapping became ambiguous. This was first discovered by TJ when he noticed that the LIGO-DAC card temperatures was being reported as -1C (not being read). We then found that model's LIGO-DAC drive counts were being reported by h1iopsusb2h34 on the wrong DAC MEDM.

A quick fix as to move the ADC inside the IO Chassis from after to before the LIGO-DACs

The original card layout (BIO not shown):

empty

empty

empty

***ADC2***

LIGO-DAC1

LIGO-DAC0

20bit-DAC1

ADC1

20bit-DAC0

ADC0

empty

Timing

A3-4

A3-3

A3-2

A3-1

A2-4

A2-3

A2-2

A2-1

A1-4

A1-3

A1-2

A1-1

The new card layout (BIO not shown):

empty

empty

LIGO_DAC1

LIGO-DAC0

empty

***ADC2***

20bit-DAC1

ADC1

20bit-DAC0

ADC0

empty

Timing

A3-4

A3-3

A3-2

A3-1

A2-4

A2-3

A2-2

A2-1

A1-4

A1-3

A1-2

A1-1

 

I was able to shuffle the cards without disconnecting any rear cables except ADC2 (SUS_HAM_209) and the MTP. The cables had enough slack to pull the chassis out to access the card screws. At the rear I was able to push the h1susauxb2h34 chassis forward to access the interface cards.

Internally only ADC2's ribbon cable needed to be disconnected.

TJ methodically started driving the top stages for MC2, PR2 and BS. We verified the model's and IOP's MEDM reported the correct channels being driven.

I trended the ADC2 channels from before and after its move to verify it is connected correctly.

DIAG_MAIN whitening checks updated

Keita pointed out that the normal DIAG_MAIN whitening check doesn't currently cover the OMC DCPD external analog whitening (state of which can be seen on H1:OMC-DCPD_{A,B}_GAINVAL, 0=high 1=low), and the corresponding digital antiwhitening for it at FM2 on H1:OMC-DCPD_{A,B}0. This is now in DIAG_MAIN's whitening test.

While I was looking at this test, I noticed it was quite out of date with the whitening overview screen.  I pull all of the names from that screen and made a new list for the DAIG_MAIN test. There were only additions from the old list, so I don't think we dropped any sensors. I commented out some BHD sensors until they are fully ready. 

DAS system, tap testing to calibrate distances along fiber

Jenne, Wanda, Shoshana, Gizem

We started very early this morning to beat the heat, and performed 'tap tests' at various points along the length of the arms to identify positions along the length of the fiber (according to DAS) with GPS positions (according to Wanda's phone).

Spreadsheet attachment is information about where along the fiber Shoshana and Gizem identified our tapping in the data, and associated time stamps.  I've also copied the spreadsheet data to a google sheet, so that we can add to it.  

We tried to do more dense taps on the 'inside of the vertex' side of the arms, where the fiber actually is.  This required some walking out in the sand along the arms, so we had sun hats and knee-high boots and plenty of water.   We also did a few taps along the road during our drive back, to get some more coarse information along the length of the arms.  At the Xend station, there were too many tumbleweeds for us to walk along the inside-vertex side of the arm, so there at EX we did our more dense taps along the road side, with the exception of the EX vault location. 

In the attached photos, you can see some examples of our tap test method.  Gerardo provided us with a scrap piece of steel that we used as a strike plate.  We also borrowed a sledgehammer from the mechanical / vacuum parts lab.  For most of the Yarm, we only did 3 heavy strikes and Gizem and Shoshana chose the strongest signal from that as our time.  Once we were on the road-side of Yarm, and then for all of Xarm, we added some light tapping that seems to have helped identify which distance bin (called a "channel" in DAS terminology) we were closest to.  The photos are also labeled with X/Yarm, and position number, which corresponds to the position numbers in the google sheet - I tried to capture features in the photo to help identify where exactly the strike plate was.

Wanda has added to the google sheet the GPS locations that she identified while we were at each tap point.

The last photo shows our used (but can be used again in the future!) strike plate. 

OPLEV charge measurement, ETMX & ETMY

I ran the charge measurements for both ETMs this morning. Unfortunately a medium (5.2 from Mexico) earthquake rumbled through from 16:38 to 16:58 UTC, but it didn't seem to effect the errors and coherences noticeably enough for me to restart the measurement.

For ETMX:

I was consistently getting low coherence for LL in Yaw, the charge appears to have plateaued or seen small increases (the error bars from the last measurement overlap with this one on all DOF/quadrants) on most DOFs/quadrants. The charge is above +\-50[V] on UR_P and LR_P, LR is still slowly trending to zero. ETMX is also quite misaligned as seen by its OPLEV.

For ETMY:

I was also seeing the worst coherence and largest errors on LL, both P and Y. The charge appears stable, there's not much difference from this measurement to the second to last (most recent one has large errors so it's not as valuable of a comparison). The charge is above +\-50[V] on LL P and Y, and UL_Y, there were some small increase of charge on LL as well.

Distributed acoustic sensing (DAS) setup, day 4

Late post for DAS setup work on Thursday yesterday.  Jenne, Wanda, Gizem, with Jonathan and Nyath helping get our remote access set up.

* EX, Fil connected a patch cable between fiber chassis and our termination fiber.  This patch cable was FC-APC on the end that was connected to the fiber chassis, and FC-UPC connected to the FC-UPC termination cable.  We were trying this mismatch between the FC-UPC of the chassis to a patch cable, in hopes that that would reduce any etalon effect between the UPC-UPC connectors.  It didn't seem to help much.

* Xarm, Did a test for a while where  Fil connected both our DAS channels to 2 fibers (#23 and #24) on the Xarm, to see if that made any difference in the noise; no impact, so eventually went back to just using #24 on X and #24 on Y.

Eventually we realized that the reason the data looked noisier is that we had changed some acquisition settings (effectively, changing our averaging).  So, nothing actually changed in the overall performance of the system, which is good (we had thought that somehow we made it much worse between Mon and Wed).  

We started an overnight data set using 500 Hz sample rate, to see that hopefully this is an okay balance between big file sizes and going up to high enough frequency to be useful.  Xarm was connected to the Febus instrument, and Yarm to the Sintela instrument.

Also in the afternoon we got the remote connectivity working for both the Sintela and the Febus instruments.  

• Febus is the IP adress that ends in xxx.xxx.xxx.xx3
• Sintela is the IP address that ends in xxx.xxx.xxx.xx4

 

h1sush12 IPC ethernet fiber and switch SFP swap

Jonathan, EJ, Erik, Dave:

The first step in fixing h1sush12's IPC errors is to replace the fiber and switch SFP. I've put SEI SWWD for HAM1 and HAM2 into bypass mode in preparation for this.

Jonathan will perform the swap in the MSR.

This will be a hot-swap, no models need to be fenced, stopped or computers rebooted. Very different from the "Dolphin crash" days.

h1sush12 low level send errors following the link restart Thursday 16:55

At 16:55 the IPC-ETH link on h1sush12 went offline for about 5 seconds, stopping all IPC traffic into and out of this front end. Since that time we have had a low rate of send errors from models running on h1sush12, but no receive errors.

Two models on h1sush12 send IPCs to remote models:
 

chan	send model	rate	receive model
H1:IOP-SUS_H12_RM1RM2PM1_WDIPC	h1iopsush12	2k	h1iopseih16
H1:IOP-SUS_H12_MC1MC3PRMPR3_WDIPC	h1iopsush12	2k	h1iopseih23
H1:SUS-PRM_M1_LOCK_L	h1susprm	16k	h1seiproc

Note that the SWWD IPC send at 2k even though the models run at 64k.

The receivers on h1iopseih16, h1iopseih23 and h1seiproc reported a low level of receive errors overnight. Each time these were one error in the second. Most of the time the IOP error times were not the same as the PRM times, except for the last one (see trend)

In the past 14 hours, IOP had 5 errors and PRM 9 errors.

EJ recommends as a first step to replace h1sush12's ethernet fiber and the switch SFP (not replacing h12 Mellanox SFP at this time).

Ops Day Shift Start

TITLE: 07/31 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 0mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s 
QUICK SUMMARY: SEI BS is in ISI_DAMPED_HEPI_OFFLINE, CDS overview is Green, Temps and dust look good.

h1sush12 IPC ethernet link reset itself

At 16:55 h1sush12's IPC ethernet port reset itself, taking all IPC to and from h1sush12 down for 5 seconds.

No in rack work was ongoing in the MSR at the time.

Thurs Ops Eve report.

TITLE: 07/31 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:
ITMX ISI stage 1 & 2 WD tripped and there seems to be some ground motion there that may have tripped them.

But The BS HEPI and ISI seems to be fine as of writing!

WP 13415 migrate IOCs to newer hosts

WP 13415.  I migrated the EX, EY, and Mid Y weather station from h0epics to the new container system.  This just gets us closer to turning off h0epics.

Thursday Ops Eve* report.

TITLE: 07/30 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 15mph Gusts, 9mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.09 μm/s 
QUICK SUMMARY:
Not much going on today as the CDS team was working on the file system today.

Testing WFS on JAC table

Jennie W, Khanh V,

 

Since Keita and I realised that the WFS A and B quadrants 2 and 3 are swapped between the RF channels and the DC channels, I have been trying to trace down where the error is. Last week we fixed the problem by swapping the cables for WFSA segment  2 and 3 and WFS B segment 2 and 3 cables at the IOT1 feedthrough panel.

I don't think it is in the RF signal chain as when we unplugged WFS HF segment 2 from the outside of the feedthrough, the channel that corresponds to WFS segment 2 went dark in EPICS.

Since we are still laser SAFE in the corner, Khanh and I took a laser pointer onto the JAC table and tried to see if we could only shine it on one quadrant of the QPD. Since the QPD is as large as the beam this was not really practical. We instead moved a beam card in front of the PD and slowly drew it downwards.

Since quadrants 2 and 3 are the two we suspect are swapped, we looked at which of these showed some light first.

We used the response to the table lights being blocked with the card to see if there was a difference in the readout from each QPD segment. Using the laser point for this did not work as the beam reflects off the card and the QPD casing.

Segment 3 showed light before segment 2 when moving the card down from the top of the diode.This implies the readout channels for 2 and 3 are swapped. The step in power was not as obvious as with a laser beam so I would like to recheck this once we go laser hazard.

Khanh and I also took the side panel off the table and traced the individual WFS cables from the QPD boxes to the feedthrough. These cable are all plugged in correctly on the inside of the feedthrough.

Next step is to get Fil's help to either check the DC pin outs on the WFS boxes or check the channels on the RF PD chassis at ISC-R1.

Summary: We think the JAC WFS QPDs have two segments wired incorrectly but only on the DC readouts, not the RF readouts.

HAM2 Annulus Ion Pump Work

(Travis S., Jordan V., Gerardo M.)
Today we replaced the ion pump for HAM2, but since the annulus system is shared with HAM1, both annulus ion pumps were powered off (HAM1 and HAM2), and the entire annulus system was vented with nitrogen gas. The AIP for HAM2 was removed, along with an elbow, the elbow was replaced with an isolation valve, we used a second hand O-ring valve because the new valve turned out to be short by 1".  After torquing all bolts on the ion pump and on the "new valve", the system was pumped down by a can turbo at the isolation valve and backed by an aux-cart, no issues pumping the annulus system down, since the aux-cart gauge already reports a vacuum pressure of 4.5X10^-05 Torr.  BTW, the can turbo at the top, flex hose and aux-cart will remain pumping on the annulus system until good vacuum pressure is achieved, and they will be a noise source.
Note for future work here: The pipes that make up this annulus system does not allow for an easy installation, during installation a person has to lift up on the pipes, while a second person pulls outward on the pipe to give space at the conflats to insert the copper gasket.  The tension on the pipes is something that we noted during the removal of the old ion pump, when the conflats were being unbolted, the gap between both increased on its own, showing us how much the pipes were flexed upward.

extremely large coil output in the user model will flip the DAC output from positive to negative rail

This doesn't seem to be related to our problem of locking IMC, but anyway I found a funny DAC behavior for MC2 M3 DAC.

See attached, I'm giving a huge offset to the coil output filter so the coil master output (bottom left) becomes larger than 2**27~134million counts most of the time. The user model DAC output (pink on the bottom right) rails as the master out increases, and so does the IOP DAC output (green on the bottom right), but when the master output goes larger than about 2 billion, the DAC outputs flip the sign and so do VOLTMON and FAST_IMON.

If you look at the transition at around 2 billion master out (between two markers), there seems to be a range where the IOP DAC flips the sign while the user model DAC doesn't, and the IMON as well as VOLTMON drops to zero. Here the driver voltage is not following IOP DAC.