J. Kissel, R. Mittleman

While at the X-end investigating the HPI pump failure, WE remembered to put the jumper back. This is inside the Corner 1 T240 Interface chassis, S1201388 in U21. I attach a few pictures.

Pg 1 shows the internal board and the jumper in the upper left corner marked GND Shield. It's engaged in the picture.
Pg 2 shows the rack number
Pg 3 shows the serial number

We experienced a power glitch at Jun 12 2014 01:08:29 UTC (18:08PDT). The lights in the control room flickered for several seconds, the MSR UPS reported a switch to battery and back. Vacuum controls was not affected. Here is what I have found:

All models on h1lsc0 and h1asc0 stopped running

Both mid station PEM systems have DAQ timing errors

Right hand monitors for workstations opsws0,1,2 went blank

I'm going to restart the affected models. The workstations were power cycled.

R. Mittleman, J. Kissel

Rich found H1 ETMX HEPI unable to drive, and after a little bit of investigation, he found the "Pressure OK" indicator on the pump controller screen indicating badness. We drive to the end station and did the following:
- Confirmed that we heard the pumps not running.
- Found the "Heartbeat" LED blinking green, the "Level Alarm" LED solid red, and no light from the "Pressure OK" LED.
- Confirmed that the power supply was feeding the pump servo box +/-15 [V] (as indicated on the box itself)
- Confirmed with a DVM that power cable from the power supply at the pump servo box was outputing +/-15 [V]
- Disconnected and reconnected the power cable to the pump servo. No change, but all LEDs turned off.
- When into the pump controller box, which sounded as though it was whirring with power, and the digital readout said "OU3," which we presumed to mean "0.03"
- Rich hit the "RESET" button on the control panel, then hit "FWD," and the digital readout went to 0, but no action from the pump.
- Of the 4 The "OUT" spigots on the pump servo box, #1 was is sent to the pump controller, and all four spigots were sending out ~0.05 [V] which is ~0.
- With power on, the pump servo box's fan did spin up.
- Just to check, with everything re-powered on, we physically lowered the fluid level threshold a smidge, but this also failed to resurrect the pump.

We conclude that the pump servo box has let out its smoke in some unknown fashion. The serial number of the box is S1301037.

Once we got back to the control room, we found the pump controller screen white as well...

Pictures of the various things attached.
Pg 1 -- Fluid Level indicator. Looks like plenty of fluid.
Pg 2 -- Back of pump servo showing heartbeat, level alarm, pressure OK LEDs (off at this point)
Pg 3 -- Fluid level fluid lever threshold that was lower a smidge
Pg 4 -- display and buttons of control panel
Pg 5 -- front of pump servo box
Pg 6 -- Close up of serial number

J. Kissel, F. Clara

Upon hearing news that LLO suspects their recent discovery of excess noise in their ESD signal chain (see LLO aLOG 13017 and 13034) is due to an older revision (V4) of AI chassis not being able to drive long signal chains (where the unique), Fil and I checked what documentation exists for ours. The serial numbers and version for our ESD AI chassis are as follows:

Optic   Chassis S/N       Board(s) S/N      Version 
ETMY    S1103819          S1101716            V4
                          S1101720            V4
ETMY    S1103818          S1101884            V4
                          S1101890            V4


So we'll need to swap these out for V6 at the earlier.

Per bugzilla integration issue 463, looked at AA/AI chassis to verify units have the proper insulating film used to isolate the -15V regulators to the chassis metal wall. 

All ISC AA/AI chassis in the CER have been modified or verified to have the appropriate insulating film.

S1102793 S1102790 S1102782 S1102794 S1102770 S1102786 S1102766 S1102789 S1102759 S1102760 S1102788 S1102784 S1301308 S1102761

All electronics have been reconnected and powered up. All units were placed back in appropriate rack and slot location.

Filiberto / Aaron / Vanessa

Day Shift Summary
LVEA Laser Safe 

08:30 Corey – Going to End-X Lab
08:50 Jodi – Working on 3IFO hardware near the H2 enclosure
09:00 Filiberto & Aaron – Working on AA/AI chassis repair in H1 electronics room
09:26 Robert & John – Checking high dust counts at HAM6
09:30 Ski and Mount Lock & Key  - Checking emergency egress doors at Mid-X
09:35 Keita – At HAM6 testing for ground loops
10:05 Karen – Going to End-Y
10:15 Rick, Peter, Olli, & Patrick – Working in H1-PSL enclosure
10:38 Jim – Going to End-Y to swap WD chassis
10:44 SUS crew – Working on ITM-Y swap at BSC1
11:04 Aaron – Going to End-Y to work on AA/AI chassis repair
11:24 Filiberto – Working in BSC3 on cabling problem
13:04 Mitch – In LVEA west bay looking for parts
13:12 Gerardo – Working CPB assembly
15:00 Safety meeting in LSB auditorium  

Corey, Keita

We were able to get rid of a ground loop on the "solo" QPD by simply moving/touching it's Seismically Responsible Cable.

Moved to Tip Tilts and discovered that two of the three Tip Tilts had grounding loops to the table.  After much investigation, Keita appeared to discover the issue.  It's related to the BOSEMS and how the connector is close to the aluminum mounting plate which holds them.  He was able to do some "surgery" to move the connectors in a way such that they didn't short to this plate.  One Tip Tilt (OM3) was repaired, re-installed, and tested good.  OM2 still needs to be fixed.

There were three runs of cables (the two WFS cables which run to Flange D3 & the OM2 cable) which were run "under" Stage1.  According to Hugh, this is a no-no, so these cables re-run and routed around the perimeter of Stage0.

Believe Keita still had more ground loop issues on his list, and he also wanted to look at the capacitance of the OMC PZT.

Travis, Betsy, Gary, Margot, Apollo

After Apollo finished mounting the arm, elevator and 5-axis lift table on the BSC 1 flange first thing this morning, the ITMy was deinstalled from the chamber.

Recall from my alog yesterday, https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=12292 that we had a broken receptacle 5 on the ESD connector connected to the CP optic which we made a repair on.  Today, Filiberto and I tested each of the 5 ESD connections from the CP solder joints on the gold traces to the air side of the feed thru at the chamber flange.  After another minor fix to a pin, we can

1) confirm continuity from the optic connector to the air feed thru on all 5 ESD channels

and

2) confirm that the test method is quite painless considering it is at the optic (likely Filiberto had the harder job juggling meters up on a ladder in cramped quarters outside the chamber).

Details:

Yesterday we noticed receptacle 5 broken, and we were able to pull it through and pop it into place.  During that fix, we realigned some of the other pins that looked to be buckled or too tight a little.  I think we over-open receptacle 3, which then made it fail the continuity test today.  Closing receptacle 3 down and reseating the connector made all 5 continuity tests pass.  I touched one end of a piece of cleaned copper wire to each ESD solder joint on top of the optic while holding the other end to the metal structure.  Simultaneously, FIl looked at each of the 5 pins outside to find the one with continuity.

Here's the mapping, which conforms to Kissel's G1400578 report:

LR -> Pin 1

UR -> Pin 2

DC Bias -> Pin 3

UL -> Pin 4

LL -> Pin 5

ESD LL, UR, LL, UL Quadrants are specified as viewed from the back of the CP.

ESD pins are specified as viewed from the 5 pin feedthru connector on the outside of the chamber.

As a first level of troubleshooting communication problems with the hardware watchdog chassis at EY, I replaced the EY chassis with a known good chassis from the DTS.  I then tested the (former EY) chassis in the DTS, it works properly.

Presently, hardware watchdog chassis S1301709 (formerly from DTS) is installed at EY, while hardware watchdog chassis S1301712 (formerly from EY) is installed in the DTS.

Given that the S1301712 chassis communicates well in the DTS, I don't expect that this solved the problem at EY.

Apollo – Move the installation arm from BSC3 to BSC1
Apollo – Remove the spool on the X-Arm for alignment
SUS crew – Getting ready to remove ITM-Y lower structure
Keita & Corey – Working in HAM6
Gerardo – CPB assembly work
EE – Repair work on AA and AI chassis 
 

model restarts logged for Tue 10/Jun/2014
2014_06_10 09:21 h1susbs
2014_06_10 11:09 h1fw0
2014_06_10 11:09 h1fw1
2014_06_10 11:09 h1nds0
2014_06_10 11:09 h1nds1
2014_06_10 11:12 h1broadcast0
2014_06_10 11:26 h1iopsusey
2014_06_10 11:35 h1iopseiey
2014_06_10 11:41 h1iopseiey
2014_06_10 11:58 h1isietmy
2014_06_10 11:59 h1hpietmy
2014_06_10 12:01 h1susetmy
2014_06_10 12:01 h1sustmsy
2014_06_10 12:39 h1broadcast0
2014_06_10 12:39 h1dc0
2014_06_10 12:39 h1fw0
2014_06_10 12:39 h1fw1
2014_06_10 12:39 h1nds0
2014_06_10 12:39 h1nds1
2014_06_10 12:57 h1iopsusey
2014_06_10 12:58 h1susetmy
2014_06_10 12:58 h1sustmsy
2014_06_10 13:01 h1iopseiey
2014_06_10 14:01 h1susetmy
2014_06_10 15:23 h1sustmsy
2014_06_10 16:20 h1hpietmy
2014_06_10 16:22 h1isietmy
2014_06_10 16:29 h1susetmy
2014_06_10 16:30 h1susetmy
2014_06_10 16:30 h1sustmsy

no unexpected restarts. SUS BS work green, SUS EY watchdog yellow, supporting DAQ restarts blue.

Baffled to find many ground loops, then I remembered that I was baffled by this same thing in all chambers.

Tested some (not all) of the HAM6 ISC cables for ground loop according to T1200131.

Each in-air cable was disconnected from the chassis while still connected to the chamber feed through (other in-air cables were still connected to both the chassis and feed through), and the continuity between the cable shell and the chassis ground was tested using DVM beeper (any continuity=bad).

FYI, for all of the above cables, there was a continuity between pin13 and shell (it's supposed to be like that).

WFS (both DC and RF), OMC PZTs, OMCS and SEI cables were not tested but I'll test at least WFS and OMC PZT.

Anyway, this is not acceptable for the chamber containing our main read out, and should be fixed in chamber.

Many more trips today, many of which are dtt sessions ending, some come from ramping the blends from TBetter (40Mhz) to Start (750mHz, no t240) which seems to trip us every time

  There was some speculation that we might have a ground issue in the T240s, similar to what we had earlier with the GS13s.

To test this we lifted the shield to ground jumper in the corner #1 T240 electronics box on the ETMX-ISI. We then drove the system in the z direction (loops on, blends = start)

and looked at the individual T240 channels

  the first thing is that all of the Z channels look the same, so lifting the shield ground didn't effect anything here.

  the other thing is the horizontal sensors which have some coherence and a flat response which is kind of odd

  WE should remember to put the jumper back

  The second plot is the same set up except now I'm plotting the transfer functions to the modal T240 signals. We expect some cross coupling at the level of a 1-3% due to gain mismatch in the electronics, which

is kind of what you see in the vertical signals, the rX and rY are smaller then the Z by about the right factor and show the same slope. While the horizontal signals show something completely different, the 1/f^2 coupling that we have been working on

This morning, we used the Genie "duct jack", the install arm, elevator, and 5-axis table to install the newly assembled ITMx Monolithic lower suspension into the chamber and mate it to it's upper structure.  All went as expected.  We then proceeded to start reconnecting wire segments and cabling.  When we mated up the lowest stage of the ESD cable, we had a small hiccup.  One of the 5 pin receptacles had fallen into the connector back shell on the end that is soldered directly to the CP.  With some care, we were able to gently fish it back out and push it into place with the wire which had been tugging on the other end of the pin receptacle.  We expanded the receptacle such that it seemed to hold itself in place better.  As well, all 5 receptacle "houses" were twisted such that they seemed to seat better in the connector.  We then connected this female connector to it's upper male cable counterpart and it "seemed" like it the connection seated properly.  We have no way of visually knowing though.  WE NEED TO CHECK THIS AT THE FEEDTHRU, OR SOMETHING.

[Mark B Jeff B Arnaud P]

Yesterday we did some software debugging in the staging building in order to run transfer functions on the assembled quad 08.
When driving the top mass, we usually monitor the lower stages osems, in both osem and euler basis. For some reason the model running for the quad doesn't record those channels (particularly the ones in the euler basis "WIT_{L/P/Y}_DQ"). We decided not to spend too much time understanding what the model situation was, and instead simply not monitor the lower stages channels during the top mass TFs.

The undamped transfer function measurements for the main and the reaction chain are attached.