We continued our investigation by disconnecting everything under the ISI table from outside.

As it turned out, all TMS and ISC cables are fine, but using DVM with alcohol-wiped tips we identified problems in SUS BOSEMs and TCS ring heater:

1. L2 (penultimate mass) BOSEM cable shield,
2. L1 (second mass from the top) BOSEM cable shield,
3. Ring heater cable sheild

are all locally (i.e. in-vac) connected to the metal structures, and if you look closely it's really obvious (Rich said "How can this NOT be grounded?!").

ETM/ERM:

The first picture shows the cable connector clamp on the penultimate mass. Can you see that the clamp doesn't have any insulation? It's just two metal plates sandowitching the metal connector shell. Of course the penultimate mass is electrically connected to ISI via suspension wire.

Also, a pinching clamp for the cable (only the bottom half of which is shown at the top of the photo) doesn't have insulation, though the cable itself has an insulation mesh.

I couldn't take a good picture of the L1 connector clamp, but it's the same design.

This is a serious ground loop right there for SUS (but not for ISC and TMS) and we should discuss any possible fix.

TCS:

The second picture shows the elaborately designed cable bracket thingie that directly connects the ring heater cable shell to the mirror cage.

ISI:

We haven't disconnected ISI cables simply because there are many,  and we know that the ISI cables are going to be revisited after the cartridge insertion. But somebody should do that to see if there's some funny thing is going on like SUS and TCS (and ISC/TMS before we fixed one clamp problem).

We just looked around and saw two cable shield rings resting on metal bolt of ISI (third and forth picture).

Attached are plots of dust counts > .5 microns.

Rolf and Dave

Report on ISI driving of ITMY March 1st 2012

The problem is introduced by the addition of a fourth ADC into the hepi/isi models. There is a hard limit set in the code for the decimation filtering history arrays to only support up to 3 ADC modules. The addition of a fourth caused the ADC decimation filtering history to write into the space used for DAC upsampling filer history data, thereby causing some form of this ADC input to be output to the first DAC module. I believe this has been in the code since the early blue box days. While other arrays used by ADC/DAC data have been updated to use MAX_ADC_MODULES and MAX_DAC_MODULES some time ago to set array sizes, as defined in cdsHardware.h (presently 12 ea), these history arrays apparently got overlooked. This hard limit would not effect IOP tasks, or any models running at 64K, as they do not perform up/down sampling.

a new rcg tag 2.4.2 was created as a copy of 2.4.1 with the MAX ADC/DAC substitution installed. All LHO models were recompiled and restarted against 2.4.2. with the exception of:

h2susb6: h2sustmsy model would not compile, missing links. Running 2.4.1 on this front end for now

h2seib6: would not reboot, needs further investigation

h2psll0: hold off until friday, running 2.4.1 on this front end for now

[Continuing from Corey]
- Retractable arm inserted on BSC8, H2ITMY Quad removed.
- Many high (10k-30k) dust count alarms in the Biergarten, but the offending dust monitors could not be located. The only dusty activity during much of this time was vacuum cleaning in the area and retractable arm installation by the door of BSC8
- CDS worked on cornering the H2ITMY fiber-breaking issue. Many reboots of h2seib6, h2seib8, and the entire DAQ as code and models were updated.
- H2 PSL and CDS are working into the evening.
- OSB doors locked at 1630.

The SEI Engineers at MIT were a bit concerned about letting the horizontal actuators take the full shear of the vertical motion when the Quad payload is removed.  So, I disconnected them.  Almost completely that is...  The SE unit has a bolt that I fear is galling.  Everything else that can has been loosened and there are slots so we feel this will be OK.

• Flurry of activity continues around BSC8
• Apollo is surveying/laying foundation for Mode Cleaner Tube display
• Vern/Vincent physically disconnected power cables from ISI Coil Chasis for continuing diagnosis
• Dave implemented a model change for ISI and this required a DAQ reboot
• Brazing of refrigerant lines is taking place in Kyle's Mechanical shop
• Have had Dust Monitor #15 Major Alarms (nothing over 500 counts this morning)

Attached are plots of dust counts > .5 microns. The dust monitor in the clean room over BSC8 (H0:PEM-LVEA_DST15_5) was momentarily disconnected, but this does not really show up in the included plot of H0:PEM-LVEA_DST15_MODE.

I and Dan confirmed that shells of ALL in-vac cables that are connected to CDS side are grounded to the metal parts of ISI and TMS. I'm talking about all cables related to ETMY, ERMY and TMSY. There is a huge potential risk of ground loop(s).

The cause of this is NOT in four ISC cable chains (picomotor, beam diverter, and two QPD cables). This was confirmed by disconnecting all ISC cable chains from the CDS side at the moc feedthrough: When we did this, the shells of disconnected in-vac cables were isolated from the metal parts of ISI/TMS.

We haven't exonerated the two BOSEM cables of TMS yet, as well as all ETMY and ERMY cables.

What is likely happening here is:

• There is one or more crappy cable clamp which physically grounds the cable shielding to the metal in-vac parts.
• Cable shields are connected to pin 1 in-vac, which is connected to pin 13 out-of-vac.
• Pins 13 are connected to each other on CDS side.
• Thus, the shell of any good cable is connected to the metal in-vac parts via CDS and crappy cable.

Even though there seems to be no DC-coupled "loop" as far as ISC table/cable is concerned in the above picture, basically the ISC cable shields on the ISC table are within 1/32 inches or so of the table surface for the entire length and there certainly is a capacitive coupling. And if there are two crappy cables, they form DC-coupled loop, even though that's not ISC. This is bad.

One crappy cable might be the cause of the entire thing.  I can continue testing the remaining two TMS cables, but if we're unlucky we might need to check each and every in-vac cable including ETMY, ERMY, TCS and ISI cables, and I need cooperation from all parties.

After pulling back the ACB and removing a few panels of flooring, QUAD vibration absorbers, sleeve and Fiber Guards we made more inspections of the ITMy weld horns.  The following collection on ResourceSpace is of the shortest broken horn:

https://ligoimages.mit.edu/?c=1002

We do not see any cracks further down into the horn or ear.  We used a swab wipe to brush away some of the debris on this ear in order to facilitate further inspection, hence why some of the pictures may look a little different when compared.

HEPAs are installed, ceiling and several wall panels have been left off for electrical inspection. Gang plates were installed. Gerbig has left but will be coming back in late March.

Brian, Fabrice and I have found that there is an odd discrepancy between the value of the watchdog recorded by the framebuilder and the value recorded by EPICS. The EPICS value reads 4, while the framebuilder value reads 0. These should be reading the same value, since they are both a part of the same filter module (H2:ISI-ITMY_ST1_WDMON_STATE). 

In the attached plot, you can see the discrepancy in the top two charts (all full data). The top left is the DQ channel, which says the watchdog is 0 (??), while the top right is the EPICS channel, which says the watchdog is at 4 (fully tripped). The watchdog C code has no '0' state, so it's odd that this value would be recorded at all when the model is running.

We show that the model is running by showing the GPS time as recorded from the model in the bottom right. The value for the GPS time drops to 0 when the model is killed (1014668665) and comes back when the model is started again (1014668677). 

We also get full data from CPS H1 at the bottom left when the model is restarted. The data from 1014668665-677 in this channel is not 'real' - it is reflections of previous data at 1 second intervals. This can be casually observed by the apparent aliasing effect from 1014668655-70 in what look like dark diagonal stripes. We can see the CPS signal grow when the model starts.

Brian and I have plotted the beginning of the (bad) excitation in dataviewer in both the ISI and IOP models. We have also plotted the GPS time according to the ISI model (DCU_ID 75). As you can see, the GPS time drops out when the model is being off, but comes back up shortly thereafter. This data is for the 11:42 UTC restart when the excitation starts. 

We plot the same for the second restart, twenty minutes after which the fiber break occurs. We see the same thing - the model is shut down, then starts again. We see a similar drop in the GPS time to 0 before the model restarts. The excitations also stop when the model stops, and start again when the model restarts. We see a growth in the amplitude of the coil drive signal as well as the DAC signal from the IOP.

This is not inconsistent with a loop instability within the model.

Yesterday, we measured the resistances of the 12 actuators on ISI-BSC8. The table below shows the measured resistance when the ISI was:

- in the staging building

- on the teststand in the LVEA

- in the chamber (March 6, 2012)

Field cables in the LVEA are about 1 ohm. Resistances measured after the important drive are similar to those measured when the ISI was on the LVEA teststand. There are no obvious signs of damage on these actuators.

Betsy B. Travis S. Thomas V.

After reinstalling the swing back plate and support brackets, we swung the ACB back so that SUS can access the quad somewhat freely.

200W beam

35W beam